RSSM on WorldSense Tech Blog

When World Models Meet Transformers: From RSSM to Large-Scale Sequence Modeling

Wed, 12 Aug 2026 00:00:00 +0000

In previous articles, we covered the RSSM architecture and training techniques in DreamerV3 in depth. RSSM is a classic design in reinforcement learning world models, but if you follow recent research, you’ll notice a clear trend: world models are becoming Transformer-based.

From Google’s UniSim to Wayve’s GAIA-1, from NVIDIA’s Cosmos to solutions from domestic embodied AI teams, the Transformer is emerging as a key technical approach for large-scale world models.

DreamerV3 Training Tips: Lessons from Real-World Debugging

Tue, 11 Aug 2026 00:00:00 +0000

In the previous article, we walked through four representation approaches for world models. Today, we shift back to the practical side of DreamerV3 and talk about the pitfalls and tricks you encounter during training. This article is based on local experiments using DreamerV3 commit e3f02248, JAX + Haiku, and MuJoCo + DM Control. Parameter names and configurations may differ across versions.

DreamerV3 is currently one of the most open-source and mature world model implementations available. But if you’ve actually trained it, you know the process is far from easy — environment setup, hyperparameter tuning, training instability, slow convergence… the list of gotchas goes on.

Deep Dive into RSSM: The Core Engine of World Models

Sun, 02 Aug 2026 00:00:00 +0000

In the previous article, we covered the basic concepts of world models and the overall architecture of DreamerV3. Some readers asked for a deeper explanation of how RSSM actually works. This article dissects the core component of the Dreamer family of world models.

I’ll try to make the math clear without being overly formal. After all, our goal is to understand the principles, not prove theorems.

Why We Need State-Space Models

Before discussing RSSM, let’s step back and ask: why do we need state-space models at all? Can’t we just use an RNN or Transformer directly?