Robots that work in homes need models that understand space: how far apart two objects are, which way a doorway is, how big a room is. We trained a video language model on spatial questions generated from digital twins of 29 real apartments. With the model, recipe, and data budget held fixed, it scores 43.8 on VSI-Bench, 5.3 points above the same model trained on procedurally generated houses.
Simulation is a scalable way to teach these skills. SIMS-V showed that spatial question-answer (QA) data from procedurally generated ProcTHOR houses improves a model's score on VSI-Bench, a benchmark of 5,130 spatial reasoning questions over real indoor scan videos. But in Cambrian-S's comparison of spatial data sources, ProcTHOR was the least effective source with exact 3D annotations.
Procedural houses follow the rules of a random layout sampler, not the way people actually build and furnish homes. We wanted to know whether spatial data generated from real home layouts gives a model a stronger learning signal.
Building RealHomes5K
We started from GRScenes-100, a set of artist-built Isaac Sim scenes that follow the floor plans and furniture arrangements of real apartments. These scenes give exact 3D ground truth, and they share no scans with VSI-Bench's test set.
We converted 29 apartments into the SIMS-V scene format and reused its question generator, so that only the scenes changed:
- Ground truth from the meshes. We compute object bounding boxes from the rendered USD meshes and fit each scene's room polygons to its footprint.
- Robot camera tours. We replay SIMS-V's Stretch robot camera through each apartment and render 58 tours (27,755 frames) in Isaac Sim 5.1. Rendering took about 1.5 GPU-hours.
- Verified answers. An independent checker recomputes every answer from the 3D geometry. All 2,378 unique questions agree.
The result is RealHomes5K: 2,378 questions, each in multiple-choice and open-ended form, for 4,756 training conversations.
A controlled comparison
We fine-tuned LLaVA-Video-7B-Qwen2 with the SIMS-V recipe and compared four variants:
- Base: the model with no extra training.
- + SIMS-VSI (5K): 4,756 conversations from ProcTHOR houses, with the same number of questions per task and format as RealHomes5K.
- + RealHomes5K: our 4,756 conversations from real apartment layouts.
- + RealHomes5K (w/o video): the same questions and answers, with every video replaced by black frames. This blind control measures how much a model learns from the question and answer text alone.
We evaluated every variant on VSI-Bench and on VSI-Bench-Debiased, a 2,362-question subset that removes questions a model can answer without looking.
Real layouts beat procedural houses
Training on RealHomes5K raises VSI-Bench from 35.0 to 43.8. The matched SIMS-VSI subset reaches 38.5. The 5.3-point gap holds on the debiased benchmark, where RealHomes5K scores 37.2 and SIMS-VSI scores 31.7, a 5.5-point gap.
The result is also sample-efficient. Our gain of 8.8 points from 4,756 conversations is similar to prior procedural-house runs that used four to five times more data:
| Training data | Samples | VSI-Bench gain |
|---|---|---|
| Cambrian-S, ProcTHOR only | 20,092 | +7.9 |
| SIMS-V, 25K mix | 25,000 | +8.4 |
| SIMS-VSI, matched subset | 4,756 | +3.5 |
| RealHomes5K | 4,756 | +8.8 |
The prior runs use different base checkpoints and training budgets, so treat this comparison as indicative rather than controlled.
How much of the gain needs the video
The blind control scores 39.8 on VSI-Bench, 4.8 points above the base model, without seeing a single frame. Fine-tuning teaches the model answer formats and plausible numeric ranges, and that alone accounts for about half of the total gain. We think any synthetic spatial dataset needs a control like this.
The other half depends on the video: RealHomes5K beats its own blind control by 4.0 points on VSI-Bench and 3.3 on the debiased set. Of the 5.3-point advantage over SIMS-VSI, only 1.2 points come from the text, and 4.0 come from the rendered video.
Room size questions explain part of the gap. Both synthetic datasets ask about the size of the whole apartment, and the SIMS-VSI model learned to overestimate on VSI-Bench's single rooms. Even with room size excluded, RealHomes5K still leads by 2.5 points, and all of that lead comes from the video.
Limitations
- Focused scale. The study uses 29 apartments, 2,378 unique questions, and a single base model,
LLaVA-Video-7B-Qwen2. There is room to scale both scene diversity and model families. - Spatial-only training. We fine-tuned on spatial QA alone. Future runs should mix in general video instruction data and check that the model keeps its broader video understanding.
- Non-commercial license. GRScenes-100 uses a CC BY-NC-SA 4.0 license, so RealHomes5K and our checkpoints are limited to non-commercial research. Moving the pipeline to permissively licensed 3D homes is an important practical step.
What's next
We plan to scale to more homes and more model families, isolate layout from rendering and assets, and measure a full scaling curve against procedural data. Most importantly for robotics, we want to test whether these spatial skills transfer to downstream tasks such as language-guided navigation and mobile manipulation.
The full paper covers the conversion pipeline, per-task results, and every pairwise comparison. Read the full paper (PDF).