YouTube System Design for Robotics Data Infrastructure
When we first started building a data platform for robotics, we weren’t thinking about YouTube.
Robotics teams wanted to be able to upload their LeRobot datasets, search across demonstrations, inspect synchronized camera feeds and robot states, and export curated episodes for training. That sounded like a fairly specialized problem in robotics. So we built Pareto.
Once we got into the infrastructure work, the requirements started to feel familiar. We needed to reliably upload large media files, preserve the originals, process them in the background, generate thumbnails and preview videos, index their metadata, and stream them to a browser without downloading the entire dataset.
“Design YouTube” is a classic system-design exercise. It can feel a little pointless: YouTube dominates, and almost none of us will ever need to build a global consumer video platform at that scale. If we reframe thousands of users uploading single videos to your platform as a single robotics dataset uploading thousands of videos, the similarities become much clearer.
ByteByteGo scopes the exercise to upload and playback: original files go to blob storage, background workers transcode them, a completion flow updates metadata, and the CDN delivers the finished versions. That is the simplified shape above.
An episode of robot data
The classic video platform starts with one dominant object: a video, perhaps with audio and maybe subtitles. A robot-learning dataset has a more complicated unit of data.
An episode of robot data is a multimodal record: it can contain several camera feeds (wrists, top, chest, etc.) alongside joint positions, actions, force readings, timestamps, task descriptions, and other sensor streams. The cameras may be the largest part of the dataset, but they are not useful in isolation. The data only makes sense when it remains connected to what the robot was sensing and doing at that moment.
Pareto searches across episodes, so someone searching for an “orange block” can scrub the surrounding trajectory, compare cameras, and decide whether a demonstration belongs in a training set.
LeRobot v3 makes this concrete. Episodes can share Parquet data files and MP4 shards, so reading an episode means resolving both its structured row range and its timestamp range inside each camera’s video. Once we wrote the data path down, it looked familiar: upload -> process -> store -> index -> play.
Pareto is open source, so you can inspect the system and run it yourself.
1. Reliable uploading
When it comes to multimedia, we get into the art of troublesome large uploads. Connections fail, processes restart, and retrying an entire dataset because the final file was interrupted is expensive. Video platforms address this with resumable uploads and an explicit transition from uploading to ready-for-processing.
In Pareto, we keep receiving source files separate from converting and indexing them. Source files are stored under versioned storage paths, completed batches are checkpointed, and the final manifest is written last. Downstream jobs use that manifest as the completion marker rather than guessing whether a directory is complete. It also lets us trace any generated output to the exact source files and batch that produced it.
Keeping the original dataset also means future processing can change without asking the user to upload the data again.
2. Processing becomes a pipeline
For any site that streams video, uploads have to be processed into multiple versions: different resolutions and codecs, thumbnails, previews, captions, and other assets. The original is one input to a larger background pipeline.
For robot data, we generate sampled frames, thumbnails, preview videos, downsampled state and action series, vector embeddings, search indexes, and Rerun recordings. Each serves a different use: search, browsing, synchronized playback, analysis, or export.
The browser can show a search result without opening the full dataset, just as YouTube can start playback without serving the original upload.
3. Running long jobs
Pareto runs different jobs in the pipeline as durable workflows using Temporal. They have a hosted service but it is also open source and self-hostable.
Workers are separated by capability and by the shape of the work. We shard independent episode ranges across GPU embedding workers, then fan in at a finalization step that merges the shard outputs and runs the global work once. Other jobs that don’t require a preloaded warm model are split into independent chunks that can run across elastic GPU or CPU pools. That keeps bursts of one job class from consuming the capacity another kind of work needs.
Not every generated output needs to be ready before a dataset is useful. We moved many lightweight, recoverable computations to lazy, on-demand paths, keeping the critical path focused on making data searchable and viewable. That lowers time-to-first-use and avoids spending cycles on artifacts nobody asked for yet.
Text queries make the same point in a different form. Image embedding is a high-throughput batch job and belongs on the GPU, while a live text search needs just one small embedding before nearest-neighbor search. We developed a CPU SigLIP 2 server for text embeddings with Rust and ONNX. It keeps GPUs focused on ingestion and can make the online path easier to provision.
4. Separate files, metadata, and indexes
YouTube does not place video metadata, search indexes, and media into one database, and Pareto follows the same division. Source datasets and derived data live in URI-addressed object storage. Postgres holds metadata. LanceDB stores multimodal vectors for retrieval. The server itself is stateless. A text query is embedded, searched against vectors, and returned as episodes.
5. Streaming only what the viewer needs
A useful video platform does not require someone to download an entire file before watching the first few seconds. It supports seeking and serves the byte ranges needed by the player. This matters even more when a robotics dataset contains multiple camera feeds and structured signals.
Pareto’s browser UI uses previews and requests only the bytes the player needs instead of opening the original dataset for every interaction. Multiple cameras share one synchronized transport, and the user can scrub while viewing the corresponding state and actions. Preview quality can be selected automatically or pinned across the camera grid.
The interface looks familiar: a timeline, previews, and seeking. But the job is different. This is not passive viewing. The user needs to understand why an episode was retrieved, inspect the evidence, compare behavior, annotate it, and decide whether it should influence a model.
Design for scale without starting at scale
Early in my career, I started as a Software Engineer at Jane Street. While building infrastructure to support processing millions of messages per second, I learned to separate thinking about scale from operating at scale. Early on, the useful decisions are structural: where durable state lives, whether work can be retried, and whether adding capacity requires redesigning the system.
You may have heard of the idea of “cattle, not pets”. Rather than having a few large nodes that we keep scaling up and manage by hand, it’s often better to have small identical nodes that you can bring down and up quickly to scale out at any time. Durable state lives outside the processing machines, while API replicas and workers are replaceable. A media worker can disappear, have its job redelivered, and be replaced without becoming a recovery project. That keeps horizontal scaling available: we can run one worker today and more later without changing the execution model.
Unfortunately, GPU capacity does not always make this easy. If a provider cannot give us enough quota, raising a herd of GPU-powered cattle is harder.
Yet, “scalable” can become an excuse for unnecessary complexity. Pareto’s direct CLI can index, search, and export synchronously, without Postgres or Temporal. We add distributed pieces only where we need durable execution over long time horizons.
XKCD’s “Is It Worth the Time?” asks how much effort an optimization can consume before it costs more time than it will ever save. Infrastructure has the same tradeoff: every abstraction must be continuously deployed, monitored, upgraded, and maintained.
Temporal may look like more infrastructure than a young project needs, but the requirement it solves for us is not hypothetical traffic. Individual conversions and ingests can run for hours on preemptible workers and need to recover from interruption. Durability is a present requirement; building our own scheduler, retry system, and reconciler is not.
Ok so, it’s not really YouTube
Robotics data is not video streaming with extra columns. A single episode can contain camera feeds, actions, joint states, annotations, and sensor streams with different sampling rates, all of which need to remain aligned. It also has schemas and model identity to preserve. Mixing vectors built with different models would make search results meaningless.
But the YouTube thought experiment is useful. It gives us a familiar starting point for upload, processing, storage, indexing, and playback. Not too bad for interview practise.
Try Pareto to search and curate robotics data, or explore and self-host it from the open-source repository.