HFlow: open-source data quality infrastructure for Physical AI
We've been speaking with teams who deeply care about data quality, and we noticed that every data team eventually builds similar pipelines for quality checks (QC).
Teams collecting data want to own their quality checks, such as detecting camera blackouts, choppy joint states, and occluded hands, because they deeply understand their end-to-end data collection process.
However, what keeps coming up is that everything around the quality checks is tedious: managing one-off pipeline scripts, fragmented buckets, and a spreadsheet or Slack thread tracking which checks ran on which data.
This motivated us to build HFlow, an open-source SDK for data teams that collect, transform, and curate Physical AI data.
HFlow is for data teams that have the ambition to process a million hours of Physical AI data and are starting today. Point it at your MCAP episodes, write your quality checks as ordinary Python functions, and curation becomes an optimized OLAP SQL query instead of manually aggregating directories of data.
Dyna's recent Dyna-2 infrastructure post shows what scaling such a quality control pipeline to a million hours looks like. Their ingestion throughput went from 14,000 episode-hours per week to 440,000 with the infrastructure they built internally, and their post has the full details.
We're aligned with their belief that infrastructure is a core blocker for advancing robotics. We experienced the same data challenges in our previous work, which is what first led us to build Pareto and the Hebbian APIs.
HFlow's vision is to let data teams define their own bespoke quality check code, while we provide the durability, observability, and auditability around it.
After many conversations with data teams, we decided the path to building the best version of HFlow is open source, because QC infrastructure compounds when the edge cases one team catches become checks for everyone else. We're excited to build this in public alongside our existing partners and grow our network of contributors.
High-quality data is the bottleneck for the next frontier of robot intelligence, and we want teams to focus on their data, not the infrastructure. When getting started takes a few lines of code and a weekend, more checks get done, and iteration cycles accelerate.
If you're collecting robot or egocentric data and your pipeline is currently a folder of scripts and fragmented buckets of data, we'd love to get your feedback.