As frontier models improve, we've increasingly been using coding agents to help fine-tune models for our own tasks. We wanted to share how HFlow fits into that work, and open source an example for anyone to explore.
HFlow is an open source SDK for data teams that collect, transform, and curate Physical AI data. In this workflow, we use HFlow to prepare and track data for an AI agent to train and evaluate models for use with HFlow's existing evaluation workflow.
Prepare the data before training
We use HFlow to process the source media and prepare it for training. It identifies duplicate images across the datasets, preserves where each sample came from, and creates reproducible training, validation, and test splits. This gives the agent a consistent dataset to work with while it explores changes to the training code.
In our experiments, we had cases where the model's answers made us question whether we had converted the labels correctly. Before trying another training approach, we checked the original labels against the prepared examples, and checked that the images still matched their labels. The conversion checks passed. But the investigation also exposed contradictory source labels, duplicate data across releases, and gaps in how we recorded where the data came from.
These are different problems with different fixes. If we had kept asking the agent to improve training without checking the data, it would be experimenting without understanding what the model was learning from. This is why we preserve where every duplicate came from, rather than just keeping the first image and dropping the rest of its history.
Google Research's study of data cascades describes how unresolved data issues can compound into downstream failures in AI systems. [1] That is a useful reason to investigate the data before asking an agent to spend more compute on training.
In our open source example, we combine datasets and deduplicate before creating the splits. Identical images with different encodings can still be identified by their decoded pixels. If duplicate images have conflicting labels, we exclude them and record the conflict instead of choosing a label for the agent.
Exact-image deduplication still has limits. Two neighbouring frames from the same recording may look almost identical without having identical pixels. If the source doesn't provide reliable recording identities, we can't claim that the splits contain independent recordings just because there are no exact duplicates.
Train and compare candidates
The usual model research loop is to train a baseline, change the training approach, and compare the result. If the data or evaluation changes between runs, we can't tell whether a better score came from the code change or from the experiment itself.
Andrej Karpathy's autoresearch project gives an agent a tight version of this loop. [2] First, we measure the untouched baseline. Then the agent edits code, runs an experiment, and compares the candidate's score with the baseline. It keeps a useful change or tries another one. Unlike your traditional hyperparameter sweep, the agent can iterate on training code itself.
Google DeepMind's AlphaEvolve uses a related pattern for algorithm discovery: language models propose programs, automated evaluators score them, and those results guide further search. [3] Our example applies a much smaller loop to training code, where the evaluation also depends on the quality and separation of the data.
To compare runs, we keep the data split, evaluation code, and base model fixed. The agent changes only the training file. Each candidate trains from the same base model, and a separate evaluator checks its saved adapter. We keep a record of every attempt, including runs that fail or run out of time.
The validation score guides the search. After the agent's attempts are over, we evaluate the selected candidate once on the held-out test partition. These limits on exact-image deduplication matter when we interpret both scores.
What we learnt from our own experiments
Some practical lessons came from deciding what to investigate next, and whether a result was actually good enough to keep. An agent can make these iterations faster, but we still need to be clear about what we are asking it to improve.
Agents need the right tools for research
The research loop depends heavily on the tools around the agent. Coding agents like Codex and Claude Code are built for software engineering, while research involves reading relevant prior work, working with large data and models on remote storage, running experiments on remote CPU/GPU compute, and using the results to decide what to try next. If those things are hard to reach, the agent spends time working around the setup instead of investigating the question. The research process matters, and so do the tools that let the agent carry it through.
A better score isn't always a better model
We had candidates that answered more examples correctly, but also made more unsupported guesses when the input was ambiguous. The overall score looked better, while a behaviour we cared about became worse.
If the workflow needs the model to avoid guessing, that needs to be part of the evaluation. We started looking at correct answers, unsupported answers, and how often the model answered separately. A model that refuses to answer everything isn't useful either, so simply reducing the number of wrong answers isn't enough.
OpenAI's research on why language models hallucinate makes a related point: accuracy-only evaluations can reward guessing over acknowledging uncertainty. [4] That work concerns language models; in our experiments, the practical question is whether the evaluation rewards the behaviour we need from a vision-language model.
The lesson for autoresearch is to decide what a useful improvement means before the agent starts. Otherwise, it can improve the metric we gave it while moving away from the behaviour we wanted. Keeping the individual predictions also helps us understand the tradeoff instead of just comparing two numbers.
A test set doesn't stay untouched by itself
We also had to think about evaluation across experiments. A run can keep its training and test rows separate, but if we look at the test failures and use them to decide what to train next, those results have already influenced the research.
Dwork and colleagues formalise this problem in their work on adaptive data analysis: repeatedly using holdout results to choose the next hypothesis can overfit the holdout itself. [5] Their work studies methods for controlled reuse. Here, we use the simpler separation between validation for search and held-out data for final evaluation.
We started separating the search from final evaluation more explicitly. The agent can inspect validation results and use them to propose the next change. Once we select a candidate, a separate evaluation checks it on the held-out data. If those results guide another round of research, we need to recognise that the data has become part of development.
This becomes more relevant as agents run more experiments. We can freeze the files and keep their hashes, but that doesn't make examples we've already learnt from untouched again.
Check what actually reaches the model
Another lesson was that using the same weights and images wasn't enough to make two evaluations equivalent. We observed differences when evaluating adapters before and after merging them into the base model, and when using different inference runtimes.
That led us to check the actual inputs, including image preprocessing and the rendered prompt, rather than just checking that the configuration looked the same. A change in the prompt format or image processing can affect the result even if we haven't changed the training.
For autoresearch, this means keeping the evaluation path consistent while the agent changes training. If we want to change the inference setup too, we treat that as a separate comparison. Otherwise, it becomes difficult to tell what caused an improvement.
Try the open source example
We kept the task small: count whether 0, 1, or 2 of the camera wearer's hands are visible in an image. It uses public Build AI evaluation releases and their published Gemini-generated labels. These are teacher labels, not human ground truth, and you don't need to call Gemini yourself to prepare them.
The example fine-tunes a tiny vision-language model (VLM) with low-rank adaptation (LoRA), which trains a small set of adapter parameters while keeping the base weights fixed. [6] You can run the baseline and starter on CPU before giving an agent the training file to edit. It can explore sampling, adapter placement, loss weighting, or training-loop efficiency under the same time allowance. There isn't a particular coding agent built into the runner, and you can also make the changes manually.
HFlow handles manifest deduplication, provenance, and reproducible splitting. The example contains the training and evaluation runner. You can follow the whole process from preparing the public data to evaluating and exporting the selected model, without adopting our internal workflow.
The recorded CPU results show two separate experiments. Macro-F1 averages the F1 score for each of the three classes equally, so the score can improve even when one class remains at zero. In the fine-tuning pilot, development macro-F1 rose from 0.1667 to 0.4889, but the one-hand class still scored zero. That is a partial improvement on the development metric, not evidence that the model solved the task. In a separate pair of short code trials, both runs stayed at their baseline score of 0.1667. Those trials showed that the loop ran, but not that it improved the model. None of these are held-out test results.
Coding agents are starting to take on more of the work in model research: proposing changes, running experiments, and learning from the results. Reliable data and evaluation make that work useful. HFlow helps teams prepare the data before the agent explores changes to the training code. Try our open source autoresearch example and let us know what you learn.
References
[1] Sambasivan, Nithya, et al. Data Cascades in High-Stakes AI. Google Research. CHI, 2021.
[2] Karpathy, Andrej. autoresearch. GitHub repository.
[3] Google DeepMind. AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. 2025.
[4] Kalai, Adam Tauman, et al. Why Language Models Hallucinate. 2025.
[5] Dwork, Cynthia, et al. Generalization in Adaptive Data Analysis and Holdout Reuse. 2015.
[6] Hu, Edward J., et al. LoRA: Low-Rank Adaptation of Large Language Models. 2021.