Getting video models to learn better, faster

Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3. In our experience, most of the gains are directly attributable to 3 flavors of data improvements:- Data Filtering & Rebalancing: Remove noisy data and resample your data strategically so your model learns more effectively - Data Annotation: Gather better annotations like richer captions, bounding boxes, and font details so that it's easier for your model to disambiguate visual concepts - Synthetic Data Generation: Finetune an ensemble of existing generative models to create training data for which there is little-to-no naturally occurring data (e.g. image editing / reference-conditioning for Nano-Banana style models) A couple of years ago, the prevailing wisdom across all generative models (be it text, image, audio) was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data (e.g. heavily compressed JPEGs) into pre-training, your model is going to waste a significant amount of its capacity learning how to mimic this slice of data. If you filter your dataset well, your model will have a lot easier time learning what you want it to learn. We know this sounds obvious, but it's a lot harder to do in practice. Today we're going to walk you through how our approach to data filtering has evolved since 2024. And, hopefully we'll save you from a couple of headaches if you end up training your own generative models down the line. [2024] Filtering on a budget — Traditional CV on CPUs On the first go around, we decided to push our raw dataset through old-school computer vision algorithms. This way we could get away with a cluster of cheap CPU instances instead of an unholy number of GPUs running a multimodal LLM. Scene detection We need to filter down tens of billions of images and videos to create our pre-training dataset. Images don't really require any specific pre-processing, but raw videos do. Next time you watch a television show or movie, track how often the camera cuts. If you're watching something made in the last twenty years, more likely than not you'll see a cut every 5 seconds. When to cut and how to cut is an authorial decision, not something a generative video model should do arbitrarily. So, we need to slice n' dice our videos on shot boundaries into video clips before we can filter them down. With our cheapskate CPU-only agenda, we picked up PySceneDetect. At a high level it maintains a rolling window of K-frames and if the K+1 frame has significantly different image statistics, it categorizes the frame as a cut. There's no underlying machine learning model. It runs really fast but struggles with common transitions like dissolves, fades, and jitter cuts (which low key is a huge issue). Getting to know your data Whenever you get new data, you should spend a few days reviewing random samples, listing what you'd like to keep and what you'd like to throw out. Ideally, you take the time to draft an ontology of categories within "good" and "bad" and track the relative sizes of these categories. At some point during the data filtering process, your engineer brain will take over, and you'll spend way too much time tuning the knobs of your heuristics (or LLMs), chasing that "perfect" decision boundary. These notes are going to save you from yourself down the line. They'll give you the facts you'll need to talk yourself out of trying "one more idea", when the answer is clearly "no". Plus, understanding the shape of the data distribution will really help with dataset rebalancing. Certain categories are overrepresented in the natural distribution of all videos. We need to subsample and suppress this signal, otherwise it will dominate training and our model will struggle to learn the long-tail of people/places/things/actions that we need in order to generate anything. Sieving out the un-captionable Generative video models are primarily limited by what we can describe correctly and consistently in words. Text provides a pretty good scaffold to understand the visual world, but it's by no means the correct conditioning mechanism for all aspects of video generation. Details like camera trajectories in space-time and the nuances of an actor's performance are simply indescribable in natural language.For now, we need to filter out clips where the primary "thing" that makes the video clip interesting is un-captionable. Without a crystal clear text description, it's just noise to our text-to-video model. Text-heavy For example, we want to filter out text-heavy videos. It's still hard for LLMs to caption motion graphics that are constantly changing on screen. We don't want to waste capacity in our 2B parameter model learning motion graphics when it could be allocated instead to learning actions.To do this, we sampled frames from each video and ran a tiny EAST Detector to extract bounding boxes for text. From there, we filtered out text heavy videos based on the percentage of the frames that had text and the percentage of each frame covered in text. Using a CNN for this task was a good idea, but the specific choice was wrong. In order to run tens of billions of frames on CPUs, we had to resize the frames aggressively. So, a lot of text-heavy samples with small fonts fell through the cracks. Small text goes undetected, after EAST image pre-processing EAST is a pretty old model from 2017. It's small and far from the state of the art on text detection. Getting it to run efficiently on CPUs without cache-thrash and thread oversubscription was a challenge. Even after performance optimizations, it was still the largest bottleneck for this version of the data pipeline. Indescribable actions When there's not much happening on the screen (e.g. close-up on a person's face), it's hard to describe the specific action taking place. If there's too much happening (e.g. extremely shaky camera, a soccer match with a bunch of folks moving across the pitch at once), LLMs struggle to caption the clip correctly. We lumped these categories of videos together as "indescribable action" clips to be thrown out. Videos are typically serialized on disk in a compressed format. Codecs like H.264 reduce file size by storing keyframes and motion vectors that describe how the keyframes change over time, rather than RGB values for each pixel over time. H.264 motion vectors Keep this clip. Lots of moderately sized motion vectors. We used the motion vectors stored within the mp4 files themselves to isolate and filter out the "indescribable action" videos.mv-extractor and computed two heuristics per clip: Specifically, we usedaverage_frame_energy : L2-norm of all motion vectors averaged across the videomin(sub_clip_average_frame_energy) : Split each clip into a variable number of chunks depending on the video's length, calculate average frame energy for each chunk, and take the minimum across these L2-norms Then came the decision tree: average_frame_energy < 0.1: Throw the clip away. These were essentially static videos (e.g. slideshows, still frames, freeze-frames).average_frame_energy > 25: Throw the clip away. The footage was incredibly chaotic.min(sub_clip_average_frame_energy) < 0.03: Throw the clip away. A portion of the clip has nothing happening (e.g. a fade, transition to a still image in a documentary).- Keep everything else. This works well as a cheap first filter, but it has mediocre recall (i.e., a lot of indescribable action clips are kept in the dataset). Subsampling talking head video clips From our initial review of the raw data distribution, it was pretty obvious that talking head clips where folks talk straight to camera were dramatically over-represented. If we let the dataset be, it would have been significantly biased towards this sort of clip. Our model would get disproportionately good at creating them (likely at the expense of others), so we needed to find them and subsample them. Living in an old-school CV world, we naturally burrowed deeper down the engineering tunnel and introduced additional heuristics. We sampled frames from each video clip and ran a Haar-cascade face detector to extract bounding boxes for faces and calculated two numbers: average_face_frame_energy : L2-norm of motion vectors within face bounding boxesaverage_background_frame_energy : L2-norm of motion vectors, just in the corners of the frame (as a proxy for background motion) And from there, another decision tree: - Moderate average_frame_energy +average_face_frame_energy >=average_background_frame_energy : Keep the clip. Usually a really good close-up. - Low/Moderate average_frame_energy : Subsample these. [Early 2025] rm -rf — Replacing hand-crafted heuristics with finetuned LLMs We're starting to sketch a rather complicated decision tree. It's full of lossy proxies that only kind of work, and it's very incomplete. This approach simply doesn't scale. Every time you have a new idea for a filter you have to re-examine how the new node in the decision tree impacts all the other branches. Everything is intertwined and eventually you end up with a pipeline that's both un-interpretable and uneditable. When we started in 2024, we were staring down the barrel of tens of billions of samples. Given our limited budget, our gut was to construct the cheapest filters possible. This was fundamentally wrong. Our video model struggled to learn basic motions like guitar strumming after training for several weeks on our 2024 dataset but was able to learn these very actions in less than 24 hours of training, after applying our 2025 filters. Instead of looking for the cheapest filters possible, you should optimize for the best possible filters you can afford. Migrating to 1000s of GPUs This brings us to our second takeaway: throw away your "principled" computer vision techniques and adopt black box neural networks wherever you can. There are patterns that humans simply can't describe well, no matter how hard we try. Old school CV methods were the best hand-crafted approximations of their era. They're truly impressive feats of engineering, but a well-trained neural network will learn a non-linear function that will win on precision and recall in 99% of cases. Once you re-orient yourself around this reality, your job should shift from crafting cheap heuristics to optimizing models for GPU throughput and engineering resilient, parallelizable workloads to run on SPOT instances across providers. Concretely, we replaced PySceneDetect's heuristics with AutoShot and TransNetV2, accelerating inference with custom CUDA kernels. We stopped running the ancient EAST Detector on CPUs. Instead, we deployed PaddleOCR with TensorRT across thousands of Nvidia A10Gs and L4s. And, we ripped out the tangled web of computer vision algorithms (Haar Face Detectors, Lucas-Kanade, monocular depth estimators, etc.), replacing them with a set of fine-tuned LLMs. Iterative dataset labeling (aka self-consistency is harder than you think) Finetuning is pretty straightforward thanks to the folks at Unsloth. This means the labeling is the work. The hard part of training LLMs for data filtration is that the categories are always somewhat fuzzy. You'll have to answer questions like: - If a sample fits several categories to different degrees, which label do I assign it? - Should I simplify my categories, so I can label the data more quickly and consistently? Or, do I need to split my category into pieces to make it clearer? - How easy is this concept for the LLM to learn? How much data do I need for each category? More often than not, the biggest problem you'll run into is one of self-consistency. Over the course of labeling a couple hundred samples, it's only natural that you'll relax your criteria, mislabel samples, and muddy the signal in your dataset. In a CMU study from 2025, researchers hired cinematography experts to annotate camera motion in online video clips and train other laypeople to make similar annotations. Even with the criteria in hand, the experts disagreed with "ground truth" ~24% of the time. Only through repeated trials were they able to converge on 96% agreement. It's painful to spend days labeling and re-labeling a dataset, but them's the breaks. At least, we're lucky to live in an era where you only need 2-3K samples to train a good filter. If you find yourself working on data filtration, we'd recommend you hack together a simple labeling tool like the one we show below. We just slapped together a super simple React app with Supabase to store labels and R2 to store the samples. Our internal web-app for data labeling & LLM evals Here, we load an existing dataset and thumb through a couple of examples of images graded "1" (ugliest on our aesthetic scale). The tool needs to make it easy to grep through your existing labels, version control your labels, and edit them, since you're going to take multiple passes on the dataset. - Use Hotkeys: You'll want to label as fast as possible (or you'll go crazy). Make sure you can label via hotkeys and that you can edit hotkey mappings easily within the app itself. - Make Datasets Forkable: You'll be taking several turns on your dataset, so it's helpful to have a fork feature, where you seed a new dataset from your old labels. Even better if you can quickly drill down to the training samples your model misclassified. These are especially problematic. You'll need to review them to iterate on your criteria effectively. Plus, you'll want to relabel them first. - Add tools for label mapping: As you iterate on your ontology, categories will come and go. So, you'll need to make it easy to assign samples that were labeled A to another categoryB , as you add, merge, and delete groupings. Turning LLMs into categorical classifiers Ultimately, we supervise-finetuned (SFT) Qwen-2-VL-2B to tag: - Image Categories: Ugly Product Image ,Diagram / Screenshot ,Collage ,Watermarked ,Bad Lighting ,Pixelated ,Drawing / Illustration ,Keep - Video Categories: Animation ,PoV ,Bars ,Motion Graphics ,Ken Burns ,Shaky Camera ,Little to No Motion ,Weird Transition ,Keep Samples across image categories Ugly Product Image: Product images with messy backgrounds or really flat gray backgrounds (think bad Amazon catalog images). We only kept images that our filter predicted as Drawing / Illustration or Keep . For videos, we retained Animation and Keep clips wholesale, while subsampling PoV . [Late 2025] Reinforcement Learning with Verifiable Rewards (RLVR) for aesthetic filtering At the start of the data labeling process, we tried to get extremely specific about the properties of the images and videos that divvied up samples into ugly vs. pretty (e.g. overexposed lighting, muted color grades). We thought it would be easier for the LLM to learn the precise reasons why we considered an image ugly than learn an arbitrary "ugliness score". Once again, our initial intuition turned out to be wrong. The properties that make a particular sample ugly tend to be correlated; you end up assigning K different aesthetic tags to the same sample. And in turn, this poses two significant challenges: - Sparse Data Signal: The combinatorial explosion of tags makes it harder for the model to disentangle the categories, especially with a small dataset of a few thousand labels. - Slow, Inconsistent Labeling: It takes a lot longer to label samples (and it's a lot harder to be self-consistent) when you have the cognitive load of weighing several possible tags per sample. Fine-grained aesthetic scorers Hang with us, as we work through a short history lesson. The primary way that folks traditionally finetuned LLMs is supervised-finetuning (SFT).Categorize this as A, B, or C , true label) and used the same objective from pre-training (next-word-prediction-with-cross-entropy) to update the model. With SFT, you're essentially extending pre-training, so your model learns your new task. At the start of 2025, DeepSeek popularized a different finetuning technique called "Reinforcement Learning with Verifiable Rewards" (RLVR) in their R1 paper. Instead of using next-word prediction, they used reinforcement learning (RL) where the model's response is graded for accuracy using a rubric. It turns out that if your LLM is able to solve a problem even 1 in 1,000 times, we can nudge the network towards finding this solution more consistently. These network updates are much smaller than those provided during SFT, so RLVR allows us to exert much finer grained control on extending the LLM to our tasks. Bluntly put, SFT is model surgery with a butcher knife, while RLVR is model surgery with a scalpel. Ingredients for aesthetic scoring (RLVR) Please grade the image on a scale of 1 to 4, where 1 is the lowest quality and 4 is the highest quality: - 1Disgusting(Lowest Quality) - 2Ugly(Bad Quality) - 3Good(Acceptable Quality) - 4Beautiful(Highest Quality) Disgusting images are often blurry (out of focus) or pixelated (old image, grainy). The details are hard to see. These images are often overexposed (so much light that details are washed out) or underexposed (too dark to see details). Infographics are disgusting. Ugly images suffer from the same problems as disgusting images, but are qualitatively not as bad. They have low contrast, are poorly lit, or have "flat" looking subjects that do not stand out much from their backgrounds. Small watermarks are ugly if they go across the entire image (e.g. in center). Good images are well lit, have clear contrast, and clear subjects. Beautiful images have high dynamic range (crisp, colorful, sharp contrast, or stylized). They are often portraits or action shots. what the model answered (ŷ) what we labeled it (y) | 1 | 2 | 3 | 4 | non-numeric | | |---|---|---|---|---|---| | 1 | +1 | −1 | −2 | −3 | −4 | | 2 | −1 | +1 | −1 | −2 | −4 | | 3 | −2 | −1 | +1 | −1 | −4 | | 4 | −3 | −2 | −1 | +1 | −4 | By the time we had our aesthetic dataset ready to go, RLVR was gaining adoption by the major labs; so we gave it a go for our aesthetic scorers and it smoked its SFT counterparts.Group Sequence Policy Optimization (GSPO) as the RLVR algorithm on Qwen-2.5-VL-3B. Our prompt explicated the different reasons why we'd grade a sample from 1-4, while our rubric penalized the model on the absolute difference between the ground truth and predicted labels. Specifically, we usedWhen you introduce reasoning to an LLM, it's somewhat standard to first SFT the model with human-written reasoning traces, so it understands how to use the logic before learning to generate its own reasoning. We struggled to explain our aesthetic grades succinctly. So, we nixed the SFT phase altogether and jumped straight to DeepSeek-R1-Zero's techniques for training LLMs how to reason with zero human traces. That didn't work either. Turns out, it's just really hard to describe in words why something is 2 vs. 3 aesthetically. Filtering end-to-end After RLVR, we adopted WAFT (a SOTA optical flow predictor) to filter out even more of the long tail of low-motion videos. Instead of relying on heuristics like Haar-cascades, we converted our video captions into tags and used these to edit our training data distribution (e.g., subsample talking heads, oversample animal videos). Data Filtering Pipeline - 01Candidate pool~15 billion images~15B100% - 02Aspect ratio + min size (256px+)keep images that match 5 target aspect-ratios~7.2B48% - 03Solid color backgroundsdrop low quality product images~5.0B33% - 04PaddleOCR on TensorRTdrop text-heavy images~3.3B22% - 05SFT Filterscategorical Qwen-2-VL-2B classifiers~950M6.3% - 06P-Hash Deduplicationcache perceptual hashes in FAISS, drop duplicates with hamming < 5649M4.3% - 07RLVR Filtersfine-grained aesthetic Qwen-2.5-VL-3B scorers~250M1.7% - kept - thrown out - 01Candidate pool~250 years of footage (~1 billion clips)~1B*100% - 02Aspect ratioretain 16:9 only~800M*80% - 03TransNetV2 & AutoShot shot detectiondrop clips under 2s or over 10s~700M70% - 04H.264 motion vectorsdrop low motion clips~520M52% - 05PaddleOCR on TensorRTdrop text-heavy clips~375M37.5% - 06SFT Filterscategorical Qwen-2-VL-2B classifiers~150M15% - 07RLVR Filtersfine-grained aesthetic Qwen-2.5-VL-3B scorers~65M6.5% - 08WAFT optical flowdrop long tail low motion clips~50M5% - kept - thrown out * Extrapolated from the clip count, before we throw out clips that are too long or too short. Nowadays, there is a lot of jargon being tossed around the internet when it comes to "pre-training", "mid-training", and "post-training". The existence of distinct pre-training and mid-training phases is just a function of the impossibility of creating the perfect filter. During pre-training, folks are more permissive. They're open to letting some low-quality data into the stack, so that they can guarantee that they cover all the necessary modes of their target data distribution. And by the time they get to mid-training, they're willing to tighten the distribution, even if it means throwing away some good stuff. Regardless, data filtering is the single biggest lever you can pull outside of scaling to improve the quality of your model. If you take one thing away from this journey, never skimp on your data filtering pipeline. These things are what they eat after all. Who are we? We're two brothers training text-to-video models from scratch, trying to make animation accessible to everyone. Stay tuned for more blogs on captioning, synthetic data generation, and our ongoing work exploring better manifolds for pixel-space generative models. Get Field Notes Technical deep dives on building generative video models from the ground up, plus updates on new releases from Linum.