User:TacticBucket/Optical flow

Instead of seeking to model optical flow directly, one can train a machine learning system to estimate optical flow. Since 2015, when FlowNet^[1] was proposed, learning based models have been applied to optical flow and have gained prominence. Initially, these approaches were based on Convolutional Neural Networks arranged in a U-Net architecture, often utilizing encoder-decoder or feature pyramid structures, such as PWC-Net^[2], which integrated cost volumes (a 4D tensor representing the matching costs between all pairs of pixels in two feature maps) and warping (the process of spatially transforming one image based on a predicted flow field) to refine flow estimates across multiple scales. However, with the advent of transformer architecture in 2017, transformer based models have gained prominence.^[3] A significant shift occurred with the introduction of RAFT^[4] (Recurrent All-Pairs Field Transforms), which replaced coarse-to-fine pyramids with a single GRU-based state that iteratively updates the flow field. By maintaining a constant feature resolution at 1/8 of the input, RAFT significantly improved the preservation of fine details and robustness to fast motion compared to previous bottleneck-heavy designs, influencing a wide range of subsequent models that adopt similar iterative update mechanisms.

However, the all-pairs correlation used in such models is computationally expensive; for high-resolution content like FullHD or 4K, global matching can require more than 32 GB of VRAM^[5], making it impractical for consumer-grade GPUs. To address this, efficiency-focused methods such as Flow1D^[5], MeFlow^[6], and Memfof^[7] have been developed. While these approaches generally optimize memory usage by decomposing the 2D search space, the latter optimizes correlation volumes for high-resolution multi-frame sequences, providing a practical implementation for standard GPUs.

Most learning-based approaches to optical flow use supervised learning. In this case, many frame pairs of video data and their corresponding ground-truth flow fields are used to optimise the parameters of the learning-based model to accurately estimate optical flow. This process often relies on vast synthetic training datasets, such as FlyingChairs^[1] and FlyingThings3D^[8], due to the number of parameters involved.^[9] The models are then evaluated on benchmarks like MPI Sintel^[10], KITTI^[11], and the high-resolution Spring^[12] dataset. However, models trained exclusively on synthetic data often struggle with the domain gap when applied to real-world footage.

To address this, some learning-based optical flow approaches use self-supervised learning (sometimes called unsupervised learning) to reduce the need for large datasets with ground-truth data and leverage real-world footage without labels during training. Instead of training models to minimise the differences between estimated and ground-truth flow fields, they are trained to achieve learning objectives such as brightness constancy and smoothness of the flow field.^[13] More recently, methods like CroCo^[14] have introduced cross-view completion pre-training. Forcing the network to predict the masked regions of one image using the full second image teaches the model a strong geometric understanding and better generalization than models trained solely on task-specific labels.