Continual Unsupervised Learning for Optical Flow Estimation

We present an extensive study on how neural networks can learn to estimate optical flow in a continual manner while observing a long video stream and reacting online to the streamed information without any further data buffering. To this end, we rely on photo-realistic video streams that we specifically created using 3D virtual environments, as well as on a real-world movie.

Our analysis considers important model selection issues that might be easily overlooked at a first glance, comparing different neural architectures and also state-of-the-art models pretrained in an offline manner.

Our results not only show the feasibility of continual unsupervised learning in optical flow estimation, but also indicate that the learned models, in several situations, are comparable to state-of-the-art offline-pretrained networks. Moreover, we show how common issues in continual learning, such as catastrophic forgetting, do not affect the proposed models in a disruptive manner, given the task at hand.

• Stream A: the simplest one, it features translation and rotation of a single object that never leaves the view, similar to FlyingChairs but with pseudo-realistic rendering.
• Stream B: it features four different objects randomly moving in a realistic scenario.
• Stream C: similar to B, even if the camera is now moving. This stream is much more complex, since it features both very slow movements of objects far from the camera and fast motion patterns from close objects/surfaces.

In our work, we also consider “1917”, a 2019 British war film directed and produced by Sam Mendes. The film has a duration of 119 minutes (approx. 103 without credits) and it appears as a single long continuous take, without artificial cuts.

Resources

• Technical report
• Code repository
• Pre-rendered continual streams

@techreport{marullo2022colof,
      title={Continual Unsupervised Learning for Optical Flow Estimation with Deep Networks}, 
      author={Simone Marullo and Matteo Tiezzi and Alessandro Betti 
      and Lapo Faggi and Enrico Meloni and Stefano Melacci},
      year={2022}
}