Introduction

Oflibpytorch cover image

Oflibpytorch: a handy python optical flow library, based on PyTorch tensors, that enables the manipulation and combination of flow fields while keeping track of valid areas (see “Usage”). It is mostly code written from scratch, but also contains useful wrappers for specific functions from libraries such as PyTorch’s grid_sample, to integrate them with the custom flow field class introduced by oflibpytorch. If you use this code, please acknowledge us with the following citation:

@article{ravasio_oflib,
  title     = {oflibnumpy {\&} oflibpytorch: Optical Flow Handling and Manipulation in Python},
  author    = {Ravasio, Claudio S. and Da Cruz, Lyndon and Bergeles, Christos},
  journal   = {Journal of Open Research Software (JORS)},
  year      = {2021},
  volume    = {9},
  publisher = {Ubiquity Press, Ltd.},
  doi       = {10.5334/jors.380}
}

An equivalent flow library based on NumPy arrays exists. Its code is on GitHub, and the documentation can be found on ReadTheDocs. Oflibpytorch is aimed at allowing the same operations to be performed with torch tensors instead of numpy arrays as far as currently feasible, and on the GPU if required.

Features:

  • Provides a custom flow field Flow class for both forwards and backwards (‘source’ / ‘target’ based) flow fields

  • Provides a number of class methods to create flow fields from lists of affine transforms, or a transformation matrix

  • Provides a number of functions to resize the flow field, visualise it, warp images, find necessary image padding

  • Provides a class method to process three different types of flow field combination operations

  • Keeps track of valid flow field areas through said operations

  • Provides alternative functions to avoid the explicit use of the custom flow class, with slightly limited functionality

Installation:

In order for oflibpytorch to work, the python environment needs to contain a PyTorch installation. To enable GPU usage, the CUDA Toolkit is required as well. As it is difficult to guarantee an automatic installation via pip will use the correct versions and work on all operating systems, it is left to the user to install pytorch and cudatoolkit independently. The easiest route is a virtual conda environment and the recommended install command from the PyTorch website, configured for the user’s specific system. To install oflibpytorch itself, use the following command:

pip install oflibpytorch

Testing:

In the command line, navigate to the folder oflibpytorch/test and run:

python -m unittest discover .

Code example:

import oflibpytorch as of
# Make a flow field and display it
shape = (300, 400)
flow = of.Flow.from_transforms([['rotation', 200, 150, -30]], shape)
flow.show()
Visualisation of optical flow representing a rotation
# Combine sequentially with another flow field, display the result
flow_2 = of.Flow.from_transforms([['translation', 40, 0]], shape)
result = flow.combine_with(flow_2, mode=3)
result.show(show_mask=True, show_mask_borders=True)
Visualisation of optical flow representing a rotation, translated to the right
result.show_arrows(show_mask=True, show_mask_borders=True)
Visualisation of optical flow representing a rotation, translated to the right