Extending soft segmentation for images to video sequences

Proposed by Mairéad Grogan
Email: mgrogan at tcd.ie

Decomposing an image into layers is an important tool when applying further processing to an image, and allows different segments of the image to be processed independently.  For example, recolouring can be applied to each layer independently, some image layers can be removed and others overlaid onto a new background etc. One soft segmentation approach that has been proposed recently decomposes an image into layers, with each layer associated with a dominant colour in the image [1]. For a given input image, n dominant colours in the image are first estimated. Each pixel in the image is then analysed, and it’s RGB colour defined as a linear combination of the n dominant colours. This is used to compute the contribution of each pixel to the n image layers.

While this approach has been investigated for images, its extension to video content has not been explored. This project would investigate ways that this soft-segmentation approach could be extended to video data. Temporal consistency of layers would need to be maintained, as well as the introduction of new colours, changing lighting, changing content etc.

Reference: Unmixing-Based Soft Color Segmentation for Image Manipulation, Yagiz Aksoy, Tunc Ozan Aydin, Aljosa Smolic, Marc Pollefeys, ACM Transactions on Graphics 2017.