Mobile Video Color Transfer Project

For the creation of online visual media, various multimedia software products can be used. Although these editing tasks were previously carried out by artists using software applications such as Adobe Photoshop and Premiere Pro, recent improvements in the affordances of mobile devices and high-speed mobile data networks mean that these editing capabilities are more readily available for mobile devices enabling a broader consumer-base. However, the precise role of the user in creative practice is often neglected in favor of reporting faster, more streamlined device functionality. In this project, using a prototype mobile color transfer application, we seek to identify high-level human-computer interaction issues concerning video recoloring interfaces that are driven by the needs of different user-types via a methodological and explorative process. In this webpage, we provide a collection of additional information for our recent papers:

• M. Grogan*, E. Zerman*, G. W. Young*, A. Smolic. “A pilot study on video color transfer: A survey of user-type opinions.”, in 14th International Conference on Interfaces and Human Computer Interaction, July 2020, Zagreb, Croatia.
• M. Grogan*, E. Zerman*, G. W. Young*, A. Smolic. “A Case Study on Video Color Transfer: Exploring User Motivations, Expectations, and Satisfaction.”, arXiv:2007.08948 [cs.HC], 2020.

Mobile Video Color Transfer Application

For this study, we develop a prototype mobile video color transfer application for devices (e.g. smartphones and tablets) with Android operating system, as described in the paper. This application lets user select the source video and target color palette from pre-selected options (see below). For this pilot study, the prototype was developed only to enable participants to carry out the color transfer operation on mobile devices. Therefore, the selected video and palette are uploaded to a dedicated server, where the color transfer operation is completed. The resulting video is downloaded to the device after the process is done. The user interface can be seen in the table below:

Video selection	Target palette selection	Target palettes	Preview screen	Result screen

Selection of the Source Videos

The source videos were selected from stock video footage websites, for example https://coverr.co/, https://pexels.com/, and https://pixabay.com/. For the replicability of this study, we selected videos with free-to-use or public domain licenses. 16 source videos (see below) were selected with consideration to various qualitative and production-related parameters, such as the type of focus, camera movement, and semantics. Specifically, consideration was given to three common mobile phone camera use scenarios: Personal (i.e., similar to personal recordings which are taken to remember a moment), Professional (i.e., similar to those who are captured for institutions or businesses), and Touristic. These parameters are reported in the table below, including a link to their source. As the videos’ durations were all different only the first 10 seconds of each video was taken to avoid disparities caused by these variations in processing duration. Therefore, the source videos were all 10 seconds in duration.

Name	Snapshot	Video	Theme	Speed	Type of Focus	Camera Movement	Link
Balloon			Touristic	Realtime	All-in-focus	Camera in hand	[Link]
Beach			Professional	Realtime	All-in-focus	Drone	[Link]
Church			Touristic	Realtime	All-in-focus	Tripod	[Link]
City			Professional	Time Lapse	All-in-focus	Still	[Link]
Concert			Personal	Slow Motion	All-in-focus	Camera in hand	[Link]
Dance			Touristic	Realtime	All-in-focus	Tripod	[Link]
Flower			Personal	Realtime	Shallow DoF	Camera in hand	[Link]
Kick			Personal	Realtime	All-in-focus	Camera in hand	[Link]
Meeting			Professional	Realtime	Focused	Camera in hand	[Link]
Road			Personal	Realtime	All-in-focus	Camera in hand	[Link]
Run			Professional	Realtime	All-in-focus	Drone	[Link]
Ski			Personal	Slow motion	All-in-focus	Tripod	[Link]
Sky			Personal	Realtime	All-in-focus	Still	[Link]
Street			Touristic	Realtime	All-in-focus	Still	[Link]
Temple			Touristic	Realtime	Out of focus	Camera in hand	[Link]
Town			Professional	Realtime	All-in-focus	Drone	[Link]

Aside from the parameters reported in the table, we also predetermined that the selected videos would have various quantitative (see the figure below) parameters which would ensure variation over the considered feature space, such as Spatial Perceptual Information (SI) and Temporal Perceptual Information (TI) (ITU-T Rec. P.910), colorfulness (Hasler and Susstrunk, 2013), and key (Valenzise et al., 2014). The results in figure below show that the selected source videos have different features that cover a wide range of scales.

Selection of the Target Palettes

The target palettes were selected with consideration to the color distributions of the color palette representation of the presented (see table below). These color palettes were selected as they have various color themes and various color combinations. Additionally, these color palettes were selected among movies that had specialized pipelines for color grading to convey a specific feeling or mood to the viewer.

300	Alice 1	Alice 2
Alice 3	Amelie	Game of Thrones
Kung Fu Panda	Martian	Saving Pr. Ryan 1
Saving Pr. Ryan 2	The Revenant	Theory of Everything

To analyze the color distribution, we looked at two distinct parameters:

• Variation of the brightness of the pixels in the target image
• Variation in the chroma values of the striped color palettes attached.

For this purpose, the colors were converted from red-gree-blue (RGB) to CIE Lab color space (Robertson, 1976). In the CIE Lab color space, L describes the luma of the pixels whereas a and b describes the chroma of each pixel. To find out the variation of the brightness of the pixels in the target image, the L values found in the target images (i.e., the screenshot frame in the target color palette) are drawn in a boxplot, see the figure below. As can be seen from these boxplots, the target images had varying levels of brightness.

To analyze the variation in the chrominance values of the striped color palettes, we plotted the distribution of a and b values in CIE Lab color space, see the figure below, where each different color and marker combination corresponded to a different color palette. As can be seen from this plot, each of these color palettes has a different color combination which provides users the option to choose from the different color themes. In application, the users have the choice to add other target images from their image gallery. However, to keep the parameters controlled in this study, we only allowed users to select from these predefined 12 target color palettes.

Additional Results

In this part, we provide you with additional color transfer results to give a better idea for the mobile video color transfer. These visual results show how different combinations of source videos and target palettes would result, see the table below.

Source Video Name	Source Video	Target Palette Name	Target Palette	Resulting Video
Balloon		Amelie
Concert		Kung Fu Panda
Dance		Kung Fu Panda
Kick		Amelie
Road		Alice 3
Road		Martian
Run		The Revenant

Acknowledgement

This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under the Grant Number 15/RP/2776.

Contact

If you have any questions, send an e-mail to mgrogan@tcd.ie or zermane@scss.tcd.ie or youngga@tcd.ie.