SVFR: A Unified Framework for Realizing Video Face Repair, Repairing Black and White, Blurry Portrait Old Videos

General Introduction

SVFR (Stable Video Face Restoration) is a unified framework for video face restoration that supports Basic Face Restoration (BFR), colorization, repair, and their combination tasks. The framework utilizes generative and motion priors to integrate task-specific information through a unified face restoration framework to improve restoration quality and temporal stability.SVFR is designed to address the challenges of temporal consistency, motion artifacts, and limited high-quality video data in video face restoration. The project was developed by Zhiyao Wang et al. and is open-sourced on GitHub, where the code and pre-trained models are available for academic and non-commercial research.

Function List

• Basic Face Restoration (BFR): Fixes face details in low-quality videos.
• coloring: Add color to black and white video.
• fixes: Fix the missing parts of the video.
• Portfolio of tasks: Supports any combination of BFR, coloring, and restoration.
• time consistency: Enhancing the temporal consistency of video restoration through generation and motion prior.
• Pre-trained models: Provide a variety of pre-trained models to facilitate users to get started quickly.

Using Help

Installation process

1. Creating a Conda Environment::

   conda create -n svfr python=3.9 -y
conda activate svfr

2. Installing PyTorch: Choose the right CUDA version for your hardware, for example:

   pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2

3. Installation of dependencies::

   pip install -r requirements.txt

4. Download Checkpoints: Manually download and place in the specified directory.

Usage Process

1. single-task reasoning::

   python3 infer.py --config config/infer.yaml --task_ids 0 --input_path . /assert/lq/lq1.mp4 --output_dir . /results/

• task_id: 0 Denotes Basic Face Restoration (BFR)
• task_id: 1 Indicates coloring
• task_id: 2 Indicates restoration

1. multitask combinatorial reasoning::

   python3 infer.py --config config/infer.yaml ---task_ids 0,1,2 --input_path . /assert/lq/lq3.mp4 --output_dir . /results/

• task_ids: 0,1 Indicates BFR and coloring
• task_ids: 0,1,2 Indicates BFR, coloring and restoration

1. Reasoning with repair masks::

   python3 infer.py --config config/infer.yaml ---task_ids 0,1,2 --input_path . /assert/lq/lq3.mp4 --output_dir . /results/ ---mask_path . /assert/mask/lq3.png

Detailed function operation flow

1. Basic Face Restoration (BFR)::
   • Select the input video path and output directory.
   • Set the task ID to 0.
   • Run the reasoning command to generate the repaired video.
2. coloring::
   • Select the input video path and output directory.
   • Set the task ID to 1.
   • Run the inference command to generate the colorized video.
3. fixes::
   • Select the input video path and output directory.
   • Set the task ID to 2.
   • Run the reasoning command to generate the repaired video.
4. Portfolio of tasks::
   • Select the input video path and output directory.
   • Set the task ID to the ID of the combined task, e.g. 0,1 for BFR and coloring.
   • Run the inference command to generate the video after the combined task processing.
5. Reasoning with repair masks::
   • Select the input video path, output directory and mask file path.
   • Set the task ID to the ID of the combined task, e.g. 0,1,2 for BFR, Coloring and Repair.
   • Run the inference command to generate the video with the repair mask processed.