Go-with-the-Flow: Controls the movement of objects in the video, adding or subtracting any moving objects in the video.

General Introduction

Go-with-the-Flow is an open source project developed by the Netflix Eyeline Studios research team to control the motion patterns of video diffusion models by distorting the noise. The project allows users to determine how cameras and objects in a scene move, and even transfer motion patterns from one video to another.Go-with-the-Flow accomplishes this by fine-tuning the underlying model without changing the original pipeline or architecture. The project's code and models are open source and can be freely used and modified by users.

ComfyUI node: https://github.com/kijai/ComfyUI-VideoNoiseWarp

Function List

• Shear and drag motion control: Users can create videos by cutting different parts of an image and dragging them around.
• Video proliferation: Converting rough animation to beautiful animation requires the use of a GPU.
• noise distortion: Use distorted noise instead of pure Gaussian noise for more natural motion effects.
• Motion Mode Transfer: Apply motion patterns from one video to another.
• Graphical User Interface (GUI): Provides a native GUI for user-created animations.
• open source: All code and models are open source and can be freely accessed and modified by users.

Using Help

Installation process

1. Cloning Warehouse:

   git clone https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.git
cd Go-with-the-Flow

2. Install local dependencies:

   pip install -r requirements_local.txt

3. Run the GUI:

   python cut_and_drag_gui.py

Follow the instructions in the GUI to complete the operation, and the generated MP4 files need to be transferred to a computer with a GPU for further processing.

Usage Process

1. Clone the repository and install the dependencies on a computer with a GPU:

   git clone https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.git
cd Go-with-the-Flow
pip install -r requirements.txt

2. Distortion Noise:

   python make_warped_noise.py 

3. Running Reasoning:

   python cut_and_drag_inference.py noise_warp_output_folder \
--prompt "A duck splashing" \\
--output_mp4_path "output.mp4" \\
--device "cuda" \
--num_inference_steps 5

Adjust folder paths, cue words, and other hyperparameters as needed. The output will be saved as output.mp4The

Main Functions

• Shear and drag motion control: Select different parts of the image in the GUI and drag them to create rough animations.
• Video proliferation: Transfer rough animation files to a computer with a GPU and run inference scripts to generate beautiful animations.
• noise distortion: Use make_warped_noise.py Script generates distortion noise for more natural motion effects.
• Motion Mode Transfer: Apply motion patterns from one video to another by adjusting cues and hyperparameters.

Detailed Operation Procedure

1. Cut and Drag: Load an image in the GUI and use the mouse to select and drag different parts of the image to create rough animations.
2. Generation of distortion noise: Runs on a computer with a GPU make_warped_noise.py script that generates the distortion noise file.
3. running inference: Use cut_and_drag_inference.py Scripts that convert rough animations into beautiful animations for output to MP4 files.
4. Adjustment parameters: Adjust cues, hyperparameters, and file paths as needed to get the best results.

With the above steps, users can easily create and control the motion patterns of the video for high quality video generation and animation.