HelloMeme: Generate localized high-fidelity expression-action-consistent images or videos, Runway Act one open-source ping-pong!

General Introduction

HelloMeme is an open source project developed by HelloVision to generate high-quality images and videos by integrating Spatial Knitting Attentions to embed high-level and high-fidelity conditions in diffusion models. The project's code and model implementation are hosted on GitHub and can be freely downloaded and used by users. helloMeme provides a variety of functional modules, including image generation, video generation, and integration with Gradio and ComfyUI The integration is user-friendly for a variety of experiments and applications.

Related:

LivePortrait: Animation tool for generating dynamic portraits from still images and videos(open source)

ConsisID: a portrait reference map to generate character-consistent video, rapid multi-terminal integration(open source)

Reface: face replacement and video generation using AI to create fun animated images (paid)

Function List

• Image Generation: Generate high quality images from reference and driver images.
• Video Generation: Generate high-fidelity videos based on reference and driver videos.
• Gradio Integration: Provides a Gradio interface for user interaction.
• ComfyUI Integration: Supports ComfyUI interface to simplify the user operation process.
• Experimental Modules: Includes a variety of experimental codes to facilitate users to conduct different experiments and tests.
• efficient algorithm: Optimize algorithms to reduce VRAM usage and improve generation efficiency.

Using Help

Installation process

1. Creating a Conda Environment::

   conda create -n hellomeme python=3.10.11
conda activate hellomeme

2. Installing PyTorch and FFmpeg: Please refer to the official PyTorch and FFmpeg websites for detailed installation.
3. Installation of dependencies::

   pip install diffusers transformers einops scipy opencv-python tqdm pillow onnxruntime onnx safetensors accelerate peft

4. clone warehouse::

   git clone https://github.com/HelloVision/HelloMeme
cd HelloMeme

5. running code::
   • Image Generation: bash
python inference_image.py

   • Video Generation: bash
python inference_video.py

6. Install the Gradio application::

   pip install gradio
pip install imageio[ffmpeg]
python app.py

Functional operation flow

Image Generation

1. Preparing the input image: Prepare the reference image and drive image.
2. Run the image generation script::

   python inference_image.py --reference_image path/to/reference.jpg --drive_image path/to/drive.jpg

3. View Generated Results: The generated image will be saved in the specified directory.

Video Generation

1. Preparing to input video: Prepare reference videos and driver videos.
2. Run the video generation script::

   python inference_video.py --reference_video path/to/reference.mp4 --drive_video path/to/drive.mp4

3. View Generated Results: The generated video will be saved in the specified directory.

Using the Gradio Interface

1. Launching the Gradio Application::

   python app.py

2. Access via browser: Open your browser and visit http://localhost:7860The Gradio interface can be used for image and video generation.

Using the ComfyUI Interface

1. Install ComfyUI: Please refer to the official ComfyUI website for installation.
2. Run HelloMeme::

   python comfyui_hellomeme.py

3. Operation via ComfyUI interface: Select the image or video generation function as required, upload the reference and driver files and click the Generate button.

caveat

• dependency version: Please note the diffusers version requirements, currently supported versions are diffusers==0.31.0The
• VRAM Usage: When generating a video, the longer the drive video is, the more VRAM is required, so please adjust it according to the actual situation.
• Model Download: On the first run, all models will be downloaded automatically, which may take a long time.

The layman's guide to getting started with AI

Help you learn how to utilize AI tools at a low cost and from a zero base.AI, like office software, is an essential skill for everyone. Mastering AI will give you an edge in your job search and half the effort in your future work and studies.

View Details>