Director: Intelligent Video Agent Framework for Performing Video Search, Editing, and Generation Workflows with Natural Language Descriptions

General Introduction

Director is an open source framework designed to simplify and optimize video interactions and workflows by building intelligent video agents. The framework is based on VideoDB's Video-as-Data infrastructure and is capable of handling complex video tasks such as searching, editing, compiling, and generating, and instantly streaming the results. Users can operate the video agent with simple natural language commands, such as uploading a video and sending highlights to Slack. director is for developers, creators and teams that want to use AI to streamline media workflows and unlock new possibilities.

Function List

• Video Summary: Generate video summaries in seconds.
• Video Generation: Generate full movies with narration from scripts.
• Video Search: Search and index specific moments in the media library.
• Video editing: easily organize and edit content.
• Audio and video editing: easy audio and video dubbing and editing.
• Subtitle Translation: Translate and add subtitles in any language.
• Integration with GenAI projects and APIs: create and edit content on-the-fly.
• Add overlays and generate thumbnails.

Using Help

Installation process

1. Cloning Warehouse:

   git clone https://github.com/video-db/Director.git
   cd Director
   

2. Run the installation script:

   . /setup.sh
   

   This script will install Node.js and Python and set up virtual environments for the front-end and back-end.

3. Configure environment variables:
   compiler .env file, adding API keys and other configuration options.

Usage Process

1. Launch the application:

   make run
   

   • Backend server: http://127.0.0.1:8000
   • Front-end server: http://127.0.0.1:8080
2. Create a new agent:
   • Copying template files sample_agent.py until (a time) director/backend/director/agents/ and renamed.
   • Update the class name, agent name and description.
   • Implementing proxy logic, updating run() Methods.
   • Handles output and status updates using the push_update() Sends a progress event.
   • Register a new agent at Director/backend/director/handler.py Import the new proxy class in the self.agents List.
3. Documentation Services:
   • Local Service Documentation:

     source backend/venv/bin/activate
     make install-be
     mkdocs serve -a localhost:9000
     

   • Build the documentation:

     mkdocs build
     

Guidelines for use

1. Video summary: After uploading a video, use natural language commands to generate a summary of the video. For example, enter "Generate a summary of this video" and the system will automatically process and generate a summary.
2. Video Search: Enter a keyword in the search bar and the system will search and display relevant clips in the video library.
3. video clip: Select a video clip and edit it using the editing tools to generate the desired video clip.
4. Integration with GenAI projects and APIs: Extend video processing capabilities by integrating other AI tools through API interfaces.
5. Adding overlays and generating thumbnails: In the video editing interface, select the option of adding an overlay layer or generating a thumbnail, and the system will handle it automatically.
6. Voice dubbing and subtitle translation: After uploading the video, select the dubbing or subtitle translation function, and the system will automatically generate the corresponding dubbing or subtitle.

Detailed Operation Procedure

• Video Summary Generation::
  1. Upload a video file.
  2. Enter "Generate Video Summary" in the command input box.
  3. After processing by the system, the generated video summary will be displayed in the result area.
• Video Search::
  1. Go to the video search page.
  2. Enter keywords in the search box.
  3. The system searches through the video library and displays matching clips.
• video clip::
  1. Select the video to be edited.
  2. Use the clip tool to select the start and end times.
  3. Click the "Generate Clip" button, the system will generate and save the clip.
• Integration with GenAI projects and APIs::
  1. Go to the API integration page.
  2. Enter the required API key and configuration.
  3. The system automatically integrates and enables new features.
• Adding overlays and generating thumbnails::
  1. On the video editing page, select "Add Overlay" or "Generate Thumbnail".
  2. The system automatically processes and applies the changes.
• Voice dubbing and subtitle translation::
  1. Upload a video file.
  2. Select the "Voice Dubbing" or "Subtitle Translation" function.
  3. The system automatically generates and applies dubbing or subtitling.