ComfyUI-Copilot: an AI assistant for text description generation ComfyUI workflows

General Introduction

ComfyUI-Copilot is an AI-driven custom node designed for the ComfyUI framework, aiming to improve the efficiency of AI algorithm debugging and deployment through natural language interaction. It is developed by the AIDC-AI team (Alibaba) and open sourced from GitHub, and is suitable for a variety of tasks such as generating text, images or audio. Whether you are a beginner or a professional user, ComfyUI-Copilot significantly lowers the threshold of use and accelerates the development process through intelligent node recommendations, workflow building assistance, and real-time question answering. The tool combines large language modeling technology to provide model query, parameter optimization and error diagnosis, making it a powerful assistant for ComfyUI users. Currently, it has gained wide attention on GitHub and welcomes community contributions of code and suggestions.

Search for ComfyUI-Copilot in ComfyUI Manager Installation

Function List

• natural language interaction:: Users can communicate with Copilot in everyday language to quickly access node suggestions or workflow scenarios.
• Smart Node Recommendation: Automatically recommend suitable ComfyUI nodes based on task requirements to improve work efficiency.
• Workflow building assistance:: Help users design and optimize complex generation task processes with fewer manual operations.
• Model Query Service:: Support query base model and LoRA model to meet diversified generation needs.
• Automation parameterization:: Use machine learning algorithms to analyze and optimize key workflow parameters.
• Bug diagnosis and fixing suggestions: Provide detailed error analysis and solutions to quickly locate problems.
• Real-time technical support: 24/7 online answers to user questions in development.

Using Help

Installation process

The installation process of ComfyUI-Copilot is simple and straightforward, here are the detailed steps:

1. pre-conditions
   Make sure the ComfyUI main program is installed. If it is not already installed, go toComfyUI Official GitHubDownload and follow its instructions to complete the configuration.
2. clone warehouse
   Open a terminal and go to ComfyUI'scustom_nodesdirectory, then run either of the following commands to clone the ComfyUI-Copilot repository:

cd ComfyUI/custom_nodes
git clone https://github.com/AIDC-AI/ComfyUI-Copilot.git

or use SSH:


git clone git@github.com:AIDC-AI/ComfyUI-Copilot.git

3. Installation via ComfyUI Manager (optional)
   If ComfyUI Manager is already installed, you can open "Custom Nodes Manager" in the interface, search for "ComfyUI-Copilot", and click "Install" to finish the installation automatically. Click "Install" to finish the installation automatically.
4. Start Copilot.
   Once the installation is complete, launch ComfyUI and find the "Copilot Activation Button" in the upper right corner of the interface. Click on it and Copilot service will be activated.
5. Getting the API key
   Visit the official designated link (provided in the repository README), enter your email address, and you will receive the API key automatically sent to you later. Enter the key into the Copilot settings screen to activate full functionality.
6. Verify Installation
   Restart ComfyUI and confirm that the Copilot interface displays normally and responds to natural language input, indicating successful installation.

Function Operation Guide

1. Natural language interaction

• procedure:: Enter a question or requirement into the Copilot interface, such as "How do I generate high-definition images?" or "Recommend a text generation node". or "Recommend a text generation node".
• Function Details: Copilot parses the input and quickly returns node suggestions or operation guidelines. Ideal for beginners to get started quickly or advanced users to optimize complex tasks.
• an actual example: Enter "Help me design an image super-resolution workflow" and Copilot will recommend relevant nodes and provide connection suggestions.

2. Intelligent node recommendation

• procedure:: Describe the mission objective in the Copilot dialog box, e.g. "Need an audio generation node".
• Function Details:: Based on the understanding of the Big Language Model, Copilot will list the adapted nodes with a brief description. For example, the "AudioGen" node is recommended and its usage is suggested.
• an actual example: Input "What is the optimal node for image denoising?" Copilot may suggest "LatentUpscale" and describe its parameter configuration.

3. Workflow construction assistance

• procedure: Enter the complete requirement in the dialog box, e.g. "Create a workflow that generates images from text".
• Function Details: Copilot will guide you step by step to add nodes and connect them, e.g. add "TextInput" first, then connect "StableDiffusion" and "ImageOutput". ImageOutput".
• an actual example: Upon user input, Copilot returns a visual suggestion, and the user can drag and drop the recommended node directly to the ComfyUI canvas.

4. Model query service

• procedure: Enter "Query LoRA models suitable for portrait generation".
• Function Details: Copilot searches and lists available models such as "portraitLoRA" and provides download links or local path suggestions.
• an actual example:: Query results may include model parameter requirements and recommended application scenarios.

5. Automated parameterization

• procedure: Enter "Optimize my image generation parameters" in Copilot.
• Function Details: Copilot analyzes the current workflow, adjusts key parameters (such as the number of sampling steps or CFG values) based on machine learning algorithms, and shows before and after comparisons.
• an actual example:: After the user uploads a workflow, Copilot recommends adjusting the "denoise" value from 0.8 to 0.6 to improve clarity.

6. Error diagnosis and repair recommendations

• procedure: When ComfyUI reports an error, paste the error message into the Copilot dialog box.
• Function Details: Copilot parses the error logs and provides cause analysis and repair recommendations. For example, "insufficient video memory" may suggest lowering the batch size.
• an actual example: Enter "RuntimeError: CUDA out of memory" and Copilot will recommend adjusting the resolution or switching models.

7. Real-time technical support

• procedure: Feel free to ask questions in dialog boxes such as "Why is my output image black?". .
• Function Details: Copilot provides instant answers that may point out that the input node is not configured correctly and give a solution to adjust it.
• an actual example:: Following a user question, Copilot suggests checking that the "CLIP" node is connected to ensure that the text is encoded correctly.

Tips for use

• Ask more questions.: Copilot supports a continuous dialog that allows for step-by-step refinement of requirements.
• Saving workflows: Remember to save the workflow file after completing the design to avoid duplication of operations.
• Community Feedback: If you encounter problems, you can submit an issue on GitHub, and the team and community will respond positively.

With the above steps and functions, users can easily get started with ComfyUI-Copilot and enjoy the convenience of intelligent assistance.