Dify v1.1.0 New "Metadata" Smart Filter for Knowledge Bases

In the era of information explosion, how to quickly and accurately locate key information from massive data has become the core challenge of enterprise and personal knowledge management. Recently, Dify product team released v1.1.0 version, and innovatively launched the "metadata" as the core of the knowledge filter function. This update is like installing an intelligent navigation system for the knowledge base, which can significantly improve the efficiency and accuracy of data retrieval, and bring users a smoother and more efficient information access experience.

In the past, when faced with a huge knowledge base, users were often like looking for a needle in a haystack, making it difficult to quickly find the information they needed. The introduction of metadata filtering has changed this situation. Metadata, in short, is "data about data". It adds additional descriptive tags and attributes to the original data, such as the creator of the document, date of creation, keywords, etc. With this structured information, the user can find the information they need quickly. With this structured information, users can set specific filtering conditions according to their own needs, so as to quickly lock the target content.

How can metadata filtering empower RAG applications?

For dependencies RAG Metadata filtering is of particular significance for the application of Retrieval-Augmented Generation (RAG) technology. It not only improves the accuracy of information retrieval for RAG applications, but also plays a key role in data security and access control. Imagine an enterprise scenario where different users may need to access information with different privilege levels. With metadata filtering, administrators can easily implement fine-grained permission management to ensure that sensitive information is only available to authorized users.

Further, metadata filtering also optimizes search performance and saves computing resources. With pre-set metadata tags, the system is able to locate relevant documents more quickly and reduce ineffective searches, thus improving overall efficiency. This customization capability is undoubtedly an important efficiency improvement tool for enterprises with massive knowledge bases.

The following diagram visualizes the benefits of metadata filtering for access control. By setting metadata conditions such as privacylevel, uploader and update_date, the system can precisely control the access rights of different users to specific information, e.g. for RAG 2.0 roadmaps.

In a nutshell, metadata filtering is like an intelligent knowledge steward, which enables smarter, secure and efficient information retrieval by adding contextual attributes and access control to data. Especially in RAG systems, the importance of metadata filtering is emphasized when both privacy and relevance of knowledge are crucial.

How is metadata filtering applied in Dify?

exist Dify In v1.1.0, users can easily add and manage metadata for documents in the knowledge base and configure metadata filtering rules for more accurate knowledge retrieval.

Step 1: Adding Metadata to Knowledge Base Documents

In Dify's knowledge base management interface, users can add custom metadata for each document. The system automatically generates some default metadata when a document is created, such as filename, uploader, and date of upload, and you can manually add new metadata fields and customize field names and data types as needed. In addition, users can manually add new metadata fields and customize the field names and data types as needed.Dify supports batch editing and modification of document metadata, making it easy for users to quickly manage and update their knowledge base. This way of "tagging" documents lays the foundation for subsequent refined search and management.

Step 2: Configure metadata filtering in your application

Dify v1.1.0 adds the "Context" section to Chatbot, as well as the Chatflow and Workflow The configuration portal for metadata filtering is provided in all of the knowledge retrieval nodes. Users can choose either automatic or manual filtering modes according to their actual needs. In automatic mode, the system can intelligently analyze user queries and automatically extract and generate filter conditions. The manual mode allows users to customize filtering conditions based on metadata field types (string, numeric, time) and flexibly set "AND" or "OR" relationships between multiple conditions.

The three main metadata types and their application scenarios

Dify v1.1.0 currently supports three types of metadata - string, numeric and time - to meet the knowledge management needs of different scenarios.

• String metadata ： improves search context relevance For example, when a user searches for "project report", the system can prioritize the return of documents tagged with "Marketing" or "R&D", etc., thus filtering out a large amount of irrelevant information and improving the accuracy of search results. For example, when a user searches for "project report", the system can prioritize the return of documents tagged with "Marketing" or "R&D", etc., thus filtering out a large amount of irrelevant information and improving the accuracy of search results.
• Numeric Metadata ： Enhanced Access Control With numeric metadata, access control based on permission levels can be realized. For example, only users with specific permissions can retrieve documents with a privacy level higher than a set threshold, ensuring data security and compliance.
• Temporal Metadata ： Effectively Manage Document Versions Temporal metadata helps users distinguish between old and new versions of documents. After a document is updated and re-uploaded, users can quickly retrieve the latest version through temporal filtering. In addition, combined with the uploader information, time metadata can also facilitate users to compare and analyze different historical versions of the same document to ensure consistency in document processing.

Overall, the metadata filtering feature introduced in Dify v1.1.0 is an important upgrade to the existing knowledge base management solution. It not only improves the efficiency and accuracy of information retrieval, but also provides stronger support in data security and access control. Dify v1.1.0 is a noteworthy update for organizations and developers looking to build intelligent RAG applications. To learn more about the operational details, we recommend checking out Dify's official knowledge base documentation and experiencing the convenience and efficiency of metadata filtering for yourself.

Reference Document ： https://docs.dify.ai/zh-hans/guides/knowledge-base