📃 License • 👨💻 Github • 📑 Paper
🎯 Introduction
Youtu-HiChunk is a hierarchical document chunking framework developed by Tencent Youtu Lab. Combined with the Auto-Merge retrieval algorithm, it can dynamically adjust the semantic granularity of retrieval fragments, mitigating issues of incomplete information caused by chunking.
-
Hierarchical Document Structuring HiChunk is a hierarchical document structuring framework designed to address the limitations of traditional linear chunking methods in RAG systems. It focuses on modeling multi-level semantic granularity (e.g., sections, subsections, paragraphs) rather than flat text sequences, enabling RAG systems to retrieve information at contextually appropriate abstraction levels.
-
Auto-Merge Retrieval Algorithm Auto-Merge Retrieval Algorithm dynamically adjusts chunk granularity via three complementary conditions, balancing semantic completeness and retrieval quality for both evidence-dense and sparse tasks.
📊 Performance
1. RAG piepline performance
2. Performance in various retrieval size
🚀 Quick Start
Install packages
uv venv hichunk --python 3.12
source hichunk/bin/activate
uv pip install torch==2.7.0 vllm==0.9.1 transformers==4.53.0 liger_kernel
uv pip install nltk
python -c "import nltk; nltk.download('punkt_tab')"
Then, you can deploy HiChunk model according link.
Usage
import os
os.environ['OPENAI_BASE_URL'] = "http://{serve_ip}:{serve_port}"
from HiChunk import HiChunkInferenceEngine, PROMPT
engine = HiChunkInferenceEngine(window_size=16*1024, line_max_len=100, max_level=10, prompt=PROMPT)
document_text = open('doc.txt', 'r').read()
chunked_document, chunks = engine.inference(document_text, recurrent_type=2)
print(chunked_document)
🎨 Visualization
Case 1
Case 2
🤝 Acknowledgements
The project is based on the excellent work of several open source projects:
📚 Citation
If you find our work useful in your research, please consider citing the following paper:
@misc{hi-chunk-2025,
title={HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking},
author={Tencent Youtu Lab},
year={2025},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/TencentYoutuResearch/HiChunk.git}},
}