【解密源码】WeKnora 文档切分与 Chunk 构建解析:腾讯生产级 RAG 的底层设计
引言
WeKnora 是腾讯开源的一套生产级 RAG 框架,定位非常明确:解决真实业务场景下“文档复杂、类型多样、规模可控但质量要求极高”的知识增强问题。社区中有人将其视为 ima 的开源实现之一,虽然这一说法无从官方考证,但可以确定的是,WeKnora 在工程完整度、边界处理和异常降级策略上,是一套经过实战打磨的系统方案。
从文档接入、解析、切分、向量化、多模态增强,到知识图谱、问题生成与摘要生成,WeKnora 几乎覆盖了一个完整 RAG 系统在生产环境中可能遇到的所有关键问题。尤其是在文档解析与 Chunk 构建这一最容易被低估、却最影响检索与生成质量的环节,WeKnora 给出了一套相当成熟且可复用的设计。
本文将聚焦 WeKnora 的文档接入与解析体系,从文件/URL/手动创建三种入口开始,深入拆解其解析器架构、Markdown 统一中间表示、语义切分策略、多模态处理,以及最终如何构建可用于检索与推理的 Chunk 数据结构。
上传方式
文件上传
上传文件 → 计算Hash去重 → 存储文件 → 异步解析
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line// Service 层 - 核心逻辑func (s *knowledgeService) CreateKnowledgeFromFile(ctx context.Context, kbID string,file *multipart.FileHeader, metadata map[string]string, enableMultimodel *bool, customFileName string,) (*types.Knowledge, error) {// 1. 计算文件 Hash(去重检测)fileContent, _ := file.Open()fileHash := calculateFileHash(fileContent)// 2. 检查是否已存在相同文件existing, _ := s.repo.FindByFileHash(ctx, kbID, fileHash)if existing != nil {return existing, &types.DuplicateKnowledgeError{ExistingID: existing.ID,Message: "相同文件已存在",}}// 3. 上传文件到存储服务(MinIO/COS)filePath, _ := s.storage.Upload(fileContent, fileName)// 4. 创建 Knowledge 记录knowledge := &types.Knowledge{ID: uuid.New().String(),Type: "file",FileName: fileName,FileType: getFileType(fileName), // pdf, docx, xlsx...FileHash: fileHash,FilePath: filePath,FileSize: file.Size,ParseStatus: types.ParseStatusPending, // 等待异步处理Metadata: metadata,}s.repo.CreateKnowledge(ctx, knowledge)// 5. 入队异步解析任务payload := types.DocumentProcessPayload{KnowledgeID: knowledge.ID,EnableMultimodel: enableMultimodel,}task := asynq.NewTask(types.TypeDocumentProcess, payload)s.task.Enqueue(task)return knowledge, nil}
URL 创建
接收URL → URL去重 → 异步抓取解析
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line// Service 层 - 核心逻辑func (s *knowledgeService) CreateKnowledgeFromURL(ctx context.Context,kbID, url string, enableMultimodel *bool, title string,) (*types.Knowledge, error) {// 1. 验证 URL 格式if !isValidURL(url) {return nil, errors.NewBadRequestError("无效的URL格式")}// 2. 检查是否已存在相同 URLexisting, _ := s.repo.FindBySourceURL(ctx, kbID, url)if existing != nil {return existing, &types.DuplicateKnowledgeError{ExistingID: existing.ID,Message: "相同URL已存在",}}// 3. 创建 Knowledge 记录knowledge := &types.Knowledge{ID: uuid.New().String(),Type: "url",SourceURL: url,Title: title,FileName: extractFileNameFromURL(url), // 从 URL 提取文件名FileType: "html",ParseStatus: types.ParseStatusPending, // 等待异步处理}s.repo.CreateKnowledge(ctx, knowledge)// 4. 入队异步抓取任务payload := types.DocumentProcessPayload{KnowledgeID: knowledge.ID,SourceURL: url, // 传递 URL 供异步任务抓取EnableMultimodel: enableMultimodel,}task := asynq.NewTask(types.TypeDocumentProcess, payload)s.task.Enqueue(task)return knowledge, nil}
手动创建
接收Markdown内容 → 无去重 → 同步处理
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line// Service 层 - 核心逻辑(同步处理)func (s *knowledgeService) CreateKnowledgeFromManual(ctx context.Context,kbID string, req *types.ManualKnowledgePayload,) (*types.Knowledge, error) {// 1. 创建 Knowledge 记录knowledge := &types.Knowledge{ID: uuid.New().String(),Type: "manual",Title: req.Title,FileName: req.Title + ".md",FileType: "md",ParseStatus: types.ParseStatusProcessing, // 直接开始处理}s.repo.CreateKnowledge(ctx, knowledge)// 2. 同步解析 Markdown 内容(无需异步任务)chunks := s.parseMarkdown(req.Content)// 3. 同步处理 Chunks(生成 embedding、索引)s.processChunks(ctx, kb, knowledge, chunks)// 4. 更新状态为完成knowledge.ParseStatus = types.ParseStatusCompleteds.repo.UpdateKnowledge(ctx, knowledge)return knowledge, nil}
解析模式
FirstParser - 链式尝试模式
尝试多个解析器,直到第一个成功
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineclass FirstParser(BaseParser):_parser_cls: Tuple[Type["BaseParser"], ...] = ()def parse_into_text(self, content: bytes) -> Document:"""顺序尝试每个解析器"""for p in self._parsers:try:document = p.parse_into_text(content)if document.is_valid():return document # ← 第一个成功就返回except Exception:continue # ← 失败就继续下一个return Document() # ← 都失败就返回空
PipelineParser - 管道链式模式
多个解析器串联处理,前一个的输出是后一个的输入
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineclass PipelineParser(BaseParser):_parser_cls: Tuple[Type["BaseParser"], ...] = ()def parse_into_text(self, content: bytes) -> Document:"""依次调用每个解析器,累积图片"""images: Dict[str, str] = {}document = Document()for p in self._parsers:document = p.parse_into_text(content)content = endecode.encode_bytes(document.content) # ← 转换为下一个解析器的输入images.update(document.images) # ← 累积图片document.images.update(images)return document
文档解析
pdf 文档处理
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# 尝试顺序:# 1. MinerUParser (主解析器)# 2. MarkitdownParser (备选解析器)class PDFParser(FirstParser):_parser_cls = (MinerUParser, MarkitdownParser)# 管道流程:# PDF → StdMinerUParser (调用 MinerU API)# → MarkdownTableFormatter (格式化表格)# → 最终 Documentclass MinerUParser(PipelineParser):_parser_cls = (StdMinerUParser, MarkdownTableFormatter)class MarkitdownParser(PipelineParser):"""使用管道处理模式的 Markdown 解析器数据流:PDF 字节流↓StdMarkitdownParser (第一阶段)├─ 调用 MarkItDown 库解析├─ 返回 Markdown 文本 + 内嵌数据 URI└─ Document(content, images={})↓MarkdownParser (第二阶段)├─ 处理 Markdown 内容├─ 提取数据 URI 中的图片├─ 上传到存储└─ 返回最终 Document(content, images)"""_parser_cls = (StdMarkitdownParser, MarkdownParser)# 两个解析器按顺序管道处理
优先使用 MinerU 将文件转换成 markdown 格式,这里需要用户自行配置 MinerU token 才能正常使用。降级采用微软开源框架 MarkItDown,将 pdf 转换成 markdown 格式,MarkItDown 插件处理不了 pdf 扫描件。转换成 markdown 之后,对 markdown 进行初步格式化处理:
对 markdown 文档的表格空行进行规范化处理,标准化列对齐 将 markdown 文档中的 base64 图片转换为二进制上传到 COS,将 COS 链接替换 markdown 文档中的 base64 图片路径
docx 文档处理
ounter(lineounter(lineounter(lineounter(lineounter(line# 尝试顺序:# 1. MarkitdownParser (主解析器,与 pdf 备选方案一致)# 2. DocxParser (备选解析器)class Docx2Parser(FirstParser): _parser_cls = (MarkitdownParser, DocxParser)
降级策略 DocxParser 采用 python-docx,最终输出 markdown 格式的文本内容。处理流程如下:
通过 python-docx 库解析 docx 文件,提取纯文本内容,表格和图片。 若开启了图片处理 enable_multimodal,则对提取的图片进行过滤,缩放等处理后上传至 COS,再将 COS 链接替换 markdown 文档中的图片路径
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# 过滤小图片(装饰元素)if image.width < 50 or image.height < 50:return None# 缩放大图片if image.width > max_image_size:image = image.resize((new_width, new_height))
将表格转换为 HTML 格式
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(linedef _convert_table_to_html(self, table): html = "<table>" for r in table.rows: html += "<tr>" for c in r.cells: # 处理合并单元格 html += f"<td colspan='{span}'>{c.text}</td>" html += "</tr>" html += "</table>" return html保持原文内容顺序进行 markdown 格式文档输出。
ounter(lineounter(lineounter(lineounter(line@dataclassclass DocumentModel: content: str # Markdown 格式文本(含图片链接) images: Dict[str, str] # {url: base64_data}doc 文档处理
doc 文档处理优先使用 _parse_with_docx 方法,采用 LibreOffice 库将 .doc 文件转换为 .docx 格式,再按照 Docx2Parser 流程解析文件。
若 _parse_with_docx 方法失败,将会使用降级方案 _parse_with_antiword,采用 antiword 库直接提取 .doc 文本内容。antiword 只能提取纯文本,不支持图片和表格的提取。
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineclass DocParser(Docx2Parser):"""DOC document parser"""def parse_into_text(self, content: bytes) -> Document:logger.info(f"Parsing DOC document, content size: {len(content)} bytes")handle_chain = [# 1. Try to convert to docx format to extract imagesself._parse_with_docx,# 2. If image extraction is not needed or conversion failed,# try using antiword to extract textself._parse_with_antiword,# 3. If antiword extraction fails, use textract# NOTE: _parse_with_textract is disabled due to SSRF vulnerability# self._parse_with_textract,]
csv 文档处理
使用 pandas 库读取 csv 文件,将其转换为 DataFrame 格式,再将 DataFrame 转换为文本内容。会默认将 DataFrame 的第一行作为表头,将其他行作为数据行进行组合。
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# Read CSV content into a pandas DataFrame, skipping malformed linesdf = pd.read_csv(BytesIO(content), on_bad_lines="skip")# Process each row in the DataFramefor i, (idx, row) in enumerate(df.iterrows()):# Format row as "column: value" pairs separated by commascontent_row = (",".join(f"{col.strip()}: {str(row[col]).strip()}" for col in df.columns)+ "\n")# Update end position for this chunkend += len(content_row)text.append(content_row)# Create a chunk for this row with position trackingchunks.append(Chunk(content=content_row, seq=i, start=start, end=end))# Update start position for next chunkstart = endreturn Document(content="".join(text),chunks=chunks,)
最终输出格式示例如下:
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineDocument( content="姓名: 张三,年龄: 25,城市: 北京\n姓名: 李四,年龄: 30,城市: 上海\n...", chunks=[ Chunk(content="姓名: 张三,年龄: 25,城市: 北京\n", seq=0, start=0, end=25), Chunk(content="姓名: 李四,年龄: 30,城市: 上海\n", seq=1, start=25, end=50), Chunk(content="姓名: 王五,年龄: 28,城市: 深圳\n", seq=2, start=50, end=75), ])
Tips: 这种默认第一行为表头的策略能够最大程度的保持 csv 文档的结构。但对于第一行不是表头的情况,也会有混乱语义的危险。
excel 文档处理
使用 openpyxl 库读取 excel 文件,将每个工作表转换为文本内容。与 csv 处理策略相同,会默认将每个工作表的第一行作为表头,将其他行作为数据行进行组合。
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# Load Excel file from bytes into pandas ExcelFile objectexcel_file = pd.ExcelFile(BytesIO(content))# Process each sheet in the Excel filefor excel_sheet_name in excel_file.sheet_names:# Parse the sheet into a DataFramedf = excel_file.parse(sheet_name=excel_sheet_name)# Remove rows where all values are NaN (completely empty rows)df.dropna(how="all", inplace=True)# Process each row in the DataFramefor _, row in df.iterrows():page_content = []# Build key-value pairs for non-null valuesfor k, v in row.items():if pd.notna(v): # Skip NaN/null valuespage_content.append(f"{k}: {v}")# Skip rows with no valid contentif not page_content:continue# Format row as comma-separated key-value pairscontent_row = ",".join(page_content) + "\n"end += len(content_row)text.append(content_row)# Create a chunk for this row with position trackingchunks.append(Chunk(content=content_row, seq=len(chunks), start=start, end=end))start = end# Combine all text and return as Documentreturn Document(content="".join(text), chunks=chunks)
Tips: pandas 处理 excel 文档,如果遇见多级表头,多个合并单元格的复杂文档,会丢失结构信息,导致语义混乱。
图片处理
图片处理很简单,上传图片到存储服务, 将返回的图片 URL 转换为 Markdown 图片语法,并生成 base64 编码。
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(linedef parse_into_text(self, content: bytes) -> Document:logger.info(f"Parsing image content, size: {len(content)} bytes")# Get file extensionext = os.path.splitext(self.file_name)[1].lower()# Upload image to storageimage_url = self.storage.upload_bytes(content, file_ext=ext)logger.info(f"Successfully uploaded image, URL: {image_url[:50]}...")# Generate markdown texttext = f""images = {image_url: base64.b64encode(content).decode()}# Create image object and add to mapreturn Document(content=text, images=images)
网页文档处理
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# 管道流程:# URL → StdWebParser (获取网页内容)# → MarkdownParser (格式化 Markdown 内容)# → 最终 Documentclass WebParser(PipelineParser): _parser_cls = (StdWebParser, MarkdownParser)
使用 Playwright 抓取页面内容
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineasync def scrape(self, url: str) -> str:async with async_playwright() as p:# 配置代理(可选)kwargs = {}if self.proxy:kwargs["proxy"] = {"server": self.proxy}# 启动 WebKit 浏览器browser = await p.webkit.launch(**kwargs)page = await browser.new_page()# 访问页面,30秒超时await page.goto(url, timeout=30000)# 获取完整 HTMLcontent = await page.content()await browser.close()return content
使用 Trafilatura 提取正文
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(linedef parse_into_text(self, content: bytes) -> Document:url = endecode.decode_bytes(content)# 抓取 HTMLchtml = asyncio.run(self.scrape(url))# 提取正文,转为 Markdownmd_text = extract(chtml,output_format="markdown", # 输出格式with_metadata=True, # 保留元数据include_images=True, # 保留图片include_tables=True, # 保留表格include_links=True, # 保留链接)return Document(content=md_text)
通过 StdWebParser 解析器,输出 markdown 文档,包含元数据(如标题、作者、发布日期等)和图片、表格、链接等。进入后续 MarkdownParser 解析器,将 markdown 文档转换为 Document 格式。
Markdown 文档处理(核心)
以下是对文档处理的核心方案,因为以上所有文档类型都是先转换成对应的 Markdown 文档结构,再进行统一处理。
核心架构
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line原始 Markdown 文本↓TextSplitter.split_text(text)├─ Step 1: _split(text)│ └─ 递归分割,保证每个 split ≤ chunk_size│├─ Step 2: _split_protected(text)│ └─ 提取所有受保护内容的位置和范围│├─ Step 3: _join(splits, protect)│ └─ 合并 splits 和 protected,保证受保护内容完整│└─ Step 4: _merge(splits)├─ 合并 splits 成为最终的 chunks├─ 处理 overlap(重叠)└─ 返回 List[Tuple[start, end, text]]返回:List[(start_pos, end_pos, chunk_text), ...]
主函数 split_text()
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(linedef split_text(self, text: str) -> List[Tuple[int, int, str]]:if text == "":return []# Step 1: Split text by separators recursivelysplits = self._split(text)# Step 2: Extract protected content positionsprotect = self._split_protected(text)# Step 3: Merge splits with protected content to ensure integritysplits = self._join(splits, protect)# Verify that joining all splits reconstructs the original textassert "".join(splits) == text# Step 4: Merge splits into final chunks with overlapchunks = self._merge(splits)return chunks
关键文本保护
受保护的 Regex 模式(6 种),分别是LaTeX 数学公式,Markdown 图片链接,Markdown 普通链接,Markdown 表格表头(Header + 分隔行),Markdown 表格内容,代码块头(带语言标识),对于这些场景不进行截断
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineprotected_regex: List[str] = [ # math formula - LaTeX style formulas enclosed in $$ r"\$\$[\s\S]*?\$\$", # image - Markdown image syntax  r"!\[.*?\]\(.*?\)", # link - Markdown link syntax [text](url) r"\[.*?\]\(.*?\)", # table header - Markdown table header with separator line r"(?:\|[^|\n]*)+\|[\r\n]+\s*(?:\|\s*:?-{3,}:?\s*)+\|[\r\n]+", # table body - Markdown table rows r"(?:\|[^|\n]*)+\|[\r\n]+", # code header - Code block start with language identifier r"```(?:\w+)[\r\n]+[^\r\n]*",],语义递归粗切 -- _split()
ounter(lineounter(lineounter(lineounter(lineounter(lineounter(lineounter(line# 语义规则self._split_fns = [ split_by_sep("\n"), # [1] 换行符(优先级最高) split_by_sep("。"), # [2] 句号 split_by_sep(" "), # [3] 空格 split_by_char() # [最后] 逐字分割(fallback)]ounter(lineounter(lineounter(l…