Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

A Practical Workflow for Fine-Tuning ChatGLM for Vertical Domains

AI summary

A Juejin tutorial presents an end-to-end workflow for adapting ChatGLM3-6B to industry-specific business scenarios using PEFT-LoRA. It covers requirements analysis, model configuration, business-data collection and cleaning, ChatGLM conversation templating, tokenization, train-validation splitting, and PyTorch DataLoader construction, with runnable code intended to prepare data for model training. The article also recommends aligning training and inference prompts and discusses evaluation with human review and Rouge.

In-depth insight

The article consolidates the data-preparation and model-adaptation steps needed to begin domain-specific ChatGLM fine-tuning, including practical code for cleaning, tokenizing, and loading proprietary business data.

ChatGLM行业垂直领域微调实战:从需求分析到数据集加载完整流程

摘要:开源大模型ChatGLM在通用场景表现优秀,但直接落地行业业务往往效果不尽人意。本文完整记录垂直领域微调全流程:项目业务需求分析、技术选型、ChatGLM模型改造思路、数据集建设、原始数据获取、文本预处理Tokenize、DataLoader数据加载器封装,附带完整可运行代码,帮助完成从业务需求到模型训练输入链路的搭建。

前言

ChatGLM系列作为国内开源较早的对话大模型,凭借不错的中文能力,被广泛用于企业内部知识库、客服问答、行业咨询等垂直业务。 通用预训练模型掌握通识知识,但缺少行业专有术语、业务逻辑、内部业务话术,直接提问经常出现答非所问、幻觉、回答不符合业务规范等问题。想要适配业务,就需要基于自有业务数据做微调。

微调完整流水线大致分为: 业务需求分析 → 技术方案选型 → ChatGLM模型针对性改进 → 业务数据获取 → 数据清洗处理 → Tokenizer文本编码 → 构建数据加载器 → 送入模型训练。 本文聚焦前面七大环节,完成训练数据侧完整工程实现。

#ChatGLM3-6B#PyTorch#Hugging Face Transformers#Hugging Face datasets#PEFT#TRL SFTTrainer#bitsandbytes

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.