Overview
What is Liu?
Liu is a general‑purpose, flexible and easy‑to‑use framework for reinforcement fine‑tuning (RFT) of large language models. It supports diverse application scenarios and serves as a unified platform for exploring advanced RL paradigms, with capabilities for synchronous/asynchronous, on‑policy/off‑policy, and online/offline training.
How to use Liu?
Install Liu via pip install trinity-rft==0.2.1 or from source (Python ≥3.10, CUDA ≥12.4, at least 2 GPUs required). Prepare a model and dataset from Huggingface or ModelScope, then launch a web interface with trinity studio --port 8080 to configure and run the RFT process, or use trinity run --config <config_path> on the command line after starting a Ray cluster.
Key features of Liu
- Unified RFT core supporting multiple training modes.
- First‑class agent‑environment interaction handling.
- Optimized data pipelines with active management.
- Modular, decoupled architecture for easy adoption.
- Web‑based graphical interfaces for low‑code usage.
Use cases of Liu
- Adapting LLMs to new multi‑turn or multi‑step agentic scenarios.
- Developing custom RL algorithms with plug‑and‑play classes.
- Low‑code monitoring and tracking of the learning process.
- Fine‑tuning models on reasoning tasks (e.g., GSM8k with GRPO).
- Running offline learning via DPO or SFT.
FAQ from Liu
What are the system requirements for Liu?
Python version ≥3.10 and ≤3.12, CUDA ≥12.4 and ≤12.8, and at least 2 GPUs are required.
Which models and datasets are supported?
Liu supports most datasets and models from Huggingface and ModelScope, downloaded via huggingface-cli or modelscope.
How do I run a typical RFT process?
After installing and starting a Ray cluster, run trinity run --config <config_path> or use the web interface’s “Run” button after configuration.
Does Liu support multi‑turn and agentic interactions?
Yes, it supports first‑class agent‑environment interaction with lagged feedback, long‑tailed latencies, and graceful failure handling for multi‑turn tasks.
Can I use Liu for offline learning?
Yes, Liu supports offline learning modes, for example through DPO or SFT, as described in its tutorials.
