Where On-Device and Cloud LLMs Meet

What is Where On-Device and Cloud LLMs Meet?

Where On-Device and Cloud LLMs Meet is a communication protocol (named Minions) that enables small on-device language models to collaborate with frontier cloud models. It is a Python package tested on Mac and Ubuntu with Python 3.10–3.11, aimed at developers and researchers looking to reduce cloud costs by processing long contexts locally.

How to use Where On-Device and Cloud LLMs Meet?

Clone the repository, install the Python package (pip install -e .), then install a local model server (Ollama or Tokasaurus) and set an API key for at least one cloud LLM provider (OpenAI or TogetherAI). Use the provided Streamlit demo app (streamlit run app.py), the CLI (minions --context <path>), or write Python code using the Minion/Minions classes.

Key features of Where On-Device and Cloud LLMs Meet

• Collaborates on-device and cloud LLMs to reduce costs
• Reads long contexts locally, sends short messages to cloud
• Supports Ollama and Tokasaurus for local models
• Supports OpenAI and TogetherAI for cloud models
• Provides Minion (single)