vllm-project/semantic-router

Registry 信息

项目简介

Semantic Router 是一个面向模型混合架构的系统级智能路由器，旨在高效调度云端、数据中心和边缘环境中的AI工作负载。它是 vLLM 生态系统的关键组件，主要使用 Go 语言构建。

要点

• 面向AI模型的系统级智能路由
• 支持模型混合（MoM）架构
• 可部署于云端、数据中心和边缘环境
• 使用 Go 语言构建，注重性能与可扩展性
• 与 vLLM 和 Hugging Face 生态系统集成
• 支持 Kubernetes 原生部署

使用场景

• 作为路由LLM请求的AI网关
• 基于语义上下文动态选择模型
• 跨多个AI模型实例的负载均衡
• 具备智能路由的边缘AI推理
• 多租户AI服务平台
• PII检测与内容过滤路由

README 摘要

<div align="center"> <img src="website/static/img/artworks/vllm-sr-logo.dark.png" alt="vLLM Semantic Router" width="50%"/> <p><strong>System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge</strong></p> <p> <a href="https://vllm-semantic-router.com">Documentation</a> | <a href="https://play.vllm-semantic-router.com">Playground</a> | <a href="https://vllm-semantic-router.com/blog/">Blog</a> | <a href="https://vllm-semantic-router.com/publications/">Publications</a> | <a href="https://huggingface.co/LLM-Semantic-Router">Hugging Face</a> </p> </div> --- ## About In the LLM era, the number of models is exploding. Different models vary across capability, scale, cost, and privacy boundaries. Choosing and connecting the right models to build semantic AI infrastructure is a system problem. **vLLM Semantic Router** is a **signal-driven** intelligent router for that problem. It helps teams build model systems that are more **efficient**, **safer**, and more **adaptive** across cloud, data center, and edge environments.  It delivers three core values: - **Token economics**: reduce wasted tokens, increase effective output, and maximize the value of every token. - **LLM safety**: detect jailbreaks, sensitive leakage, and hallucinations so agents remain controllable, trustworthy, and auditable. - **Fullmesh intelligence**: build personal AI at the edge and intelligent MaaS in the cloud by coordinating local, private, and frontier models across cost, privacy, and capability boundaries. ## Getting Started ### Install ```bash curl -fsSL https://vllm-semantic-router.com/install.sh | bash ``` For platform notes, detailed setup options, and troubleshooting, see the **[Installation Guide](https://v

话题