ros/claude-code-router
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

support plugins

Browse Source
This commit is contained in:
jinhui.li
2025-03-31 22:28:02 +08:00
parent 93fcd77544
commit 089654871c
11 changed files with 304 additions and 478 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
README.md
View File

@@ -4,23 +4,11 @@
![demo.png](https://github.com/musistudio/claude-code-router/blob/main/screenshoots/demo.png)
## Warning! This project is for testing purposes and may consume a lot of tokens! It may also fail to complete tasks!
## Implemented
- [x] Normal Mode and Router Mode
- [x] Support writing custom plugins for rewriting prompts.
- [x] Using the qwen2.5-coder-3b model as the routing dispatcher (since it’s currently free on Alibaba Cloud’s official website)
- [x] Using the qwen-max-0125 model as the tool invoker
- [x] Using deepseek-v3 as the coder model
- [x] Using deepseek-r1 as the reasoning model
- [x] Support proxy
Thanks to the free qwen2.5-coder-3b model from Alibaba and deepseek’s KV-Cache, we can significantly reduce the cost of using Claude Code. Make sure to set appropriate ignorePatterns for the project. See: https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview
- [x] Support writing custom plugins for implementing routers.
## Usage
@@ -30,16 +18,18 @@ Thanks to the free qwen2.5-coder-3b model from Alibaba and deepseek’s KV-Cache
npm install -g @anthropic-ai/claude-code
```
1. Install claude-code-router
1. Clone this repo and install dependencies
```shell
npm install -g @musistudio/claude-code-router
git clone https://github.com/musistudio/claude-code-router
cd claude-code-router && pnpm i
npm run build
```
2. Start claude-code-router server
```shell
claude-code-router
node dist/cli.js
```
3. Set environment variable to start claude code
@@ -51,21 +41,20 @@ export API_TIMEOUT_MS=600000
claude
```
## Normal Mode
## Plugin
The initial version uses a single model to accomplish all tasks. This model needs to support function calling and must allow for a sufficiently large tool description length, ideally greater than 1754. If the model used in this mode does not support KV Cache, it will consume a significant number of tokens.
The plugin allows users to rewrite Claude Code prompt and custom router. The plugin path is in `$HOME/.claude-code-router/plugins`. Currently, there are two demos available:
1. [custom router](https://github.com/musistudio/claude-code-router/blob/dev/custom-prompt/plugins/deepseek.js)
2. [rewrite prompt](https://github.com/musistudio/claude-code-router/blob/dev/custom-prompt/plugins/gemini.js)
![normal mode](https://github.com/musistudio/claude-code-reverse/blob/main/screenshoots/normal.png)
You need to move them to the `$HOME/.claude-code-router/plugins` directory and configure 'usePlugin' in `$HOME/.claude-code-router/config.json`，like this:
## Router Mode
Using multiple models to handle different tasks, this mode requires setting ENABLE_ROUTER to true and configuring four models: ROUTER_AGENT_MODEL, TOOL_AGENT_MODEL, CODER_AGENT_MODEL, and THINK_AGENT_MODEL.
ROUTER_AGENT_MODEL does not require high intelligence and is only responsible for request routing. A small model is sufficient for this task (testing has shown that the qwen-coder-3b model performs well).
TOOL_AGENT_MODEL must support function calling and allow for a sufficiently large tool description length, ideally greater than 1754. If the model used in this mode does not support KV Cache, it will consume a significant number of tokens.
CODER_AGENT_MODEL and THINK_AGENT_MODEL can use the DeepSeek series of models.
The purpose of router mode is to separate tool invocation from coding tasks, enabling the use of inference models like r1, which do not support function calling.
![router mode](https://github.com/musistudio/claude-code-router/blob/main/screenshoots/router.png)
```json
{
"usePlugin": "gemini",
"LOG": true,
"OPENAI_API_KEY": "",
"OPENAI_BASE_URL": "",
"OPENAI_MODEL": ""
}
```

97
src/deepseek.ts → plugins/deepseek.js
View File

@@ -1,15 +1,10 @@
import { OpenAI } from "openai";
import { createClient } from "./utils";
import { log } from "./utils/log";
export interface BaseRouter {
name: string;
description: string;
run: (
args: OpenAI.Chat.Completions.ChatCompletionCreateParams
) => Promise<any>;
}
const {
log,
streamOpenAIResponse,
createClient,
} = require("claude-code-router");
const thinkRouter: BaseRouter = {
const thinkRouter = {
name: "think",
description: `This agent is used solely for complex reasoning and thinking tasks. It should not be called for information retrieval or repetitive, frequent requests. Only use this agent for tasks that require deep analysis or problem-solving. If there is an existing result from the Thinker agent, do not call this agent again.你只负责深度思考以拆分任务，不需要进行任何的编码和调用工具。最后讲拆分的步骤按照顺序返回。比如\n1. xxx\n2. xxx\n3. xxx`,
run(args) {
@@ -18,13 +13,13 @@ const thinkRouter: BaseRouter = {
baseURL: process.env.THINK_AGENT_BASE_URL,
});
const messages = JSON.parse(JSON.stringify(args.messages));
messages.forEach((msg: any) => {
messages.forEach((msg) => {
if (Array.isArray(msg.content)) {
msg.content = JSON.stringify(msg.content);
}
});
let startIdx = messages.findIndex((msg: any) => msg.role !== "system");
let startIdx = messages.findIndex((msg) => msg.role !== "system");
if (startIdx === -1) startIdx = messages.length;
for (let i = startIdx; i < messages.length; i++) {
@@ -46,14 +41,12 @@ const thinkRouter: BaseRouter = {
return client.chat.completions.create({
...args,
messages,
model: process.env.THINK_AGENT_MODEL as string,
model: process.env.THINK_AGENT_MODEL,
});
},
};
export class Router {
routers: BaseRouter[];
client: OpenAI;
class Router {
constructor() {
this.routers = [thinkRouter];
this.client = createClient({
@@ -61,37 +54,37 @@ export class Router {
baseURL: process.env.ROUTER_AGENT_BASE_URL,
});
}
async route(
args: OpenAI.Chat.Completions.ChatCompletionCreateParams
): Promise<any> {
async route(args) {
log(`Request Router: ${JSON.stringify(args, null, 2)}`);
const res: OpenAI.Chat.Completions.ChatCompletion =
await this.client.chat.completions.create({
...args,
messages: [
...args.messages,
{
role: "system",
content: `## **Guidelines:**
- **Trigger the "think" mode when the user's request involves deep thinking, complex reasoning, or multi-step analysis.**
- **Criteria:**
- Involves multi-layered logical reasoning or causal analysis
- Requires establishing connections or pattern recognition between different pieces of information
- Involves cross-domain knowledge integration or weighing multiple possibilities
- Requires creative thinking or non-direct inference
### **Format requirements:**
- When you need to trigger the "think" mode, return the following JSON format:
\`\`\`json
{
"use": "think"
}
\`\`\`
`,
},
],
model: process.env.ROUTER_AGENT_MODEL as string,
stream: false,
});
const res = await this.client.chat.completions.create({
...args,
messages: [
...args.messages,
{
role: "system",
content: `## **Guidelines:**
- **Trigger the "think" mode when the user's request involves deep thinking, complex reasoning, or multi-step analysis.**
- **Criteria:**
- Involves multi-layered logical reasoning or causal analysis
- Requires establishing connections or pattern recognition between different pieces of information
- Involves cross-domain knowledge integration or weighing multiple possibilities
- Requires creative thinking or non-direct inference
### **Special Case:**
- **When the user sends "test", respond with "success" only.**
### **Format requirements:**
- When you need to trigger the "think" mode, return the following JSON format:
\`\`\`json
{
"use": "think"
}
\`\`\`
`,
},
],
model: process.env.ROUTER_AGENT_MODEL,
stream: false,
});
let result;
try {
const text = res.choices[0].message.content;
@@ -102,13 +95,13 @@ export class Router {
text.slice(text.indexOf("{"), text.lastIndexOf("}") + 1)
);
} catch (e) {
(res.choices[0] as any).delta = res.choices[0].message;
res.choices[0].delta = res.choices[0].message;
log(`No Router: ${JSON.stringify(res.choices[0].message)}`);
return [res];
}
const router = this.routers.find((item) => item.name === result.use);
if (!router) {
(res.choices[0] as any).delta = res.choices[0].message;
res.choices[0].delta = res.choices[0].message;
log(`No Router: ${JSON.stringify(res.choices[0].message)}`);
return [res];
}
@@ -138,3 +131,9 @@ export class Router {
return router.run(args);
}
}
const router = new Router();
module.exports = async function handle(req, res, next) {
const completions = await router.route(req.body);
streamOpenAIResponse(res, completions, req.body.model);
};

23
plugins/gemini.js Normal file
View File

@@ -0,0 +1,23 @@
module.exports = async function handle(req, res, next) {
if (Array.isArray(req.body.tools)) {
// rewrite tools definition
req.body.tools.forEach((tool) => {
if (tool.function.name === "BatchTool") {
// HACK: Gemini does not support objects with empty properties
tool.function.parameters.properties.invocations.items.properties.input.type =
"number";
return;
}
Object.keys(tool.function.parameters.properties).forEach((key) => {
const prop = tool.function.parameters.properties[key];
if (
prop.type === "string" &&
!["enum", "date-time"].includes(prop.format)
) {
delete prop.format;
}
});
});
}
next();
};

15
src/constants.ts
View File

@@ -5,24 +5,11 @@ export const HOME_DIR = path.join(os.homedir(), ".claude-code-router");
export const CONFIG_FILE = `${HOME_DIR}/config.json`;
export const PROMPTS_DIR = `${HOME_DIR}/prompts`;
export const PLUGINS_DIR = `${HOME_DIR}/plugins`;
export const DEFAULT_CONFIG = {
log: false,
ENABLE_ROUTER: true,
OPENAI_API_KEY: "",
OPENAI_BASE_URL: "https://openrouter.ai/api/v1",
OPENAI_MODEL: "openai/o3-mini",
CODER_AGENT_API_KEY: "",
CODER_AGENT_BASE_URL: "https://api.deepseek.com",
CODER_AGENT_MODEL: "deepseek-chat",
THINK_AGENT_API_KEY: "",
THINK_AGENT_BASE_URL: "https://api.deepseek.com",
THINK_AGENT_MODEL: "deepseek-reasoner",
ROUTER_AGENT_API_KEY: "",
ROUTER_AGENT_BASE_URL: "https://api.deepseek.com",
ROUTER_AGENT_MODEL: "deepseek-chat",
};

27
src/index.ts
View File

@@ -1,8 +1,11 @@
import { existsSync } from "fs";
import { writeFile } from "fs/promises";
import { initConfig, initDir } from "./utils";
import { getOpenAICommonOptions, initConfig, initDir } from "./utils";
import { createServer } from "./server";
import { rewriteToolsPrompt } from "./middlewares/rewriteToolsPrompt";
import { formatRequest } from "./middlewares/formatRequest";
import { rewriteBody } from "./middlewares/rewriteBody";
import OpenAI from "openai";
import { streamOpenAIResponse } from "./utils/stream";
async function initializeClaudeConfig() {
const homeDir = process.env.HOME;
@@ -29,7 +32,25 @@ async function run() {
await initDir();
await initConfig();
const server = createServer(3456);
server.useMiddleware(rewriteToolsPrompt);
server.useMiddleware(formatRequest);
server.useMiddleware(rewriteBody);
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
baseURL: process.env.OPENAI_BASE_URL,
...getOpenAICommonOptions(),
});
server.app.post("/v1/messages", async (req, res) => {
try {
if (process.env.OPENAI_MODEL) {
req.body.model = process.env.OPENAI_MODEL;
}
const completion: any = await openai.chat.completions.create(req.body);
await streamOpenAIResponse(res, completion, req.body.model);
} catch (e) {
console.error("Error in OpenAI API call:", e);
}
});
server.start();
}
run();

101
src/middlewares/formatRequest.ts Normal file
View File

@@ -0,0 +1,101 @@
import { Request, Response, NextFunction } from "express";
import { ContentBlockParam } from "@anthropic-ai/sdk/resources";
import { MessageCreateParamsBase } from "@anthropic-ai/sdk/resources/messages";
import OpenAI from "openai";
import { streamOpenAIResponse } from "../utils/stream";
export const formatRequest = async (
req: Request,
res: Response,
next: NextFunction
) => {
let {
model,
max_tokens,
messages,
system = [],
temperature,
metadata,
tools,
}: MessageCreateParamsBase = req.body;
try {
const openAIMessages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] =
messages.map((item) => {
if (item.content instanceof Array) {
return {
role: item.role,
content: item.content
.map((it: ContentBlockParam) => {
if (it.type === "text") {
return typeof it.text === "string"
? it.text
: JSON.stringify(it);
}
return JSON.stringify(it);
})
.join(""),
} as OpenAI.Chat.Completions.ChatCompletionMessageParam;
}
return {
role: item.role,
content:
typeof item.content === "string"
? item.content
: JSON.stringify(item.content),
};
});
const systemMessages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] =
Array.isArray(system)
? system.map((item) => ({
role: "system",
content: item.text,
}))
: [{ role: "system", content: system }];
const data: OpenAI.Chat.Completions.ChatCompletionCreateParams = {
model,
messages: [...systemMessages, ...openAIMessages],
temperature,
stream: true,
};
if (tools) {
data.tools = tools
.filter((tool) => !["StickerRequest"].includes(tool.name))
.map((item: any) => ({
type: "function",
function: {
name: item.name,
description: item.description,
parameters: item.input_schema,
},
}));
}
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
req.body = data;
} catch (error) {
console.error("Error in request processing:", error);
const errorCompletion: AsyncIterable<OpenAI.Chat.Completions.ChatCompletionChunk> =
{
async *[Symbol.asyncIterator]() {
yield {
id: `error_${Date.now()}`,
created: Math.floor(Date.now() / 1000),
model: "gpt-3.5-turbo",
object: "chat.completion.chunk",
choices: [
{
index: 0,
delta: {
content: `Error: ${(error as Error).message}`,
},
finish_reason: "stop",
},
],
};
},
};
await streamOpenAIResponse(res, errorCompletion, model);
}
next();
};

43
src/middlewares/rewriteBody.ts Normal file
View File

@@ -0,0 +1,43 @@
import { Request, Response, NextFunction } from "express";
import Module from "node:module";
import { streamOpenAIResponse } from "../utils/stream";
import { log } from "../utils/log";
import { PLUGINS_DIR } from "../constants";
import path from "node:path";
import { access } from "node:fs/promises";
import { OpenAI } from "openai";
import { createClient } from "../utils";
// @ts-ignore
const originalLoad = Module._load;
// @ts-ignore
Module._load = function (request, parent, isMain) {
if (request === "claude-code-router") {
return {
streamOpenAIResponse,
log,
OpenAI,
createClient,
};
}
return originalLoad.call(this, request, parent, isMain);
};
export const rewriteBody = async (
req: Request,
res: Response,
next: NextFunction
) => {
if (!process.env.usePlugin) {
return next();
}
const pluginPath = path.join(PLUGINS_DIR, `${process.env.usePlugin}.js`);
try {
await access(pluginPath);
const rewritePlugin = require(pluginPath);
rewritePlugin(req, res, next);
} catch (e) {
console.error(e);
next();
}
};

34
src/middlewares/rewriteToolsPrompt.ts
View File

@@ -1,34 +0,0 @@
import { Request, Response, NextFunction } from "express";
import { readFile, access } from "node:fs/promises";
import { join } from "node:path";
import { PROMPTS_DIR } from "../constants";
const getPrompt = async (name: string) => {
try {
const promptPath = join(PROMPTS_DIR, `${name}.md`);
await access(promptPath);
const prompt = await readFile(promptPath, "utf-8");
return prompt;
} catch {
return null;
}
};
export const rewriteToolsPrompt = async (
req: Request,
res: Response,
next: NextFunction
) => {
const { tools } = req.body;
if (!Array.isArray(tools)) {
next();
return;
}
for (const tool of tools) {
const prompt = await getPrompt(tool.name);
if (prompt) {
tool.description = prompt;
}
}
next();
};

206
src/router copy.ts
View File

@@ -1,206 +0,0 @@
import { OpenAI } from "openai";
import { createClient } from "./utils";
import { log } from "./utils/log";
export interface BaseRouter {
name: string;
description: string;
run: (
args: OpenAI.Chat.Completions.ChatCompletionCreateParams
) => Promise<any>;
}
const coderRouter: BaseRouter = {
name: "coder",
description: `This agent is solely responsible for helping users write code. This agent could not call tools. This agent is used for writing and modifying code when the user provides clear and specific coding requirements. For example, tasks like implementing a quicksort algorithm in JavaScript or creating an HTML layout. If the user's request is unclear or cannot be directly translated into code, please route the task to 'think' first for clarification or further processing.`,
run(args) {
const client = createClient({
apiKey: process.env.CODER_AGENT_API_KEY,
baseURL: process.env.CODER_AGENT_BASE_URL,
});
delete args.tools;
args.messages.forEach((item) => {
if (Array.isArray(item.content)) {
item.content = JSON.stringify(item.content);
}
});
return client.chat.completions.create({
...args,
messages: [
...args.messages,
{
role: "system",
content:
"You are a code writer who helps users write code based on their specific requirements. You create algorithms, implement functionality, and build structures according to the clear instructions provided by the user. Your focus is solely on writing code, ensuring that the task is completed accurately and efficiently.",
},
],
model: process.env.CODER_AGENT_MODEL as string,
});
},
};
const useToolRouter: BaseRouter = {
name: "use-tool",
description: `This agent can call user-specified tools to perform tasks. The user provides a list of tools to be used, and the agent integrates these tools to complete the specified tasks efficiently. The agent follows user instructions and ensures proper tool utilization for each request`,
run(args) {
const client = createClient({
apiKey: process.env.TOOL_AGENT_API_KEY,
baseURL: process.env.TOOL_AGENT_BASE_URL,
});
return client.chat.completions.create({
...args,
messages: [
...args.messages,
{
role: "system",
content:
"You need to select the appropriate tool for the task based on the user’s request. Review the requirements and choose the tool that fits the task best.",
},
],
model: process.env.TOOL_AGENT_MODEL as string,
});
},
};
const thinkRouter: BaseRouter = {
name: "think",
description: `This agent is used solely for complex reasoning and thinking tasks. It should not be called for information retrieval or repetitive, frequent requests. Only use this agent for tasks that require deep analysis or problem-solving. If there is an existing result from the Thinker agent, do not call this agent again.你只负责深度思考以拆分任务，不需要进行任何的编码和调用工具。最后讲拆分的步骤按照顺序返回。比如\n1. xxx\n2. xxx\n3. xxx`,
run(args) {
const client = createClient({
apiKey: process.env.THINK_AGENT_API_KEY,
baseURL: process.env.THINK_AGENT_BASE_URL,
});
const messages = JSON.parse(JSON.stringify(args.messages));
messages.forEach((msg: any) => {
if (Array.isArray(msg.content)) {
msg.content = JSON.stringify(msg.content);
}
});
let startIdx = messages.findIndex((msg: any) => msg.role !== "system");
if (startIdx === -1) startIdx = messages.length;
for (let i = startIdx; i < messages.length; i++) {
const expectedRole = (i - startIdx) % 2 === 0 ? "user" : "assistant";
messages[i].role = expectedRole;
}
if (
messages.length > 0 &&
messages[messages.length - 1].role === "assistant"
) {
messages.push({
role: "user",
content:
"Please follow the instructions provided above to resolve the issue.",
});
}
delete args.tools;
return client.chat.completions.create({
...args,
messages,
model: process.env.THINK_AGENT_MODEL as string,
});
},
};
export class Router {
routers: BaseRouter[];
client: OpenAI;
constructor() {
this.routers = [coderRouter, useToolRouter, thinkRouter];
this.client = createClient({
apiKey: process.env.ROUTER_AGENT_API_KEY,
baseURL: process.env.ROUTER_AGENT_BASE_URL,
});
}
async route(
args: OpenAI.Chat.Completions.ChatCompletionCreateParams
): Promise<any> {
log(`Route: ${JSON.stringify(args, null, 2)}`);
const res: OpenAI.Chat.Completions.ChatCompletion =
await this.client.chat.completions.create({
...args,
messages: [
...args.messages,
{
role: "system",
content: `You are an AI task router and executor, responsible for understanding user requests and directing them to the appropriate processing mode or tool based on the task type and requirements. Your main responsibility is to determine the nature of the request, execute the task when possible, and respond appropriately.
### **Guidelines:**
- **If an external tool is required to complete the task (such as searching for information, generating images, or modifying code), route the task to \`use-tool\` rather than handling it directly.**
- If the task requires generating an image, route to \`use-tool\` and specify the image generation tool.
- If the task requires searching for information, route to \`use-tool\` and specify the search tool.
- If the task requires modifying or executing code, route to \`use-tool\` and specify the code handling tool.
- **Do NOT execute the tool action directly; always trigger it through \`use-tool\`.**
- **If the user is chatting casually or having a general conversation, respond naturally and conversationally. Improving the user experience through friendly interactions is one of your main responsibilities.**
- **If the user's request involves deep thinking, complex reasoning, or multi-step analysis, use the "think" mode to break down and solve the problem.**
- **If the user's request involves coding or technical implementation, use the "coder" mode to generate or modify code.**
- **After generating the code, if the task requires applying or integrating the code, route to \`use-tool\` and specify the code execution tool.**
- **Do NOT re-trigger "coder" to apply code — route to \`use-tool\` instead.**
### **Format requirements:**
- When you need to trigger a specific mode (such as "think", "coder", or "use-tool"), return the following JSON format:
### IMPORTANT:
- 你不能也不会调用BatchTool，如果你需要使用工具请路由到\`use-tool\`，由\`use-tool\`来调用BatchTool。
\`\`\`json
{
"use": "<mode-name>",
}
\`\`\`
`,
},
],
model: process.env.ROUTER_AGENT_MODEL as string,
stream: false,
});
let result;
try {
const text = res.choices[0].message.content;
if (!text) {
throw new Error("No text");
}
result = JSON.parse(
text.slice(text.indexOf("{"), text.lastIndexOf("}") + 1)
);
} catch (e) {
(res.choices[0] as any).delta = res.choices[0].message;
return [res];
}
const router = this.routers.find((item) => item.name === result.use);
if (!router) {
(res.choices[0] as any).delta = res.choices[0].message;
log(`No Router: ${JSON.stringify(res.choices[0].message)}`);
return [res];
}
log(`Use Router: ${router.name}`);
if (router.name === "think" || router.name === "coder") {
const agentResult = await router.run({
...args,
stream: false,
});
try {
args.messages.push({
role: "user",
content:
`${router.name} Agent Result: ` +
agentResult.choices[0].message.content,
});
log(
`${router.name} Agent Result: ` +
agentResult.choices[0].message.content
);
return await this.route(args);
} catch (error) {
console.log(agentResult);
throw error;
}
}
return router.run(args);
}
}

136
src/server.ts
View File

@@ -1,18 +1,4 @@
import express, { RequestHandler } from "express";
import {
ContentBlockParam,
MessageCreateParamsBase,
} from "@anthropic-ai/sdk/resources/messages";
import { OpenAI } from "openai";
import { Router } from "./deepseek";
import { getOpenAICommonOptions } from "./utils";
import { streamOpenAIResponse } from "./utils/stream";
interface Client {
call: (
data: OpenAI.Chat.Completions.ChatCompletionCreateParams
) => Promise<any>;
}
interface Server {
app: express.Application;
@@ -23,128 +9,6 @@ interface Server {
export const createServer = (port: number): Server => {
const app = express();
app.use(express.json({ limit: "500mb" }));
let client: Client;
if (process.env.ENABLE_ROUTER && process.env.ENABLE_ROUTER === "true") {
const router = new Router();
client = {
call: (data) => {
return router.route(data);
},
};
} else {
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
baseURL: process.env.OPENAI_BASE_URL,
...getOpenAICommonOptions(),
});
client = {
call: (data) => {
if (process.env.OPENAI_MODEL) {
data.model = process.env.OPENAI_MODEL;
}
return openai.chat.completions.create(data);
},
};
}
app.post("/v1/messages", async (req, res) => {
try {
let {
model,
max_tokens,
messages,
system = [],
temperature,
metadata,
tools,
}: MessageCreateParamsBase = req.body;
const openAIMessages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] =
messages.map((item) => {
if (item.content instanceof Array) {
return {
role: item.role,
content: item.content
.map((it: ContentBlockParam) => {
if (it.type === "text") {
return typeof it.text === "string"
? it.text
: JSON.stringify(it);
}
return JSON.stringify(it);
})
.join(""),
} as OpenAI.Chat.Completions.ChatCompletionMessageParam;
}
return {
role: item.role,
content:
typeof item.content === "string"
? item.content
: JSON.stringify(item.content),
};
});
const systemMessages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] =
Array.isArray(system)
? system.map((item) => ({
role: "system",
content: item.text,
}))
: [{ role: "system", content: system }];
const data: OpenAI.Chat.Completions.ChatCompletionCreateParams = {
model,
messages: [...systemMessages, ...openAIMessages],
temperature,
stream: true,
};
if (tools) {
data.tools = tools
.filter((tool) => !["StickerRequest"].includes(tool.name))
.map((item: any) => ({
type: "function",
function: {
name: item.name,
description: item.description,
parameters: item.input_schema,
},
}));
}
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
try {
const completion = await client.call(data);
await streamOpenAIResponse(res, completion, model);
} catch (e) {
console.error("Error in OpenAI API call:", e);
}
} catch (error) {
console.error("Error in request processing:", error);
const errorCompletion: AsyncIterable<OpenAI.Chat.Completions.ChatCompletionChunk> =
{
async *[Symbol.asyncIterator]() {
yield {
id: `error_${Date.now()}`,
created: Math.floor(Date.now() / 1000),
model: "gpt-3.5-turbo",
object: "chat.completion.chunk",
choices: [
{
index: 0,
delta: {
content: `Error: ${(error as Error).message}`,
},
finish_reason: "stop",
},
],
};
},
};
await streamOpenAIResponse(res, errorCompletion, "gpt-3.5-turbo");
}
});
return {
app,
useMiddleware: (middleware: RequestHandler) => {

47
src/utils/index.ts
View File

@@ -1,11 +1,12 @@
import { HttpsProxyAgent } from "https-proxy-agent";
import OpenAI, { ClientOptions } from "openai";
import fs from "node:fs/promises";
import readline from "node:readline";
import {
CONFIG_FILE,
DEFAULT_CONFIG,
HOME_DIR,
PROMPTS_DIR,
PLUGINS_DIR,
} from "../constants";
export function getOpenAICommonOptions(): ClientOptions {
@@ -26,7 +27,29 @@ const ensureDir = async (dir_path: string) => {
export const initDir = async () => {
await ensureDir(HOME_DIR);
await ensureDir(PROMPTS_DIR);
await ensureDir(PLUGINS_DIR);
};
const createReadline = () => {
return readline.createInterface({
input: process.stdin,
output: process.stdout,
});
};
const question = (query: string): Promise<string> => {
return new Promise((resolve) => {
const rl = createReadline();
rl.question(query, (answer) => {
rl.close();
resolve(answer);
});
});
};
const confirm = async (query: string): Promise<boolean> => {
const answer = await question(query);
return answer.toLowerCase() !== "n";
};
export const readConfigFile = async () => {
@@ -34,8 +57,24 @@ export const readConfigFile = async () => {
const config = await fs.readFile(CONFIG_FILE, "utf-8");
return JSON.parse(config);
} catch {
await writeConfigFile(DEFAULT_CONFIG);
return DEFAULT_CONFIG;
const useRouter = await confirm(
"No config file found. Enable router mode? (Y/n)"
);
if (!useRouter) {
const apiKey = await question("Enter OPENAI_API_KEY: ");
const baseUrl = await question("Enter OPENAI_BASE_URL: ");
const model = await question("Enter OPENAI_MODEL: ");
const config = Object.assign({}, DEFAULT_CONFIG, {
OPENAI_API_KEY: apiKey,
OPENAI_BASE_URL: baseUrl,
OPENAI_MODEL: model,
});
await writeConfigFile(config);
return config;
} else {
const router = await question("Enter OPENAI_API_KEY: ");
return DEFAULT_CONFIG;
}
}
};