One API for every LLM provider

NodeLLM is the backend runtime for building reliable, testable, provider-agnostic AI systems in Node.js. Nine providers, one predictable API — with normalized streaming, automatic tool loops, deterministic testing, and first-class observability built in.

npm install @node-llm/core

import { createLLM } from "@node-llm/core";

const llm = createLLM({ provider: "anthropic" });
const chat = llm.chat("claude-sonnet-5");

// Same code works for OpenAI, Gemini, Bedrock, Ollama, and more —
// switch providers with config, not a rewrite.
const response = await chat.ask("Explain event-driven architecture");
console.log(response.content);

// Streaming is a standard AsyncIterator on every provider
for await (const chunk of chat.stream("Now explain it to a five-year-old")) {
  process.stdout.write(chunk.content);
}

Unified support for

Quick Start View on GitHub

---

Why NodeLLM?

Most AI SDKs optimize for getting a response to the user fast (frontend/edge). NodeLLM optimizes for system reliability — it’s built for API servers, workers, cron jobs, and agents, where the hard problems are infrastructure problems:

• Decoupling — isolate business logic from the rapid churn of model versions. Switch providers via config, not code rewrites.
• Production safety — timeouts that protect the event loop, circuit breaking, redaction, and audit logging.
• Determinism — record/replay your AI interactions with VCR cassettes; unit-test agents with a fluent mocker.
• One mental model — identical behavior for streaming, tool loops, structured output, and vision across every provider.

Feature	NodeLLM	Official SDKs	Impact
Provider Logic	Transparently handled	Exposed to your code	Low coupling
Streaming	Standard AsyncIterator	Vendor-specific events	Predictable data flow
Tool Loops	Automated recursion	Manual implementation	Less boilerplate
Files/Vision	Intelligent path/URL handling	Base64/Buffer management	Cleaner service layer
Configuration	Centralized & global	Per-instance initialization	Easier lifecycle mgmt

NodeLLM is not a thin wrapper around vendor SDKs, a UI streaming library like Vercel AI SDK, or a prompt-engineering framework. It sits between your app and the providers, and owns the messy middle:

                Your App
                   ↓
NodeLLM (Unified API + State + Security)
                   ↓
 OpenAI | Anthropic | Gemini | Bedrock | xAI | Ollama | ...

---

What you can build with it

Auto-executing tools

Define a tool once with the class-based DSL; NodeLLM runs the recursive execution loop for you — no manual “check for tool_calls, execute, re-send” plumbing.

import { Tool, z } from "@node-llm/core";

class WeatherTool extends Tool {
  name = "get_weather";
  description = "Get current weather";
  schema = z.object({ loc: z.string() });

  async handler({ loc }) {
    return `Sunny in ${loc}`;
  }
}

await chat.withTool(WeatherTool).ask("Weather in Tokyo?");

Model Context Protocol (MCP)

Connect to external data sources and tools over the industry-standard MCP. Discover tools, resources, and prompt templates from any server dynamically.

import { MCP } from "@node-llm/mcp";

const mcp = (await MCP.connect({
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-github"]
})).onLog(e => console.log(e.message));

const tools = await mcp.discoverTools();
await chat.withTools(tools).ask("List my repos");

Extended thinking

Direct, unified access to the reasoning of models like Claude, DeepSeek R1, and OpenAI’s o-series — one interface, every provider.

const res = await chat
  .withThinking({ budget: 16000 })
  .ask("Solve this logical puzzle");

console.log(res.thinking.text); // Full chain-of-thought

Persistence with your database

Track chat history, tool executions, and API metrics automatically with @node-llm/orm — Postgres, MySQL, or SQLite via Prisma, including extended-thinking persistence.

import { createChat } from "@node-llm/orm/prisma";

const chat = await createChat(prisma, llm, { model: "claude-sonnet-5" });
await chat.withThinking({ budget: 16000 }).ask("Develop a strategy");

Deterministic testing

No more flaky, expensive AI tests. Record real interactions once with VCR cassettes, replay them forever; mock tool-calling flows with the fluent mocker — powered by @node-llm/testing.

import { vcr, Mocker } from "@node-llm/testing";

// Integration tests: record once, replay deterministically
await vcr.useCassette("pricing_flow", async () => {
  const res = await chat.ask("How much?");
  expect(res.content).toContain("$20/mo");
});

// Unit tests: no network at all
const mock = new Mocker()
  .chat("Next step?")
  .respond("Login User")
  .callsTool("getCurrentUser", { id: 1 });

Multi-provider parallelism

Run providers side by side with isolated contexts — no global-config side effects.

const [gpt, claude] = await Promise.all([
  NodeLLM.withProvider("openai").chat("gpt-5").ask(prompt),
  NodeLLM.withProvider("anthropic").chat("claude-sonnet-5").ask(prompt)
]);

Images, embeddings, audio, and more

The same llm handle covers image generation, embeddings, transcription, moderation, and structured output — see the full Core Features section.

Security & compliance

Plug custom security, PII detection, and compliance logic into asynchronous beforeRequest and afterResponse hooks — enforce policy at the framework level, not per call site.

---

Supported providers

Provider	Supported Features
OpenAI	Chat, Streaming, Tools, Vision, Audio, Images, Transcription, Reasoning, Smart Developer Role
Anthropic	Chat, Streaming, Tools, Vision, PDF, Structured Output, Extended Thinking, Prompt Caching
Gemini	Chat, Streaming, Tools, Vision, Audio, Video, Embeddings
DeepSeek	Chat, Extended Thinking (R1), Tools, Streaming
Bedrock	Chat, Streaming, Tools, Image Gen, Embeddings, Prompt Caching
OpenRouter	Aggregator: Chat, Streaming, Tools, Vision, Embeddings, Reasoning
xAI	Chat, Streaming, Tools, Vision, Images, Reasoning
Ollama	Local Inference: Chat, Streaming, Tools, Vision, Embeddings
Mistral	Chat, Streaming, Tools, Vision, Embeddings, Transcription, Moderation, Reasoning

The model registry tracks current model IDs, capabilities, and pricing across all providers — kept in sync automatically.

---

Next steps

• Quick Start — chat, images, and embeddings in 5 minutes
• Configuration — API keys, defaults, and per-request overrides
• Tool Calling — give your AI the ability to execute code
• Testing — reliable, zero-cost integration tests
• Examples — complete, runnable scripts for every feature

---

Contributing & credits

We welcome contributions — see the Contributing Guide to get started.

NodeLLM is heavily inspired by the elegant design of RubyLLM. 🫶