Moderation v1.0.0+

Protect your users and your brand by checking text content against safety policies for violence, hate speech, and harassment.

Table of contents

  1. Basic Usage
  2. Understanding Results
    1. Common Categories
  3. Batch Moderation
  4. Integration Patterns
    1. Pre-Chat Moderation
    2. Custom Risk Thresholds

Check if text content violates safety policies using `NodeLLM.moderate`. This is crucial for user-facing applications to prevent abuse.

Basic Usage

The simplest check returns a flagged boolean and categories.

const result = await NodeLLM.moderate("I want to help everyone!");

if (result.flagged) {
  console.log(`❌ Flagged for: ${result.flaggedCategories.join(", ")}`);
} else {
  console.log("✅ Content appears safe");
}

Understanding Results

The moderation result object provides detailed signals:

  • flagged: (boolean) Overall safety check. if true, content violates provider policies.
  • categories: (object) Boolean flags for specific buckets (e.g., sexual: false, violence: true).
  • category_scores: (object) Confidence scores (0.0 - 1.0) for each category.
const result = await NodeLLM.moderate("Some controversial text");

// Check specific categories
if (result.categories.hate) {
  console.log("Hate speech detected");
}

// Check confidence levels
console.log(`Violence Score: ${result.category_scores.violence}`);

Common Categories

  • Sexual: Sexual content.
  • Hate: Content promoting hate based on identity.
  • Harassment: Threatening or bullying content.
  • Self-Harm: Promoting self-harm or suicide.
  • Violence: Promoting or depicting violence.

Batch Moderation

NodeLLM.moderate also accepts an array of strings in a single call. The returned Moderation object aggregates every input’s result:

const result = await NodeLLM.moderate(["First message", "Second message"]);

console.log(result.length); // 2
console.log(result.flagged); // true if ANY input was flagged
console.log(result.flaggedCategories); // Categories flagged across ALL inputs

// Iterate per-input results
for (const item of result) {
  console.log(item.flagged, item.categories);
}

// Or access by index
console.log(result.results[0].flagged);

result.categories and result.category_scores (used in the examples above) are shorthand for the first result — reach for result.results when moderating a batch.

Integration Patterns

Pre-Chat Moderation

We recommend validating user input before sending it to a Chat model to save costs and prevent jailbreaks.

async function safeChat(input: string) {
  const mod = await NodeLLM.moderate(input);

  if (mod.flagged) {
    throw new Error(`Content Unsafe: ${mod.flaggedCategories.join(", ")}`);
  }

  // Only proceed if safe
  return await chat.ask(input);
}

Custom Risk Thresholds

Providers have their own thresholds for “flagging”. You can implement stricter (or looser) logic using raw scores.

const result = await NodeLLM.moderate(userInput);

// Custom strict policy: Flag anything with > 0.1 confidence
const isRisky = Object.entries(result.category_scores).some(([category, score]) => score > 0.1);

if (isRisky) {
  console.warn("Potential risk detected (custom strict mode)");
}