Observability & Monitoring for AI Agents

The Nevermined Payment Libraries include built-in observability capabilities that allow you to monitor, track, and analyze your AI agent’s performance, usage patterns, and costs. This integration provides comprehensive logging and analytics for your AI operations.

Overview

The observability API provides:

Request/Response Logging: Automatic logging of all AI API calls with full context
Usage Tracking: Token usage, costs, and performance metrics
Custom Properties: Add custom metadata to your logs for better analysis
Real-time Monitoring: View logs and metrics in the Nevermined dashboard

Basic Integration

1. Initialize the Observability API

The observability functionality is automatically available through the Payments class:

TypeScript
Python

import { Payments } from "@nevermined-io/payments";

const payments = Payments.getInstance({
  nvmApiKey: "your-api-key",
  environment: "sandbox",
});

// Access observability through payments.observability
const observability = payments.observability;

2. Configure OpenAI with Observability Logging

Use the withOpenAI method to wrap your OpenAI client with automatic logging:

TypeScript
Python

import OpenAI from "openai";

// Create OpenAI client with observability
const openai = new OpenAI(
  payments.observability.withOpenAI(
    "your-openai-api-key",
    agentRequest, // From payment validation
    {
      userid: "your-user-id", // optional custom property
      operation: "financial_advice", // optional custom property
    }
  )
);

// All OpenAI calls will be automatically logged
const completion = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: messages,
  temperature: 0.3,
  max_tokens: 250,
});

3. Configure Langchain with Observability Logging

Use the withLangchain method to wrap your Langchain client with automatic logging:

TypeScript
Python

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI(
  payments.observability.withLangchain(
    "gpt-4o-mini",
    "your-openai-api-key",
    agentRequest,
    {
      userid: "your-user-id", // optional custom property
      operation: "financial_advice", // optional custom property
    }
  )
);

Advanced Usage

Manual Operation Logging

Documentation for Manual Operation Logging is coming soon. This section will cover how to wrap custom operations with observability logging for non-OpenAI or Langchain services.

Custom Properties and Metadata

Add custom properties to track additional context:

TypeScript
Python

const customProperties = {
  sessionid: sessionId,
  operation: "investment_analysis",
  userId: "user-123",
  requestType: "portfolio_review",
  timestamp: new Date().toISOString(),
};

Usage Calculation Helpers

Documentation for usage calculation helpers for video and audio operations is coming soon. This section will cover how to calculate usage metrics for different types of AI operations.

Batch vs Regular Requests

You can process requests in two modes:

Regular Requests: Process one request at a time. Each request gets its own unique agent request ID.
Batch Requests: Process multiple requests together using the same agent request ID. This is useful when you need to make multiple AI calls (e.g., multiple OpenAI requests) within a single user request, and you want to redeem credits once at the end for all operations combined.

TypeScript
Python

// Regular request - process one request at a time
const agentRequest = await payments.requests.startProcessingRequest(
  agentId,
  authHeader,
  requestedUrl,
  httpVerb
);

// Batch request - process multiple requests together
const agentRequest = await payments.requests.startProcessingBatchRequest(
  agentId,
  authHeader,
  requestedUrl,
  httpVerb
);

Credit Redemption Strategies

There are two ways to redeem credits after processing a request:

Important: The redemption method you use must match how you started the request:

If you used startProcessingRequest(), use redeemCreditsFromRequest() or redeemWithMarginFromRequest()
If you used startProcessingBatchRequest(), use redeemCreditsFromBatchRequest() or redeemWithMarginFromBatchRequest()

Fixed Credit Redemption

Charges a specific number of credits per request. Useful for predictable, pay-per-use models. The credit amount can be a static value or calculated dynamically (e.g., based on token usage, API calls made, or other metrics).

TypeScript
Python

// Credit amount can be static or dynamically computed
const creditAmount = BigInt(10); // Static value
// OR
const creditAmount = calculateCreditAmount(tokensUsed, maxTokens); // Computed value

// Redeem fixed credits from a regular request
const redemptionResult = await payments.requests.redeemCreditsFromRequest(
  agentRequestId,
  requestAccessToken,
  creditAmount
);

// Redeem fixed credits from a batch request
const redemptionResult = await payments.requests.redeemCreditsFromBatchRequest(
  agentRequestId,
  requestAccessToken,
  creditAmount
);

// Extract credits redeemed
const creditsRedeemed = redemptionResult.data?.amountOfCredits || 0;

Margin-based Redemption

Charges the actual API cost plus a margin percentage. Useful for adding a service fee on top of API costs. The margin percentage can be a static value or calculated dynamically (e.g., based on business logic, user tier, or market conditions). For example, if an API call costs 10 cents and you set a 20% margin (0.2), the total charge will be 10 + (10 × 0.2) = 12 cents in dollar-equivalent credits.

TypeScript
Python

// Margin percentage can be static or dynamically computed
const marginPercent = 0.2; // Static 20% margin
// OR
const marginPercent = calculateMargin(userTier, apiCost); // Computed value

// Redeem with margin from a regular request
const redemptionResult = await payments.requests.redeemWithMarginFromRequest(
  agentRequestId,
  requestAccessToken,
  marginPercent
);

// Redeem with margin from a batch request
const redemptionResult = await payments.requests.redeemWithMarginFromBatchRequest(
  agentRequestId,
  requestAccessToken,
  marginPercent
);

// Extract credits redeemed
const creditsRedeemed = redemptionResult.data?.amountOfCredits || 0;

Complete Example

Here’s a complete example showing how to integrate observability into an AI agent:

TypeScript
Python

import express from "express";
import OpenAI from "openai";
import { Payments } from "@nevermined-io/payments";

const app = express();
app.use(express.json());

const payments = Payments.getInstance({
  nvmApiKey: process.env.NVM_API_KEY!,
  environment: "sandbox",
});

app.post("/ask", async (req, res) => {
  try {
    // Validate payment and get agent request
    const agentRequest = await payments.requests.startProcessingRequest(
      process.env.AGENT_ID!,
      req.headers.authorization as string,
      req.url,
      req.method
    );

    // Check user has sufficient balance
    if (
      !agentRequest.balance.isSubscriber ||
      agentRequest.balance.balance < 1n
    ) {
      return res.status(402).json({ error: "Payment Required" });
    }

    const { input_query, sessionId, userId } = req.body;

    // Set up observability metadata
    const customProperties = {
      sessionid: sessionId,
      operation: "financial_advice",
      userid: userId,
    };

    // Create OpenAI client with observability
    const openai = new OpenAI(
      payments.observability.withOpenAI(
        process.env.OPENAI_API_KEY!,
        agentRequest,
        customProperties
      )
    );

    // Make the AI call (automatically logged)
    const completion = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "You are a financial advisor..." },
        { role: "user", content: input_query },
      ],
      temperature: 0.3,
      max_tokens: 250,
    });

    const response = completion.choices[0]?.message?.content;

    // Redeem credits
    await payments.requests.redeemCreditsFromRequest(
      agentRequest.agentRequestId,
      req.headers.authorization?.replace("Bearer ", "")!,
      BigInt(10) // Credit amount
    );

    res.json({
      output: response,
      sessionId: customProperties.sessionid,
      tokensUsed: completion.usage?.total_tokens,
    });

  } catch (error) {
    console.error("Error:", error);
    res.status(500).json({ error: "Internal server error" });
  }
});

Additional Examples

Batch Mode

Complete example using batch request processing. This example shows making multiple AI calls within a single batch request, all sharing the same agent request ID:

TypeScript
Python

// Use batch processing - all operations will share the same agent request ID
const agentRequest = await payments.requests.startProcessingBatchRequest(
  process.env.AGENT_ID!,
  req.headers.authorization as string,
  req.url,
  req.method
);

// Check user has sufficient balance
if (!agentRequest.balance.isSubscriber || agentRequest.balance.balance < 1n) {
  return res.status(402).json({ error: "Payment Required" });
}

const requestAccessToken = req.headers.authorization?.replace("Bearer ", "")!;

// Create OpenAI client with observability
const openai = new OpenAI(
  payments.observability.withOpenAI(
    process.env.OPENAI_API_KEY!,
    agentRequest,
    { operation: "batch_analysis" }
  )
);

// Make multiple AI calls in the same batch
// Call 1: Analyze user query
const analysis = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Analyze this investment strategy..." }],
  max_tokens: 100,
});

// Call 2: Generate recommendations
const recommendations = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Provide investment recommendations..." }],
  max_tokens: 100,
});

// Call 3: Create summary
const summary = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Summarize the analysis..." }],
  max_tokens: 50,
});

// Calculate total credits for all operations
const totalCredits = BigInt(30); // Or calculate based on actual token usage

// Redeem credits once for all operations in the batch
await payments.requests.redeemCreditsFromBatchRequest(
  agentRequest.agentRequestId,
  requestAccessToken,
  totalCredits
);

res.json({
  analysis: analysis.choices[0]?.message?.content,
  recommendations: recommendations.choices[0]?.message?.content,
  summary: summary.choices[0]?.message?.content,
});

Margin-based Pricing

Complete example using margin-based credit redemption:

TypeScript
Python

// Regular request processing
const agentRequest = await payments.requests.startProcessingRequest(
  process.env.AGENT_ID!,
  req.headers.authorization as string,
  req.url,
  req.method
);

// ... rest of your agent logic ...

// Redeem with margin percentage (e.g., 20% markup on API costs)
const marginPercent = 0.2;
await payments.requests.redeemWithMarginFromRequest(
  agentRequest.agentRequestId,
  requestAccessToken,
  marginPercent
);

Batch Mode + Margin-based Pricing

Complete example combining batch processing with margin-based pricing. All operations share the same agent request ID, and credits are redeemed once at the end based on actual API costs plus margin:

TypeScript
Python

// Use batch processing - all operations will share the same agent request ID
const agentRequest = await payments.requests.startProcessingBatchRequest(
  process.env.AGENT_ID!,
  req.headers.authorization as string,
  req.url,
  req.method
);

// Check user has sufficient balance
if (!agentRequest.balance.isSubscriber || agentRequest.balance.balance < 1n) {
  return res.status(402).json({ error: "Payment Required" });
}

const requestAccessToken = req.headers.authorization?.replace("Bearer ", "")!;

// Create OpenAI client with observability
const openai = new OpenAI(
  payments.observability.withOpenAI(
    process.env.OPENAI_API_KEY!,
    agentRequest,
    { operation: "batch_analysis" }
  )
);

// Make multiple AI calls in the same batch
// Call 1: Analyze user query
const analysis = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Analyze this investment strategy..." }],
  max_tokens: 100,
});

// Call 2: Generate recommendations
const recommendations = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Provide investment recommendations..." }],
  max_tokens: 100,
});

// Call 3: Create summary
const summary = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Summarize the analysis..." }],
  max_tokens: 50,
});

// Redeem with margin using batch method
// The margin is applied to the total API cost of all three calls
const marginPercent = 0.2; // 20% markup on total API costs
await payments.requests.redeemWithMarginFromBatchRequest(
  agentRequest.agentRequestId,
  requestAccessToken,
  marginPercent
);

res.json({
  analysis: analysis.choices[0]?.message?.content,
  recommendations: recommendations.choices[0]?.message?.content,
  summary: summary.choices[0]?.message?.content,
});

Monitoring and Analytics

Events Log Table

The frontend provides a comprehensive events log table showing:

Screenshot of the events log table showing detailed request information, timestamps, costs, and status

Screenshot of the events log table showing details of a specific request

Key Metrics Tracked

The observability integration automatically tracks:

Timestamp: When each request was made
User: Account address and session information
Agent: Agent name and operation type
Request: Input query and context
Response: AI-generated response
Cost Analysis: Credit consumption and cost per request
Status: Success/failure status
Performance Metrics: Response times, success rates, error rates, token usage, and more

Data Analytics Dashboard

Once your agent is running with observability enabled, you can view detailed data analytics in the Nevermined dashboard:

Screenshot showing the Nevermined analytics dashboard

Data Analytics Dashboard Cumulative Analysis

Screenshot showing the Nevermined analytics dashboard cumulative analysis

Data Analytics Dashboard Summmary Analysis

Screenshot showing the Nevermined analytics dashboard summary analysis

Key Metrics Tracked

The observability integration automatically tracks:

Per Request Analysis: Cost, credit revenue, PnL, by agent, model, and more
Cumulative Analysis: Cost, credit revenue, PnL, by agent, model, and more
Summary Analysis: Cost, credit revenue, PnL, by agent, model, and more

Filtering and Search

Use the built-in filtering capabilities to analyze specific patterns:

Screenshot showing filtering options for date range, agent type, user, and more

Screenshot showing filtering options for agent, model, provider, and more

Next Steps

Process Requests

Learn how to handle and validate paid requests in your agents

Query Agents

Learn how to query AI agents programmatically

Getting Started

Development Guide

Integrations

Observability & Monitoring

Observability & Monitoring for AI Agents

Overview

Basic Integration

1. Initialize the Observability API

2. Configure OpenAI with Observability Logging

3. Configure Langchain with Observability Logging

Advanced Usage

Manual Operation Logging

Custom Properties and Metadata

Usage Calculation Helpers

Batch vs Regular Requests

Credit Redemption Strategies

Fixed Credit Redemption

Margin-based Redemption

Complete Example

Additional Examples

Monitoring and Analytics

Events Log Table

Key Metrics Tracked

Data Analytics Dashboard

Key Metrics Tracked

Filtering and Search

Next Steps

Process Requests

Query Agents

Getting Started

Development Guide

Integrations

​Observability & Monitoring for AI Agents

​Overview

​Basic Integration

​1. Initialize the Observability API

​2. Configure OpenAI with Observability Logging

​3. Configure Langchain with Observability Logging

​Advanced Usage

​Manual Operation Logging

​Custom Properties and Metadata

​Usage Calculation Helpers

​Batch vs Regular Requests

​Credit Redemption Strategies

​Fixed Credit Redemption

​Margin-based Redemption

​Complete Example

​Additional Examples

​Monitoring and Analytics

​Events Log Table

​Key Metrics Tracked

​Data Analytics Dashboard

​Key Metrics Tracked

​Filtering and Search

​Next Steps

Process Requests

Query Agents

Observability & Monitoring for AI Agents

Overview

Basic Integration

1. Initialize the Observability API

2. Configure OpenAI with Observability Logging

3. Configure Langchain with Observability Logging

Advanced Usage

Manual Operation Logging

Custom Properties and Metadata

Usage Calculation Helpers

Batch vs Regular Requests

Credit Redemption Strategies

Fixed Credit Redemption

Margin-based Redemption

Complete Example

Additional Examples

Monitoring and Analytics

Events Log Table

Key Metrics Tracked

Data Analytics Dashboard

Key Metrics Tracked

Filtering and Search

Next Steps