No entries match these filters. Clear filters
Openlayer adds a governance API endpoint to list projects associated with a compliance framework︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ USE IT
Audit which projects are enrolled under a specific compliance framework and inspect their monitoring goal counts.
python
copy
from openlayer import Openlayer
client = Openlayer()
projects = client.governance.frameworks.list_projects("5e6f7a8b-9c0d-4e1f-8a2b-3c4d5e6f7a8b")
for project in projects.items:
print(project.id, project.name)
List all projects linked to a governance framework via client.governance.frameworks.list_projects(<framework-id>) in the Python SDK, returning items with fields including goalCount, developmentGoalCount, monitoringGoalCount, dataRetentionDays, and taskType.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet
1 detail in Everything
Openlayer's Retrieve Workspace API now returns a samlOnlyAccess field in the workspace response object︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Check whether a workspace enforces SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. -only access before provisioning users programmatically.
copy
$ curl -X GET https://api.openlayer.com/v1/workspaces/3c90c3cc-0d44-4b50-8888-8dd25736052a \
-H 'Authorization: Bearer <token>' | jq '.samlOnlyAccess'
Retrieve workspace details via GET /workspaces/{id} now includes the samlOnlyAccess boolean field, letting callers detect whether a workspace enforces SAML-only login.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet
1 detail in Everything
Openlayer adds time-range and filter options to the export CLI command︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Export the last 7 days of data from a pipeline to investigate a recent drift in model outputs.
copy
$ openlayer export my-pipeline --last 7d
Export a precise date range from a pipeline using ISO 8601 ISO 8601 An international standard for representing dates and times as text, using formats like 2025-04-23T14:00:00Z. Tools adopt it to ensure timestamps are unambiguous and sortable across systems and locales. timestamps for a post-incident audit.
copy
$ openlayer export my-pipeline --from 2025-08-01T10:00:00 --to 2025-08-10T18:00:00
Export this week's pipeline data with a filter applied to narrow results.
copy
$ openlayer export my-pipeline --range this-week --filter
Export pipeline data within a time window using --from and --to flags, accepting ISO ISO A widely adopted international standards body that publishes technical specifications; cyber tools cite ISO standards to signal compliance with recognized benchmarks for quality, security, or interoperability. 8601 timestamps or Unix epoch integers (e.g.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ --from 2025-08-01T10:00:00 --to 2025-08-10T18:00:00). Not checked yet Export the last N hours or days of pipeline data using --last with durations like 7d or 2h30m. Not checked yet Export a named calendar period using --range (e.g. --range this-week). Not checked yet Filter exported pipeline data using the --filter flag. Not checked yet
4 details in Everything
Openlayer adds project and governance webhook events covering lifecycle, risk, intake, and state changes with full project payloads︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Mirror Openlayer's AI project inventory into a GRC GRC Governance, Risk, and Compliance — a framework for aligning IT controls with business policy, regulatory requirements, and risk management so organizations can audit and report on their posture systematically. tool by upserting on any project event — the full record is always present so no incremental diffing is needed.
copy
$ POST /your-webhook-endpoint
Content-Type: application/json
{
"event": "project.risk.assessed",
"data": {
"project": {
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"risk": {
"level": "critical",
"totalScore": 90,
"maxScore": 100,
"scores": [
{"dimensionId": "6f1b2c3d-1111-4a2b-9c3d-000000000010", "label": "Patient harm", "score": 50}
]
}
},
"actorId": "user-abc"
}
}
Detect when a project advances or is retired in the AI lifecycle so a downstream compliance system can trigger the appropriate review workflow.
python
copy
# In your webhook handler, inspect data.lifecycle.kind
# kind values: advance | revert | retire | reopen
if event['event'] == 'project.lifecycle.changed':
lifecycle = event['data']['lifecycle']
if lifecycle['kind'] == 'retire':
trigger_retirement_review(event['data']['project']['id'])
elif lifecycle['kind'] == 'advance':
notify_next_stage_owner(lifecycle['toStage']['name'])
Adds new webhook events for project changes: project.created (with the project's full current state in data.project and actor in data.actorId), project.updated (with data.changes listing each changed field's previous and new value), project.deleted and project.restored (setting or clearing dateDeleted), project.intake.updated (with data.intake.changedFieldIds and full updated answers in data.project.intake), project.risk.assessed (with level, totalScore, maxScore, and per-dimension scores in data.project.risk), project.risk_exception.updated (with data.riskException.action of set or revoked and a note when set), project.lifecycle.changed (with data.lifecycle containing kind of advance, revert, retire, or reopen, plus fromStage, toStage, and reason), and project.lifecycle.approved (with data.approval naming the approved stage and the reviewer's note, after which Openlayer sends project.lifecycle.approved followed by project.lifecycle.changed when it is the final approval needed).︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Every project.* event carries the project's full current state in data.project, enabling upsert-by-ID sync to external systems such as GRC tools or AI inventories without incremental change tracking. Not checked yet
2 details in Everything
Openlayer adds a governance API covering frameworks, rules, rule results, and evidence endpoints︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Create a governance framework in your workspace, then post evidence against a failing rule result in a CI pipeline.
copy
$ # 1. Create a framework
curl --request POST \
--url https://api.openlayer.com/v1/workspaces/<WORKSPACE_ID>/frameworks \
--header 'Authorization: Bearer <TOKEN>' \
--header 'Content-Type: application/json' \
--data '{"name": "EU AI Act"}'
# 2. Attach evidence to a rule result that failed
curl --request POST \
--url https://api.openlayer.com/v1/rule-results/<RULE_RESULT_ID>/evidence \
--header 'Authorization: Bearer <TOKEN>' \
--header 'Content-Type: application/json' \
--data '{"note": "Drift threshold exceeded in production batch 42"}'
Poll an async governance operation until it completes — useful after creating or updating a framework that triggers background processing.
copy
$ curl --request GET \
--url https://api.openlayer.com/v1/background-tasks/<TASK_ID> \
--header 'Authorization: Bearer <TOKEN>'
(new) Adds framework management endpoints: list documents with GET /v1/frameworks/<FRAMEWORK_ID>/documents, retrieve a single document with GET /v1/frameworks/<FRAMEWORK_ID>/documents/<DOCUMENT_ID>, create a framework in a workspace with POST /v1/workspaces/<WORKSPACE_ID>/frameworks, and update a framework with PUT /v1/frameworks/<FRAMEWORK_ID>.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet (new) Adds rule result evidence endpoints: fetch evidence with GET /v1/rule-results/<RULE_RESULT_ID>/evidence and attach evidence with POST /v1/rule-results/<RULE_RESULT_ID>/evidence. Not checked yet (new) Adds rule management endpoints: create a rule in a workspace with POST /v1/workspaces/<WORKSPACE_ID>/rules, update a rule with PUT /v1/rules/<RULE_ID>, and delete a rule with DELETE /v1/rules/<RULE_ID>. Not checked yet Poll the status of an async governance operation with GET /v1/background-tasks/<TASK_ID>. Not checked yet
4 details in Everything
Openlayer adds user rules to its OpenAI compliance integration to route or suppress individual users' logs by email, user ID, or domain︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Route a specific user's ChatGPT and Codex logs to a restricted project so only an access group can read them, keeping them out of the main compliance project.
📍 In the Openlayer integration page, go to the OpenAI compliance connection › General tab › User rules. Add a rule: match by email (e.g. contractor@example.com), action: Route to project , select the target data source in the same workspace.
Suppress all logs from an entire email domain (e.g. internal testers) so they are never ingested into Openlayer.
📍 In the Openlayer integration page, go to the OpenAI compliance connection › General tab › User rules. Add a rule: match by email domain (e.g. testdomain.com), action: Ignore . Rules take effect from the next sync.
Add user rules on the General tab of the OpenAI compliance connection to either ignore a user's logs entirely or route them to a specific project data source, matched by email address, user ID, or email domain (case-insensitive).︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet When multiple rules match a user, ignore takes precedence over route , and an exact email or user ID takes precedence over a domain rule — giving admins precise control over log routing priority. Not checked yet Route rules require the destination data source to be in the same workspace and use Openlayer's default storage; if that data source is deleted, matching sessions are skipped and an error is shown on the integration page until the rule is updated or removed. Not checked yet
3 details in Everything
Openlayer adds an Open WebUI integration with configurable tracing valves for LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. calls, tool calls, sources, attachments, and multi-pipeline routing.
New integration Configure the Open WebUI integration via Valves settings including openlayer_api_key, inference_pipeline_id, capture_tool_calls, capture_sources, upload_attachments, max_attachment_mb, routes, and debug to control what gets traced to Openlayer. Not checked yet Run the Open WebUI container with OPENLAYER_API_KEY and OPENLAYER_INFERENCE_PIPELINE_ID environment variables to connect it to an Openlayer data source. Not checked yet Openlayer now traces LLM chat completions via StepType.CHAT_COMPLETION, retrieval steps via StepType.RETRIEVER, tool calls via StepType.TOOL, and user turns via StepType.USER_CALL inside Open WebUI sessions. Not checked yet Route traces to multiple Openlayer inference pipelines by setting the routes valve to a JSON mapping of model-name keys to pipeline-ID values. Not checked yet Attach audio and file content from Open WebUI messages to Openlayer traces, with per-attachment size capped by max_attachment_mb (default in MB) and deduplication by MD5 checksum. Not checked yet
+1 more
show less
Feed back ratings from Open WebUI to Openlayer inference pipelines via an outlet action that calls update_current_trace with an inferenceId tied to the message. Not checked yet
6 details in Everything
Openlayer adds per-user routing and ignore rules for Microsoft Teams bot integrations︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Route a contractor's conversations to a restricted project so only the right access group can read them, while keeping the main bot projects clean.
📍 In the Microsoft Teams integration, go to the User rules tab, click Add rule , choose Email domain , enter the contractor domain (e.g. contractor.example.com), select Route to another project , and pick the restricted project's data source.
Prevent a specific executive's chats from ever being ingested by Openlayer, identified reliably by Entra object ID rather than a changeable email.
📍 In the Microsoft Teams integration, go to the User rules tab, click Add rule , choose Entra object ID , enter the user's object ID (e.g. 00000000-0000-0000-0000-000000000000), and select Ignore their logs .
Add user rules on the Microsoft Teams integration to keep a user's conversations out of Openlayer entirely with Ignore their logs , or send them to a chosen project with Route to another project .︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Match users by email, email domain (such as example.com), or Entra object ID (such as 00000000-0000-0000-0000-000000000000) when creating a user rule — matching is case-insensitive. Not checked yet Ignored users do not count toward the 200-users-read-per-sync limit. Not checked yet With multiple tenants connected, select a tenant on the connections tab to view and manage that tenant's rules independently. Not checked yet
4 details in Everything
Openlayer: Route Microsoft 365 Copilot user logs to a separate project by matching email, email domain, or Entra object ID , keeping them out of default projects.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Exclude a contractor's Copilot activity from all Openlayer ingestion by creating an ignore rule matched on their exact email address.
📍 In the Microsoft 365 Copilot connection settings, go to the User rules tab › Add rule › Match: Email › enter the contractor's address › Action: Ignore their logs › Save. The rule takes effect from the next sync.
Route an entire department's Copilot logs to a restricted project whose access group limits readers, without affecting other users.
📍 In the Microsoft 365 Copilot connection settings, go to the User rules tab › Add rule › Match: Email domain › enter the domain (e.g. example.com) › Action: Route to another project › select the target project and data source › Save.
Recover held-back logs after a route rule's data source was deleted, by replacing the rule to point at a new project.
📍 In the Microsoft 365 Copilot connection settings: 1) Turn sync off. 2) Delete the rule showing Data source deleted, logs held back . 3) Add a new rule with the same match type, same value, and the replacement project. 4) Turn sync back on. Held-back logs are written to the new project on the next sync.
Exclude a user's Copilot logs from Openlayer entirely with an Ignore their logs rule, so nothing of theirs is stored. Not checked yet Replace a rule without data loss by turning sync off, deleting the old rule, adding a new one with the same match type and value, then turning sync back on. Not checked yet Deleted route rules whose data source was removed surface a Data source deleted, logs held back warning; held-back logs are recovered by adding a replacement rule pointing to a new project. Not checked yet
3 details in Everything
Openlayer adds an Island integration with configurable project routing, data source labeling, and automatic app discovery︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
When multiple teams share one LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. project, use Data source label to keep each connection's Island data separate without creating per-app projects.
📍 In the Island connection settings, set Project for new products to your shared LLM project (e.g. AI apps ) and set Data source label to Sandbox . Discovered apps will write to data sources named ChatGPT (Sandbox) , Claude (Sandbox) , etc., separate from unlabeled connections.
When rolling out Island to a new environment without enabling all discovered apps immediately, turn off automatic enablement so you can review and map each app yourself.
📍 In the Island connection settings, toggle Enable new products automatically off. Then open the Products tab and manually enable each app, choosing an existing project and data source for each one.
Route discovered Island apps to a shared project using the Project for new products setting, where each app gets a data source named after it (such as ChatGPT) inside the chosen LLM project.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Append a label to data sources a connection creates using the Data source label setting — for example ChatGPT (Sandbox) — to keep one connection's data separate from others sharing the same project. Not checked yet Control whether newly discovered Island apps are automatically enabled with the Enable new products automatically setting, which is on by default; turn it off to leave discovered apps disabled until you map them manually. Not checked yet Map an Island app to an existing project and data source when enabling it, instead of letting the connection create a new one. Not checked yet Connections that send an app to the same project with the same Data source label — or both with no label — share one data source for that app; a different label keeps them apart. Not checked yet
+2 more
show less
Per-tenant routing names each tenant's data source as <data source> · <tenant>. Not checked yet Data sources backed by BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. or Snowflake cannot receive Island rows; choose or create a data source on Openlayer's default storage instead. Not checked yet
7 details in Everything
Openlayer: Rules apply from the next sync only; past logs stay put, and routed users' future sessions go to the routed data source.
└──▷ HOW TO FIND IT
Prevent a specific user's Claude sessions from ever being ingested into Openlayer — useful for excluding internal test accounts or privileged users from compliance logging.
📍 In the Claude compliance integration page, go to the User rules tab › Add rule › match by email address (e.g. testuser@example.com) › set action to Ignore › Save. The rule takes effect on the next sync.
Route a sensitive user's Claude logs to a restricted project with an access group, so only authorised reviewers can read them.
📍 In the Claude compliance integration page, go to the User rules tab › Add rule › match by email domain (e.g. executives@example.com) › set action to Route to project › select the restricted project data source (must be in the same workspace, using default storage) › Save.
Openlayer adds a TypeScript traceOpenAI wrapper from openlayer/lib/integrations/openAiTracer to stream OpenAI call traces to an inference pipeline.
New integration Trace OpenAI client calls to an Openlayer inference pipeline by wrapping the OpenAI client with traceOpenAI from openlayer/lib/integrations/openAiTracer, configured via the OPENLAYER_API_KEY and OPENLAYER_INFERENCE_PIPELINE_ID environment variables. Not checked yet
1 detail in Everything
Openlayer adds a custom metrics system with openlayer metrics push and a BaseMetric class for user-defined scoring logic︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Push a custom exact-match metric to Openlayer after implementing it as a BaseMetric subclass.
copy
$ openlayer metrics push -d metrics
Define a parameterised custom metric that reads a threshold and target column from runtime params rather than hard-coded values.
python
copy
class MyMetric(metrics.BaseMetric):
def __init__(self):
super().__init__()
params = load_params()
self.threshold = float(params.get('threshold', 0.5))
self.column_name = params.get('column_name')
def compute_on_dataset(self, dataset: metrics.Dataset) -> metrics.MetricReturn:
score = dataset.df.apply(
lambda x: self.compute_on_row(x, dataset.config), axis=1
).mean()
return metrics.MetricReturn(value=score, unit='fraction')
Push custom metrics to Openlayer using the openlayer metrics push -d <metric_name> CLI command, pointing at a single metric or a metrics directory.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Define custom scoring logic by subclassing metrics.BaseMetric and implementing compute_on_dataset or compute_on_row, which receive a metrics.Dataset and return a metrics.MetricReturn. Not checked yet Load per-metric runtime parameters such as thresholds and column names inside a custom metric via load_params() , enabling configuration without code changes. Not checked yet
3 details in Everything
Openlayer adds a user-level data export to the monitoring Users tab, producing one row per user with usage and cost columns︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Export a filtered view of users by cost to audit which users are driving the most spend in a monitored project.
📍 In Monitoring, open the Users tab, apply a filter on Total cost, then select the Export users (download icon) button in the page header.
Export per-user monitoring data from the Users tab by selecting the Export users download icon in the page header, producing a CSV with columns userId, dateOfFirstRecord, dateOfLastRecord, sessions, records, tokens, and cost.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Users export respects the tab's current filters (including filters on totals such as Total records or Total cost ), so the downloaded file contains exactly the users currently shown in the table. Not checked yet
2 details in Everything
Openlayer now publishes an API — 63 endpoints across 10 areas: Governance, Monitoring, Workspaces, …︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Openlayer adds renewal_cadence_days and assignee_id parameters to the governance rules update API︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Set a recurring review schedule for governance rules by passing renewal_cadence_days to client.governance.rules.update() , with the cadence returned on the rule object as rule.renewal_cadence_days.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Assign an owner to a governance rule by passing assignee_id to client.governance.rules.update() . Not checked yet
2 details in Everything
Openlayer adds a Governance Frameworks API to create and track AI policy compliance controls︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ USE IT
Bootstrap an internal AI governance framework and retrieve its ID for downstream policy enforcement workflows.
python
copy
from openlayer import Openlayer
client = Openlayer()
framework = client.governance.frameworks.create(
name="Internal AI policy",
description="Controls every production AI system must meet.",
tags=["internal"]
)
print(framework.id)
Create governance frameworks programmatically via client.governance.frameworks.create() , specifying a name, description, and tags such as 'internal'.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Track per-framework compliance status across fields including total, totalPassing, totalFailing, totalSkipped, totalRunning, totalError, totalPending, and totalDueSoon. Not checked yet Inspect risk-level breakdowns (high, mid, low) within each framework's status response. Not checked yet
3 details in Everything
Openlayer adds a -d flag to the openlayer metrics push CLI command︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Push a single named metric instead of all metrics, useful in CI pipelines where only one metric has changed.
copy
$ openlayer metrics push -d metric_name
Openlayer adds an API key rotation endpoint with a configurable grace period for workspace API keys︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ USE IT
Rotate a workspace API key while keeping the old key alive for a grace period, so in-flight requests using the previous key still succeed.
python
copy
from openlayer import Openlayer
client = Openlayer()
api_key = client.workspaces.api_keys.rotate(
workspace_id="182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e",
grace_period_hours=24
)
print(api_key.secret)
Rotate workspace API keys via client.workspaces.api_keys.rotate(workspace_id=..., grace_period_hours=...) , with the response including secret, expiresAt, lastRotatedAt, and previousKeyExpiresAt fields to manage key transitions.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet
1 detail in Everything
Openlayer adds an Open WebUI integration that traces chat completions, tool calls, retrieval sources, and attachments to inference pipelines︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Start Open WebUI with Openlayer tracing enabled so every chat completion is sent to your inference pipeline.
copy
$ docker run -d -p 3000:8080 \
-e OPENLAYER_API_KEY=YOUR_OPENLAYER_API_KEY \
-e OPENLAYER_INFERENCE_PIPELINE_ID=YOUR_DATA_SOURCE_ID \
-v open-webui:/app/backend/data \
--name open-webui --restart always \
ghcr.io/open-webui/open-webui:main
Configure the Open WebUI Docker container with OPENLAYER_API_KEY and OPENLAYER_INFERENCE_PIPELINE_ID environment variables to route traces to an Openlayer inference pipeline.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Control tracing behaviour with Valves settings including capture_tool_calls, capture_sources, upload_attachments, max_attachment_mb (default 20), routes, and debug. Not checked yet Openlayer's Open WebUI filter now captures audio, images, and file attachments (via base64 data URLs or /api/v1/files/ file IDs), deduplicating by MD5 checksum and skipping files that exceed max_attachment_mb. Not checked yet Openlayer's Open WebUI filter records chat completions as StepType.CHAT_COMPLETION traces, tool calls as StepType.TOOL, and retrieval context as StepType.RETRIEVER, with per-step latency derived from step.latency = (step.end_time - start) * 1000. Not checked yet Openlayer's Open WebUI feedback handler now propagates user feedback events across multiple inference pipelines defined in the routes valve, using the inference_pipeline_id and inference_id from the trace. Not checked yet
5 details in Everything
Openlayer adds a Google Gemini integration, letting practitioners wrap the Gemini client with traceGoogleGenAI to capture model call traces.
New integration Trace Google Gemini model calls by wrapping the Gemini client with traceGoogleGenAI to send observability data to Openlayer. Not checked yet
1 detail in Everything
Openlayer adds workspace- and project-level environment variables to pass credentials to LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. judges, custom metrics, and development mode runs︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Route an LLM judge through an internal OpenAI-compatible gateway by setting the base URL and any required auth headers as project-level environment variables.
📍 Settings → Environment (project page) → OpenAI group:
OPENAI_API_KEY=<your-key>
OPENAI_BASE_URL=https://internal-gateway.corp/v1
OPENAI_CUSTOM_HEADERS=X-Gateway-Token:abc123,X-Team:security
Trust an internal CA certificate so Openlayer's LLM judge and dev-mode runs can reach a model behind a corporate TLS gateway.
📍 Settings → Environment (workspace page) → add a plain variable:
Name: SSL_CERT_GATEWAY
Value: -----BEGIN CERTIFICATE-----
<PEM-encoded certificate>
-----END CERTIFICATE-----
Read a custom workspace variable inside a dev-mode batch script or custom metric without storing it in the repo.
python
copy
import os
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY injected by Openlayer
retriever_url = os.environ["RETRIEVER_URL"] # custom variable set under Settings → Environment
Store API keys and configuration in Openlayer at the workspace or project level as environment variables, so credentials never appear in your repository or in openlayer.json.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Development mode runs and custom metrics now receive every workspace and project environment variable as a process environment variable during installCommand, batchCommand, and runCommand execution, readable via os.environ in Python or process.env in Node. Not checked yet Openlayer automatically injects OPENLAYER_API_KEY, OPENLAYER_PROJECT_ID, and OPENLAYER_WORKSPACE_ID into every development mode run, so those do not need to be stored manually. Not checked yet Project-level environment variables now override workspace-level variables of the same name within that project. Not checked yet Add custom CA certificates by creating environment variables whose names start with SSL_CERT_, followed by a descriptive suffix (for example, SSL_CERT_GATEWAY), with the certificate in PEM format as the value; Openlayer trusts them in development mode runs, custom metrics, and LLM judge connections. Not checked yet
5 details in Everything
Openlayer adds governance lifecycle stages, per-step test evaluation, EU data residency, Gateway routing/allow-block lists, new API endpoints, and multiple project dashboards.
Adds evaluationScope: step config field for per-step evaluation of tests on each trace step. Not checked yet Adds openlayer tests delete CLI command — archives by default, --force permanently deletes. Not checked yet Extends openlayer init with Copilot CLI and Gemini CLI support, custom CA bundles, a 'debug with an agent' option, and .openlayerignore for smaller uploads. Not checked yet Adds Gateway GET /v1/models (per key) and OpenAI-compatible POST /v1/embeddings API endpoints. Not checked yet Adds governance registry and framework endpoints to the public API and SDKs, writable with a workspace API key. Not checked yet
+26 more
show less
Adds a new batch-delete endpoint for asynchronous, batched, idempotent row deletes. Not checked yet Adds EU data region — choose United States or European Union at sign-in/sign-up; account and data stay in the selected region. Not checked yet Adds project lifecycle stages — define stages, assign reviewers per stage, and receive warnings when an earlier stage's requirements are no longer met. Not checked yet Adds audit-ready framework export — export a compliance report for any framework. Not checked yet Adds custom intake and risk-assessment forms with AI-assisted completion. Not checked yet Adds redesigned framework activation — pick projects explicitly, add criteria for future projects, preview tests before creation, and set custom framework rules during creation. Not checked yet New detection New integration Adds Custom columns — a named JSONPath JSONPath A query language for extracting values from JSON documents, modeled after XPath for XML. Cyber tools use it to pinpoint specific fields in structured data without parsing the whole document manually. into structured outputs structured outputs Forcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. , defined once per project and usable in tests, filters, and tables across every data source and development commit. Not checked yet Adds Jev-as-a-judge test that scores every row in the window and explains its score on demand. Not checked yet Adds support for upper-only or lower-only automatic thresholds. Not checked yet Adds AI-powered insights controls — workspace and project admins can enable/disable insights, override the judge model, and lock settings. Not checked yet Adds ability to duplicate a test or set of tests into other projects, and export test configs as JSON from the results page. Not checked yet Adds workspace-wide project tags with names and colors, and test tags settable from a pushed tests.json. Not checked yet Adds multiple dashboards per project — create from the default layout or blank, rename, duplicate, delete, set visibility, and switch via breadcrumb or ⌘K. Not checked yet Adds mode-aware usage, cost, and judge-cost widgets on the project dashboard; token consumption on the workspace dashboard; and provider grouping in cost-by-model. Not checked yet Adds LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. cost table — view and edit token costs with workspace-level overrides. Not checked yet Adds Observations view — inspect and compare steps across traces. Not checked yet Adds row comments with threaded replies, @mentions, and a Feedback filter on the rows table. Not checked yet Adds custom metric run logs and errors surfaced in the UI. Not checked yet Adds ability to recover monitoring data after a project is deleted. Not checked yet Adds API key expiry dates, automatic rotation, and full API key management. Not checked yet Updated supported platform Adds Gateway routing by deployment, group, user, or key with narrowest-match precedence. Not checked yet Adds Gateway model access allow and block lists, scoped to key, user, group, or deployment. Not checked yet Adds Gateway All traffic observability destination that publishes every request. Not checked yet Adds Gateway high availability — safe on multiple replicas with health endpoints and graceful shutdown; admin changes and usage limits apply within seconds. Not checked yet Adds development-mode test result-over-time graph; development-mode tests can be paused. Not checked yet Adds test result filters and open-row state persisted in the URL, so reloads and shared links restore them. Not checked yet
31 details in Everything
└──▷ BREAKING ON UPGRADE
!
Gateway 'Teams' are renamed to 'Groups' — any configuration, API calls, or documentation referencing 'Teams' in the Gateway admin UI must be updated to 'Groups'.
Our best guess
Openlayer gateway documents error formats, check ordering, 503 causes, and X-Request-ID tracing across all endpoints.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Attach a correlation ID to every API call so your operator can pinpoint the exact request behind a 422 guardrail block or 429 limit error in gateway logs.
copy
$ curl https://your-gateway.example.com/v1/responses \
-H "Authorization: Bearer sk-olga-..." \
-H "X-Request-ID: checkout-7f3a2c" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-4o-mini", "input": "Hello!"}'
Adds X-Request-ID request header support so operators can trace exact gateway requests in logs and as the Openlayer trace ID.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Documents error format per endpoint: /v1/messages returns Anthropic shape; /v1/responses and /v1/embeddings return OpenAI shape; /v1/models returns shape based on path (/anthropic/v1/models or /openai/v1/models) or SDK headers on bare /v1/models. Not checked yet Adds error type mapping from HTTP status code: invalid_request_error (400/422), authentication_error (401), permission_error (403), not_found_error (404), rate_limit_error (429), api_error (5xx). Not checked yet Documents 503 Retry-After headers per message prefix: gateway at capacity and no upstream capacity for carry 1 s; guardrail loading and database unavailable carry 5 s; provider config errors carry none. Not checked yet Documents /v1/images/generations error body as {"detail": ...} with guardrail blocks carrying type, stage, and reason fields rather than the OpenAI shape. Not checked yet
+2 more
show less
Documents gateway check order (API key → model access → usage limits → input guardrails → routing/provider → output guardrails) determining which error a caller sees when multiple checks could fail. Not checked yet Documents that 422 guardrail blocks on the way out still count tokens toward usage limits. Not checked yet
7 details in Everything
Governance lifecycle stages, bias and fairness tests, and multiple project dashboards
seen Oct 21, 2026
·
issue 115
Openlayer adds governance lifecycle stages, bias/fairness tests, 8 new integrations, Gateway routing, and a public governance API.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Delete a specific test permanently from the CLI, bypassing the default archive behavior.
copy
$ openlayer tests delete <test-id> --force
Enable per-step evaluation so tests run against each trace step independently, not just the final output.
tests.json
copy
{
"evaluationScope": "step"
}
Adds evaluationScope: step config field for per-step evaluation — run tests on each individual trace step.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds openlayer tests delete CLI command — archives tests by default; --force flag permanently deletes them. Not checked yet Extends openlayer init to support Copilot CLI and Gemini CLI, custom CA bundles, a 'debug with an agent' option, and .openlayerignore for smaller uploads. Not checked yet Adds governance registry and framework endpoints to the public API and SDKs, writable with a workspace API key. Not checked yet Adds API key expiry dates, automatic rotation, and full API key management via the API. Not checked yet
+5 more
show less
Adds bias and fairness tests — four test types check protected groups and their intersections for outcome, score, performance, and metric disparities, with an Analysis tab for charts, drill-down, and CSV export. Not checked yet Adds prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. tests — choose the detector and a sensitivity level (multilingual included), see why a detection fired, and pick the judge model. Not checked yet Adds Freshness and Column freshness tests — alert when a data source stops receiving data or a datetime column gets too old, with manual or automatic thresholds. Not checked yet Adds Custom columns — a named JSONPath JSONPath A query language for extracting values from JSON documents, modeled after XPath for XML. Cyber tools use it to pinpoint specific fields in structured data without parsing the whole document manually. into structured outputs structured outputs Forcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. , defined once per project and usable in tests, filters, and tables across every data source and development commit. Not checked yet Adds Jev-as-a-judge test that scores every row in the window and explains its score on demand. Not checked yet
Adds evaluationScope: step config field for per-step evaluation — run tests on each individual trace step.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds openlayer tests delete CLI command — archives tests by default; --force flag permanently deletes them. Not checked yet Adds governance registry and framework endpoints to the public API and SDKs, writable with a workspace API key. Not checked yet Adds API key expiry dates, automatic rotation, and full API key management via the API. Not checked yet Adds bias and fairness tests — four test types check protected groups and their intersections for outcome, score, performance, and metric disparities, with an Analysis tab for charts, drill-down, and CSV export. Not checked yet
+4 more
show less
Adds prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. tests — choose the detector and a sensitivity level (multilingual included), see why a detection fired, and pick the judge model. Not checked yet Adds Freshness and Column freshness tests — alert when a data source stops receiving data or a datetime column gets too old, with manual or automatic thresholds. Not checked yet Adds Custom columns — a named JSONPath JSONPath A query language for extracting values from JSON documents, modeled after XPath for XML. Cyber tools use it to pinpoint specific fields in structured data without parsing the whole document manually. into structured outputs structured outputs Forcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. , defined once per project and usable in tests, filters, and tables across every data source and development commit. Not checked yet Adds Jev-as-a-judge test that scores every row in the window and explains its score on demand. Not checked yet
1 more in Everything
Openlayer: Adds Export test configs from the table header overflow menu to export every test at once.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Replicate a working test suite across projects when onboarding a new LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. task — use bulk duplicate to push all tests at once and review compatibility warnings before confirming.
📍 In the Manage Tests table, select all rows, open the header overflow menu, and choose Duplicate selected to project . Select one or more target projects (incompatible task types appear greyed out), then confirm — copies begin running immediately.
Export the full test config from one project to back it up or import it elsewhere.
📍 In the Manage Tests table header overflow menu, choose Export test configs to download all test definitions in the table.
Copies carry over thresholds, cadence, and data settings; data-source selections and tags are not transferred, and column-specific settings are copied as-is but skipped at run time if target columns are missing. Not checked yet
1 detail in Everything
Openlayer adds bulk and single-test duplication across projects, with compatibility filtering and pre-copy editing.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Copy a configured test — thresholds, cadence, and data settings intact — into a different project in your workspace without re-building it from scratch.
📍 In a project's Tests view, open the overflow menu on any test and choose Duplicate to project , select the target project, then use Edit before creating to adjust any settings before confirming.
Adds Duplicate selected (bulk) action to copy selected tests within the same project.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds Duplicate selected to project (bulk) and Duplicate to project (single-test overflow) actions to copy tests to one or more other projects in the workspace, with incompatible project targets shown as unavailable and partially compatible targets receiving only supported tests. Not checked yet Adds Edit before creating option when duplicating a single test to one project, letting you adjust settings before the copy is made. Not checked yet
3 details in Everything
Openlayer adds governance/compliance REST APIs REST APIs A convention for building HTTP-based interfaces where resources are addressed by URL and manipulated with standard verbs like GET, POST, and DELETE, giving cyber tools a predictable way to integrate with external services. , Jev-as-a-judge test, and new integrations for Claude, Microsoft 365 Copilot, and Palantir Foundry.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Pull a compliance audit-ready archive for a governance framework to hand off to auditors without manual exports.
copy
$ curl -X GET 'https://api.openlayer.com/api-reference/rest/governance/export-framework' \
-H 'Authorization: Bearer <your-api-key>' \
--output framework-audit.zip
Rotate a compromised API key in an automated pipeline without touching the Openlayer UI.
copy
$ curl -X POST 'https://api.openlayer.com/api-reference/rest/api-keys/rotate-api-key' \
-H 'Authorization: Bearer <your-api-key>' \
-H 'Content-Type: application/json'
Check whether a background governance or ingestion job has completed before proceeding in a CI pipeline.
copy
$ curl -X GET 'https://api.openlayer.com/api-reference/rest/background-tasks/retrieve-background-task' \
-H 'Authorization: Bearer <your-api-key>'
Adds GET /api-reference/rest/workspaces/retrieve-workspace endpoint to retrieve workspace details.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds Palantir Foundry integration to monitor AIP Agents and AIP Logic functions in Openlayer. Not checked yet
2 details in Everything
Openlayer adds workspace-level AI feature configuration with per-project overrides, model selection, and cost tracking︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Set up the Intake assistant for your workspace so it can draft project details and risk scores using your own model provider key.
📍 In Workspace Settings, go to AI Features › Intake assistant, select your model, then navigate to Environment and add your model provider API key.
Lock a workspace-wide AI feature so individual projects cannot toggle it off, while still permitting per-project model overrides.
📍 In Workspace Settings, go to AI Features › [feature], enable the lock that 'stops projects from turning the feature on or off'.
Adds workspace-level AI feature configuration with per-project override support, covering Intake assistant , Trace summaries , and other AI-powered features.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds Intake assistant to draft project details, intake answers, and risk scores during project registration or from the project's Lifecycle page. Not checked yet Adds a per-feature enable/disable lock that stops individual projects from turning a feature on or off, while still allowing per-project model overrides. Not checked yet Adds model selection per AI feature, allowing each feature to use a different configured model. Not checked yet Adds AI spend tracking broken down by feature — including Intake assistant — over a configurable time range, estimated using token prices from LLM costs . Not checked yet
+1 more
show less
Requires adding the selected model provider's API key under the workspace or project Environment settings for Intake assistant ; Openlayer does not supply its own provider keys. Not checked yet
6 details in Everything
Openlayer adds governance lifecycle stages, AI-assisted intake drafting, and a workspace-level review approval queue.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Enable the AI intake assistant so teams can auto-draft risk scores and intake answers when registering a new AI project.
📍 1. Go to Settings → AI features and enable Intake assistant , then select your model.
2. Go to Settings → Environment and add your model provider API key.
3. In Register your project , complete the optional context step and select AI Assistant to generate draft answers.
4. Review each suggestion and accept only the fields you want before saving.
Configure workspace lifecycle stages with ordered reviewers so each stage requires sequential sign-off before a project advances.
📍 1. Go to Settings › Governance › Lifecycle stages.
2. Select the Openlayer template or start from scratch.
3. For each stage, choose Ordered approval order and add reviewers in the desired sequence.
4. Set per-stage requirements (e.g. owner assigned, passing tests, minimum record count) from the requirements catalog.
Handle a blocked project by reviewing risk exceptions and sending it back to an earlier stage for remediation.
📍 1. Open the project page and select Risk exceptions to review risks blocking the current lifecycle requirement.
2. If rework is needed, click Send back in the approval view and choose an earlier stage.
3. Monitor progress via the Review queue in the workspace sidebar under Pending other approvers .
Supports PDF, DOCX, MD, and TXT documents up to 25 MB as context for AI-assisted intake drafting (browser extracts text; up to ~60,000 characters used; file is not stored by Openlayer). Not checked yet Adds workspace-level lifecycle stages configured under Settings › Governance › Lifecycle stages , with an Openlayer-provided four-stage template (Requirements, development, pipeline, and production stages) or a blank starting point. Not checked yet Adds two approval-order modes per stage — Concurrent (all reviewers approve simultaneously) and Ordered (reviewers notified and approve in sequence). Not checked yet Adds a Review queue in the workspace sidebar to manage pending lifecycle approvals, with a count badge and separate Needs your review and Pending other approvers views. Not checked yet Adds Risk exceptions view on the project page to review exceptions associated with risks that block a lifecycle requirement. Not checked yet
+4 more
show less
Sends Lifecycle review requested and Lifecycle stage changed notifications on stage submission and transition; reminders are rate-limited to one per hour by default. Not checked yet Supports per-project stage overrides to customize requirements or approvers for any stage without allowing advancement past unmet requirements. Not checked yet Adds Send back action to return a project to an earlier stage for additional review. Not checked yet Adds Retire option in the project options menu (⋯) to remove a project from its active lifecycle, with a Reopen action to restore it. Not checked yet
9 details in Everything
Openlayer adds API key expiry, rotation with grace periods, and full programmatic key management via REST API REST API A web API design style where clients interact with resources over HTTP using standard methods like GET, POST, and DELETE, giving cyber tools a predictable, language-agnostic way to integrate with external services. .︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Automate zero-downtime key rotation from a scheduled job or secret manager hook, giving applications 24 hours to pick up the new secret before the old one expires.
copy
$ curl --request POST \
--url https://api.openlayer.com/v1/workspaces/<workspace-id>/api-keys/<api-key-id>/rotate \
--header 'Authorization: Bearer <token>' \
--data '{"gracePeriodHours": 24}'
Adds expiresAt field to API key creation and rotation API calls, with UI options of 7, 30, 60, or 90 days, or 1 year; keys default to never expiring.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds gracePeriodHours parameter to the POST /v1/workspaces/<workspace-id>/api-keys/<api-key-id>/rotate endpoint, allowing the previous secret to remain valid for up to 7 days after rotation to enable zero-downtime rollover. Not checked yet Adds secret field in rotate API responses, returning the new secret exactly once. Not checked yet Adds status and expiresAt fields to List API keys and Retrieve API key endpoints; secrets are never returned in listing or retrieval calls. Not checked yet Adds Update an API key endpoint to rename existing keys. Not checked yet
+4 more
show less
Adds Delete an API key endpoint that immediately invalidates all secrets for the key. Not checked yet Expired keys return 401 Unauthorized on every request and cannot be renewed or rotated — a new key must be created. Not checked yet Read-only credentials (such as MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. client connections) can list and retrieve keys but cannot create, change, rotate, or delete them. Not checked yet Keys created or rotated using an expiring API key cannot outlive the caller's key; omitting expiresAt inherits the caller's expiry, and a later expiry or null is rejected. Not checked yet
9 details in Everything
Openlayer adds three new integrations: Claude Code trace collection via managed org settings, Microsoft 365 Copilot prompt/response import via Microsoft Graph Microsoft Graph Microsoft's unified REST API for accessing data and services across Microsoft 365, Azure AD, and related cloud products, giving cyber tools a single authenticated endpoint to query users, emails, files, and sign-in logs. , and Claude Compliance session ingestion.
New integration Adds Claude Code integration to send Claude Code traces to Openlayer via managed organizational settings. Not checked yet New integration Adds Microsoft 365 Copilot integration to import first-party prompts and responses from Microsoft Graph. Not checked yet New integration Adds Claude Compliance integration to ingest claude.ai chats and Cowork / Claude Code sessions into Openlayer. Not checked yet
3 details in Everything
Openlayer: Adds GET /api-reference/rest/workspaces/retrieve-workspace endpoint to retrieve workspace details.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Adds API key management endpoints: List API keys , Create API key , Retrieve API key , Update API key (rename), Delete API key , and Rotate API key (replace the secret). Not checked yet Adds List invites endpoint to list invites in a workspace. Not checked yet Adds test management endpoints: Create test , List tests , and Update tests for a project. Not checked yet Adds Create an upload url (presigned URL) and Retrieve a download url (short-lived) endpoints for file storage operations. Not checked yet Adds Retrieve project commit endpoint to fetch a specific project commit. Not checked yet
+4 more
show less
Adds a full governance REST API REST API A web API design style where clients interact with resources over HTTP using standard methods like GET, POST, and DELETE, giving cyber tools a predictable, language-agnostic way to integrate with external services. surface: list/create/retrieve/update/export frameworks, list per-project compliance stats, list framework documents with sections and rules, list section and subsection rules, and full CRUD CRUD The four basic database operations — Create, Read, Update, Delete — used as shorthand for any system that manages persistent data records, giving cyber tools a standard model for interacting with stored information. for rules and rule results including evidence attachment and workspace-level compliance statistics. Not checked yet Adds Retrieve a background task endpoint to poll a task's status and outputs. Not checked yet Adds Jev-as-a-judge test type to the test catalog for threshold-based LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. evaluation. Not checked yet Adds project lifecycle configuration to define workspace lifecycle stages and guide projects through requirements and review. Not checked yet
9 details in Everything
Openlayer adds a full Governance REST API REST API A web API design style where clients interact with resources over HTTP using standard methods like GET, POST, and DELETE, giving cyber tools a predictable, language-agnostic way to integrate with external services. , prompt-injection prompt-injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. detection, EU data residency, and integrations for Azure AI Speech, Google Gemini, and OpenAI Compliance Logs.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Pull a workspace-level compliance roll-up to see how many governance rules are passing, failing, pending, or due for renewal — useful for a daily compliance dashboard.
copy
$ curl -X GET 'https://api.openlayer.com/governance/rule-stats' \
-H 'Authorization: Bearer <token>' \
-H 'Accept: application/json'
Export a governance framework as an audit-ready zip archive — attach it directly to a compliance review or regulator submission.
copy
$ curl -X POST 'https://api.openlayer.com/governance/frameworks/<framework-id>/export' \
-H 'Authorization: Bearer <token>' \
-H 'Accept: application/zip' \
-o framework-audit.zip
Attach evidence to a rule result to satisfy an evidence rule programmatically — for example, after an automated test run completes.
copy
$ curl -X POST 'https://api.openlayer.com/governance/rule-results/<rule-result-id>/evidence' \
-H 'Authorization: Bearer <token>' \
-H 'Content-Type: application/json' \
-d '{"description": "Automated test suite passed on 2025-07-01", "url": "https://ci.example.com/runs/42"}'
Adds GET /governance/frameworks/{id}/rules, GET /governance/frameworks/{id}/projects, and GET /governance/frameworks/{id}/rule-stats to list a framework's rules, projects, and per-project compliance roll-ups. Not checked yet Adds GET /governance/rules, POST /governance/rules, GET /governance/rules/{id}, PATCH /governance/rules/{id}, and DELETE /governance/rules/{id} REST endpoints for full CRUD CRUD The four basic database operations — Create, Read, Update, Delete — used as shorthand for any system that manages persistent data records, giving cyber tools a standard model for interacting with stored information. on governance rules. Not checked yet Adds GET /governance/rule-results, GET /governance/rule-results/{id}, PATCH /governance/rule-results/{id}, GET /governance/rule-results/{id}/evidence, and POST /governance/rule-results/{id}/evidence to track and satisfy evidence requirements against rules. Not checked yet Adds support for managing and exporting test configurations from the Results page. Not checked yet Adds governance intake and risk forms, letting workspace admins define intake questions and risk dimensions shared across all projects. Not checked yet
+1 more
show less
Adds a project lifecycle view to review intake and risk information and advance projects through each governance lifecycle stage. Not checked yet
6 details in Everything
Openlayer adds Azure AI Speech and OpenAI Compliance integrations for speech and enterprise ChatGPT observability.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Adds Azure AI Speech integration to observe speech-to-text, translation, and text-to-speech workloads with Openlayer.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds OpenAI Compliance integration to ingest ChatGPT Enterprise and Codex activity into Openlayer. Not checked yet
2 details in Everything
Openlayer: Adds GET /governance/frameworks to list governance frameworks in a workspace.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Adds POST /governance/frameworks to create a custom governance framework in a workspace. Not checked yet Adds GET /governance/frameworks/{id} to retrieve a governance framework by its id. Not checked yet Adds PATCH /governance/frameworks/{id} to update a governance framework. Not checked yet Adds GET /governance/frameworks/{id}/export to export a framework's evidence and progress as an audit-ready zip archive. Not checked yet Adds GET /governance/frameworks/{id}/rules to list the rules that belong to a framework. Not checked yet
+23 more
show less
Adds GET /governance/frameworks/{id}/projects to list the projects a framework applies to. Not checked yet Adds GET /governance/frameworks/{id}/documents to list the documents attached to a framework. Not checked yet Adds GET /governance/frameworks/{id}/documents/{docId} to retrieve a framework document including its sections, subsections, and mapped rules. Not checked yet Adds GET /governance/documents/{docId}/sections/{sectionId}/rules to list the rules mapped to a section of a framework document. Not checked yet Adds GET /governance/documents/{docId}/subsections/{subsectionId}/rules to list the rules mapped to a subsection of a framework document. Not checked yet Adds GET /governance/rules to list all governance rules in a workspace. Not checked yet Adds POST /governance/rules to create a governance rule in a workspace. Not checked yet Adds GET /governance/rules/{id} to retrieve a governance rule by its id, including the frameworks it belongs to and its tags. Not checked yet Adds PATCH /governance/rules/{id} to update a governance rule (only fields sent are changed). Not checked yet Adds DELETE /governance/rules/{id} to delete a governance rule and its rule results. Not checked yet Adds GET /governance/rule-results to list rule results across a workspace. Not checked yet Adds GET /governance/rule-results/{id} to retrieve a rule result by its id. Not checked yet Adds PATCH /governance/rule-results/{id} to update a rule result (only fields sent are changed). Not checked yet Adds GET /governance/rule-results/{id}/evidence to list the evidence attached to a rule result. Not checked yet Adds POST /governance/rule-results/{id}/evidence to attach evidence to a rule result, satisfying an evidence rule. Not checked yet Adds GET /governance/rule-stats to retrieve a workspace-level compliance roll-up (counts of passing, failing, pending, and renewal-due rule results). Not checked yet Adds GET /governance/rule-tags to list all rule tags in a workspace. Not checked yet Adds GET /development/retrieve-download-url to exchange a storageUri for a short-lived presigned download URL. Not checked yet Adds GET /background-tasks/{id} to retrieve a background task's status, progress, and results. Not checked yet Adds has-prompt-injection test to detect prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. attempts, with optional LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. judge explanations. Not checked yet Supports choosing United States or European Union data residency regions at login or sign-up. Not checked yet Adds Openlayer Agent Skills to teach AI coding assistants (Claude Code, Cursor, GitHub Copilot) how to integrate a codebase with Openlayer. Not checked yet Adds Google Gemini integration to evaluate Gemini models and use them as LLM judges with Openlayer. Not checked yet
28 details in Everything
Openlayer adds a Lifecycle page with stage advancement, intake overview, risk scorecard, and lifecycle history for governance-mode projects.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Advance a high-risk AI project through its governance stages and audit who moved it and when.
📍 In the project sidebar, select Lifecycle . Check the Lifecycle stage section for 2/4 criteria met. Once all required criteria are satisfied, use the next-stage control to advance. Then select View history to open Lifecycle history and confirm the Stage , Date , and Actor of each transition.
Adds a Lifecycle page (reachable from the project sidebar or command menu) that consolidates lifecycle stage, intake information, and risk scorecard for governance-mode projects.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds a Lifecycle stage section with a segmented progress bar, a stage counter (e.g. Stage 2 of 5 ), per-stage criteria with Expand /Collapse controls, and progress reporting such as 2/4 criteria met. Not checked yet Adds Send back for re-review control on the Lifecycle page to return a project for another review, restricted to workspace admins and project access-group admins. Not checked yet Adds View history on the Lifecycle page to open a Lifecycle history modal listing each stage transition with Stage , Transition , Date , and Actor columns. Not checked yet Adds a Legacy lifecycle value banner prompting migration when a project carries a lifecycle value from an earlier workflow. Not checked yet
+3 more
show less
Adds an Intake overview tab on the Lifecycle page that renders one card per section of the workspace intake form, with a Search intake information… field, and supports a user-arranged two-column card layout saved by section ID. Not checked yet Adds a Risk tab on the Lifecycle page that opens a Risk scorecard driven by the workspace's risk template, with categories, dimensions, and descriptions pulled from that template. Not checked yet Adds a Project details card shown before workspace-defined intake sections during project registration, surfacing name, description, owner, and tags as fixed project properties. Not checked yet
8 details in Everything
Openlayer adds workspace-level governance with intake forms and risk templates for AI project registration.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Set up a workspace intake form and risk template so every new AI project is assessed consistently at registration time.
📍 In Openlayer, go to Settings › Governance, open the Intake tab, then choose Start from Openlayer template or Start from scratch . Organize questions into sections and save. Switch to the Risk tab to define risk categories and dimensions, then save. Project teams will now see these forms on their project's Lifecycle page.
Adds a Settings › Governance area with Intake and Risk tabs where workspace admins can define intake forms (organized into sections) and risk templates (categories and dimensions) used across all projects.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Intake forms and risk assessments are answered per-project from the project's Lifecycle page during registration or afterward. Not checked yet Workspace admins can start from the Openlayer template or from scratch when building intake forms and risk dimensions; workspace members can view but not edit these definitions. Not checked yet Removing an intake field or risk dimension after projects have responded does not erase recorded data — historical answers and scores are retained. Not checked yet
4 details in Everything
Openlayer: New Export a framework endpoint (/api-reference/rest/governance/export-framework) generates an audit-ready zip archive of a framework's evidence and progress.
└──▷ TRY IT
Check workspace-wide compliance health — how many rules are passing, failing, pending, or due for renewal — as part of a scheduled reporting job.
copy
$ curl -X GET 'https://api.openlayer.com/api-reference/rest/governance/retrieve-rule-stats' \
-H 'Authorization: Bearer <your_api_key>'
New Attach evidence to a rule result endpoint (/api-reference/rest/governance/create-rule-result-evidence) lets you satisfy evidence rules programmatically. Not checked yet New List a framework's per-project compliance endpoint (/api-reference/rest/governance/list-project-rule-stats) surfaces a compliance roll-up per project for a given framework. Not checked yet Adds EU data region support — choose United States or European Union when signing up or logging in to Openlayer. Not checked yet Adds intake and risk forms to define intake questions and risk dimensions shared across a workspace. Not checked yet Adds project lifecycle governance view to review intake and risk, then advance projects through each governance stage. Not checked yet
+1 more
show less
Adds test export capability from the Results page for managing and exporting test configurations. Not checked yet
6 details in Everything
Openlayer launches jevals, an open-source Python library for typed agent evals and guardrails, plus Jev as a native judge inside Openlayer.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ USE IT
Run security and behavior checks on a live agent trace to catch indirect injection and scope violations before acting on the result.
python
copy
from jevals import evaluate
from jevals.agent import ToolChoice, Grounded, StayedInScope
from jevals.security import IndirectInjection
result = evaluate(
{"messages": messages, "tools": tools},
[
ToolChoice(),
Grounded(),
StayedInScope(),
IndirectInjection(),
],
)
print(result.table())
Install jevals to start writing typed agent evals and guardrails in a new project.
copy
$ pip install jevals
Adds pip install jevals open-source Python library for agent evals and runtime guardrails built around typed decision models (Jev-style), replacing freeform LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -judge text generation with structured choices, scores, and probabilities.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds evaluate() function from jevals accepting agent messages and tool schemas, returning typed results via .table() . Not checked yet Adds 37 prebuilt evaluations in jevals covering tool selection, groundedness, task completion, scope adherence, prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. , and sensitive information — importable as ToolChoice, Grounded , StayedInScope, IndirectInjection, and more from modules jevals.agent and jevals.security. Not checked yet Supports custom evaluations written in Python or YAML alongside the prebuilt checks. Not checked yet Adds gate mechanism in jevals that applies a policy to evaluation results and routes agent actions to proceed, block, or require human review — enabling the same eval definition to run in both development testing and production guardrails. Not checked yet
+2 more
show less
Supports hosted Jev, local Kev and Laya backends, and conventional LLM backends, with evaluation definitions remaining stable across backend switches. Not checked yet Adds Jev as a native judge directly inside Openlayer, alongside existing classifier evaluation tooling (F1 scores, confusion matrices, drift, explainability). Not checked yet
7 details in Everything
Openlayer adds EU data residency with a dedicated eu.openlayer.com host alongside the existing US region.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Access Openlayer under EU data residency — navigate directly to the EU host so your account data stays in the European Union.
📍 In the browser, go to eu.openlayer.com. On the login or signup form, select the Data region field and choose European Union (EU data region) , then complete authentication.
Adds a European Union data region accessible at eu.openlayer.com, separate from the existing United States region at app.openlayer.com.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds a Data region selector on the login and signup forms, letting users choose between US data region and EU data region before authenticating. Not checked yet
2 details in Everything
Openlayer adds EU data region support, letting users choose between United States and European Union at login.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Select the EU data region at account creation so your data stays within European Union infrastructure.
📍 On the Openlayer login or sign-up page, locate the Data region field and select European Union before completing sign-up. Your account and all session data will be scoped to the EU region.
Openlayer adds bulk test config export from the Results page, downloading definitions as a ready-to-use tests.json array.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Export all selected test definitions at once via the Results page bulk action, then review the downloaded file.
📍 Results page › select tests › overflow menu › Export selected test configs
Adds Export test config from a single test's row overflow menu (⋯) on the Results page to download its definition as a tests.json array.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds Export selected test configs bulk action on the Results page to download multiple test definitions as a single tests.json array, also copying the JSON to the clipboard. Not checked yet Exported tests.json array can be checked into a repository and used with openlayer push to keep tests in sync — distinct from the openlayer tests command output, which wraps definitions in _meta and items. Not checked yet
Adds Export test config from a single test's row overflow menu (⋯) on the Results page to download its definition as a tests.json array.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds Export selected test configs bulk action on the Results page to download multiple test definitions as a single tests.json array, also copying the JSON to the clipboard. Not checked yet
1 more in Everything
Openlayer adds OpenAI Compliance integration to ingest ChatGPT Enterprise and Codex activity.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
Openlayer adds a full Governance API, prompt-injection prompt-injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. detection, data residency choice, OpenAI Compliance Logs ingestion, and Google Gemini integration.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ TRY IT
Export a governance framework as an audit-ready zip archive after all evidence has been attached to its rule results.
copy
$ curl -X GET 'https://api.openlayer.com/governance/frameworks/{id}/export' \
-H 'Authorization: Bearer <token>' \
--output framework-audit.zip
Attach evidence to a rule result to satisfy a compliance requirement programmatically.
copy
$ curl -X POST 'https://api.openlayer.com/governance/rule-results/{ruleResultId}/evidence' \
-H 'Authorization: Bearer <token>' \
-H 'Content-Type: application/json' \
-d '{"name": "Bias evaluation report", "url": "https://storage.example.com/report.pdf"}'
Check workspace-wide compliance health — how many rules are passing, failing, pending, or due for renewal — in a single call.
copy
$ curl -X GET 'https://api.openlayer.com/governance/rule-stats' \
-H 'Authorization: Bearer <token>'
Adds GET /governance/frameworks (List frameworks ), POST /governance/frameworks (Create a framework ), GET /governance/frameworks/{id} (Retrieve a framework ), PATCH /governance/frameworks/{id} (Update a framework ), and GET /governance/frameworks/{id}/export (Export a framework ) REST endpoints to programmatically manage governance frameworks and download audit-ready zip archives.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Adds GET /governance/frameworks/{id}/rules (List a framework's rules ), GET /governance/frameworks/{id}/projects (List a framework's projects ), GET /governance/frameworks/{id}/project-rule-stats (List a framework's per-project compliance ), GET /governance/frameworks/{id}/documents (List a framework's documents ), and GET /governance/frameworks/{id}/documents/{docId} (Retrieve a framework document ) REST endpoints for deep inspection of framework structure and compliance roll-ups. Not checked yet Adds GET /governance/frameworks/{id}/documents/{docId}/sections/{sectionId}/rules (List a section's rules ) and GET /governance/frameworks/{id}/documents/{docId}/sections/{sectionId}/subsections/{subsectionId}/rules (List a subsection's rules ) REST endpoints to enumerate rules mapped to specific document sections. Not checked yet Adds GET /governance/rules (List rules ), POST /governance/rules (Create a rule ), GET /governance/rules/{id} (Retrieve a rule ), PATCH /governance/rules/{id} (Update a rule ), and DELETE /governance/rules/{id} (Delete a rule ) REST endpoints to fully manage individual governance rules. Not checked yet Adds GET /governance/rule-results (List rule results ), GET /governance/rule-results/{id} (Retrieve a rule result ), PATCH /governance/rule-results/{id} (Update a rule result ), GET /governance/rule-results/{id}/evidence (List a rule result's evidence ), and POST /governance/rule-results/{id}/evidence (Attach evidence to a rule result ) REST endpoints to track and satisfy evidence requirements. Not checked yet
+8 more
show less
Adds GET /governance/rule-stats (Retrieve workspace compliance statistics ) REST endpoint returning a roll-up of how many rules are passing, failing, pending, or due for renewal across a workspace. Not checked yet Adds GET /governance/rule-tags (List rule tags ) REST endpoint for enumerating tags across workspace governance rules. Not checked yet Adds GET /development/retrieve-download-url REST endpoint to exchange a storageUri for a short-lived presigned download URL. Not checked yet Adds GET /background-tasks/{id} (Retrieve a background task ) REST endpoint to poll status, progress, and results of async operations. Not checked yet Adds a Has prompt injection test (catalog entry has-prompt-injection-count) that detects prompt injection attempts and supports optional LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. judge explanations. Not checked yet Adds a Google Gemini integration to evaluate Google Gemini models and use them as LLM judges. Not checked yet Adds Openlayer Agent Skills to teach AI coding assistants (Claude Code, Cursor, GitHub Copilot) how to integrate a codebase with Openlayer. Not checked yet Adds a Governance API guide covering how to create frameworks and rules, attach evidence, and pull compliance status and audit-ready exports. Not checked yet
13 details in Everything
Openlayer adds audit-ready evidence export from compliance framework pages, with SHA-256 SHA-256 A cryptographic hash function that turns data into a fixed 256-bit fingerprint. It is used to check integrity; hashing does not encrypt the original data. checksums and scoped project exports.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄
└──▷ HOW TO FIND IT
Package all compliance evidence for an auditor across every project in a framework.
📍 In the Openlayer UI, go to the workspace framework page, open the options menu (⋯) next to the framework, and select Export report . The download includes uploaded evidence files, a markdown control-status report, and CSV manifests with SHA-256 checksums. Ensure your role has the Export framework evidence permission.
Scope an audit evidence package to a single project when only one project is under review.
📍 In the Openlayer UI, navigate to the specific project, open its framework page, open the options menu (⋯), and select Export report . The resulting zip is scoped to that project only.
Adds Export report action (via the ⋯ options menu) on workspace and project framework pages to download a zip containing uploaded evidence files, a markdown progress/control-status report, and CSV manifests with SHA-256 checksums — requires the Export framework evidence permission.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet Supports scoping the export to a single project by triggering Export report from the project's framework page instead of the workspace framework page. Not checked yet
Adds Export report action (via the ⋯ options menu) on workspace and project framework pages to download a zip containing uploaded evidence files, a markdown progress/control-status report, and CSV manifests with SHA-256 checksums — requires the Export framework evidence permission.︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄ Not checked yet
1 more in Everything
Openlayer's Amazon Bedrock integration now traces embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. invocations for Titan and Cohere models
Openlayer MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. connector adds OAuth 2.1 OAuth 2.1 A consolidated draft of the OAuth authorization framework that merges OAuth 2.0 with its most important security BCP updates, dropping unsafe flows so tools can delegate user access without handling credentials directly. resource-server mode, configurable via environment variables, with tools for pipelines, traces, and governance.
Adds OPENLAYER_MCP_OAUTH_ISSUER environment variable to set the Authorization Server base URL, used as token issuer and advertised sign-in endpoint. Not checked yet Adds OPENLAYER_MCP_PUBLIC_URL environment variable to set the connector's public base URL; MCP is served at <OPENLAYER_MCP_PUBLIC_URL>/mcp. Not checked yet Adds OPENLAYER_MCP_OAUTH_JWKS_URI environment variable to optionally override the jwks_uri advertised by the Authorization Server. Not checked yet Adds OPENLAYER_MCP_AUTH_MODE environment variable to control caller authentication; defaults to resource-server, with oidc and none as narrower alternatives. Not checked yet Adds MCP tools to list, create, and update inference pipelines, check data ingestion status, and re-evaluate monitoring tests over a time range. Not checked yet
+4 more
show less
Adds MCP tools to fetch and filter production traces by inference ID and patch rows to backfill ground-truth labels for scored tests. Not checked yet Adds MCP tools to investigate sessions and users, including session- and user-level aggregations, AI summaries, and sessions behind failing tests. Not checked yet Adds MCP tools to list, create, update, and delete tests (integrity, consistency, performance, and LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge), trigger evaluation, and drill into failed rows with per-row scores and explanations. Not checked yet Adds MCP tools to review and edit governance posture: frameworks, documents, rules, rule tags, rule results, and evidence attached to results. Not checked yet
9 details in Everything
Openlayer adds configurable data retention periods for monitoring data, with options from 7 days to indefinite.
└──▷ HOW TO FIND IT
Set a 90-day rolling retention window so monitoring data is automatically purged nightly and does not accumulate indefinitely.
📍 1. In your project, go to Project settings → General.
2. Scroll to the Data retention section.
3. Under Retention period, open the selector (placeholder: Select retention period ) and choose 90 days .
4. Confirm the toast 'Data retention updated — Monitoring data older than 90 days will be permanently deleted.'
Openlayer adds project-scoped Secret and Plain environment variables with workspace inheritance and LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge connection testing.
└──▷ HOW TO FIND IT
Store an OpenAI API key as a secret at the project level so it is encrypted at rest and never visible to teammates, then verify the connection to the judge model.
📍 1. Open your project and go to Settings › Environment.
2. Under Secret variables , click Add variable , enter the key name (e.g. OPENAI_API_KEY) and its value, then save.
3. Navigate to Settings › LLM-as-a-judge, select OpenAI / gpt-4.1-mini-2025-04-14 in the Default LLM picker, then click Test connection to confirm Openlayer can reach the model.
Store a non-sensitive endpoint or region as a Plain variable so teammates can look it up without needing admin access.
📍 1. Open your project and go to Settings › Environment.
2. Under Plain variables , click Add variable , enter the key (e.g. AWS_REGION) and value (e.g. us-east-1), then save.
3. Confirm the value is visible to workspace members on the same page.
Adds Secret and Plain variable kinds under Settings > Environment: secrets are encrypted at rest and value-masked to workspace members; plain variables are readable and suited for client IDs, regions, and endpoints. Not checked yet Adds project-level Settings > Environment page for variables scoped to a single project, with the same Secret variables and Plain variables sections as the workspace page. Not checked yet Workspace-level variables are automatically inherited at the project level, shown with an INHERITED FROM WORKSPACE badge, and can be overridden by setting a different value at the project level. Not checked yet Adds Settings > LLM-as-a-judge page with Default LLM judge , Default LLM model picker, and Test connection button to verify that Openlayer can reach the selected provider model. Not checked yet
4 details in Everything
Openlayer adds opt-in default-deny project visibility via Hide projects without access group workspace setting
└──▷ HOW TO FIND IT
Enable default-deny project visibility before creating access groups so non-admins never see unassigned projects
📍 1. Go to Workspace settings → Security → Access control
2. Enable Hide projects without access group
3. Create your access groups and add users to them before creating new projects
Adds Hide projects without access group under Workspace settings → Security → Access control to make projects invisible to non-admins unless they are explicitly added via an access group, inverting the default open-visibility posture Not checked yet
1 detail in Everything
SAML SSO SAML SSO An authentication protocol that lets a user log in once with an identity provider and access multiple services without re-entering credentials, by passing signed XML assertions between parties. directory sync gains a No access group mapping and ongoing reconciliation that corrects role drift and removes stale members.
└──▷ HOW TO FIND IT
Suppress workspace access for a company-wide IdP IdP A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. group that directory sync must see, without removing it from the sync.
📍 In the SAML SSO configuration screen, under group mappings, map your company-wide group (e.g. Everyone ) to the No access role. Confirm that at least one other group is still mapped to Admin . The mapped group appears in the No access section, separate from groups with real roles.
Adds No access as a mappable role in SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. group configuration, letting you include IdP-pushed groups (such as a company-wide directory group) without granting any workspace permissions or surfacing the workspace in members' switchers. Not checked yet Reconciliation events are recorded in the workspace audit trail for review. Not checked yet
Adds No access as a mappable role in SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. group configuration, letting you include IdP-pushed groups (such as a company-wide directory group) without granting any workspace permissions or surfacing the workspace in members' switchers. Not checked yet
1 more in Everything
Openlayer adds a No access role for IdP IdP A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. directory sync group mapping, letting synced groups exist without granting workspace permissions.
Openlayer adds workspace security controls: domain restrictions, project visibility, connected apps management, and audit trail export
└──▷ HOW TO FIND IT
Lock down workspace sprawl so every employee with a company email joins the single governed workspace instead of creating their own.
📍 1. Go to Workspace settings → Security → Domains
2. Add and verify your organization's email domain
3. Enable Restrict workspace creation by domain
Respond to an auditor request by exporting a time-scoped CSV of all security-related workspace actions.
📍 1. Go to Workspace settings → Security → Compliance & audit
2. Set the desired time range
3. Click Export — the CSV is generated in the background and downloaded when ready
Adds Restrict workspace creation by domain toggle under Workspace settings → Security to prevent users with a verified organization domain from creating new workspaces, routing them into the existing governed workspace instead. Not checked yet Adds Hide projects without access group control under Access control to limit non-admins to only projects they have been explicitly added to via an access group (off by default). Not checked yet Adds Connected apps panel listing authorized applications (such as the Openlayer MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. connector) with full or read-only access, with per-connection revocation to immediately cut off access. Not checked yet Adds audit trail export under Compliance & audit that produces a CSV of security-related workspace events, optionally filtered by time range, available to workspace admins. Not checked yet
4 details in Everything
Openlayer gateway gains workspace connections, traffic-routing destinations, and multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. media attachment support in traces.
└──▷ HOW TO FIND IT
Mirror all gateway traffic into one shared data source while routing a specific group to its own, using All traffic and a group destination together.
📍 In the portal, go to Settings › Connect a workspace, enter your Openlayer API key and leave Base URL blank for the public API. Then add two destinations: one with target All traffic pointing to your audit data source, and one with target set to your specific Group pointing to that team's project data source.
Temporarily pause trace delivery to a destination without losing its data source mapping, for example during a data source migration.
📍 In the portal, open the destination you want to pause and enable the mute toggle. The destination retains its workspace, project, and data source configuration and can be re-enabled when ready.
Adds workspace connection management under Settings, where you supply an Openlayer API key and optional Base URL for on-premises installs; the gateway verifies, encrypts, and stores the credential and reflects back project and member counts. Not checked yet Supports connecting more than one workspace so a single gateway can publish traces into separate Openlayer workspaces simultaneously. Not checked yet Adds destinations that route traces by target — Group , User , All traffic , or Unclaimed — to a chosen workspace, project, and data source, with fan-out semantics so multiple matching destinations each receive a copy. Not checked yet Introduces All traffic destination type that mirrors every trace into one data source regardless of other matching destinations. Not checked yet Introduces Unclaimed fallback destination that receives only traces no more specific destination claimed. Not checked yet
+4 more
show less
Adds mute toggle on destinations to pause trace delivery without losing the data source configuration. Not checked yet Supports creating a project or data source directly from the destination picker without leaving the portal. Not checked yet Carries images and documents sent to or returned by a model into traces as attachments uploaded to Openlayer storage, avoiding inline base64 and keeping large-image traces readable. Not checked yet Provides two deployment-wide attachment options: whether to upload attachment bytes at all, and whether to upload media supplied as a URL (attachment uploads on by default, URL uploads off by default). Not checked yet
9 details in Everything
Openlayer Gateway adds model access rules to allow or block specific models per API key, user, group, or deployment
└──▷ TRY IT
Verify that a blocked model is refused and not surfaced — confirm a 403 permission_error and that GET /v1/models omits the blocked model for the key.
copy
$ curl -s -o /dev/null -w '%{http_code}' https://<gateway>/v1/chat/completions \
-H 'Authorization: Bearer <api-key>' \
-H 'Content-Type: application/json' \
-d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "hello"}]}'
# Expect: 403
curl -s https://<gateway>/v1/models \
-H 'Authorization: Bearer <api-key>'
# Expect: gpt-4o absent from the list
Navigate to the Gateway console to create an allow-only rule that restricts a specific API key to a single model family using a prefix wildcard.
📍 1. In the Gateway console, go to Policies › Model access.
2. Click Create a rule .
3. Set Who it's for to the target API key.
4. Set the rule type to Allow only .
5. Enter the model name pattern, e.g. claude-opus-* (prefix wildcard).
6. Save — the key can now call only models matching that prefix; all others return 403.
Adds model access rules under Policies > Model access to allow or block specific model names (e.g. gpt-4o, claude-sonnet-4-5) per API key, user, group, or all traffic — before routing occurs. Not checked yet Supports prefix wildcard matching (e.g. claude-opus-*) so rules continue to cover a model family when vendors append new date suffixes to names. Not checked yet Refused requests return 403 mapped to permission_error in the caller's dialect, and GET /v1/models is filtered by the same policy so blocked models never appear in a client's model picker. Not checked yet
3 details in Everything
Openlayer Gateway gains stackable guardrails for PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. , prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. , and content safety on LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. traffic
└──▷ HOW TO FIND IT
Set up a PII guardrail that redacts email addresses and phone numbers in LLM responses before they reach end users, while blocking credit card numbers outright.
📍 In the Openlayer console, go to Policies › add a guardrail, then: set target to the relevant API key or group; choose PII as the type; set stage to Output ; under Redacted entities add Email and Phone number ; under Blocked entities add Credit card number ; raise the confidence threshold to reduce false positives; save.
Add a prompt injection guardrail scoped to a user group to block jailbreak attempts on input traffic only, without affecting system prompts.
📍 In the Openlayer console, go to Policies › add a guardrail, then: set target to the relevant group; choose Prompt injection as the type; set stage to Input only ; set the confidence threshold to your desired sensitivity; save.
Enable content safety scanning on both input and output for a specific API key, monitoring only Threat and Identity hate categories to minimise false positives.
📍 In the Openlayer console, go to Policies › add a guardrail, then: set target to the relevant API key; choose Content safety as the type; set stage to Both ; deselect all categories except Threat and Identity hate ; raise the per-category threshold as needed; save.
Adds PII guardrail type with Blocked entities (reject on match) and Redacted entities (replace matched spans with placeholders) lists, covering universal types like emails, phone numbers, and credit card numbers as well as country-specific identifiers, with a configurable confidence threshold to reduce false positives. Not checked yet Adds prompt injection guardrail type that scores user turns only (skipping system prompts and assistant/tool turns) and always blocks, with a configurable confidence threshold. Not checked yet Adds content safety guardrail type scoring text across six categories — Toxic , Severe toxic , Obscene , Threat , Insult , and Identity hate — with a per-category threshold probability and block-or-allow outcome (no redaction path). Not checked yet Guardrails can be scoped to a group, a user, or a single API key (including keys with No group assigned), and multiple matching guardrails stack so a key can be screened simultaneously by its own guardrail and its group's. Not checked yet Guardrail stage can be set to Output (scan responses only) or Both (scan input and output); prompt injection offers Input only . Not checked yet
+1 more
show less
Guardrails table shows each guardrail's stage, violation action, violation count over the last 7 days, and last-triggered time — storing only metadata, never the underlying text. Not checked yet
6 details in Everything
Custom metric parameters now support enum to restrict inputs to a dropdown of fixed values in the test creation modal.
└──▷ USE IT
Restrict a comparison parameter to exact or fuzzy so testers pick from a dropdown instead of typing a free-form value.
json
copy
{
"comparison": {
"enum": ["exact", "fuzzy"]
}
}
Openlayer adds bulk test management — run, pause, duplicate, tag, archive, and delete multiple tests at once from the Results page.
└──▷ HOW TO FIND IT
Bulk-pause noisy tests during an incident so they stop computing results until you re-enable them.
📍 On the Results page, check the boxes next to all affected tests, then open the overflow menu and choose Pause selected .
Adds bulk actions to the Results page overflow menu: Run selected , Pause selected , Duplicate selected , Tag selected , Archive selected , Unarchive selected , and Delete selected for managing multiple tests at once. Not checked yet Adds Archive selected bulk action that hides tests while preserving result history, reversible from the platform. Not checked yet Adds openlayer tests delete CLI command to remove or archive tests outside the Results page UI. Not checked yet
3 details in Everything
Openlayer adds automatic thresholds, filter value suggestions, and test pausing to test configuration
└──▷ HOW TO FIND IT
Pause all tests in a project while replaying historical data so results are not polluted during the replay window.
📍 In the tests list, open the overflow menu on a test row and select Pause. While paused, a banner reads 'Tests are paused. You can still create tests, but they will not compute results.' Select Unpause in the banner when ready to resume.
Fail a cost metric test only when costs rise — not when they drop — by using a one-sided automatic threshold.
📍 On the test threshold tab, select Auto, then choose Upper only so the test fails only when the observed metric exceeds the predicted upper bound.
Limit LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge scoring to a random 20% sample of rows, capped at 50 rows, to control cost on large datasets.
📍 In Judge settings, set Sample size (%) to 20, Data limit (rows) to 50, and Sampling method to Random. To apply the same defaults project-wide, go to Settings → LLM-as-a-judge and set the same values.
Adds an Auto threshold tab to tests so Openlayer learns the historical metric distribution and fails a test when the observed value falls outside predicted bounds, with Both bounds , Upper only , or Lower only directionality controls. Not checked yet Adds Sample size (%), Data limit (rows), and Sampling method controls under Judge settings for LLM-as-a-judge tests, configurable per-test or project-wide under Settings → LLM-as-a-judge . Not checked yet Adds Evaluation windows settings in monitoring mode — Cadence (Scheduled or Manual ), Evaluation delay , Include historical data in each run , and Anchor evaluation windows toggles. Not checked yet Adds filter value suggestions in the column filter picker, showing values Openlayer has already seen, each with a match count, scoped to equality, membership, contains, or ordering comparisons; high-cardinality columns show a Type to search values prompt instead. Not checked yet Adds the ability to pause and unpause tests via the overflow menu (Pause / Unpause ) so tests stop computing results during data replays or upstream fixes. Not checked yet
Adds an Auto threshold tab to tests so Openlayer learns the historical metric distribution and fails a test when the observed value falls outside predicted bounds, with Both bounds , Upper only , or Lower only directionality controls. Not checked yet Adds Evaluation windows settings in monitoring mode — Cadence (Scheduled or Manual ), Evaluation delay , Include historical data in each run , and Anchor evaluation windows toggles. Not checked yet Adds filter value suggestions in the column filter picker, showing values Openlayer has already seen, each with a match count, scoped to equality, membership, contains, or ordering comparisons; high-cardinality columns show a Type to search values prompt instead. Not checked yet Adds the ability to pause and unpause tests via the overflow menu (Pause / Unpause ) so tests stop computing results during data replays or upstream fixes. Not checked yet
1 more in Everything
Openlayer adds data export from records and test-result tables in CSV, XLSX, and JSON formats with score filtering
└──▷ TRY IT
Download a monitoring pipeline directly from the terminal without opening the UI.
Export records with test scores attached via the platform UI.
📍 1. Go to Records (or a test-result table) in monitoring or development.
2. Open the table options overflow menu and select Export data .
3. In the modal, choose a format: CSV, XLSX, or JSON.
4. Under Scores, select All scores or Scores for {testName} .
5. Select Export data — the download starts automatically when the file is ready.
Adds an Export data control to the Records table and test-result tables in both monitoring and development, supporting CSV, XLSX, and JSON output formats. Not checked yet Adds score filtering in the export modal — choose No scores , All scores , or Scores for {testName} to control which test scores are attached to the exported file. Not checked yet Adds openlayer export CLI command to download a monitoring pipeline from the command line. Not checked yet Export jobs are tracked as activity states Exporting data , Exported data , or Failed to export data , giving visibility into export progress. Not checked yet Export access is gated by an Export data permission — Viewer role cannot export. Not checked yet
5 details in Everything
Openlayer adds AI-powered trace, session, and test summaries with workspace- and project-level controls and semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs.
└──▷ HOW TO FIND IT
Automatically summarize test results as they finish computing so your team gets an immediate failure-mode digest without manual review.
📍 In the workspace, go to Settings → AI features → Test summaries, enable the feature, and toggle on Generate automatically .
Lock AI summary settings workspace-wide so projects cannot override them, while still letting individual projects swap the underlying model.
📍 In the workspace, go to Settings → AI features, open the desired summary section (Trace summaries, Session summaries, or Test summaries), enable Locked for all projects , and set the workspace-level Model.
Adds AI summary button in the trace details view to summarize, analyze, and get up to speed with a trace without reading line-by-line spans. Not checked yet Adds Session summaries feature that summarizes user sessions, configurable under Settings → AI features . Not checked yet Adds Test summaries feature that summarizes test result sets, with an optional Generate automatically control to summarize results as they finish computing, configurable under Settings → AI features . Not checked yet Adds Trace summaries feature for individual traces, configurable under Settings → AI features at workspace or project level. Not checked yet Adds workspace-level Locked for all projects control per AI summary feature, preventing projects from enabling, disabling, or changing automatic generation (per-project model overrides still apply). Not checked yet
+3 more
show less
Adds AI usage section under Workspace settings → AI features showing spend and volume broken down by feature over a selectable time range, estimated using token prices from Workspace settings → LLM costs . Not checked yet Adds semantic search for goal subpopulations that matches by meaning instead of exact text, requiring a ClickHouse-backed pipeline; the embedding model embedding model Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. is set per deployment via the EMBEDDING_MODEL environment variable. Not checked yet Enables per-project override of workspace AI summary defaults via Settings → AI features within each project, mirroring workspace controls for Trace summaries , Session summaries , and Test summaries . Not checked yet
8 details in Everything
└──▷ BREAKING ON UPGRADE
! Changing the EMBEDDING_MODEL value requires re-embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. all stored data because existing vectors are not comparable across models.
Openlayer adds four new integrations: Google Conversational Search, Portkey, Palantir Foundry AIP Agents, and Databricks AI/BI Genie.
Adds tracing for Google Cloud Discovery Engine conversational search calls via the Openlayer Ruby SDK. Not checked yet Adds tracing for Portkey completions with Openlayer. Not checked yet Adds monitoring support for Palantir Foundry AIP Chatbots with Openlayer. Not checked yet Adds discovery and conversation monitoring for Databricks AI/BI Genie spaces in Openlayer. Not checked yet
4 details in Everything
Openlayer adds gateway guardrails, PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. redaction, A2A protocol A2A protocol An open protocol from Google for communication between AI agents, letting them discover each other's capabilities and exchange tasks without custom integration code. tracing, bulk delete API, and integrations with Portkey, Databricks, and Palantir.
└──▷ HOW TO FIND IT
Enforce a guardrail on every gateway request to block prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. attempts before they reach your model.
📍 In the Openlayer console, go to Gateway › Guardrails, enable Stop prompt injection , and apply the policy to the target key, user, or group.
Export test-result records as JSON for offline analysis or compliance archiving.
📍 In the Openlayer console, open a project's test-result table, click Export data , and select JSON (CSV and XLSX are also available).
Adds Guardrails gateway feature to block or redact PII, stop prompt injection, and screen for unsafe content on requests and responses. Not checked yet Adds Redact PII capability to strip personally identifiable information from trace data before it is stored. Not checked yet Adds bulk-delete REST endpoint to delete multiple records by their IDs from a data source (formerly inference pipeline ). Not checked yet Adds Set usage limits gateway feature to cap spend, requests, or tokens per key, user, or group, enforced on every request. Not checked yet Adds Assign groups gateway feature to group API keys so they share usage limits and guardrails. Not checked yet
+16 more
show less
Adds Send invites gateway feature to let people create their own API key from a link, pre-bound to a group and usage limits. Not checked yet Adds Model access gateway control to allow or block models per key, user, group, or the whole deployment. Not checked yet Adds Data retention project setting to configure how long a project keeps monitoring data, with nightly cleanup of data older than the retention period. Not checked yet Adds Update existing traces capability to update or delete traces previously published to the Openlayer platform. Not checked yet Adds Export data capability to download records from the Records table or a test-result table as CSV, XLSX, or JSON. Not checked yet Adds A2A A2A Agent2Agent, an open protocol started by Google for AI agents from different vendors to discover each other, delegate tasks and exchange results. It complements MCP, which connects an agent to tools. Protocol integration to trace multi-agent systems communicating over the A2A protocol as one merged trace or as linked per-project traces. Not checked yet Adds Portkey integration to trace and monitor Portkey completions with Openlayer. Not checked yet Adds Databricks AI/BI Genie integration to discover Genie spaces and monitor their conversations in Openlayer. Not checked yet Adds Palantir Foundry AIP Agents integration to connect Palantir Foundry AIP Chatbots for trace ingestion, evaluation, and monitoring. Not checked yet Adds Project lifecycle governance feature to advance a project through Intake, Development, Staging, Production, and Retired stages. Not checked yet Adds AI-generated summaries for sessions, traces, and test results. Not checked yet Adds Workspace dashboard to scan recently viewed items, portfolio health, project status, compliance, and cost across a workspace. Not checked yet Adds Project dashboard to track a project's metrics, tests, cost, and activity. Not checked yet Adds In-app notifications center to read workspace and project events and configure which app events are received. Not checked yet Adds Track usage gateway view to see what each key, user, and group is consuming and what it costs. Not checked yet Adds bulk test management on the Results page to run, pause, duplicate, tag, archive, or delete multiple tests at once. Not checked yet
21 details in Everything
Openlayer MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server launches with remote and local connectors, OAuth, and purpose-built tools for LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. pipeline management.
└──▷ HOW TO FIND IT
Add the remote MCP connector in any MCP client to get instant access to Openlayer tools without installing anything locally.
📍 https://mcp.openlayer.com/mcp
Point the local server at a self-hosted Openlayer instance instead of Openlayer Cloud.
copy
$ OPENLAYER_API_KEY=YOUR_OPENLAYER_API_KEY OPENLAYER_BASE_URL=https://openlayer.example.com uvx openlayer-mcp
Adds a remote MCP connector at https://mcp.openlayer.com/mcp using Streamable HTTP Streamable HTTP An MCP transport protocol that lets a client send JSON-RPC requests over HTTP POST while the server streams responses back using Server-Sent Events, replacing the older stdio and SSE transports for remote tool communication. transport with OAuth sign-in — no local install or API key required. Not checked yet Adds a local MCP server via uvx openlayer-mcp using stdio transport, authenticated with the OPENLAYER_API_KEY environment variable. Not checked yet Adds OPENLAYER_BASE_URL environment variable to the local server to point at a self-hosted Openlayer instance instead of Openlayer Cloud. Not checked yet Exposes push_commit and call_openlayer_api tools exclusively on the local connector — push_commit reads a local directory, call_openlayer_api provides freeform HTTP API access. Not checked yet Exposes create_presigned_upload_url and finalize_commit on the remote connector as the upload path for clients with their own file/shell access (e.g. Claude Code), using a storage_uri returned from the presigned upload. Not checked yet
+6 more
show less
Adds list_workspaces tool to identify which workspace a given MCP connection is bound to, supporting multiple simultaneous workspace connections. Not checked yet Supports self-hosted deployments serving the remote MCP connector at <OPENLAYER_URL>/mcp, authenticating against the on-premise OAuth2 OAuth2 An open authorization framework (RFC 6749) that lets a service grant limited access to a user's resources on another service without exposing credentials, giving cyber tools a standard way to authenticate against third-party APIs. Authorization Server with no external identity provider identity provider A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. . Not checked yet Tools are annotated as read-only or destructive, enabling MCP clients to auto-approve reads and prompt confirmation for writes. Not checked yet Supports listing, creating, and inspecting projects, commits, data sources (inference pipelines), production traces, sessions, and per-user activity with filters. Not checked yet Supports listing and creating tests — integrity, consistency, performance, and LLM-as-a-judge — plus triggering evaluation, reading results, and drilling into failed rows. Not checked yet Connected apps (OAuth sessions) are reviewable and revocable per workspace under Settings → Security → Connected apps. Not checked yet
11 details in Everything
Openlayer CLI gains a tests command to export, archive, and delete project tests (goals)
Openlayer adds row-level test result API, project deletion API, MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. custom functions, and metric bundle downloads
New API endpoint to pull individual rows from specific test results, enabling programmatic drill-down into evaluation data. Not checked yet New API endpoint to delete projects, available via the TypeScript SDK. Not checked yet MCP integration now supports custom functions callable from Openlayer. Not checked yet Custom metrics bundles are now viewable and downloadable directly from the UI. Not checked yet
4 details in Everything
└──▷ BREAKING ON UPGRADE
! Openlayer system tags are renamed to categories ; user-defined labels remain tags .
Openlayer adds remote MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. connector, Agent Skills, semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. , webhooks for test results, and 25+ new capabilities in its August 2026 release.
└──▷ TRY IT
Bootstrap a new project from scratch — tracing, links, and monitoring configured in one shot — without manual setup steps.
Adds openlayer init — one guided command takes a fresh directory to a pushable, linked, monitored project with tracing instrumentation. Not checked yet Adds non-interactive CLI mode — link, login, and resource creation run headless, enabling use from AI coding agents and CI pipelines. Not checked yet Expands openlayer export to support CSV and XLSX formats (previously JSON only) and adds the ability to include scores in exports. Not checked yet Adds webhooks for test results — subscribe a webhook endpoint to test-suite outcomes and get notified when tests pass or fail. Not checked yet Adds multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. support to the /responses and /messages Gateway endpoints — media sent and received, not just text. Not checked yet
+29 more
show less
Adds a new PATCH endpoint to update a project's name and description via API, picked up by the Python, Go, Java, and Ruby SDKs and the MCP server. Not checked yet Expands workspace API key surface — keys now work for updating projects, cost-override endpoints, test retrieval/update/deletion, and seven previously session-only read routes. Not checked yet Adds sorting and filtering to more list endpoints — workspace rules, project members, and user sessions; rules tables also gain server-side group filtering. Not checked yet Adds a remote, OAuth-protected MCP connector — the MCP server runs hosted instead of local-only, usable in Claude Code and claude.ai, with in-chat workspace switching. Not checked yet Adds governance and lifecycle tools via MCP — frameworks, rules, and rule results are reachable through MCP, growing the total toolset from 44 to 53 tools. Not checked yet Adds Openlayer Agent Skills — packaged skills that teach AI coding assistants to integrate a codebase with Openlayer (tracing, offline evals, tests, guardrails, CI gating); ships as a Claude Code plugin, Cursor plugin, or via the skills CLI. Not checked yet Adds dedicated Gemini and Vertex tracing in Python and TypeScript for the unified Gemini SDK including Vertex, where previously only the legacy SDK was auto-instrumented. Not checked yet Adds Claude Agent SDK tracing for TypeScript — drop-in import auto-traces agent runs. Not checked yet Adds native IBM watsonx Orchestrate integration, modeled on the existing Agentforce and Copilot Studio integrations. Not checked yet Adds semantic (model-based) routing — routing rules can score each prompt in-process and route it to a strong or weak target by threshold, alongside static first-match rules. Not checked yet Adds per-user usage limits and guardrails in the Gateway — limits and guardrails can target an individual user, including per-user caps within a team. Not checked yet Adds a content safety guardrail type for Gateway requests and responses. Not checked yet Adds a missing-guardrails governance rule — a new integrity test that flags traces missing an expected guardrail. Not checked yet Adds US National Provider Identifier (NPI) as a native PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. check — detectable in Contains-PII tests and redactable in traces. Not checked yet Adds per-project data retention — set a rolling retention window (7/30/90/180/365 days or indefinite); monitoring data older than the window is deleted. Not checked yet Adds the ability to pause and resume individual monitoring tests without disabling the whole project. Not checked yet Adds bulk session deletion — multi-select sessions and delete with a full cascade to their traces, scores, and embeddings embeddings Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. . Not checked yet Adds workspace-level dashboards — at-a-glance cost, usage, and project health across the workspace with drill-in widgets. Not checked yet Adds AI-generated session, trace, and test-result summaries — session and trace detail views open with an AI summary; test results include a summary of dominant failure modes with drill-down to exact rows. Not checked yet Adds semantic search filters — embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. -based subpopulation filtering for tests, beyond exact and keyword matching. Not checked yet Adds an official Helm chart for the Gateway for Kubernetes deployments. Not checked yet Adds a deployment admin area — operators get background-job queue health, metrics, history, and license/feature status, gated by a dedicated deployment-admin role. Not checked yet Adds granular token usage and cost tracking — cached, reasoning, and audio token categories are priced and stored; LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-judge evaluator runs are cost-tracked per test result. Not checked yet Adds Gateway spend visibility — per-model breakdown on the usage endpoint, a per-user spend aggregation endpoint, projected savings when creating a limit, key rotation, and provider secrets settable from the admin UI. Not checked yet Adds browser-based CLI login — sign in with a device-code browser flow instead of pasting an API key. Not checked yet Adds suggested filter values — filter fields suggest likely values as you type, served from the backend and respecting other active filters. Not checked yet Adds bulk test actions — duplicate several tests at once and schedule runs for several tests at once; view, download, and delete uploaded custom metric files. Not checked yet Adds on-prem deployment options — CSI CSI An ambiguous abbreviation — it most commonly means Container Storage Interface (a Kubernetes storage plugin standard) or Crime Scene Investigation — without more context the correct meaning cannot be determined reliably. Secrets Store support, external/managed ClickHouse, native Oracle Object Storage as a storage backend, and a multi-arch Gateway image for ARM ARM A family of RISC processor architectures designed by Arm Holdings and licensed to chip makers worldwide; tools target it to run natively on mobile, embedded, and increasingly server and desktop hardware. hosts. Not checked yet Adds a free public LLM cost explorer tool at llm-costs.openlayer.com for browsing and comparing LLM pricing across providers, including per-token-category pricing (cached, reasoning, audio). Not checked yet
34 details in Everything
Openlayer Skills package conditions AI coding agents (Claude Code, Cursor, Codex, Copilot, Gemini CLI) to follow Openlayer's current best practices end to end.
└──▷ TRY IT
Install the Openlayer skill into all detected coding agents on your machine in one command.
copy
$ npx skills add openlayer-ai/openlayer-skills --skill "openlayer"
Authenticate your agent to Openlayer, or point it at a self-hosted instance.
copy
$ export OPENLAYER_API_KEY=<your-api-key>
export OPENLAYER_BASE_URL=<your-instance-url> # self-hosted only
Adds npx skills add openlayer-ai/openlayer-skills --skill "openlayer" via the skills CLI to install the Openlayer agent skill into Codex, GitHub Copilot, Gemini CLI, Windsurf, opencode, and dozens of other coding agents in one step. Not checked yet Supports manual installation for any agent that reads a skills directory by symlinking or copying the skills/openlayer folder from github.com/openlayer-ai/openlayer-skills. Not checked yet Skill covers tracing/monitoring via the Python or TypeScript SDK, offline evals with openlayer.json + tests.json, test and guardrail creation from the catalog, and CI/CD build gating on eval regressions. Not checked yet Integrates the Openlayer MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server so agents can query the workspace and search Openlayer documentation directly. Not checked yet Supports both LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. /agent applications and traditional ML models (scikit-learn, XGBoost, and similar). Not checked yet
5 details in Everything
Openlayer MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server now available as a hosted, OAuth-protected remote connector at https://mcp.openlayer.com — no local install required.
└──▷ TRY IT
Register the Openlayer remote MCP server in Claude Code so the assistant can access your Openlayer workspace without managing an API key.
copy
$ claude mcp add --transport http openlayer https://mcp.openlayer.com
Add the Openlayer MCP server to a VS Code project for use with GitHub Copilot or other MCP-aware extensions.
json
copy
{
"servers": {
"openlayer": {
"type": "http",
"url": "https://mcp.openlayer.com"
}
}
}
Bridge the remote connector through mcp-remote for clients (e.g. Windsurf, Zed) that lack native remote-MCP + OAuth support.
json
copy
{
"command": "npx",
"args": ["-y", "mcp-remote", "https://mcp.openlayer.com"]
}
Adds a hosted, OAuth-protected remote MCP connector at https://mcp.openlayer.com — no API key or local install needed; sign-in is handled via OAuth against your Openlayer account. Not checked yet Supports workspace switching directly from the chat interface when a user belongs to multiple workspaces. Not checked yet Integrates with Claude desktop/claude.ai via Settings → Connectors → Add custom connector using https://mcp.openlayer.com. Not checked yet Integrates with Claude Code via claude mcp add --transport http openlayer https://mcp.openlayer.com. Not checked yet Integrates with Cursor via .cursor/mcp.json with "url": "https://mcp.openlayer.com". Not checked yet
+3 more
show less
Integrates with VS Code via the MCP: Add Server command palette action (HTTP type) or .vscode/mcp.json with "type": "http" and "url": "https://mcp.openlayer.com". Not checked yet Integrates with Codex via codex mcp add openlayer --url https://mcp.openlayer.com. Not checked yet Supports clients without native remote-MCP + OAuth via mcp-remote bridge: npx -y mcp-remote https://mcp.openlayer.com. Not checked yet
8 details in Everything
Openlayer adds semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. filters for test subpopulations, matching rows by meaning rather than exact wording.
└──▷ HOW TO FIND IT
Narrow a test to rows about billing issues even when users phrase the problem differently (charged twice , invoice looks wrong ).
📍 In your project, go to Tests and create or edit a test. Under subpopulation filters, select the free-text column (e.g. user message), set the match type to semantic , and enter the phrase billing problems . The test will evaluate only rows semantically similar to that phrase.
Adds a semantic match type in test subpopulation filters, using embeddings embeddings Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. to match rows by topic or intent rather than exact or keyword wording — useful for free-text columns like user messages, model outputs, and transcripts. Not checked yet Semantic filters compose with existing metadata conditions (date ranges, user IDs, tags) to narrow the exact data slice a test runs on. Not checked yet
2 details in Everything
Openlayer adds AI-generated summaries for sessions, traces, and test results to surface failure patterns without raw log reading.
New AI summaries on the session detail view explain what the user was trying to do, what the agent did, and where it struggled — accessible via Projects > Sessions > open a session. Not checked yet New AI summaries on the trace detail view replace line-by-line span reading — accessible by opening any trace from the Traces view. Not checked yet New AI summaries on test results surface dominant failure modes and link each mode to the exact matching rows — accessible by opening a test result above the row-level results. Not checked yet Summaries are generated on demand and cached so that reopening the same session or result is instant. Not checked yet
4 details in Everything
Openlayer adds support for monitoring IBM watsonx Orchestrate agents
Openlayer adds AI-generated summaries, semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. filters, agent skills for coding assistants, and IBM Watson Orchestrate integration.
Adds AI-generated summaries for sessions, traces, and test results so practitioners see a plain-language overview without reconstructing events span by span. Not checked yet Adds semantic search filters for building test subpopulations using embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. -based matching instead of exact or keyword matching. Not checked yet Adds Openlayer Agent Skills to teach AI coding assistants (Claude Code, Cursor) how to integrate a codebase with Openlayer. Not checked yet Adds IBM watsonx Orchestrate integration for agent tracing, evaluation, and monitoring. Not checked yet
4 details in Everything
Openlayer adds AI summaries, semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. , remote MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. , webhooks, per-user Gateway limits, and a Helm chart for the Gateway in its August 2026 release.
└──▷ TRY IT
Bootstrap a new project with tracing fully wired in one guided step, suitable for kicking off a new LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. app or CI environment.
Adds openlayer init CLI command to guide a fresh directory to a linked, monitored project with tracing instrumentation in one step. Not checked yet Adds non-interactive (headless) CLI mode — link, login, and resource creation now accept no user prompts, enabling use from AI coding agents and CI pipelines. Not checked yet Expands openlayer export to support CSV and XLSX output formats (previously JSON only) and adds a flag to include scores in the export. Not checked yet Adds openlayer.init() automatic instrumentation for Python — one call patches every installed supported LLM SDK (OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. , Azure); opt out with auto_instrument=False. Not checked yet Introduces webhooks for test results — subscribe a webhook endpoint to test-suite outcomes and receive notifications when tests pass or fail. Not checked yet
+39 more
show less
Adds /responses and /messages Gateway endpoints with multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. support, accepting and returning media as well as text. Not checked yet Expands API-key surface so workspace API keys now cover updating projects, cost-override endpoints, test retrieval/update/deletion, and seven previously session-only read routes. Not checked yet Adds a new PATCH endpoint to update a project's name and description, surfaced in the Python, Go, Java, and Ruby SDKs and the MCP server. Not checked yet Adds sorting and filtering to more list endpoints: workspace rules, project members, and user sessions, plus server-side group filtering on rules tables. Not checked yet Adds a per-user spend aggregation endpoint and a per-model breakdown on the usage endpoint for Gateway spend visibility. Not checked yet Adds Gemini and Vertex tracing in Python and TypeScript via a dedicated tracer for the unified Gemini SDK, including Vertex AI (previously only the legacy SDK was auto-instrumented). Not checked yet Adds Claude Agent SDK tracing for TypeScript — drop-in import auto-traces agent runs. Not checked yet Adds Claude Agent SDK tracing for Python — instruments the full agent loop, capturing tool calls, MCP tools, subagent nesting, session continuity, cost, and tokens. Not checked yet Adds a remote, OAuth-protected MCP connector — the MCP server runs hosted (not local-only), usable in Claude Code and claude.ai, with in-chat workspace switching. Not checked yet Expands MCP tooling to 53 tools (up from 44), adding governance/lifecycle access (frameworks, rules, rule results) and 12 new session, trace, and test-lifecycle tools. Not checked yet Introduces Openlayer Agent Skills — packaged skills for AI coding assistants (Claude Code plugin, Cursor plugin, or skills CLI) covering tracing, offline evals, tests, guardrails, and CI gating for both LLM and traditional/tabular ML workflows. Not checked yet Adds native IBM watsonx Orchestrate integration, modeled on the existing Agentforce and Copilot Studio integrations. Not checked yet Adds semantic (model-based) routing — routing rules can score each prompt in-process and send it to a strong or weak target by threshold, alongside existing static first-match rules. Not checked yet Adds per-project data retention with a configurable rolling window (7, 30, 90, 180, or 365 days, or indefinite); monitoring data older than the window is deleted. Not checked yet Adds per-user usage limits and guardrails in the Gateway, including per-user caps within a team. Not checked yet Adds a content safety guardrail type for Gateway requests and responses. Not checked yet Adds a missing-guardrails governance rule — a new integrity test that flags traces missing an expected guardrail. Not checked yet Adds US National Provider Identifier (NPI) as a native PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. check, detectable in Contains-PII tests and redactable in traces. Not checked yet Adds project lifecycle stages and evidence documents — attach a lifecycle stage and evidence to a project for governance tracking. Not checked yet Adds rule deactivation reasons — supply a reason when deactivating a rule; it is displayed on the rule page. Not checked yet Adds workspace-level dashboards with at-a-glance cost, usage, and project health across the workspace, including drill-in widgets. Not checked yet Adds AI-generated summaries to session, trace, and test-result detail views, with drill-down from failure modes to the exact rows behind each pattern. Not checked yet Adds semantic (embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. -based) search filters for subpopulation filtering in tests, beyond exact and keyword matching. Not checked yet Adds pause and resume for individual monitoring tests without disabling the whole project. Not checked yet Adds bulk session deletion — multi-select sessions and delete with full cascade to their traces, scores, and embeddings embeddings Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. . Not checked yet Adds a Helm chart for the Gateway for Kubernetes deployments. Not checked yet Adds a deployment admin area for on-prem operators with background-job queue health, metrics, history, and license/feature status, gated by a dedicated deployment-admin role. Not checked yet Adds browser-based CLI login via device-code browser flow instead of requiring an API key to be pasted. Not checked yet Adds bulk test actions — duplicate or schedule runs for several tests at once, and view, download, or delete uploaded custom metric files. Not checked yet Adds a timeline (Gantt-style) view for traces showing every span's start time, duration, and nesting side by side. Not checked yet Adds Gateway API key invite links — admins can generate links that let users self-provision their own Gateway API keys. Not checked yet Adds compliance test visibility to results pages — shows which tests are required for compliance, in context. Not checked yet Introduces a free public LLM cost explorer at llm-costs.openlayer.com for browsing and comparing LLM pricing across providers, including per-token-category pricing (cached, reasoning, audio). Not checked yet Adds granular token usage tracking — cached, reasoning, and audio token categories are now priced and stored, with per-API-key cost visible in the UI; LLM-as-judge evaluator runs are cost-tracked per test result. Not checked yet Adds LangChain ToolMessage.artifact capture (including document lists, promoted to trace-level context) in the Python and TypeScript SDKs. Not checked yet Unifies trace configuration around init() in the SDK — settings merge incrementally instead of resetting on each call (configure() is deprecated in favor of init() ). Not checked yet Adds on-prem support for CSI CSI An ambiguous abbreviation — it most commonly means Container Storage Interface (a Kubernetes storage plugin standard) or Crime Scene Investigation — without more context the correct meaning cannot be determined reliably. Secrets Store, external/managed ClickHouse, native Oracle Object Storage as a storage backend, and a multi-arch Gateway image for ARM ARM A family of RISC processor architectures designed by Arm Holdings and licensed to chip makers worldwide; tools target it to run natively on mobile, embedded, and increasingly server and desktop hardware. hosts. Not checked yet Adds the ability to rename data sources directly from the data source row menu. Not checked yet Adds row data download in development mode. Not checked yet
44 details in Everything
└──▷ BREAKING ON UPGRADE
! configure() is deprecated in favor of init() for trace configuration in the SDK; settings now merge incrementally instead of resetting on each call.
AI summaries, semantic search filters, and the remote MCP connector
Aug 8, 2026
· release notes
Openlayer vAI adds semantic search semantic search Searching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. filters, remote MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. connector, AI summaries, and openlayer init CLI.
└──▷ USE IT
Bootstrap a new project with tracing instrumentation using the new CLI command
shell
copy
openlayer init --guided
Update a project's name and description via the expanded API surface
shell
copy
curl -X PATCH https://api.openlayer.com/api/projects/<project_id> \
-H 'Authorization: Bearer <WORKSPACE_API_KEY>' \
-H 'Content-Type: application/json' \
-d '{
"name": "New Project Name",
"description": "Updated description for vAI release"
}'
Export test results to CSV and XLSX formats including scores using the new CLI capabilities
shell
copy
openlayer export --format csv,xlsx --include-scores > results.csv
Adds semantic search filters for embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. -based subpopulation filtering of tests Not checked yet Introduces the remote, OAuth-protected MCP connector with in-chat workspace switching Not checked yet Opens sessions, traces, and test results with AI summaries of dominant failure modes Not checked yet Adds openlayer init guided CLI command to bootstrap a monitored project Not checked yet Enables non-interactive CLI mode for link, login, and resource creation via headless execution Not checked yet
+15 more
show less
Supports native IBM watsonx Orchestrate integration through Agent Skills Not checked yet Adds /responses and /messages endpoints support for media (multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. Gateway) Not checked yet Introduces content safety guardrail type for Gateway requests and responses Not checked yet Allows setting per-user usage limits and guardrails in the Gateway Not checked yet Provides bulk session deletion with cascade to traces, scores, and embeddings embeddings Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. Not checked yet Enables pause and resume of individual monitoring tests without disabling the project Not checked yet Adds per-project data retention windows (7/30/90/180/365 days or indefinite) Not checked yet Exposes a new PATCH endpoint to update a project's name and description via API Not checked yet Expands workspace API keys to cover project updates, cost-override endpoints, and test retrieval/deletion Not checked yet Adds server-side group filtering for workspace rules, project members, and user sessions list endpoints Not checked yet Supports CSV and XLSX export formats in openlayer export, including scores Not checked yet Reveals granular token usage by category (cached, reasoning, audio) per API-key in the UI Not checked yet Adds US National Provider Identifier as a native PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. check detectable in Contains-PII tests Not checked yet Introduces Missing-guardrails governance rule to flag traces lacking expected guardrails Not checked yet Deploys an official Helm chart for Gateway Kubernetes deployments (On-Prem) Not checked yet
20 details in Everything
June 2026: Gantt-style trace timeline view makes slow spans and unexpected branches immediately visible.
New Gantt-style trace timeline view lines up every span by start time, duration, and nesting to surface slow steps and unexpected branches at a glance. Not checked yet
1 detail in Everything
Openlayer adds a guided CLI init command and multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. trace support for images, audio, and files.
New init CLI command sets up Openlayer in a project through a single guided flow. Not checked yet Supports attaching images, audio, and files to traces for multimodal LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. monitoring. Not checked yet
2 details in Everything
Openlayer CLI gains data-sources, profile, projects, tests, and update commands; adds tabular ML prediction publishing.
└──▷ TRY IT
Export all test definitions for a project to review or version-control your monitoring goals.
Switch between workspaces or CI service accounts by managing named authentication profiles.
New data-sources CLI command to manage Openlayer data sources directly from the terminal. Not checked yet New profile CLI command to manage authentication profiles for the Openlayer CLI. Not checked yet New projects CLI command to manage Openlayer projects from the CLI. Not checked yet New tests CLI command to export test definitions (goals) for an Openlayer project. Not checked yet New update CLI command to self-update the Openlayer CLI to the latest version. Not checked yet
+1 more
show less
Supports publishing traditional/tabular ML model predictions to Openlayer for monitoring, streamed in real-time or uploaded in batches. Not checked yet
6 details in Everything
Openlayer CLI gains data-sources, profile, projects, tests, and update commands; adds tabular ML prediction publishing.
└──▷ TRY IT
Export all test definitions for a project to review or version-control your monitoring goals.
Switch between workspaces or CI service accounts by managing named authentication profiles.
New data-sources CLI command to manage Openlayer data sources directly from the terminal. Not checked yet New profile CLI command to manage authentication profiles for the Openlayer CLI. Not checked yet New projects CLI command to manage Openlayer projects from the CLI. Not checked yet New tests CLI command to export test definitions (goals) for an Openlayer project. Not checked yet New update CLI command to self-update the Openlayer CLI to the latest version. Not checked yet
+1 more
show less
Supports publishing traditional/tabular ML model predictions to Openlayer for monitoring, streamed in real-time or uploaded in batches. Not checked yet
6 details in Everything
Batch test re-runs and expanded integrations
seen Jul 13, 2026
·
issue 014
Openlayer adds batch test re-runs, Ruby SDK, Gemini/Copilot Studio integrations, and new drift detection.
└──▷ TRY IT
Retrieve test results programmatically to pipe into downstream alerting or reporting pipelines.
copy
$ curl -X GET 'https://api.openlayer.com/v1/projects/<project-id>/test-results' \
-H 'Authorization: Bearer <api-key>'
Adds batch and manual test re-runs so results can be recalculated retroactively when new data is added. Not checked yet Supports automatic threshold configuration for tests built on custom metrics. Not checked yet Introduces custom time range selection for filtering test execution history. Not checked yet New primary key uniqueness constraint test type available on the platform. Not checked yet Adds Ruby SDK for instrumenting Openlayer in Ruby applications. Not checked yet
+11 more
show less
New Strands Agents SDK integration for tracing agentic workflows. Not checked yet New Microsoft Copilot Studio integration. Not checked yet New Google Gemini API integration. Not checked yet New API endpoints for retrieving test results and managing projects. Not checked yet Adds Characteristic Stability Index (CSI CSI An ambiguous abbreviation — it most commonly means Container Storage Interface (a Kubernetes storage plugin standard) or Crime Scene Investigation — without more context the correct meaning cannot be determined reliably. ) as a drift detection method. Not checked yet Adds step detection support in the LangChain integration. Not checked yet Adds model output capture in the Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. integration. Not checked yet Smoother Snowflake connection flow now supports views. Not checked yet Simplified Databricks onboarding flow. Not checked yet Redesigned monitoring test results graph for clearer visibility. Not checked yet Clearer trace visualizations in the UI. Not checked yet
16 details in Everything
Complete design system overhaul, Snowflake Integration
seen Jul 13, 2026
·
issue 014
Openlayer adds Snowflake integration, new security role, prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. tests, and expanded LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. tracing for Bedrock, OpenAI Agents, Pydantic AI, and LangGraph.
└──▷ HOW TO FIND IT
Connect a Snowflake table directly to a project to run evals against live warehouse data without manual exports.
📍 In the console, open your project, go to Data Sources › Connect, select Snowflake, and choose the target table.
Enable a prompt injection test on an LLM data source to automatically detect adversarial inputs in production traffic.
📍 In the console, open your project's Tests tab, click Add Test, and select Prompt Injection from the Evals library.
New UI with full light and dark mode support, updated design system, and improved navigation hierarchy. Not checked yet Adds priority levels for tests and more data visualizations throughout the product. Not checked yet SDK support for tracing Bedrock models. Not checked yet SDK support for tracing OpenAI Agents. Not checked yet SDK support for tracing Pydantic AI systems. Not checked yet
+12 more
show less
SDK support for tracing LangGraph systems. Not checked yet New Member restricted security role that can perform member actions without viewing data source data. Not checked yet Direct Snowflake table connections to projects via native integration. Not checked yet Project, dataset, and table dropdowns when connecting BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. tables. Not checked yet Supports hosting Openlayer on subpaths in on-premises deployments. Not checked yet Allows users to override LLM costs with custom cost values. Not checked yet Includes standard deviation score in LLM-as-a-judge and Ragas test results. Not checked yet New prompt injection test to detect adversarial attacks on LLM systems. Not checked yet Per-row scores now visible for metrics like semantic similarity and exact match in data tables. Not checked yet API endpoints that return record counts and last record date for data sources are now faster. Not checked yet Faster data source creation via API. Not checked yet Supports streaming data with numpy arrays in the API request body. Not checked yet
17 details in Everything
└──▷ BREAKING ON UPGRADE
! Inference pipelines are renamed to data sources throughout the platform
Custom metrics, rotating API keys, and new models for direct-to-API calls
seen Jul 13, 2026
·
issue 014
Openlayer adds custom metrics via openlayer.json, rotating API keys, and 11 new models for direct-to-API evaluation calls.
└──▷ USE IT
Define a custom domain-specific metric in openlayer.json so it appears as a project-wide metric and a runnable test on all your data.
json
copy
{
"metrics": [
{
"name": "toxicity_score",
"description": "Custom toxicity scorer for our domain",
"type": "custom",
"script": "metrics/toxicity.py"
}
]
}
Rotate API keys by creating a new key before revoking the old one — navigate to your account API settings to manage multiple keys.
📍 In the Openlayer console, go to Account Settings › API Keys, click Create new key , copy the new key into your services, then delete the old key.
Supports custom metrics uploaded via openlayer.json, usable as project-wide metrics or as runnable eval tests alongside standard metrics. Not checked yet Enables creation, renaming, and deletion of multiple personal API keys to support API key rotation workflows. Not checked yet Adds ability to specify desired metrics directly in openlayer.json for API-driven evaluation runs. Not checked yet Adds 11 new models for direct-to-API calls: GPT-4o, GPT-4 Turbo, Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Command R, Command R Plus, Gemini 1.0 Flash, Gemini 1.5 Flash, and Gemini 1.5 Pro. Not checked yet Adds git repository and organization linking in git settings pages for tighter version-control integration. Not checked yet
+1 more
show less
New UI button to view the processing status of a commit during project loading. Not checked yet
6 details in Everything
Improved quality control over your LLM’s responses with annotations and human feedback
seen Jul 13, 2026
·
issue 014
Openlayer adds request annotations, human feedback ratings, data export, and Git integration to LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. quality control.
└──▷ HOW TO FIND IT
Review a filtered set of low-quality responses offline by exporting only the flagged requests cohort from your workspace.
📍 In the console, go to your workspace › Requests, apply the desired filters (e.g., thumbs-down ratings), then click Export to download the filtered cohort as a file.
Connect a project to a Git repository and configure the target branch and root directory for version-controlled evaluation pipelines.
📍 In the console, go to your Project › Settings › Git Repository, enter the repository URL, set the branch and root directory, then save.
Adds per-request annotations with custom values, including step-level annotation for individual trace steps when tracing is enabled. Not checked yet Adds thumbs up/thumbs down human feedback ratings on every request to quickly surface good and bad model responses. Not checked yet Enables data export from the UI, including filtered cohorts of request data. Not checked yet Introduces updated navigation with breadcrumbs and the ability to copy project name and inference pipeline ID. Not checked yet Adds project-level Git repository settings, including editable branch and root directory configuration. Not checked yet
+4 more
show less
Adds ability to add and edit ground truths on individual requests. Not checked yet Adds filtering of test modal data by the selected evaluation window. Not checked yet Adds more project templates. Not checked yet Improves the OpenAI SDK integration. Not checked yet
9 details in Everything
Improved test diagnosis page, SAML SSO, design refreshes, + more
seen Jul 13, 2026
·
issue 014
Openlayer adds SAML SSO SAML SSO An authentication protocol that lets a user log in once with an identity provider and access multiple services without re-entering credentials, by passing signed XML assertions between parties. , Azure OpenAI evaluator, Ragas metrics, and an overhauled test diagnosis page.
└──▷ HOW TO FIND IT
Enforce org-level SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. for your team's Openlayer workspace to add a layer of authentication security.
📍 In the console, go to Settings › Collaboration › SAML SSO and configure your identity provider credentials.
Adds SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. SSO support via any major identity provider identity provider A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. , enabling teams to enforce org-level authentication for workspace access. Not checked yet Supports Azure OpenAI as an LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. evaluator integration. Not checked yet Adds additional Ragas metrics — faithfulness and answer correctness — for evaluation. Not checked yet New list view on the test diagnosis page surfaces error messages for skipped and errored tests, with filtering by result type and a status-breakdown overview. Not checked yet Test diagnosis page sections are now collapsible for a streamlined review experience. Not checked yet
+4 more
show less
Updated cost table for OpenAI models in the Observability view. Not checked yet Notifications for new inference pipelines now include the pipeline name. Not checked yet More informative tooltips on test cards. Not checked yet Groq guide now available in documentation. Not checked yet
9 details in Everything
Introducing Openlayer Governance
seen Jul 13, 2026
·
issue 014
Openlayer adds Governance mode, A/B testing, new data backends, CLI data sources, and expanded tracing support.
└──▷ HOW TO FIND IT
Enable Governance mode to define and track enforcement rules across your workspace's AI projects.
📍 In the Openlayer console, open your workspace, navigate to Governance, select New Framework , define rules (e.g. PII leakage test, monitoring required, risk level), assign owners, and track compliance across projects.
New Governance mode lets teams define and enforce custom policy frameworks (PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. leakage, prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. , monitoring requirements, ownership, risk level) across all projects and workspaces, with rule owners, comment threads, and evidence uploads. Not checked yet New workspace overview and project inventory gives a birds-eye view of project activity with categorization and labeling across the workspace. Not checked yet Adds support for Databricks, Redshift, and Postgres as third-party data backends in Monitoring mode. Not checked yet Enables A/B testing across two commits (Development mode) or two data sources (Monitoring mode) to compare performance across metrics. Not checked yet Supports configuring directory sync for SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. /SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. -enabled workspaces via the UI. Not checked yet
+12 more
show less
Adds custom onboarding flow for users purchasing Openlayer through the AWS Marketplace. Not checked yet New API endpoint to delete custom metrics. Not checked yet Adds Python tracing support for OpenAI systems using the .parse() method or Responses API. Not checked yet Adds Python tracing support for systems built with Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. . Not checked yet Allows configuring trace settings (timeouts, retries) in Python via defined settings. Not checked yet CLI now supports creating data sources directly. Not checked yet CLI now supports deleting custom metrics directly. Not checked yet On-premises deployments can now leverage internal LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. gateways. Not checked yet New UI flow for accepting Assistant-generated tests, allowing users to select a subset, refine suggestions, or retry requests. Not checked yet Enables filtering columns in the Data page by clicking on a column header. Not checked yet Improved third-party data backend experience (Snowflake, Databricks) with faster multi-table connections, schema inference, and remote test execution for large data volumes. Not checked yet Enhanced navigation header for faster movement between projects and modes. Not checked yet
17 details in Everything
Openlayer Gateway, customizable dashboards, and Salesforce Agentforce GA
seen Jul 13, 2026
·
issue 014
Openlayer adds a managed AI gateway, customizable dashboards, Salesforce Agentforce GA, Helm on-prem, new guardrails, governance frameworks, and audit trails.
└──▷ HOW TO FIND IT
Route all LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. calls through the Openlayer Gateway to get centralized API key management, rate limiting, and end-to-end observability without changing your model provider code.
📍 In the console, go to Gateway › Admin Portal, create a gateway API key, assign rate limits and team, then point your LLM client's base URL to the Openlayer Gateway endpoint instead of the provider directly.
Delete stale monitoring records programmatically to keep your monitoring dataset clean.
copy
$ curl -X DELETE https://api.openlayer.com/v1/monitoring/records \
-H 'Authorization: Bearer <your-api-key>' \
-H 'Content-Type: application/json' \
-d '{"recordIds": ["<record-id-1>", "<record-id-2>"]}'
Deploy Openlayer on-prem to your Kubernetes cluster using the new Helm chart from Docker Hub OCI OCI Open Container Initiative standard, maintained by the Linux Foundation, defining the image format and runtime spec that container tools share so images built once run anywhere OCI-compliant. .
copy
$ helm install openlayer oci://registry-1.docker.io/openlayer/openlayer \
--set storage.type=s3 \
--set database.url=<your-db-url> \
--set replicaCount=3 \
--set image.pullSecrets[0]=<pull-secret> \
--set node.heapSize=4096
New Openlayer Gateway: centralized managed control layer for all LLM calls, with admin portal, API key management, per-key rate limits and freeze, and team-based organization. Not checked yet Customizable project home pages: pin metric widgets, view project notifications inline, and navigate alerts from a new sidebar notification center. Not checked yet Salesforce Agentforce integration reaches GA, with configurable login domain (production, sandbox, or My Domain) and updated Builder schema capturing every intermediate step. Not checked yet On-prem Helm chart published to Docker Hub OCI for Kubernetes deployments, with config for storage, database, replicas, image pull secrets, and Node heap. Not checked yet Built-in governance frameworks available out of the box: EU AI Act EU AI Act EU regulation that classifies AI systems by risk level and imposes compliance obligations—documentation, transparency, human oversight—on developers and deployers, shaping how AI-based cyber tools are built and sold in Europe. , TRAIGA, E23 MRM, and the Openlayer Governance Framework. Not checked yet
+30 more
show less
Full workspace audit trails with activity log export. Not checked yet Workspace-wide PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. redaction on trace processing, with per-project entity-type configuration. Not checked yet New guardrails: Llama Prompt Guard for prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. , English and Portuguese toxicity (with chunking and category filtering), and Brazilian PII recognizers (CPF, CNPJ, CEP, phone) with checksum validation. Not checked yet New tests: Bias, Contains PHI PHI Protected Health Information: individually identifiable health data regulated under the US HIPAA law, whose strict handling requirements drive cyber tools to audit, mask, or restrict access to it. , and Unauthorized Tool Call. Not checked yet Universal search across projects, sessions, users, and tests from a single search bar. Not checked yet Dialogflow integration added. Not checked yet Azure Content Understanding tracer added to SDKs. Not checked yet PydanticAI OTel OTel An open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. GenAI attribute parsing added to SDKs. Not checked yet Supports tracing the OpenAI Responses API from the TypeScript SDK. Not checked yet openlayer-ts SDK now accepts .csv datasets. Not checked yet Trace decorator gains a promote parameter to surface nested step outputs to the parent span. Not checked yet Session-level aggregated metrics: cost, latency, and token counts rolled up per session. Not checked yet LLM cost configuration: view and edit per-model pricing used for cost calculations. Not checked yet View, edit, and download custom-metric parameter bundles from the UI. Not checked yet LLM evaluator settings are now configurable for Ragas metrics. Not checked yet Monitoring-data limit notifications added. Not checked yet Test-connection button for LLM configuration. Not checked yet BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. data sources now supported without a timestamp column. Not checked yet Browser-based openlayer login via device code flow added to the CLI. Not checked yet New API endpoint to delete records in monitoring mode. Not checked yet New API endpoint to retrieve sessions linked to a test result. Not checked yet dateUpdated filter added to the List Tests API. Not checked yet Set workspace admins via environment variables on on-prem deployment. Not checked yet License usage and version reporting from on-prem deployments to Openlayer. Not checked yet Automatic threshold mode for custom metrics. Not checked yet Batch actions (multi-select and delete) on rows, rules, and projects tables. Not checked yet Filtering and sorting on sessions, users, data sources, projects, rules, and framework progress tables. Not checked yet Tag multiple tests at once. Not checked yet Domain verification added to workspace settings. Not checked yet Workspace setting to restrict workspace creation to deployment admins. Not checked yet
35 details in Everything
Pausing tests, checks for duplicate keys, and new integrations
seen Jul 13, 2026
·
issue 014
Openlayer adds Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. integration, Snowflake remote execution, pausing tests, duplicate-key checks, and cloud marketplace availability.
Adds column distribution graphs for LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. projects on the platform. Not checked yet New SDK integration with Google Agent Development Kit (ADK) for evaluating multi-agent systems. Not checked yet Enables pausing test execution in Monitoring mode. Not checked yet New CLI command to export tests for a project. Not checked yet New test to check for duplicate unique and primary keys. Not checked yet
+8 more
show less
Supports connecting to Snowflake views and running tests via remote execution. Not checked yet Enables custom SQL tests that join multiple tables. Not checked yet Supports specifying custom TLS certificates as environment variables to access external services. Not checked yet Adds Azure managed identity support for storage connections (On-Prem). Not checked yet Openlayer now available on AWS, Azure, and Google Cloud marketplaces. Not checked yet Allows defining specific columns to check for nulls in Null Rows tests. Not checked yet Assigns static IPs to Openlayer servers to simplify allowlisting. Not checked yet Renders higher decimal precision for test result values in the UI. Not checked yet
13 details in Everything
Project-level secrets, tracing LLM requests with OpenTelemetry
seen Jul 13, 2026
·
issue 014
Openlayer adds project-level secrets, OpenTelemetry OpenTelemetry An open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. tracing, Azure OpenAI for evals, and new platform APIs.
└──▷ HOW TO FIND IT
Store a sensitive API key once at the project level so all evals and tests can reference it without hardcoding credentials.
📍 In the Openlayer console, go to your Project › Settings › Secrets and add a new secret (e.g., OPENAI_API_KEY). Reference it by name in your test configurations.
Set a deployment-wide default LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge API key for all on-prem workspaces so teams don't need to configure keys individually.
📍 In the Openlayer on-prem admin console, go to Deployment Settings › Default API Keys and set the key for LLM-as-a-judge.
Adds project-level secrets storage in the Platform so API keys and auth tokens can be securely referenced across tests without hardcoding. Not checked yet Enables setting default LLM-as-a-judge API keys for an entire on-premise deployment, eliminating per-workspace/project key configuration. Not checked yet Adds SDK support for tracing LLM requests with OpenTelemetry. Not checked yet Supports Azure OpenAI deployments as the backend for LLM-as-a-judge evaluations. Not checked yet Adds gpt-4o-mini as an available LLM evaluator model. Not checked yet
+6 more
show less
Adds JSON schema validation test type to Evals. Not checked yet New API endpoints for workspace creation/retrieval, API key creation, and member invitations. Not checked yet New /versions/{id} endpoint on the public API for retrieving a specific commit/version by ID. Not checked yet New SDK endpoint to retrieve a commit by ID. Not checked yet Adds search, sort, and filter for inference pipelines in the Platform UI and via the API. Not checked yet Adds default test cases and metrics to LLM project templates. Not checked yet
11 details in Everything
SAML Directory Sync, new LLM-as-a-judge models, and website refresh
seen Jul 13, 2026
·
issue 014
Openlayer adds SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. directory sync, Cohere/Vertex AI LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge models, new RAG RAG Retrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. metrics, Viewer role, and expanded APIs.
└──▷ TRY IT
Include project and workspace details in a single inference pipeline GET call, avoiding extra round-trips.
copy
$ curl -X GET 'https://api.openlayer.com/v1/inference-pipelines/<id>?expand=project,workspace' \
-H 'Authorization: Bearer <token>'
Check how many tests are skipped or failing on a project version without fetching full test details.
copy
$ curl -X GET 'https://api.openlayer.com/v1/projects/<project-id>/versions/<version-id>' \
-H 'Authorization: Bearer <token>'
Enable SAML directory sync so workspace membership mirrors your identity provider identity provider A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. 's security groups.
📍 In the Openlayer console, go to Workspace Settings › Security › SAML SSO and enable Directory Sync to map your IdP groups to Openlayer workspaces.
Supports SAML SSO SAML SSO An authentication protocol that lets a user log in once with an identity provider and access multiple services without re-entering credentials, by passing signed XML assertions between parties. directory sync so Openlayer workspaces can be governed by existing organizational security groups. Not checked yet Adds Cohere and Vertex AI models as judge options for LLM-as-a-judge evaluation tests. Not checked yet New Faithfulness and Answer Correctness metrics available for RAG system evaluations. Not checked yet New Viewer workspace role with read-only access — no write, update, or delete permissions on resources. Not checked yet Adds expand parameter to inference pipeline GET endpoints, embedding embedding Lists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. project and workspace data directly in the response body. Not checked yet
+8 more
show less
Adds skipped and failing test counts to project version and inference pipeline API objects. Not checked yet Lower latency for the data stream endpoint. Not checked yet Supports async data uploads and faster batch uploads via pyarrow in the Python SDK. Not checked yet Enables pushing commits to the platform via the Python SDK. Not checked yet New tabular view of test results in test modals. Not checked yet Adds pie graph for test results on the project home screen. Not checked yet Slack notifications for pipeline creation now include the pipeline name. Not checked yet Adds Java code samples to documentation. Not checked yet
13 details in Everything
Stronger security, multimodal tracing, and expanded governance controls
seen Jul 13, 2026
·
issue 014
Openlayer adds MFA MFA Multi-factor authentication, requiring proof from different categories, such as a password and a hardware key, to sign in. It reduces reliance on a password alone. , multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. traces, directory sync, MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. integration, and expanded API/CLI controls for AI governance.
└──▷ TRY IT
Force a metrics run in CI to ensure all metrics execute even when no changes are detected.
copy
$ openlayer metrics run -force
Adds platform-wide multi-factor authentication multi-factor authentication Multi-factor authentication, requiring proof from different categories, such as a password and a hardware key, to sign in. It reduces reliance on a password alone. (MFA) for enterprise security. Not checked yet Supports deployment-wide directory sync with project-level access group support, including a NoAccess group level for finer-grained access control. Not checked yet Renders images, audio, and PDFs inline in trace steps and data tables for richer multimodal observability. Not checked yet Surfaces rule-result summaries on Governance and Frameworks pages to streamline audit readiness and compliance reporting. Not checked yet New Langflow integration for connecting Langflow-based AI workflows. Not checked yet
+17 more
show less
Adds MCP support: custom functions are now callable from Openlayer. Not checked yet Adds OpenAI Responses API support in the TypeScript SDK. Not checked yet New API endpoint to pull rows from specific test results. Not checked yet New API endpoint to delete projects. Not checked yet Adds API key authentication support for test result endpoints. Not checked yet Custom metrics bundle is now viewable and downloadable from the UI. Not checked yet New On-Prem Diagnostics page for testing connections and sending test emails. Not checked yet Enables data download in development mode. Not checked yet Real-time system health now reflected on the Status page. Not checked yet CLI openlayer push wait mode is now CI-mode compatible. Not checked yet CLI openlayer metrics run gains a -force flag and now prints which metrics are being run. Not checked yet Project owners now have admin-level access for all project-level actions. Not checked yet Session and user IDs in SDKs now accept a broader range of characters. Not checked yet LangChain tracer now supports VertexAI API Keys. Not checked yet ConversationalSearchService traces now include document chunks and nested step hierarchy. Not checked yet Renames Openlayer system tags to categories ; user-defined labels remain tags . Not checked yet Detected PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. types now render as tags in test results. Not checked yet
22 details in Everything
└──▷ BREAKING ON UPGRADE
! Openlayer system tags are renamed to categories ; user-defined labels remain tags , which may break any workflows, scripts, or integrations that reference the old tags name for system tags
Support for tracking users and sessions, test tags, and more
seen Jul 13, 2026
·
issue 014
Openlayer adds user/session tracking, test tags, LiteLLM support, PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. redaction guardrails, and multi-agent LangGraph workflows.
└──▷ HOW TO FIND IT
Redact PII automatically from traces in production before they reach Openlayer.
📍 In the Openlayer SDK, enable the guardrail system for tracing by configuring the guardrails option to redact PII on your tracer initialization.
Control which AWS Signature version Openlayer uses in an on-prem deployment without changing application code.
copy
$ export AWS_SIGNATURE_VERSION=v4
Set up user and session tracking to view full trace history per user in the Openlayer app.
📍 In the Openlayer app, go to the Data section and enable user and session tracking to associate traces with individual users and sessions.
Adds user and session tracking in the Data section, enabling full session and record history per user for behavioral analysis. Not checked yet Adds predefined and custom tags on tests to browse and organize tests by outcome, domain (e.g., safety, data drift), or governance framework (e.g., EU AI Act EU AI Act EU regulation that classifies AI systems by risk level and imposes compliance obligations—documentation, transparency, human oversight—on developers and deployers, shaping how AI-based cyber tools are built and sold in Europe. ). Not checked yet New API support for user-defined tags on tests and filtering tests by those tags. Not checked yet New API support for batch LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. calls. Not checked yet New API support for user sessions to track traces across interactions. Not checked yet
+8 more
show less
New LiteLLM integration via SDKs for tracing LiteLLM-powered calls. Not checked yet New guardrail system in SDKs for Openlayer tracing to redact PII from traces. Not checked yet On-Prem: AWS Signature version is now configurable via the AWS_SIGNATURE_VERSION environment variable. Not checked yet New API filter on version goals by whether they use validation or training datasets. Not checked yet On-Prem: Openlayer now supports the base Redis image for versions 8.2.1+. Not checked yet On-Prem: Adds custom schema support for Postgres tables connected to Openlayer. Not checked yet SDK support for Python 3.13. Not checked yet New comprehensive extraction and multi-agent LangGraph workflow support in SDKs. Not checked yet
13 details in Everything
Test bundles, new tests, support for new Python runtimes
seen Jul 13, 2026
·
issue 014
Openlayer adds test bundles (EU AI Act EU AI Act EU regulation that classifies AI systems by risk level and imposes compliance obligations—documentation, transparency, human oversight—on developers and deployers, shaping how AI-based cyber tools are built and sold in Europe. , OWASP OWASP A nonprofit foundation that publishes open standards, guides, and ranked vulnerability lists—most famously the OWASP Top 10—giving cyber tools a vendor-neutral benchmark to test or report against. , agentic), new LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. tests, and expanded Python runtime support.
└──▷ HOW TO FIND IT
Apply a pre-built EU AI Act test bundle to catch compliance gaps across your LLM pipeline before deploying.
📍 In the Openlayer app, go to Tests › Test Bundles, select the EU AI Act bundle, and click Apply to your project.
Filter your test results to surface only high-priority failing tests during an incident triage.
📍 In the Openlayer app, go to Tests, then use the Filter controls to set Priority = High and Status = Failing.
Introduces test bundles grouping related tests by policy or use case — pre-built sets for EU AI Act, OWASP, agentic workflows, and data quality. Not checked yet Adds new tests to the library: groundedness for LLMs LLMs A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. , toxicity score, prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. detection, and competitor-recommendation checks. Not checked yet Supports new Python runtimes for CLI development mode. Not checked yet Adds filtering of tests by priority, status, name, and other attributes in the platform UI. Not checked yet Updates OpenTelemetry OpenTelemetry An open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. GenAI semantic conventions attribute handling in the API. Not checked yet
+2 more
show less
Enhances secret management interface in the platform. Not checked yet Adds LangChain callback handler and Bedrock Agents support via TypeScript SDK trace refactoring. Not checked yet
7 details in Everything
The Openlayer MCP server, Automatic thresholds, BigQuery Integration and Anomaly Detection, Project-level access groups
seen Jul 13, 2026
·
issue 014
Openlayer adds MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server, automatic thresholds, BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. integration, anomaly detection, and project-level access groups.
└──▷ HOW TO FIND IT
Let Openlayer automatically determine acceptable thresholds for a new test instead of guessing manually — ideal when you lack historical baselines.
📍 In the Openlayer app, go to your project › Tests › Create new test. In the threshold configuration step, select automatic instead of entering a manual value. Openlayer will derive thresholds from historical data patterns for that metric.
Connect a BigQuery table to a project for ongoing, automated data quality monitoring without manual data uploads.
📍 In the Openlayer app, go to your project › Integrations › BigQuery and follow the prompts to link a BigQuery table. Once connected, Openlayer will continuously pull data and run data quality tests against it.
Run Openlayer tests and explore results without leaving your IDE by using the MCP server.
📍 In your IDE's MCP configuration, add the Openlayer MCP server as a provider. Once connected, invoke Openlayer tests directly from the IDE workflow to get evaluation results inline during development.
New Openlayer MCP server enables use of Openlayer tests directly inside IDE workflows. Not checked yet Supports automatic, data-driven thresholds for tests that adapt over time based on historical patterns — covering cost, data quality, and GPT eval scores. Not checked yet Adds anomaly detection to flag test results that deviate from predicted norms. Not checked yet New column statistic test supports mean, median, min, max, std, sum, count, and variance. Not checked yet New raw SQL query test for custom evaluation logic. Not checked yet
+21 more
show less
Adds anomalous column detection test. Not checked yet New BigQuery integration enables directly connecting a project to BigQuery tables for continuous data quality monitoring. Not checked yet Supports creating tests programmatically via the API. Not checked yet Adds async OpenAI tracer to the SDK. Not checked yet SDK convenience function to copy tests from one project to another. Not checked yet SDK push function gains an option to wait for commit completion. Not checked yet Adds support for Gemini 2.0 models in LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-a-judge tests. Not checked yet New priority property for tests (critical, high, medium, low). Not checked yet Enables running monitoring mode tests against the entire history of data rather than moving windows. Not checked yet Supports immediately executing tests in monitoring mode. Not checked yet Parses OpenTelemetry OpenTelemetry An open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. traces from Semantic Kernel and Spring AI. Not checked yet Adds root cause analysis and segment distribution graphs to test diagnostic pages. Not checked yet Daily feature distribution graphs for tabular data projects. Not checked yet Project-level permissions through access groups. Not checked yet On-premise deployment guides for OpenShift OpenShift Red Hat's Kubernetes distribution that adds developer workflows, built-in CI/CD, and stricter security defaults on top of upstream Kubernetes, giving cyber tools a managed container platform with enterprise auth and RBAC already wired in. and AWS EKS EKS Amazon's managed Kubernetes service, run by AWS, that handles control-plane provisioning so teams can deploy containerized workloads without operating the underlying cluster infrastructure themselves. . Not checked yet Option to include or exclude inference pipelines when creating project-level tests. Not checked yet Adds record count and last record received date to inference pipeline views. Not checked yet Surfaces OpenAI refusals to users in LLM-as-a-judge tests. Not checked yet Adds notifications when batch data uploads fail. Not checked yet Test failures no longer cause a commit's status to fail. Not checked yet Adds OpenLIT integration notebook to the SDK. Not checked yet
26 details in Everything
Trace timeline, project lifecycles, and Claude Agent SDK tracing
seen Jul 13, 2026
·
issue 014
Openlayer adds Gantt trace timelines, Claude Agent SDK tracing, auto-instrumentation, and Gateway guardrails enforcement.
└──▷ USE IT
Bootstrap full automatic LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. SDK instrumentation for all supported providers in one call, then run your app — no per-SDK setup needed.
python
copy
import openlayer
openlayer.init(api_key="<your-api-key>", project_name="<project-name>")
# All installed supported SDKs (OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK, Azure) are now patched automatically.
Opt out of automatic instrumentation when you want manual control over which SDKs are traced.
python
copy
import openlayer
openlayer.init(api_key="<your-api-key>", project_name="<project-name>", auto_instrument=False)
Inspect the full agent loop — tool calls, MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. tools, subagent nesting, and token costs — by enabling Claude Agent SDK tracing in Python.
python
copy
import openlayer
import anthropic
openlayer.init(api_key="<your-api-key>", project_name="<project-name>")
# The Claude Agent SDK is now instrumented: tool calls, MCP tools, subagents,
# session continuity, and cost/tokens will appear in Openlayer traces.
New Gantt-style timeline view for traces — every span's start time, duration, and nesting shown side by side, making slow steps and unexpected branches immediately visible. Not checked yet Adds project lifecycle stages and evidence document attachments for governance tracking, living alongside tests and results on each project. Not checked yet First-party tracing for the Anthropic Claude Agent SDK in Python — captures tool calls, MCP tools, subagent nesting, session continuity, and cost and token counts per run. Not checked yet New automatic SDK instrumentation via openlayer.init() — one call patches every installed supported LLM SDK (OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. , Azure); opt out with auto_instrument=False. Not checked yet Gateway can now enforce guardrails policies inline on LLM requests routing through it. Not checked yet
+9 more
show less
Admins can generate Gateway API key invite links so users can self-provision their own Gateway API keys. Not checked yet Adds rule deactivation reasons — supply a reason when deactivating a rule; it is displayed on the rule page. Not checked yet Results pages now surface which tests are required for compliance, shown in context. Not checked yet Data sources can now be renamed directly from their row menu in the UI. Not checked yet Row data can now be downloaded directly from the development-mode view. Not checked yet LangChain callback now captures ToolMessage.artifact (including document lists, promoted to trace-level context) instead of dropping it. Not checked yet Trace configuration is now unified around init() — settings merge incrementally instead of resetting on each call; configure() is deprecated in favor of init() . Not checked yet New openlayer tests export CLI subcommand support for projects that use tags. Not checked yet Dialogflow integration now promotes relevant metadata to the row level and allows editing the agent display name. Not checked yet
14 details in Everything
└──▷ BREAKING ON UPGRADE
! The configure() function is deprecated in favor of init() ; trace configuration is now unified around openlayer.init() , with settings merging incrementally instead of resetting on each call.! The Copilot Studio Logic App and Batch API integration paths are now deprecated in favor of the pull-based native data source.
Trace timeline, project lifecycles, and Claude Agent SDK tracing
Jun 4, 2026
· release notes
New Gantt trace timeline view, Claude Agent SDK tracing, automatic init() instrumentation, Gateway guardrail enforcement, and invite link key provisioning
Adds openlayer.init() auto-instrumentation function to patch OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. , and Azure SDKs automatically Not checked yet Adds Gantt-style trace timeline view showing span start time, duration, and nesting for debugging slow steps or unexpected branches Not checked yet Adds project lifecycle tracking with evidence document attachment for governance process visibility Not checked yet Adds Gateway inline guardrails enforcement on LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. requests routing through the service Not checked yet Adds Gateway admin API key invite links allowing users to self-provision their own Gateway keys Not checked yet
5 details in Everything
Openlayer Gateway, customizable dashboards, and Salesforce Agentforce GA
May 15, 2026
· release notes
Openlayer Spring 2026 adds Gateway, Helm chart, Salesforce integration, new guardrails/tests, customizable dashboards, and API filters.
Adds openlayer Gateway, a managed control layer for LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. calls with an admin portal for per-key rate limits, freeze, and team-based organization Not checked yet Adds on-prem Helm chart published to Docker Hub OCI OCI Open Container Initiative standard, maintained by the Linux Foundation, defining the image format and runtime spec that container tools share so images built once run anywhere OCI-compliant. for Kubernetes deployments with config for storage, database, replicas, image pull secrets, and Node heap Not checked yet Adds Salesforce Agentforce integration (GA) allowing end-to-end tracing with configurable login domain (production, sandbox, or My Domain) and updated Builder schema Not checked yet Adds Llama Prompt Guard prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. guardrail, English/Portuguese toxicity guardrails, and Brazilian PII PII Personally Identifiable Information — data that can identify a specific individual, such as names, email addresses, or SSNs. Cyber tools flag or redact it to limit exposure and meet privacy regulations. recognizers (CPF, CNPJ, CEP, phone) Not checked yet Adds new platform tests: Bias , Contains PHI , and Unauthorized tool call Not checked yet
+14 more
show less
Adds workspace-level and per-project PII redaction in traces with configurable entity types Not checked yet Adds universal search across projects, sessions, users, and tests from one search bar Not checked yet Adds customizable project dashboards with pin-able widgets, inline notifications, and a sidebar notification center Not checked yet Adds Delete records in monitoring mode via API Not checked yet Adds Retrieve sessions linked to a test result via API Not checked yet Adds dateUpdated filter on the List Tests API Not checked yet Adds BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. data source support for datasets without a timestamp column Not checked yet Adds workspace settings to restrict workspace creation to deployment admins and prevent users from given domains from creating workspaces Not checked yet Adds domain verification on the workspace settings page Not checked yet Adds built-in governance frameworks: EU AI Act EU AI Act EU regulation that classifies AI systems by risk level and imposes compliance obligations—documentation, transparency, human oversight—on developers and deployers, shaping how AI-based cyber tools are built and sold in Europe. , TRAIGA, E23 MRM, and Openlayer Governance Framework Not checked yet Adds audit trails with full workspace activity log and export Not checked yet Adds session-level aggregated metrics for cost, latency, and token counts Not checked yet Adds LLM cost configuration to view and edit per-model pricing for cost calculations Not checked yet Adds ability to view, edit, and download custom-metric parameter bundles from the UI Not checked yet
19 details in Everything
Stronger security, multimodal tracing, and expanded governance controls
Mar 4, 2026
· release notes
Openlayer adds MFA MFA Multi-factor authentication, requiring proof from different categories, such as a password and a hardware key, to sign in. It reduces reliance on a password alone. , multimodal multimodal Able to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. trace rendering, API project deletion, and custom function MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. integration.
Adds platform-wide multi-factor authentication multi-factor authentication Multi-factor authentication, requiring proof from different categories, such as a password and a hardware key, to sign in. It reduces reliance on a password alone. (MFA) for enterprise security Not checked yet Images and audio now render inline in trace steps and data tables via UI/UX updates Not checked yet PDF preview is available in traces and data tables through the UI Not checked yet Enables deployment-wide directory sync with project-level access group support for granular IAM IAM Identity and Access Management — a framework of policies and services for controlling who can authenticate to systems and what resources they are authorized to use, letting cyber tools enforce least-privilege and audit access. Not checked yet Governance and frameworks pages surface rule-result summaries for audit readiness Not checked yet
+8 more
show less
Data download capability is now available in development mode Not checked yet New diagnostics page allows testing connections and sending test emails on-prem Not checked yet Adds Langflow integration for workflow interoperability Not checked yet TypeScript SDK gains OpenAI Responses API support Not checked yet Pulls rows from specific test results via a new API endpoint Not checked yet Adds the ability to delete projects via the API Not checked yet Custom metrics bundles are viewable and downloadable directly from the UI Not checked yet Supports MCP custom functions callable from Openlayer for extended agent capabilities Not checked yet
13 details in Everything
Batch test re-runs and expanded integrations
Jan 28, 2026
· release notes
Openlayer adds batch test re-runs, new Ruby/Strands/Gemini/Copilot integrations, and CSI CSI An ambiguous abbreviation — it most commonly means Container Storage Interface (a Kubernetes storage plugin standard) or Crime Scene Investigation — without more context the correct meaning cannot be determined reliably. drift detection
Adds Batch test re-runs to recalculate results when new data is added retroactively Not checked yet Adds Openlayer Ruby SDK support for programmatic interaction Not checked yet Adds Strands Agents SDK integration Not checked yet Adds Microsoft Copilot Studio integration Not checked yet Adds Google Gemini API integration Not checked yet
+5 more
show less
Adds Custom time range selection for test execution Not checked yet Adds Manual test re-runs capability Not checked yet Adds New primary key uniqueness constraint test type Not checked yet Adds API endpoints for retrieving test results and managing projects Not checked yet Adds Characteristic Stability Index (CSI) as a drift detection method Not checked yet
10 details in Everything
Pausing tests, checks for duplicate keys, and new integrations
Dec 23, 2025
· release notes
Openlayer adds Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. integration, pausable tests in Monitoring mode, duplicate key checks, CLI export, Snowflake/remote support, and marketplace availability.
Adds openlayer test export CLI command to export tests for a project Not checked yet Adds ability to pause test execution in Monitoring mode via the platform UI Not checked yet Adds new integration with Google Agent Development Kit (ADK) for evaluating multi-agent systems Not checked yet Adds new platform test type to check for duplicate unique and primary keys Not checked yet Adds support connecting to Snowflake views and running tests via remote execution Not checked yet
+5 more
show less
Adds ability to run custom SQL tests joining multiple tables Not checked yet Adds OPENLAYER_CERTS environment variables (or equivalent config) to specify custom certs for external services Not checked yet Adds Azure managed identity support for storage connections in On-Prem deployments Not checked yet Makes Openlayer available on AWS, Azure, and Google Cloud marketplaces Not checked yet Adds ability to assign static IPs to Openlayer servers for allowlisting Not checked yet
10 details in Everything
Openlayer Governance introduces policy enforcement, new data sources, and CLI/API capabilities for custom metrics.
Introduces Governance mode to define and enforce policies across projects in a workspace Not checked yet Adds a new workspace overview and project inventory for tracking activity Not checked yet Supports Databricks, Redshift, and Postgres as 3rd-party data backends in Monitoring mode Not checked yet Enables A/B testing across two commits in Development mode or data sources in Monitoring mode Not checked yet Allows configuring directory sync in workspaces with SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. /SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. enabled via the UI Not checked yet
+6 more
show less
Adds DELETE capability for custom metrics via the API Not checked yet Adds Python SDK support for tracing OpenAI systems using .parse() method or Responses API Not checked yet Adds Python SDK support for tracing systems built with Google ADK ADK Google's Agent Development Kit, an open-source framework for building and orchestrating AI agents; cyber tools use it to compose multi-agent workflows with built-in tool-calling and state management. Not checked yet Allows configuring trace settings like timeouts and retries in Python SDK Not checked yet Enables creating data sources directly from the CLI Not checked yet Enables deleting custom metrics directly from the CLI Not checked yet
11 details in Everything
Support for tracking users and sessions, test tags, and more
Oct 22, 2025
· release notes
Openlayer adds user/session tracking, test tags, LiteLLM guardrails, and AWS env var support
AWS_SIGNATURE_VERSION environment variable allows setting the AWS signature version for On-Prem deployments. Not checked yet Adds user_sessions support in the API to track traces per user session. Not checked yet Adds user-defined tags on tests with filtering capabilities via the API. Not checked yet Adds batch LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. call support to the API. Not checked yet Adds LiteLLM integration for SDK tracing. Not checked yet
5 details in Everything
Test bundles, new tests, support for new Python runtimes
Sep 20, 2025
· release notes
Openlayer introduces test bundles, new LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. evaluation tests, and Python runtime support in CLI.
Introduces test bundles to group tests for use cases like the EU AI Act EU AI Act EU regulation that classifies AI systems by risk level and imposes compliance obligations—documentation, transparency, human oversight—on developers and deployers, shaping how AI-based cyber tools are built and sold in Europe. , OWASP OWASP A nonprofit foundation that publishes open standards, guides, and ranked vulnerability lists—most famously the OWASP Top 10—giving cyber tools a vendor-neutral benchmark to test or report against. , agentic workflows, and data quality. Not checked yet Adds new test types to the library: groundedness for LLMs LLMs A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. , toxicity score, prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. checks, and competitor recommendation checks. Not checked yet Adds support for new Python runtimes in the CLI development mode. Not checked yet
3 details in Everything
Complete design system overhaul, Snowflake Integration
Aug 21, 2025
· release notes
New Snowflake integration, Bedrock/LangGraph tracing support, dark mode UI, and member-restricted role.
Add directly connect Snowflake tables to projects integration Not checked yet Add SDK support for tracing Bedrock models , OpenAI Agents, Pydantic AI systems , and LangGraph systems Not checked yet Add new Member restricted role that can perform member actions without viewing data source data Not checked yet Allow hosting Openlayer on subpaths in on-prem deployments Not checked yet Allow users to override LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. costs with custom costs via the platform Not checked yet
+2 more
show less
Include standard deviation score in LLM-as-a-judge and Ragas test results Not checked yet Add new prompt injection prompt injection An attack where text the model reads, such as a web page, file or tool result, carries instructions that override the user's. It is the main security risk for agents with tools. test to detect adversarial attacks on LLM systems Not checked yet
7 details in Everything
The Openlayer MCP server, Automatic thresholds, BigQuery Integration and Anomaly Detection, Project-level access groups
Jun 14, 2025
· release notes
Openlayer adds MCP MCP Model Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server, automatic test thresholds, BigQuery BigQuery A serverless, cloud-hosted analytical database from Google that runs SQL queries across massive datasets without managing infrastructure, giving cyber tools scalable storage and fast querying over large volumes of log or event data. integration, anomaly detection, and project-level access groups.
Releases the Openlayer MCP server for IDE workflows Not checked yet Adds automatic threshold option for tests in the app UI Not checked yet Introduces anomaly detection to flag test results deviating from automatic thresholds Not checked yet Supports creating tests via API Not checked yet Adds direct BigQuery integration for continuous data quality monitoring Not checked yet
5 details in Everything
Project-level secrets, tracing LLM requests with OpenTelemetry
Mar 15, 2025
· release notes
Add project-level secrets management, default API keys for on-prem LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-judge, and OpenTelemetry OpenTelemetry An open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. tracing support
Support project-level secrets stored and referenced directly from the Platform UI/API Not checked yet
1 detail in Everything
SAML Directory Sync, new LLM-as-a-judge models, and website refresh
Jan 16, 2025
· release notes
Openlayer adds SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. directory sync, Cohere/Vertex AI LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. -as-judge models, and expand API query param.
Directory sync with SAML Not checked yet Use Cohere, Vertex AI models as options for LLM-as-a-judge metrics Not checked yet Add expand to inference pipeline GETs so projects and workspaces are included in the response body Not checked yet New Viewer role in workspaces that doesn’t have write, update or delete permissions on resources Not checked yet
4 details in Everything
Improved test diagnosis page, SAML SSO, design refreshes, + more
Oct 31, 2024
· release notes
Openlayer adds SAML SSO SAML SSO An authentication protocol that lets a user log in once with an identity provider and access multiple services without re-entering credentials, by passing signed XML assertions between parties. support and a test diagnosis page with error visibility and filtering
Enables SAML SAML A standard for passing signed identity assertions between a sign-in provider and an application. It lets a user sign in through their organization's identity provider. SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. authentication for workspaces via major identity providers identity providers A service that authenticates users and supplies identity information to other applications. It is the sign-in authority behind many single sign-on systems. . Not checked yet Adds List view on the test diagnosis page to scan skipped/errored tests and filter by type. Not checked yet Shows an overview at the top of the test diagnosis page with total results per status and recent issues. Not checked yet Makes sections in the test diagnosis page collapsible. Not checked yet Supports Azure OpenAI as an LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. evaluator. Not checked yet
+3 more
show less
Adds Ragas metrics (faithfulness, answer correctness) to Evals. Not checked yet Updates cost table for OpenAI models in Observability. Not checked yet Lists pipeline names in notifications for new inference pipelines. Not checked yet
8 details in Everything
Custom metrics, rotating API keys, and new models for direct-to-API calls
Jul 1, 2024
· release notes
Openlayer introduces custom metric upload via config, API key rotation management, and expanded model support for direct API calls.
openlayer.json now accepts metrics configuration to upload and use custom metrics as project-wide evaluations or individual tests Not checked yet API keys can be created, renamed, and deleted directly via the platform to enable rotation Not checked yet
2 details in Everything
Improved quality control over your LLM’s responses with annotations and human feedback
May 31, 2024
· release notes
Openlayer adds request annotations, thumbs-up/down ratings, ground truth editing, and filtered test modal data.
Adds the ability to annotate any request with custom values for granular quality control Not checked yet Adds thumbs up / thumbs down rating UI to requests for scanning good and bad responses Not checked yet Adds ability to add ground truths to requests and edit existing ground truths Not checked yet Filters data in individual test modals by selected evaluation window Not checked yet
4 details in Everything
Simple, dev-focused workflow for AI evals
May 3, 2024
· release notes
Openlayer adds GitHub integration for continuous evals via CLI and REST API REST API A web API design style where clients interact with resources over HTTP using standard methods like GET, POST, and DELETE, giving cyber tools a predictable, language-agnostic way to integrate with external services. with grouped-by-column test support
Adds GitHub integration that commits to Openlayer on every GitHub commit to trigger continuous evaluation Not checked yet Adds ability to customize the evaluation workflow using the CLI and REST API Not checked yet Adds template repositories for common AI eval use cases with cloning support in the Openlayer UI Not checked yet Adds None value support in the token column via API Not checked yet Adds new test type allowing results to be grouped by a specific column Not checked yet
5 details in Everything
Openlayer adds request tracing for debugging and Azure OpenAI integration support
Adds the ability to trace every step of a request in the UI, including latency and function inputs & outputs Not checked yet Adds support for using Azure OpenAI models Not checked yet Expands request streaming capabilities to include support for function calling function calling A model's ability to answer with a structured request to run a named function or API, which the application executes and returns. It is how agents take actions. Not checked yet
3 details in Everything
Openlayer v1.0 adds granular latency percentiles (min, median, p90, p95, p99) and subpopulation filters for data integrity/consistency tests.
Adds Min Latency , Median Latency , 90th Percentile Latency , 95th Percentile Latency , and 99th Percentile Latency test types in the Performance page Not checked yet Enables subpopulation filters for Data Integrity or Data Consistency tests to target specific data clusters Not checked yet Adds project name search capability in the project overview UI Not checked yet
3 details in Everything
Go deep on test result history and add multiple criteria to GPT evaluation tests
Feb 1, 2024
· release notes
Openlayer adds multi-criteria GPT evaluation tests and deep test result history visualization
Adds ability to add multiple criteria to a single GPT evaluation test, allowing unique thresholds for each criterion in one go Not checked yet Enables selecting specific date ranges and scrubbing through graphs on individual test pages to view requests from specific time periods Not checked yet
2 details in Everything
Cost-per-request, new tests, subpopulation support for data tests, and more precise row filtering
Jan 17, 2024
· release notes
Openlayer adds cost-per-request monitoring, subpopulation filtering for data tests, and new Column average/contains string test types.
Adds Column average test to ensure column averages fall within a specified range Not checked yet Adds Column contains string test where values in one column must contain strings from another column Not checked yet Adds cost-related tests: Max cost , Mean cost , and Total cost per evaluation window Not checked yet Shows request costs on the Requests page with cost graph support Not checked yet Enables subpopulation filtering for integrity and consistency data tests using row-level metrics like conciseness Not checked yet
+1 more
show less
Returns to the timeline page to visualize commit performance against goals over time Not checked yet
6 details in Everything
Log multi-turn interactions, sort and filter production requests, and token usage and latency graphs
Dec 21, 2023
· release notes
Adds multi-turn chat history logging, token/latency graphs, Gemini model support, and request sorting/filtering
Logs full multi-turn chat history for production requests in monitoring mode Not checked yet Displays token usage and latency graphs over time for all requests Not checked yet Supports Google's Gemini model for evaluation and comparison Not checked yet Adds row-by-row explanations for tests using GPT evaluation Not checked yet Allows sorting and filtering production requests by timestamp, token usage, or latency Not checked yet
5 details in Everything
GPT evaluation, Great Expectations, real-time streaming, TypeScript support, and new docs
Nov 30, 2023
· release notes
Openlayer introduces GPT evaluation, Great Expectations integration, real-time streaming API, and TypeScript SDK
Adds built-in GPT evaluation tests for model outputs using descriptive criteria via LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. grading Not checked yet Adds support for creating and running tests from Great Expectations (GX) data integrity checks Not checked yet Adds new /stream endpoint for real-time data ingestion instead of batch uploads Not checked yet Releases V1 of the Openlayer TypeScript client library for logging requests Not checked yet Adds a new Requests page in the UI to view real-time model request streams and metadata like token count and latency Not checked yet
5 details in Everything
└──▷ BREAKING ON UPGRADE
! The term goals has been renamed to tests across the product, documentation, and website; references to the old term are no longer valid.
Enhanced onboarding, redesigned navigation, and new goals
Nov 17, 2023
· release notes
Openlayer adds new consistency/integrity goals and arbitrary column names in thresholds.
Adds size-ratio goal to enforce required size ratios between datasets Not checked yet Adds character_length goal to enforce character limits on text-based columns Not checked yet
2 details in Everything
Evals for LLMs, real-time monitoring, Slack notifications and so much more!
Oct 24, 2023
· release notes
Openlayer adds LLM LLM A large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. evals in dev/monitoring modes, Slack notifications, and workspace secrets
Adds LLMs in development mode for experimenting with prompts and models via UI prompt playground connecting to OpenAI, Anthropic and Cohere. Not checked yet Adds LLMs in monitoring mode to evaluate production responses using the same tests from development while measuring token usage, latency, drift and data volume. Not checked yet Adds Slack integration to receive notifications for goal status updates, team activity like comments, and workspace updates. Not checked yet
3 details in Everything
Regression projects, toasts, and artifact retrieval
Jun 29, 2023
· release notes
Openlayer adds tabular regression evaluation with MSE/MAE metrics and commit artifact downloads.
Adds support for evaluating tabular regression models in projects Not checked yet Enables downloading models and datasets (commit artifacts) via the platform UI Not checked yet Allows deleting commits through the platform interface Not checked yet
3 details in Everything
Sign in with Google, sample projects, mentions and more!
May 30, 2023
· release notes
Openlayer v230530 adds Google SSO SSO Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. , sample projects, user mentions, shell model uploads, ROC AUC metric, and workspace browsing.
Enabled SSO (single sign-on single sign-on Single sign-on, using one identity provider to sign in to multiple applications. It reduces separate passwords and centralizes account and access management. ) with Google for authentication Not checked yet Added sample projects to all workspaces Not checked yet Added support for mentioning users, goals, and commits in goal comments and descriptions using @ and # symbols Not checked yet Added the ability to upload shell models without a model binary for explainability and fairness analysis Not checked yet Added ROC AUC to available project metrics Not checked yet
+1 more
show less
Added an overview page to browse and navigate to projects Not checked yet
6 details in Everything