https://cybertoolchain.io/tools/openlayer New releases and features in Openlayer, tracked by The Cyber Toolchain. en Wed, 28 Oct 2026 07:53:45 GMT https://www.openlayer.com/changelog https://www.openlayer.com/changelog Wed, 28 Oct 2026 07:53:45 GMT Openlayer adds governance lifecycle stages, bias/fairness tests, EU data residency, Gateway routing controls, and 10+ new native integrations.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds `evaluationScope: step` config field for per-step evaluation of tests on each trace step.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds `openlayer tests delete` CLI command — archives by default, `--force` permanently deletes. • Extends `openlayer init` with Copilot CLI and Gemini CLI support, custom CA bundles, a 'debug with an agent' option, and `.openlayerignore` for smaller uploads. • Adds Gateway `GET /v1/models` (per key) and OpenAI-compatible `POST /v1/embeddings` API endpoints. • Adds governance registry and framework endpoints to the public API and SDKs, writable with a workspace API key. • Adds a new batch-delete endpoint for asynchronous, batched, idempotent row deletes. • Adds EU data region — choose United States or European Union at sign-in/sign-up; account and data stay in the selected region. • Adds project lifecycle stages — define stages, assign reviewers per stage, and receive warnings when an earlier stage's requirements are no longer met. • Adds three new built-in compliance frameworks: NYDFS Circular Letter No. 7 (2024), Colorado Regulation 10-1-1, and the NAIC AI Model Bulletin. • Adds audit-ready framework export — export a compliance report for any framework. Breaking changes: • Gateway 'Teams' are renamed to 'Groups' — any configuration, API calls, or documentation referencing 'Teams' in the Gateway admin UI must be updated to 'Groups'. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog https://www.openlayer.com/changelog Wed, 21 Oct 2026 07:44:07 GMT Openlayer adds governance lifecycle stages, bias/fairness tests, 8 new integrations, Gateway routing, and a public governance API.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds `evaluationScope: step` config field for per-step evaluation — run tests on each individual trace step.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds `openlayer tests delete` CLI command — archives tests by default; `--force` flag permanently deletes them. • Extends `openlayer init` to support Copilot CLI and Gemini CLI, custom CA bundles, a 'debug with an agent' option, and `.openlayerignore` for smaller uploads. • Adds governance registry and framework endpoints to the public API and SDKs, writable with a workspace API key. • Adds API key expiry dates, automatic rotation, and full API key management via the API. • Adds bias and fairness tests — four test types check protected groups and their intersections for outcome, score, performance, and metric disparities, with an Analysis tab for charts, drill-down, and CSV export. • Adds prompt injection tests — choose the detector and a sensitivity level (multilingual included), see why a detection fired, and pick the judge model. • Adds Freshness and Column freshness tests — alert when a data source stops receiving data or a datetime column gets too old, with manual or automatic thresholds. • Adds Custom columns — a named JSONPath into structured outputs, defined once per project and usable in tests, filters, and tables across every data source and development commit. • Adds Jev-as-a-judge test that scores every row in the window and explains its score on demand. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog https://www.openlayer.com/changelog Wed, 12 Aug 2026 09:34:36 GMT Openlayer adds row-level test result API, project deletion API, MCP custom functions, and metric bundle downloads • New API endpoint to pull individual rows from specific test results, enabling programmatic drill-down into evaluation data. • New API endpoint to delete projects, available via the TypeScript SDK. • MCP integration now supports custom functions callable from Openlayer. • Custom metrics bundles are now viewable and downloadable directly from the UI. Breaking changes: • Openlayer system tags are renamed to 'categories'; user-defined labels remain 'tags'. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog https://www.openlayer.com/changelog Tue, 11 Aug 2026 15:43:28 GMT Openlayer adds remote MCP connector, Agent Skills, semantic search, webhooks for test results, and 25+ new capabilities in its August 2026 release. • Adds `openlayer init` — one guided command takes a fresh directory to a pushable, linked, monitored project with tracing instrumentation. • Adds non-interactive CLI mode — `link`, `login`, and resource creation run headless, enabling use from AI coding agents and CI pipelines. • Expands `openlayer export` to support CSV and XLSX formats (previously JSON only) and adds the ability to include scores in exports. • Adds webhooks for test results — subscribe a webhook endpoint to test-suite outcomes and get notified when tests pass or fail. • Adds multimodal support to the `/responses` and `/messages` Gateway endpoints — media sent and received, not just text. • Adds a new PATCH endpoint to update a project's name and description via API, picked up by the Python, Go, Java, and Ruby SDKs and the MCP server. • Expands workspace API key surface — keys now work for updating projects, cost-override endpoints, test retrieval/update/deletion, and seven previously session-only read routes. • Adds sorting and filtering to more list endpoints — workspace rules, project members, and user sessions; rules tables also gain server-side group filtering. • Adds a remote, OAuth-protected MCP connector — the MCP server runs hosted instead of local-only, usable in Claude Code and claude.ai, with in-chat workspace switching. • Adds governance and lifecycle tools via MCP — frameworks, rules, and rule results are reachable through MCP, growing the total toolset from 44 to 53 tools. • Adds Openlayer Agent Skills — packaged skills that teach AI coding assistants to integrate a codebase with Openlayer (tracing, offline evals, tests, guardrails, CI gating); ships as a Claude Code plugin, Cursor plugin, or via the skills CLI. • Adds dedicated Gemini and Vertex tracing in Python and TypeScript for the unified Gemini SDK including Vertex, where previously only the legacy SDK was auto-instrumented. • Adds Claude Agent SDK tracing for TypeScript — drop-in import auto-traces agent runs. • Adds native IBM watsonx Orchestrate integration, modeled on the existing Agentforce and Copilot Studio integrations. • Adds semantic (model-based) routing — routing rules can score each prompt in-process and route it to a strong or weak target by threshold, alongside static first-match rules. • Adds per-user usage limits and guardrails in the Gateway — limits and guardrails can target an individual user, including per-user caps within a team. • Adds a content safety guardrail type for Gateway requests and responses. • Adds a missing-guardrails governance rule — a new integrity test that flags traces missing an expected guardrail. • Adds US National Provider Identifier (NPI) as a native PII check — detectable in Contains-PII tests and redactable in traces. • Adds per-project data retention — set a rolling retention window (7/30/90/180/365 days or indefinite); monitoring data older than the window is deleted. • Adds the ability to pause and resume individual monitoring tests without disabling the whole project. • Adds bulk session deletion — multi-select sessions and delete with a full cascade to their traces, scores, and embeddings. • Adds workspace-level dashboards — at-a-glance cost, usage, and project health across the workspace with drill-in widgets. • Adds AI-generated session, trace, and test-result summaries — session and trace detail views open with an AI summary; test results include a summary of dominant failure modes with drill-down to exact rows. • Adds semantic search filters — embedding-based subpopulation filtering for tests, beyond exact and keyword matching. • Adds an official Helm chart for the Gateway for Kubernetes deployments. • Adds a deployment admin area — operators get background-job queue health, metrics, history, and license/feature status, gated by a dedicated deployment-admin role. • Adds granular token usage and cost tracking — cached, reasoning, and audio token categories are priced and stored; LLM-as-judge evaluator runs are cost-tracked per test result. • Adds Gateway spend visibility — per-model breakdown on the usage endpoint, a per-user spend aggregation endpoint, projected savings when creating a limit, key rotation, and provider secrets settable from the admin UI. • Adds browser-based CLI login — sign in with a device-code browser flow instead of pasting an API key. • Adds suggested filter values — filter fields suggest likely values as you type, served from the backend and respecting other active filters. • Adds bulk test actions — duplicate several tests at once and schedule runs for several tests at once; view, download, and delete uploaded custom metric files. • Adds on-prem deployment options — CSI Secrets Store support, external/managed ClickHouse, native Oracle Object Storage as a storage backend, and a multi-arch Gateway image for ARM hosts. • Adds a free public LLM cost explorer tool at llm-costs.openlayer.com for browsing and comparing LLM pricing across providers, including per-token-category pricing (cached, reasoning, audio). Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog https://www.openlayer.com/changelog Tue, 11 Aug 2026 09:15:27 GMT Openlayer adds AI summaries, semantic search, remote MCP, webhooks, per-user Gateway limits, and a Helm chart for the Gateway in its August 2026 release. • Adds `openlayer init` CLI command to guide a fresh directory to a linked, monitored project with tracing instrumentation in one step. • Adds non-interactive (headless) CLI mode — `link`, `login`, and resource creation now accept no user prompts, enabling use from AI coding agents and CI pipelines. • Expands `openlayer export` to support CSV and XLSX output formats (previously JSON only) and adds a flag to include scores in the export. • Adds openlayer.init() automatic instrumentation for Python — one call patches every installed supported LLM SDK (OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK, Azure); opt out with `auto_instrument=False`. • Introduces webhooks for test results — subscribe a webhook endpoint to test-suite outcomes and receive notifications when tests pass or fail. • Adds `/responses` and `/messages` Gateway endpoints with multimodal support, accepting and returning media as well as text. • Expands API-key surface so workspace API keys now cover updating projects, cost-override endpoints, test retrieval/update/deletion, and seven previously session-only read routes. • Adds a new PATCH endpoint to update a project's name and description, surfaced in the Python, Go, Java, and Ruby SDKs and the MCP server. • Adds sorting and filtering to more list endpoints: workspace rules, project members, and user sessions, plus server-side group filtering on rules tables. • Adds a per-user spend aggregation endpoint and a per-model breakdown on the usage endpoint for Gateway spend visibility. • Adds Gemini and Vertex tracing in Python and TypeScript via a dedicated tracer for the unified Gemini SDK, including Vertex AI (previously only the legacy SDK was auto-instrumented). • Adds Claude Agent SDK tracing for TypeScript — drop-in import auto-traces agent runs. • Adds Claude Agent SDK tracing for Python — instruments the full agent loop, capturing tool calls, MCP tools, subagent nesting, session continuity, cost, and tokens. • Adds a remote, OAuth-protected MCP connector — the MCP server runs hosted (not local-only), usable in Claude Code and claude.ai, with in-chat workspace switching. • Expands MCP tooling to 53 tools (up from 44), adding governance/lifecycle access (frameworks, rules, rule results) and 12 new session, trace, and test-lifecycle tools. • Introduces Openlayer Agent Skills — packaged skills for AI coding assistants (Claude Code plugin, Cursor plugin, or `skills` CLI) covering tracing, offline evals, tests, guardrails, and CI gating for both LLM and traditional/tabular ML workflows. • Adds native IBM watsonx Orchestrate integration, modeled on the existing Agentforce and Copilot Studio integrations. • Adds semantic (model-based) routing — routing rules can score each prompt in-process and send it to a strong or weak target by threshold, alongside existing static first-match rules. • Adds per-project data retention with a configurable rolling window (7, 30, 90, 180, or 365 days, or indefinite); monitoring data older than the window is deleted. • Adds per-user usage limits and guardrails in the Gateway, including per-user caps within a team. • Adds a content safety guardrail type for Gateway requests and responses. • Adds a missing-guardrails governance rule — a new integrity test that flags traces missing an expected guardrail. • Adds US National Provider Identifier (NPI) as a native PII check, detectable in Contains-PII tests and redactable in traces. • Adds project lifecycle stages and evidence documents — attach a lifecycle stage and evidence to a project for governance tracking. • Adds rule deactivation reasons — supply a reason when deactivating a rule; it is displayed on the rule page. • Adds workspace-level dashboards with at-a-glance cost, usage, and project health across the workspace, including drill-in widgets. • Adds AI-generated summaries to session, trace, and test-result detail views, with drill-down from failure modes to the exact rows behind each pattern. • Adds semantic (embedding-based) search filters for subpopulation filtering in tests, beyond exact and keyword matching. • Adds pause and resume for individual monitoring tests without disabling the whole project. • Adds bulk session deletion — multi-select sessions and delete with full cascade to their traces, scores, and embeddings. • Adds a Helm chart for the Gateway for Kubernetes deployments. • Adds a deployment admin area for on-prem operators with background-job queue health, metrics, history, and license/feature status, gated by a dedicated deployment-admin role. • Adds browser-based CLI login via device-code browser flow instead of requiring an API key to be pasted. • Adds bulk test actions — duplicate or schedule runs for several tests at once, and view, download, or delete uploaded custom metric files. • Adds a timeline (Gantt-style) view for traces showing every span's start time, duration, and nesting side by side. • Adds Gateway API key invite links — admins can generate links that let users self-provision their own Gateway API keys. • Adds compliance test visibility to results pages — shows which tests are required for compliance, in context. • Introduces a free public LLM cost explorer at `llm-costs.openlayer.com` for browsing and comparing LLM pricing across providers, including per-token-category pricing (cached, reasoning, audio). • Adds granular token usage tracking — cached, reasoning, and audio token categories are now priced and stored, with per-API-key cost visible in the UI; LLM-as-judge evaluator runs are cost-tracked per test result. • Adds LangChain `ToolMessage.artifact` capture (including document lists, promoted to trace-level context) in the Python and TypeScript SDKs. • Unifies trace configuration around init() in the SDK — settings merge incrementally instead of resetting on each call (configure() is deprecated in favor of init()). • Adds on-prem support for CSI Secrets Store, external/managed ClickHouse, native Oracle Object Storage as a storage backend, and a multi-arch Gateway image for ARM hosts. • Adds the ability to rename data sources directly from the data source row menu. • Adds row data download in development mode. Breaking changes: • configure() is deprecated in favor of init() for trace configuration in the SDK; settings now merge incrementally instead of resetting on each call. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog https://www.openlayer.com/changelog Thu, 06 Aug 2026 10:51:17 GMT June 2026: Gantt-style trace timeline view makes slow spans and unexpected branches immediately visible. • New Gantt-style trace timeline view lines up every span by start time, duration, and nesting to surface slow steps and unexpected branches at a glance. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/batch-test-re-runs-and-expanded-integrations https://www.openlayer.com/changelog/batch-test-re-runs-and-expanded-integrations Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds batch test re-runs, Ruby SDK, Gemini/Copilot Studio integrations, and new drift detection. • Adds batch and manual test re-runs so results can be recalculated retroactively when new data is added. • Supports automatic threshold configuration for tests built on custom metrics. • Introduces custom time range selection for filtering test execution history. • New primary key uniqueness constraint test type available on the platform. • Adds Ruby SDK for instrumenting Openlayer in Ruby applications. • New Strands Agents SDK integration for tracing agentic workflows. • New Microsoft Copilot Studio integration. • New Google Gemini API integration. • New API endpoints for retrieving test results and managing projects. • Adds Characteristic Stability Index (CSI) as a drift detection method. • Adds step detection support in the LangChain integration. • Adds model output capture in the Google ADK integration. • Smoother Snowflake connection flow now supports views. • Simplified Databricks onboarding flow. • Redesigned monitoring test results graph for clearer visibility. • Clearer trace visualizations in the UI. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/complete-design-system-overhaul-snowflake-integration https://www.openlayer.com/changelog/complete-design-system-overhaul-snowflake-integration Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds Snowflake integration, new security role, prompt injection tests, and expanded LLM tracing for Bedrock, OpenAI Agents, Pydantic AI, and LangGraph. • New UI with full light and dark mode support, updated design system, and improved navigation hierarchy. • Adds priority levels for tests and more data visualizations throughout the product. • SDK support for tracing Bedrock models. • SDK support for tracing OpenAI Agents. • SDK support for tracing Pydantic AI systems. • SDK support for tracing LangGraph systems. • New 'Member restricted' security role that can perform member actions without viewing data source data. • Direct Snowflake table connections to projects via native integration. • Project, dataset, and table dropdowns when connecting BigQuery tables. • Supports hosting Openlayer on subpaths in on-premises deployments. • Allows users to override LLM costs with custom cost values. • Includes standard deviation score in LLM-as-a-judge and Ragas test results. • New prompt injection test to detect adversarial attacks on LLM systems. • Per-row scores now visible for metrics like semantic similarity and exact match in data tables. • API endpoints that return record counts and last record date for data sources are now faster. • Faster data source creation via API. • Supports streaming data with numpy arrays in the API request body. Breaking changes: • "Inference pipelines" are renamed to "data sources" throughout the platform Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/custom-metrics-rotating-api-keys-and-new-models-for-direct-to-api-calls https://www.openlayer.com/changelog/custom-metrics-rotating-api-keys-and-new-models-for-direct-to-api-calls Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds custom metrics via openlayer.json, rotating API keys, and 11 new models for direct-to-API evaluation calls. • Supports custom metrics uploaded via `openlayer.json`, usable as project-wide metrics or as runnable eval tests alongside standard metrics. • Enables creation, renaming, and deletion of multiple personal API keys to support API key rotation workflows. • Adds ability to specify desired metrics directly in `openlayer.json` for API-driven evaluation runs. • Adds 11 new models for direct-to-API calls: GPT-4o, GPT-4 Turbo, Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Command R, Command R Plus, Gemini 1.0 Flash, Gemini 1.5 Flash, and Gemini 1.5 Pro. • Adds git repository and organization linking in git settings pages for tighter version-control integration. • New UI button to view the processing status of a commit during project loading. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/improved-quality-control-over-your-llm-s-responses-with-annotations-and-human-feedback https://www.openlayer.com/changelog/improved-quality-control-over-your-llm-s-responses-with-annotations-and-human-feedback Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds request annotations, human feedback ratings, data export, and Git integration to LLM quality control. • Adds per-request annotations with custom values, including step-level annotation for individual trace steps when tracing is enabled. • Adds thumbs up/thumbs down human feedback ratings on every request to quickly surface good and bad model responses. • Enables data export from the UI, including filtered cohorts of request data. • Introduces updated navigation with breadcrumbs and the ability to copy project name and inference pipeline ID. • Adds project-level Git repository settings, including editable branch and root directory configuration. • Adds ability to add and edit ground truths on individual requests. • Adds filtering of test modal data by the selected evaluation window. • Adds more project templates. • Improves the OpenAI SDK integration. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/improved-test-diagnosis-page-saml-sso-design-refreshes-more https://www.openlayer.com/changelog/improved-test-diagnosis-page-saml-sso-design-refreshes-more Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds SAML SSO, Azure OpenAI evaluator, Ragas metrics, and an overhauled test diagnosis page. • Adds SAML SSO support via any major identity provider, enabling teams to enforce org-level authentication for workspace access. • Supports Azure OpenAI as an LLM evaluator integration. • Adds additional Ragas metrics — faithfulness and answer correctness — for evaluation. • New list view on the test diagnosis page surfaces error messages for skipped and errored tests, with filtering by result type and a status-breakdown overview. • Test diagnosis page sections are now collapsible for a streamlined review experience. • Updated cost table for OpenAI models in the Observability view. • Notifications for new inference pipelines now include the pipeline name. • More informative tooltips on test cards. • Groq guide now available in documentation. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/introducing-openlayer-governance https://www.openlayer.com/changelog/introducing-openlayer-governance Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds Governance mode, A/B testing, new data backends, CLI data sources, and expanded tracing support. • New Governance mode lets teams define and enforce custom policy frameworks (PII leakage, prompt injection, monitoring requirements, ownership, risk level) across all projects and workspaces, with rule owners, comment threads, and evidence uploads. • New workspace overview and project inventory gives a birds-eye view of project activity with categorization and labeling across the workspace. • Adds support for Databricks, Redshift, and Postgres as third-party data backends in Monitoring mode. • Enables A/B testing across two commits (Development mode) or two data sources (Monitoring mode) to compare performance across metrics. • Supports configuring directory sync for SAML/SSO-enabled workspaces via the UI. • Adds custom onboarding flow for users purchasing Openlayer through the AWS Marketplace. • New API endpoint to delete custom metrics. • Adds Python tracing support for OpenAI systems using the .parse() method or Responses API. • Adds Python tracing support for systems built with Google ADK. • Allows configuring trace settings (timeouts, retries) in Python via defined settings. • CLI now supports creating data sources directly. • CLI now supports deleting custom metrics directly. • On-premises deployments can now leverage internal LLM gateways. • New UI flow for accepting Assistant-generated tests, allowing users to select a subset, refine suggestions, or retry requests. • Enables filtering columns in the Data page by clicking on a column header. • Improved third-party data backend experience (Snowflake, Databricks) with faster multi-table connections, schema inference, and remote test execution for large data volumes. • Enhanced navigation header for faster movement between projects and modes. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/spring-2026 https://www.openlayer.com/changelog/spring-2026 Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds a managed AI gateway, customizable dashboards, Salesforce Agentforce GA, Helm on-prem, new guardrails, governance frameworks, and audit trails. • New Openlayer Gateway: centralized managed control layer for all LLM calls, with admin portal, API key management, per-key rate limits and freeze, and team-based organization. • Customizable project home pages: pin metric widgets, view project notifications inline, and navigate alerts from a new sidebar notification center. • Salesforce Agentforce integration reaches GA, with configurable login domain (production, sandbox, or My Domain) and updated Builder schema capturing every intermediate step. • On-prem Helm chart published to Docker Hub OCI for Kubernetes deployments, with config for storage, database, replicas, image pull secrets, and Node heap. • Built-in governance frameworks available out of the box: EU AI Act, TRAIGA, E23 MRM, and the Openlayer Governance Framework. • Full workspace audit trails with activity log export. • Workspace-wide PII redaction on trace processing, with per-project entity-type configuration. • New guardrails: Llama Prompt Guard for prompt injection, English and Portuguese toxicity (with chunking and category filtering), and Brazilian PII recognizers (CPF, CNPJ, CEP, phone) with checksum validation. • New tests: Bias, Contains PHI, and Unauthorized Tool Call. • Universal search across projects, sessions, users, and tests from a single search bar. • Dialogflow integration added. • Azure Content Understanding tracer added to SDKs. • PydanticAI OTel GenAI attribute parsing added to SDKs. • Supports tracing the OpenAI Responses API from the TypeScript SDK. • openlayer-ts SDK now accepts .csv datasets. • Trace decorator gains a `promote` parameter to surface nested step outputs to the parent span. • Session-level aggregated metrics: cost, latency, and token counts rolled up per session. • LLM cost configuration: view and edit per-model pricing used for cost calculations. • View, edit, and download custom-metric parameter bundles from the UI. • LLM evaluator settings are now configurable for Ragas metrics. • Monitoring-data limit notifications added. • Test-connection button for LLM configuration. • BigQuery data sources now supported without a timestamp column. • Browser-based `openlayer login` via device code flow added to the CLI. • New API endpoint to delete records in monitoring mode. • New API endpoint to retrieve sessions linked to a test result. • `dateUpdated` filter added to the List Tests API. • Set workspace admins via environment variables on on-prem deployment. • License usage and version reporting from on-prem deployments to Openlayer. • Automatic threshold mode for custom metrics. • Batch actions (multi-select and delete) on rows, rules, and projects tables. • Filtering and sorting on sessions, users, data sources, projects, rules, and framework progress tables. • Tag multiple tests at once. • Domain verification added to workspace settings. • Workspace setting to restrict workspace creation to deployment admins. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/pausing-tests-checks-for-duplicate-keys-and-new-integrations https://www.openlayer.com/changelog/pausing-tests-checks-for-duplicate-keys-and-new-integrations Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds Google ADK integration, Snowflake remote execution, pausing tests, duplicate-key checks, and cloud marketplace availability. • Adds column distribution graphs for LLM projects on the platform. • New SDK integration with Google Agent Development Kit (ADK) for evaluating multi-agent systems. • Enables pausing test execution in Monitoring mode. • New CLI command to export tests for a project. • New test to check for duplicate unique and primary keys. • Supports connecting to Snowflake views and running tests via remote execution. • Enables custom SQL tests that join multiple tables. • Supports specifying custom TLS certificates as environment variables to access external services. • Adds Azure managed identity support for storage connections (On-Prem). • Openlayer now available on AWS, Azure, and Google Cloud marketplaces. • Allows defining specific columns to check for nulls in Null Rows tests. • Assigns static IPs to Openlayer servers to simplify allowlisting. • Renders higher decimal precision for test result values in the UI. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/project-level-secrets-tracing-llm-requests-with-opentelemetry https://www.openlayer.com/changelog/project-level-secrets-tracing-llm-requests-with-opentelemetry Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds project-level secrets, OpenTelemetry tracing, Azure OpenAI for evals, and new platform APIs. • Adds project-level secrets storage in the Platform so API keys and auth tokens can be securely referenced across tests without hardcoding. • Enables setting default LLM-as-a-judge API keys for an entire on-premise deployment, eliminating per-workspace/project key configuration. • Adds SDK support for tracing LLM requests with OpenTelemetry. • Supports Azure OpenAI deployments as the backend for LLM-as-a-judge evaluations. • Adds gpt-4o-mini as an available LLM evaluator model. • Adds JSON schema validation test type to Evals. • New API endpoints for workspace creation/retrieval, API key creation, and member invitations. • New `/versions/{id}` endpoint on the public API for retrieving a specific commit/version by ID. • New SDK endpoint to retrieve a commit by ID. • Adds search, sort, and filter for inference pipelines in the Platform UI and via the API. • Adds default test cases and metrics to LLM project templates. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/saml-directory-sync-new-llm-as-a-judge-models-and-website-refresh https://www.openlayer.com/changelog/saml-directory-sync-new-llm-as-a-judge-models-and-website-refresh Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds SAML directory sync, Cohere/Vertex AI LLM-as-a-judge models, new RAG metrics, Viewer role, and expanded APIs. • Supports SAML SSO directory sync so Openlayer workspaces can be governed by existing organizational security groups. • Adds Cohere and Vertex AI models as judge options for LLM-as-a-judge evaluation tests. • New Faithfulness and Answer Correctness metrics available for RAG system evaluations. • New 'Viewer' workspace role with read-only access — no write, update, or delete permissions on resources. • Adds `expand` parameter to inference pipeline GET endpoints, embedding project and workspace data directly in the response body. • Adds skipped and failing test counts to project version and inference pipeline API objects. • Lower latency for the data stream endpoint. • Supports async data uploads and faster batch uploads via pyarrow in the Python SDK. • Enables pushing commits to the platform via the Python SDK. • New tabular view of test results in test modals. • Adds pie graph for test results on the project home screen. • Slack notifications for pipeline creation now include the pipeline name. • Adds Java code samples to documentation. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/stronger-security-multimodal-tracing-and-expanded-governance-controls https://www.openlayer.com/changelog/stronger-security-multimodal-tracing-and-expanded-governance-controls Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds MFA, multimodal traces, directory sync, MCP integration, and expanded API/CLI controls for AI governance. • Adds platform-wide multi-factor authentication (MFA) for enterprise security. • Supports deployment-wide directory sync with project-level access group support, including a NoAccess group level for finer-grained access control. • Renders images, audio, and PDFs inline in trace steps and data tables for richer multimodal observability. • Surfaces rule-result summaries on Governance and Frameworks pages to streamline audit readiness and compliance reporting. • New Langflow integration for connecting Langflow-based AI workflows. • Adds MCP support: custom functions are now callable from Openlayer. • Adds OpenAI Responses API support in the TypeScript SDK. • New API endpoint to pull rows from specific test results. • New API endpoint to delete projects. • Adds API key authentication support for test result endpoints. • Custom metrics bundle is now viewable and downloadable from the UI. • New On-Prem Diagnostics page for testing connections and sending test emails. • Enables data download in development mode. • Real-time system health now reflected on the Status page. • CLI `openlayer push` wait mode is now CI-mode compatible. • CLI `openlayer metrics run` gains a `-force` flag and now prints which metrics are being run. • Project owners now have admin-level access for all project-level actions. • Session and user IDs in SDKs now accept a broader range of characters. • LangChain tracer now supports VertexAI API Keys. • ConversationalSearchService traces now include document chunks and nested step hierarchy. • Renames Openlayer system tags to 'categories'; user-defined labels remain 'tags'. • Detected PII types now render as tags in test results. Breaking changes: • Openlayer system tags are renamed to "categories"; user-defined labels remain "tags", which may break any workflows, scripts, or integrations that reference the old "tags" name for system tags Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/support-for-tracking-users-and-sessions-test-tags-and-more https://www.openlayer.com/changelog/support-for-tracking-users-and-sessions-test-tags-and-more Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds user/session tracking, test tags, LiteLLM support, PII redaction guardrails, and multi-agent LangGraph workflows. • Adds user and session tracking in the Data section, enabling full session and record history per user for behavioral analysis. • Adds predefined and custom tags on tests to browse and organize tests by outcome, domain (e.g., safety, data drift), or governance framework (e.g., EU AI Act). • New API support for user-defined tags on tests and filtering tests by those tags. • New API support for batch LLM calls. • New API support for user sessions to track traces across interactions. • New LiteLLM integration via SDKs for tracing LiteLLM-powered calls. • New guardrail system in SDKs for Openlayer tracing to redact PII from traces. • On-Prem: AWS Signature version is now configurable via the `AWS_SIGNATURE_VERSION` environment variable. • New API filter on version goals by whether they use validation or training datasets. • On-Prem: Openlayer now supports the base Redis image for versions 8.2.1+. • On-Prem: Adds custom schema support for Postgres tables connected to Openlayer. • SDK support for Python 3.13. • New comprehensive extraction and multi-agent LangGraph workflow support in SDKs. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/test-bundles-new-tests-support-for-new-python-runtimes https://www.openlayer.com/changelog/test-bundles-new-tests-support-for-new-python-runtimes Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds test bundles (EU AI Act, OWASP, agentic), new LLM tests, and expanded Python runtime support. • Introduces test bundles grouping related tests by policy or use case — pre-built sets for EU AI Act, OWASP, agentic workflows, and data quality. • Adds new tests to the library: groundedness for LLMs, toxicity score, prompt injection detection, and competitor-recommendation checks. • Supports new Python runtimes for CLI development mode. • Adds filtering of tests by priority, status, name, and other attributes in the platform UI. • Updates OpenTelemetry GenAI semantic conventions attribute handling in the API. • Enhances secret management interface in the platform. • Adds LangChain callback handler and Bedrock Agents support via TypeScript SDK trace refactoring. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/the-openlayer-mcp-server-automatic-thresholds-bigquery-integration-and-anomaly-detection-project https://www.openlayer.com/changelog/the-openlayer-mcp-server-automatic-thresholds-bigquery-integration-and-anomaly-detection-project Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds MCP server, automatic thresholds, BigQuery integration, anomaly detection, and project-level access groups. • New Openlayer MCP server enables use of Openlayer tests directly inside IDE workflows. • Supports automatic, data-driven thresholds for tests that adapt over time based on historical patterns — covering cost, data quality, and GPT eval scores. • Adds anomaly detection to flag test results that deviate from predicted norms. • New column statistic test supports mean, median, min, max, std, sum, count, and variance. • New raw SQL query test for custom evaluation logic. • Adds anomalous column detection test. • New BigQuery integration enables directly connecting a project to BigQuery tables for continuous data quality monitoring. • Supports creating tests programmatically via the API. • Adds async OpenAI tracer to the SDK. • SDK convenience function to copy tests from one project to another. • SDK push function gains an option to wait for commit completion. • Adds support for Gemini 2.0 models in LLM-as-a-judge tests. • New priority property for tests (critical, high, medium, low). • Enables running monitoring mode tests against the entire history of data rather than moving windows. • Supports immediately executing tests in monitoring mode. • Parses OpenTelemetry traces from Semantic Kernel and Spring AI. • Adds root cause analysis and segment distribution graphs to test diagnostic pages. • Daily feature distribution graphs for tabular data projects. • Project-level permissions through access groups. • On-premise deployment guides for OpenShift and AWS EKS. • Option to include or exclude inference pipelines when creating project-level tests. • Adds record count and last record received date to inference pipeline views. • Surfaces OpenAI refusals to users in LLM-as-a-judge tests. • Adds notifications when batch data uploads fail. • Test failures no longer cause a commit's status to fail. • Adds OpenLIT integration notebook to the SDK. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/june-2026 https://www.openlayer.com/changelog/june-2026 Mon, 13 Jul 2026 10:06:58 GMT Openlayer adds Gantt trace timelines, Claude Agent SDK tracing, auto-instrumentation, and Gateway guardrails enforcement. • New Gantt-style timeline view for traces — every span's start time, duration, and nesting shown side by side, making slow steps and unexpected branches immediately visible. • Adds project lifecycle stages and evidence document attachments for governance tracking, living alongside tests and results on each project. • First-party tracing for the Anthropic Claude Agent SDK in Python — captures tool calls, MCP tools, subagent nesting, session continuity, and cost and token counts per run. • New automatic SDK instrumentation via openlayer.init() — one call patches every installed supported LLM SDK (OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK, Azure); opt out with `auto_instrument=False`. • Gateway can now enforce guardrails policies inline on LLM requests routing through it. • Admins can generate Gateway API key invite links so users can self-provision their own Gateway API keys. • Adds rule deactivation reasons — supply a reason when deactivating a rule; it is displayed on the rule page. • Results pages now surface which tests are required for compliance, shown in context. • Data sources can now be renamed directly from their row menu in the UI. • Row data can now be downloaded directly from the development-mode view. • LangChain callback now captures `ToolMessage.artifact` (including document lists, promoted to trace-level context) instead of dropping it. • Trace configuration is now unified around init() — settings merge incrementally instead of resetting on each call; configure() is deprecated in favor of init(). • New `openlayer tests export` CLI subcommand support for projects that use tags. • Dialogflow integration now promotes relevant metadata to the row level and allows editing the agent display name. Breaking changes: • The configure() function is deprecated in favor of init(); trace configuration is now unified around openlayer.init(), with settings merging incrementally instead of resetting on each call. • The Copilot Studio Logic App and Batch API integration paths are now deprecated in favor of the pull-based native data source. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/june-2026 https://www.openlayer.com/changelog/june-2026 Thu, 04 Jun 2026 00:00:00 GMT New Gantt trace timeline view, Claude Agent SDK tracing, automatic init() instrumentation, Gateway guardrail enforcement, and invite link key provisioning • Adds openlayer.init() auto-instrumentation function to patch OpenAI, Anthropic, Mistral, Groq, Gemini, LiteLLM, Portkey, Google ADK, and Azure SDKs automatically • Adds Gantt-style trace timeline view showing span start time, duration, and nesting for debugging slow steps or unexpected branches • Adds project lifecycle tracking with evidence document attachment for governance process visibility • Adds Gateway inline guardrails enforcement on LLM requests routing through the service • Adds Gateway admin API key invite links allowing users to self-provision their own Gateway keys Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/spring-2026 https://www.openlayer.com/changelog/spring-2026 Fri, 15 May 2026 00:00:00 GMT Openlayer Spring 2026 adds Gateway, Helm chart, Salesforce integration, new guardrails/tests, customizable dashboards, and API filters. • Adds `openlayer Gateway`, a managed control layer for LLM calls with an admin portal for per-key rate limits, freeze, and team-based organization • Adds on-prem Helm chart published to Docker Hub OCI for Kubernetes deployments with config for storage, database, replicas, image pull secrets, and Node heap • Adds `Salesforce Agentforce` integration (GA) allowing end-to-end tracing with configurable login domain (production, sandbox, or My Domain) and updated Builder schema • Adds `Llama Prompt Guard` prompt injection guardrail, English/Portuguese toxicity guardrails, and Brazilian PII recognizers (CPF, CNPJ, CEP, phone) • Adds new platform tests: `Bias`, `Contains PHI`, and `Unauthorized tool call` • Adds workspace-level and per-project PII redaction in traces with configurable entity types • Adds universal search across projects, sessions, users, and tests from one search bar • Adds customizable project dashboards with pin-able widgets, inline notifications, and a sidebar notification center • Adds `Delete records in monitoring mode` via API • Adds `Retrieve sessions linked to a test result` via API • Adds `dateUpdated` filter on the List Tests API • Adds BigQuery data source support for datasets without a timestamp column • Adds workspace settings to restrict workspace creation to deployment admins and prevent users from given domains from creating workspaces • Adds domain verification on the workspace settings page • Adds built-in governance frameworks: EU AI Act, TRAIGA, E23 MRM, and Openlayer Governance Framework • Adds audit trails with full workspace activity log and export • Adds session-level aggregated metrics for cost, latency, and token counts • Adds LLM cost configuration to view and edit per-model pricing for cost calculations • Adds ability to view, edit, and download custom-metric parameter bundles from the UI Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/stronger-security-multimodal-tracing-and-expanded-governance-controls https://www.openlayer.com/changelog/stronger-security-multimodal-tracing-and-expanded-governance-controls Wed, 04 Mar 2026 00:00:00 GMT Openlayer adds MFA, multimodal trace rendering, API project deletion, and custom function MCP integration. • Adds platform-wide multi-factor authentication (MFA) for enterprise security • Images and audio now render inline in trace steps and data tables via UI/UX updates • PDF preview is available in traces and data tables through the UI • Enables deployment-wide directory sync with project-level access group support for granular IAM • Governance and frameworks pages surface rule-result summaries for audit readiness • Data download capability is now available in development mode • New diagnostics page allows testing connections and sending test emails on-prem • Adds Langflow integration for workflow interoperability • TypeScript SDK gains OpenAI Responses API support • Pulls rows from specific test results via a new API endpoint • Adds the ability to delete projects via the API • Custom metrics bundles are viewable and downloadable directly from the UI • Supports MCP custom functions callable from Openlayer for extended agent capabilities Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/batch-test-re-runs-and-expanded-integrations https://www.openlayer.com/changelog/batch-test-re-runs-and-expanded-integrations Wed, 28 Jan 2026 00:00:00 GMT Openlayer adds batch test re-runs, new Ruby/Strands/Gemini/Copilot integrations, and CSI drift detection • Adds `Batch test re-runs` to recalculate results when new data is added retroactively • Adds Openlayer Ruby SDK support for programmatic interaction • Adds Strands Agents SDK integration • Adds Microsoft Copilot Studio integration • Adds Google Gemini API integration • Adds Custom time range selection for test execution • Adds Manual test re-runs capability • Adds New primary key uniqueness constraint test type • Adds API endpoints for retrieving test results and managing projects • Adds Characteristic Stability Index (CSI) as a drift detection method Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/pausing-tests-checks-for-duplicate-keys-and-new-integrations https://www.openlayer.com/changelog/pausing-tests-checks-for-duplicate-keys-and-new-integrations Tue, 23 Dec 2025 00:00:00 GMT Openlayer adds Google ADK integration, pausable tests in Monitoring mode, duplicate key checks, CLI export, Snowflake/remote support, and marketplace availability. • Adds `openlayer test export` CLI command to export tests for a project • Adds ability to pause test execution in Monitoring mode via the platform UI • Adds new integration with Google Agent Development Kit (ADK) for evaluating multi-agent systems • Adds new platform test type to check for duplicate unique and primary keys • Adds support connecting to Snowflake views and running tests via remote execution • Adds ability to run custom SQL tests joining multiple tables • Adds `OPENLAYER_CERTS` environment variables (or equivalent config) to specify custom certs for external services • Adds Azure managed identity support for storage connections in On-Prem deployments • Makes Openlayer available on AWS, Azure, and Google Cloud marketplaces • Adds ability to assign static IPs to Openlayer servers for allowlisting Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/introducing-openlayer-governance https://www.openlayer.com/changelog/introducing-openlayer-governance Mon, 24 Nov 2025 00:00:00 GMT Openlayer Governance introduces policy enforcement, new data sources, and CLI/API capabilities for custom metrics. • Introduces `Governance` mode to define and enforce policies across projects in a workspace • Adds a new workspace overview and project inventory for tracking activity • Supports Databricks, Redshift, and Postgres as 3rd-party data backends in Monitoring mode • Enables A/B testing across two commits in Development mode or data sources in Monitoring mode • Allows configuring directory sync in workspaces with SAML/SSO enabled via the UI • Adds `DELETE` capability for custom metrics via the API • Adds Python SDK support for tracing OpenAI systems using .parse() method or Responses API • Adds Python SDK support for tracing systems built with Google ADK • Allows configuring trace settings like timeouts and retries in Python SDK • Enables creating data sources directly from the CLI • Enables deleting custom metrics directly from the CLI Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/support-for-tracking-users-and-sessions-test-tags-and-more https://www.openlayer.com/changelog/support-for-tracking-users-and-sessions-test-tags-and-more Wed, 22 Oct 2025 00:00:00 GMT Openlayer adds user/session tracking, test tags, LiteLLM guardrails, and AWS env var support • `AWS_SIGNATURE_VERSION` environment variable allows setting the AWS signature version for On-Prem deployments. • Adds `user_sessions` support in the API to track traces per user session. • Adds user-defined tags on tests with filtering capabilities via the API. • Adds batch LLM call support to the API. • Adds LiteLLM integration for SDK tracing. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/test-bundles-new-tests-support-for-new-python-runtimes https://www.openlayer.com/changelog/test-bundles-new-tests-support-for-new-python-runtimes Sat, 20 Sep 2025 00:00:00 GMT Openlayer introduces test bundles, new LLM evaluation tests, and Python runtime support in CLI. • Introduces `test bundles` to group tests for use cases like the EU AI Act, OWASP, agentic workflows, and data quality. • Adds new test types to the library: groundedness for LLMs, toxicity score, prompt injection checks, and competitor recommendation checks. • Adds support for new Python runtimes in the CLI development mode. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/complete-design-system-overhaul-snowflake-integration https://www.openlayer.com/changelog/complete-design-system-overhaul-snowflake-integration Thu, 21 Aug 2025 00:00:00 GMT New Snowflake integration, Bedrock/LangGraph tracing support, dark mode UI, and member-restricted role. • Add `directly connect Snowflake tables to projects` integration • Add SDK support for tracing `Bedrock models`, `OpenAI Agents`, `Pydantic AI systems`, and `LangGraph systems` • Add new "Member restricted" role that can perform member actions without viewing data source data • Allow hosting Openlayer on subpaths in on-prem deployments • Allow users to override LLM costs with custom costs via the platform • Include standard deviation score in LLM-as-a-judge and Ragas test results • Add new prompt injection test to detect adversarial attacks on LLM systems Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/the-openlayer-mcp-server-automatic-thresholds-bigquery-integration-and-anomaly-detection-project https://www.openlayer.com/changelog/the-openlayer-mcp-server-automatic-thresholds-bigquery-integration-and-anomaly-detection-project Sat, 14 Jun 2025 00:00:00 GMT Openlayer adds MCP server, automatic test thresholds, BigQuery integration, anomaly detection, and project-level access groups. • Releases the Openlayer MCP server for IDE workflows • Adds `automatic` threshold option for tests in the app UI • Introduces anomaly detection to flag test results deviating from automatic thresholds • Supports creating tests via API • Adds direct BigQuery integration for continuous data quality monitoring Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/project-level-secrets-tracing-llm-requests-with-opentelemetry https://www.openlayer.com/changelog/project-level-secrets-tracing-llm-requests-with-opentelemetry Sat, 15 Mar 2025 00:00:00 GMT Add project-level secrets management, default API keys for on-prem LLM-as-judge, and OpenTelemetry tracing support • Support project-level secrets stored and referenced directly from the Platform UI/API Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/saml-directory-sync-new-llm-as-a-judge-models-and-website-refresh https://www.openlayer.com/changelog/saml-directory-sync-new-llm-as-a-judge-models-and-website-refresh Thu, 16 Jan 2025 00:00:00 GMT Openlayer adds SAML directory sync, Cohere/Vertex AI LLM-as-judge models, and `expand` API query param. • Directory sync with SAML • Use Cohere, Vertex AI models as options for LLM-as-a-judge metrics • Add `expand` to inference pipeline GETs so projects and workspaces are included in the response body • New "Viewer" role in workspaces that doesn’t have write, update or delete permissions on resources Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/improved-test-diagnosis-page-saml-sso-design-refreshes-more https://www.openlayer.com/changelog/improved-test-diagnosis-page-saml-sso-design-refreshes-more Thu, 31 Oct 2024 00:00:00 GMT Openlayer adds SAML SSO support and a test diagnosis page with error visibility and filtering • Enables SAML SSO authentication for workspaces via major identity providers. • Adds `List view` on the test diagnosis page to scan skipped/errored tests and filter by type. • Shows an overview at the top of the test diagnosis page with total results per status and recent issues. • Makes sections in the test diagnosis page collapsible. • Supports Azure OpenAI as an LLM evaluator. • Adds Ragas metrics (faithfulness, answer correctness) to Evals. • Updates cost table for OpenAI models in Observability. • Lists pipeline names in notifications for new inference pipelines. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/custom-metrics-rotating-api-keys-and-new-models-for-direct-to-api-calls https://www.openlayer.com/changelog/custom-metrics-rotating-api-keys-and-new-models-for-direct-to-api-calls Mon, 01 Jul 2024 00:00:00 GMT Openlayer introduces custom metric upload via config, API key rotation management, and expanded model support for direct API calls. • `openlayer.json` now accepts `metrics` configuration to upload and use custom metrics as project-wide evaluations or individual tests • API keys can be created, renamed, and deleted directly via the platform to enable rotation Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/improved-quality-control-over-your-llm-s-responses-with-annotations-and-human-feedback https://www.openlayer.com/changelog/improved-quality-control-over-your-llm-s-responses-with-annotations-and-human-feedback Fri, 31 May 2024 00:00:00 GMT Openlayer adds request annotations, thumbs-up/down ratings, ground truth editing, and filtered test modal data. • Adds the ability to annotate any request with custom values for granular quality control • Adds thumbs up / thumbs down rating UI to requests for scanning good and bad responses • Adds ability to add ground truths to requests and edit existing ground truths • Filters data in individual test modals by selected evaluation window Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/simple-dev-focused-workflow-for-ai-evals https://www.openlayer.com/changelog/simple-dev-focused-workflow-for-ai-evals Fri, 03 May 2024 00:00:00 GMT Openlayer adds GitHub integration for continuous evals via CLI and REST API with grouped-by-column test support • Adds GitHub integration that commits to Openlayer on every GitHub commit to trigger continuous evaluation • Adds ability to customize the evaluation workflow using the CLI and REST API • Adds template repositories for common AI eval use cases with cloning support in the Openlayer UI • Adds `None` value support in the token column via API • Adds new test type allowing results to be grouped by a specific column Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/trace-every-step-of-your-requests https://www.openlayer.com/changelog/trace-every-step-of-your-requests Thu, 28 Mar 2024 00:00:00 GMT Openlayer adds request tracing for debugging and Azure OpenAI integration support • Adds the ability to trace every step of a request in the UI, including latency and function inputs & outputs • Adds support for using Azure OpenAI models • Expands request streaming capabilities to include support for function calling Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/more-tests-around-latency-metrics https://www.openlayer.com/changelog/more-tests-around-latency-metrics Thu, 15 Feb 2024 00:00:00 GMT Openlayer v1.0 adds granular latency percentiles (min, median, p90, p95, p99) and subpopulation filters for data integrity/consistency tests. • Adds 'Min Latency', 'Median Latency', '90th Percentile Latency', '95th Percentile Latency', and '99th Percentile Latency' test types in the Performance page • Enables subpopulation filters for Data Integrity or Data Consistency tests to target specific data clusters • Adds project name search capability in the project overview UI Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/go-deep-on-test-result-history-and-add-multiple-criteria-to-gpt-evaluation-tests https://www.openlayer.com/changelog/go-deep-on-test-result-history-and-add-multiple-criteria-to-gpt-evaluation-tests Thu, 01 Feb 2024 00:00:00 GMT Openlayer adds multi-criteria GPT evaluation tests and deep test result history visualization • Adds ability to add multiple criteria to a single GPT evaluation test, allowing unique thresholds for each criterion in one go • Enables selecting specific date ranges and scrubbing through graphs on individual test pages to view requests from specific time periods Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/cost-per-request-new-tests-subpopulation-support-for-data-tests-and-more-precise-row-filtering https://www.openlayer.com/changelog/cost-per-request-new-tests-subpopulation-support-for-data-tests-and-more-precise-row-filtering Wed, 17 Jan 2024 00:00:00 GMT Openlayer adds cost-per-request monitoring, subpopulation filtering for data tests, and new Column average/contains string test types. • Adds `Column average` test to ensure column averages fall within a specified range • Adds `Column contains string` test where values in one column must contain strings from another column • Adds cost-related tests: `Max cost`, `Mean cost`, and `Total cost` per evaluation window • Shows request costs on the Requests page with cost graph support • Enables subpopulation filtering for integrity and consistency data tests using row-level metrics like conciseness • Returns to the timeline page to visualize commit performance against goals over time Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/log-multi-turn-interactions-sort-and-filter-production-requests-and-token-usage-and-latency https://www.openlayer.com/changelog/log-multi-turn-interactions-sort-and-filter-production-requests-and-token-usage-and-latency Thu, 21 Dec 2023 00:00:00 GMT Adds multi-turn chat history logging, token/latency graphs, Gemini model support, and request sorting/filtering • Logs full multi-turn chat history for production requests in monitoring mode • Displays token usage and latency graphs over time for all requests • Supports Google's Gemini model for evaluation and comparison • Adds row-by-row explanations for tests using GPT evaluation • Allows sorting and filtering production requests by timestamp, token usage, or latency Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/gpt-evaluation-great-expectations-real-time-streaming-typescript-support-and-new-docs https://www.openlayer.com/changelog/gpt-evaluation-great-expectations-real-time-streaming-typescript-support-and-new-docs Thu, 30 Nov 2023 00:00:00 GMT Openlayer introduces GPT evaluation, Great Expectations integration, real-time streaming API, and TypeScript SDK • Adds built-in GPT evaluation tests for model outputs using descriptive criteria via LLM grading • Adds support for creating and running tests from Great Expectations (GX) data integrity checks • Adds new `/stream` endpoint for real-time data ingestion instead of batch uploads • Releases V1 of the Openlayer TypeScript client library for logging requests • Adds a new 'Requests' page in the UI to view real-time model request streams and metadata like token count and latency Breaking changes: • The term 'goals' has been renamed to 'tests' across the product, documentation, and website; references to the old term are no longer valid. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/enhanced-onboarding-redesigned-navigation-and-new-goals https://www.openlayer.com/changelog/enhanced-onboarding-redesigned-navigation-and-new-goals Fri, 17 Nov 2023 00:00:00 GMT Openlayer adds new consistency/integrity goals and arbitrary column names in thresholds. • Adds `size-ratio` goal to enforce required size ratios between datasets • Adds `character_length` goal to enforce character limits on text-based columns Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/evals-for-llms-real-time-monitoring-slack-notifications-and-so-much-more https://www.openlayer.com/changelog/evals-for-llms-real-time-monitoring-slack-notifications-and-so-much-more Tue, 24 Oct 2023 00:00:00 GMT Openlayer adds LLM evals in dev/monitoring modes, Slack notifications, and workspace secrets • Adds `LLMs in development mode` for experimenting with prompts and models via UI prompt playground connecting to OpenAI, Anthropic and Cohere. • Adds `LLMs in monitoring mode` to evaluate production responses using the same tests from development while measuring token usage, latency, drift and data volume. • Adds Slack integration to receive notifications for goal status updates, team activity like comments, and workspace updates. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/regression-projects-toasts-and-artifact-retrieval https://www.openlayer.com/changelog/regression-projects-toasts-and-artifact-retrieval Thu, 29 Jun 2023 00:00:00 GMT Openlayer adds tabular regression evaluation with MSE/MAE metrics and commit artifact downloads. • Adds support for evaluating tabular regression models in projects • Enables downloading models and datasets (commit artifacts) via the platform UI • Allows deleting commits through the platform interface Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer https://www.openlayer.com/changelog/sign-in-with-google-sample-projects-mentions-and-more https://www.openlayer.com/changelog/sign-in-with-google-sample-projects-mentions-and-more Tue, 30 May 2023 00:00:00 GMT Openlayer v230530 adds Google SSO, sample projects, user mentions, shell model uploads, ROC AUC metric, and workspace browsing. • Enabled SSO (single sign-on) with Google for authentication • Added sample projects to all workspaces • Added support for mentioning users, goals, and commits in goal comments and descriptions using `@` and `#` symbols • Added the ability to upload “shell” models without a model binary for explainability and fairness analysis • Added ROC AUC to available project metrics • Added an overview page to browse and navigate to projects Full history on Cyber Toolchain: https://cybertoolchain.io/tools/openlayer