https://cybertoolchain.io/tools/promptfoo New releases and features in Promptfoo, tracked by The Cyber Toolchain. en Mon, 07 Sep 2026 07:33:06 GMT https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Mon, 07 Sep 2026 07:33:06 GMT Promptfoo adds Docker Model Runner, MCP server support, Google Imagen, Grok-4, and JKS/PFX certificate support for HTTP providers. • Adds HTTP provider support for JKS and PFX client certificates, enabling mTLS connections to secured endpoints. • Adds Docker Model Runner provider to run models in isolated Docker containers for better security and reproducibility. • Adds MCP (Model Context Protocol) provider to connect to MCP servers, including a red-team agent testing example with tool call results. • Adds Google Imagen provider for image generation in multimodal testing scenarios. • Adds AIMLAPI provider for accessing various AI models through a unified interface. • Adds Grok-4 model support via xAI for advanced reasoning evaluations. • Adds `o3-deep-research` and `o4-mini-deep-research` OpenAI Deep Research models for complex problem-solving evaluations. • Adds LiteLLM Embeddings support for similarity testing and semantic search evaluations. • Adds Google Vision provider for image understanding in multimodal evaluations. • Enhances Azure provider with system prompt support for better control over model behavior. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Wed, 19 Aug 2026 00:24:01 GMT Promptfoo adds an Unverifiable Claims Detection plugin to test AI systems for susceptibility to unverifiable assertions. • New Unverifiable Claims Detection plugin tests AI systems' susceptibility to unverifiable claims. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Wed, 05 Aug 2026 00:30:13 GMT • Adds test-level prompts filter and providers filter to scope test cases more precisely. • New Vercel AI Gateway and Cloudflare AI Gateway providers for request routing. • OpenAI Codex provider adds `collaboration_mode` support and integrated tracing. • Code scanning adds support for fork pull requests and comment-triggered scans. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Tue, 04 Aug 2026 10:52:00 GMT Promptfoo snapshot-20260804 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scanning. • Adds `promptfoo logs` command for viewing log files directly. • Adds `--share` and `--no-share` flags to Model Audit for controlling cloud sharing. • Adds `-d`/`--description` flag to `redteam generate` commands. • Adds `-j` flag propagation to Python worker pools for concurrency control. • Adds `numTests` config key to cap test cases generated per red team strategy. • Adds `$VAR` syntax support for environment variables in file paths. • Supports multiple `--env-file` flags to load multiple environment files. • Adds `word-count` assertion type for validating response word counts. • Adds `__count` variable for use in derived metrics when computing averages. • Surfaces `metadata.sessionId` as a variable column in eval tables and exports. • Adds test-level prompts filter and providers filter for per-test-case scoping in configuration. • Supports per-test structured output configuration at the test case level. • New Transformers.js provider — run Hugging Face models locally in Node.js or the browser. • New Vercel AI Gateway provider for routing requests through Vercel's AI gateway. • New Cloudflare AI Gateway provider for routing requests through Cloudflare's AI gateway. • New AWS Bedrock Video provider supporting Nova Reel and Luma Ray 2 video generation. • New Azure AI Foundry Video provider for Sora video generation via Azure. • Adds native session endpoint support to the HTTP provider for stateful conversations. • Adds `apiBaseUrl`, `websocketUrl` override, and function call support to the xAI Voice provider. • Adds `collaboration_mode` support and integrated tracing to the OpenAI Codex provider. • New adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • New Telecom red team plugin suite for telecommunications AI systems. • New RAG Source Attribution red team plugin to test whether RAG systems properly attribute sources. • Red team scans now support multiple input variables for testing systems with complex input structures. • Adds early scan stop behavior when plugins fail to generate test cases. • Adds user-rated filter to show only manually rated eval results. • Adds provider config hover in eval results UI to view provider configuration details. • Adds Fork PR support and comment-triggered scans to Code Scanning. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Fri, 31 Jul 2026 15:56:21 GMT Promptfoo January 2026 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers. • Adds Transformers.js provider to run Hugging Face models locally in Node.js or browser without external API calls. • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests through managed gateways. • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation. • Adds native session endpoint support to the HTTP provider for stateful multi-turn conversations. • Adds `apiBaseUrl`, `websocketUrl` override, and function call support to the xAI Voice provider. • Adds `word-count` assertion type for validating response word counts. • Adds `__count` variable usable in derived metrics for computing averages. • Adds provider config hover in eval results UI to view provider configuration details inline. • Surfaces `metadata.sessionId` as a variable column in tables and exports. • Adds user-rated filter to show only manually rated results in the UI. • Adds `promptfoo logs` command for viewing log files directly. • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • Adds test-level prompts filter, providers filter, and per-test structured output configuration at the individual test case level. • Adds `$VAR` syntax support for environment variables in file paths. • Adds support for multiple `--env-file` flags to load multiple environment files. • Adds fork PR support and comment-triggered scan capability to Code Scanning. • Adds `--share` and `--no-share` auto-sharing flags to Model Audit. • Adds RAG Source Attribution plugin to test whether RAG systems properly attribute sources. • Adds multi-input variable support to red team scans for testing systems with complex input structures. • Adds `numTests` config to cap test cases generated per red team strategy. • Adds `-d`/`--description` flag to `redteam generate` commands. • Adds early scan stop behavior when plugins fail to generate test cases. • Adds automatic retries for transient 5xx errors in Enterprise. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Wed, 29 Jul 2026 15:45:41 GMT • Adds fork PR support for code scanning pull requests. • Adds comment-triggered scans to initiate code scans via PR comments. • Adds Telecom red team plugins for industry-specific testing of telecommunications AI systems. • Red team scans now support multiple input variables for testing systems with complex input structures. • Propagates `-j` flag to Python worker pools for improved Python concurrency. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Mon, 27 Jul 2026 16:09:54 GMT Promptfoo snapshot-20260727 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers. • Adds Transformers.js provider to run Hugging Face models locally in Node.js or browser. • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests. • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) for video generation. • Adds native session endpoint support to the HTTP provider for stateful conversations. • Adds xAI Voice provider overrides for `apiBaseUrl`, `websocketUrl`, and function call support. • Adds `word-count` assertion type for validating response word counts. • Adds `__count` variable for use in derived metrics to compute averages. • Adds provider config hover in eval results UI to view configuration details inline. • Surfaces `metadata.sessionId` as a variable column in tables and exports. • Adds user-rated filter to show only manually rated results in the UI. • Adds `promptfoo logs` command for viewing log files directly. • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • Adds test-level prompts filter to filter prompts at the individual test case level. • Adds providers filter at the test case level. • Adds per-test structured output configuration at the test case level. • Supports `$VAR` syntax for environment variables in file paths. • Adds support for multiple `--env-file` flags to load multiple environment files. • Adds fork PR support and comment-triggered scans to Code Scanning. • Adds `--share` and `--no-share` auto-sharing flags to Model Audit. • Adds Telecom industry-specific red team plugins for telecommunications AI systems. • Adds RAG Source Attribution plugin to test whether RAG systems properly attribute sources. • Adds support for multiple input variables in red team scans for complex input structures. • Adds `numTests` config to cap the number of test cases generated per red team strategy. • Adds `-d`/`--description` flag to `redteam generate` commands. • Adds early scan stop when plugins fail to generate test cases. • Adds automatic retries for transient 5xx errors. • Propagates `-j` concurrency flag to Python worker pools. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Fri, 24 Jul 2026 15:22:12 GMT • Adds xAI Voice provider overrides: apiBaseUrl, websocketUrl, and function call support. • Adds test-level prompts filter, providers filter, and per-test structured output configuration at the individual test case level. • Adds --share and --no-share flags to Model Audit for controlling cloud sharing. • Adds multi-input variable support to red team scans for testing systems with complex input structures. • Adds early scan stop when plugins fail to generate test cases. • Propagates -j flag to Python worker pools for Python concurrency in Enterprise. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Wed, 22 Jul 2026 15:27:34 GMT Promptfoo snapshot-20260722 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers. • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser. • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing. • Adds AWS Bedrock Video (Nova Reel and Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation. • Adds native session endpoint support to the HTTP provider for stateful conversations. • Adds `apiBaseUrl`, `websocketUrl` override, and function call support to the xAI Voice provider. • New `word-count` assertion type for validating response word counts. • Adds `__count` variable for use in derived metrics to compute averages. • New provider config hover in eval results to view provider configuration details. • Surfaces `metadata.sessionId` as a variable column in tables and exports. • New user-rated filter to show only manually rated eval results. • New `promptfoo logs` command for viewing log files directly. • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • Enables test-level prompts and providers filtering per individual test case. • Supports per-test structured output configuration at the test case level. • Supports `$VAR` syntax for environment variables in file paths. • Supports multiple `--env-file` flags to load multiple environment files. • Adds fork PR support and comment-triggered scans to Code Scanning. • Adds `--share` and `--no-share` auto-sharing flags to Model Audit. • New Telecom red team plugin for telecommunications AI systems. • New RAG Source Attribution plugin to test whether RAG systems properly attribute sources. • Red team scans now support multiple input variables for complex input structures. • Adds `numTests` config to cap test cases generated per strategy. • Adds `-d`/`--description` flag to `redteam generate` commands. • Adds automatic retries for transient 5xx errors in Enterprise. • The `-j` flag now propagates to Python worker pools for improved concurrency. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Sat, 18 Jul 2026 14:45:20 GMT Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Fri, 17 Jul 2026 15:07:22 GMT Promptfoo snapshot-20260717 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers. • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser. • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing. • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) for video generation. • Adds native session endpoint support on the HTTP provider for stateful conversations. • Adds xAI Voice provider overrides for `apiBaseUrl`, `websocketUrl`, and function call support. • Adds `word-count` assertion type for validating response word counts. • Adds `__count` variable for use in derived metrics to compute averages. • Adds provider config hover in eval results UI to view provider configuration details. • Surfaces `metadata.sessionId` as a variable column in eval tables and exports. • Adds user-rated filter to show only manually rated results in the UI. • Adds `promptfoo logs` command for viewing log files directly. • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • Adds test-level prompts filter to filter prompts at the individual test case level. • Adds providers filter at the test case level. • Adds per-test structured output configuration at the test case level. • Supports `$VAR` syntax for environment variables in file paths. • Supports multiple `--env-file` flags to load multiple environment files. • Adds fork PR support and comment-triggered scans for code scanning in CI/CD. • Adds `--share` and `--no-share` auto-sharing flags to Model Audit for controlling cloud sharing. • Adds Telecom industry-specific red team plugins for telecommunications AI systems. • Adds RAG Source Attribution plugin to test whether RAG systems properly attribute sources. • Red team scans now support multiple input variables for testing systems with complex input structures. • Adds `numTests` config to cap test cases generated per red team strategy. • Adds `-d`/`--description` flag to `redteam generate` commands. • Adds early scan stop when plugins fail to generate test cases. • Adds automatic retries for transient 5xx errors. • Propagates `-j` flag to Python worker pools for improved Python concurrency. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Wed, 15 Jul 2026 01:22:58 GMT • New video generation providers: AWS Bedrock Nova Reel/Luma Ray 2 and Azure AI Foundry Sora. • xAI Voice provider adds `apiBaseUrl`, `websocketUrl` override, and function call support. • Provider config hover: view provider configuration details inline on hover in eval results. • Test-level prompts filter and providers filter: scope prompts and providers to individual test cases. • Support for `$VAR` syntax in file paths for environment variable interpolation. • Multiple `--env-file` flags now supported to load several environment files simultaneously. • Code scanning now supports fork PR scanning and comment-triggered scan workflows. • Model Audit adds `--share` and `--no-share` flags to control cloud sharing. • New Telecom red team plugin suite for industry-specific telecommunications AI security testing. • Red team scans now support multiple input variables for testing complex input structures. • The `-j` concurrency flag now propagates to Python worker pools. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/ https://www.promptfoo.dev/docs/releases/ Mon, 13 Jul 2026 10:06:59 GMT Promptfoo January 2026 snapshot adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scans. • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser. • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing. • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation. • Adds native session endpoint support to the HTTP provider for stateful multi-turn conversations. • Adds apiBaseUrl, websocketUrl override, and function call support to the xAI Voice provider. • Adds word-count assertion type for validating response word counts. • Adds __count variable for use in derived metrics to compute averages. • Adds provider config hover in eval results to view provider configuration details inline. • Surfaces metadata.sessionId as a variable column in tables and exports. • Adds user-rated filter to show only manually rated results. • Adds promptfoo logs command for viewing log files directly. • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers. • Adds test-level prompts filter to filter prompts at the individual test case level. • Adds providers filter at the test case level. • Adds per-test structured output configuration at the test case level. • Supports $VAR syntax for environment variables in file paths. • Supports multiple --env-file flags to load multiple environment files. • Adds fork PR support and comment-triggered scans to code scanning. • Adds --share and --no-share flags to Model Audit for controlling cloud sharing. • Adds Telecom industry-specific red team plugins for telecommunications AI systems. • Adds RAG Source Attribution plugin to test whether RAG systems properly attribute sources. • Enables multi-input variable support in red team scans for testing complex input structures. • Adds numTests config to cap test cases generated per strategy. • Adds -d/--description flag to redteam generate commands. • Adds early scan stop when plugins fail to generate test cases. • Adds automatic retries for transient 5xx errors in Enterprise. • Propagates -j flag to Python worker pools for improved Python concurrency. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#january-2026 https://www.promptfoo.dev/docs/releases/#january-2026 Thu, 01 Jan 2026 00:00:00 GMT Promptfoo January 2026 adds local inference, adaptive rate limiting, new providers, and per-test filtering. • Adds `Transformers.js` provider for running models locally in Node.js or browser • Adds `Vercel AI Gateway` and `Cloudflare AI Gateway` as new eval providers • Adds `AWS Bedrock Video` and `Azure AI Foundry Video` (Sora) providers for video generation • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider headers • Adds `--share` and `--no-share` flags to control cloud sharing in Model Audit • Adds `Telecom` red team plugins for telecommunications AI systems • Adds `word-count` assertion type for validating response word counts • Uses `$VAR` syntax to insert environment variables into file paths • Adds support for multiple `--env-file` flags to load multiple environment files • Adds `-d/--description` flag to redteam generate commands for adding descriptions • Adds `numTests` config option to cap the number of test cases generated per strategy Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#december-2025 https://www.promptfoo.dev/docs/releases/#december-2025 Mon, 01 Dec 2025 00:00:00 GMT New video and voice providers, OTLP tracing, `--extension` flag, and GPT-5.2 support added to Promptfoo CLI. • Adds OpenAI Sora and Google Veo as new video generation providers • Adds xAI Voice Agent and ElevenLabs as new audio/voice providers • Adds support for the `gpt-5.2` model in the GPT-5 series • Adds native OTLP tracing with GenAI semantic conventions via Protobuf support • Adds `--extension` CLI flag to load extensions via command line after initialization • Adds ability to generate shareable URLs from the Node.js evaluate() API • Adds Wildcard prompt filters allowing wildcards when filtering prompts in the UI • Displays total evaluation duration in the web UI during evaluations • Shows complete grading prompts directly in the UI for inspection • Adds OAuth authentication with proactive token refresh via MCP OAuth provider • Adds interactive team selection during the login flow • Adds OpenTelemetry tracing support in Hydra and IterativeMeta components • Adds Multi-modal layer strategy to chain audio and image attacks in test strategies • Updates Browser provider to support multi-turn session persistence • Adds configurable `numberOfResults` for Bedrock Knowledge Base queries • Adds F1 score metric to evaluate tool calling accuracy with Enterprise Assertions Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#november-2025 https://www.promptfoo.dev/docs/releases/#november-2025 Sat, 01 Nov 2025 00:00:00 GMT Promptfoo November release adds Hydra multi-turn strategy, code scanning, XLSX config, and SharePoint integration. • Adds the `hydra` multi-turn red teaming strategy that adapts dynamically to target responses • Adds code scanning to analyze codebases for potential AI security issues • Introduces Microsoft SharePoint integration to load datasets directly from SharePoint • Enables loading test cases from Excel files using `XLSX/XLS` support in configuration • Adds the OpenAI ChatKit provider for conversational AI interactions • Supports Claude Opus 4.5, GPT-5.1, Gemini 3 Pro, and Groq reasoning models • Adds a web search assertion to assert on web search results • Introduces `dot product` and `euclidean distance` as new similarity metrics for embeddings • Adds FERPA and Ecommerce plugins for education privacy and e-commerce vulnerability testing • Enables loading tool definitions from Python or JavaScript files via configuration Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#october-2025 https://www.promptfoo.dev/docs/releases/#october-2025 Wed, 01 Oct 2025 00:00:00 GMT Promptfoo Oct 2025 adds jailbreak red-teaming, remediation reports, SARIF export, new providers, and MCP support. • `jailbreak :meta` strategy uses multiple AI agents to generate attacks, offering up to 50% more effectiveness in single-shot mode. • Remediation reports with executive summary, prioritized action items, and guardrail recommendations accessible via "View Remediation Report" from any vulnerability report. • SARIF export for vulnerability reports and expanded CSV exports including Strategy ID, Plugin ID, and Session IDs. • New Eval Providers: OpenAI Agents SDK, Claude Agent SDK, Azure AI Foundry Agents, Ruby scripts, Snowflake Cortex, and Slack bots. • `metric filtering with operators` allows filtering eval results using `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, and `is_defined` operators. • New Model Support for Claude Haiku 4.5. • Provider Updates: Python provider gets persistent worker pools (10-100x performance improvement); WebSocket provider streams multiple responses; Ollama adds function calling and tool support. • `MCP server configuration` enables Model Context Protocol server setup. • New Plugins: Wordplay, COPPA, and data protection preset for privacy testing. • Postman/cURL import auto-populates HTTP target connection details from curl commands or Postman files. • Scan template enhancements: display estimated probe count/runtime and preview generated test cases when selecting plugins. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#september-2025 https://www.promptfoo.dev/docs/releases/#september-2025 Mon, 01 Sep 2025 00:00:00 GMT Promptfoo Sept 2025: reusable policy libraries, risk scoring, new AI models/providers, eval pause/resume, and enterprise compliance features. • Adds `pause`/`resume` workflow with `Ctrl+C` to interrupt and `promptfoo eval --resume` to continue long-running evaluations • Introduces 'Risk Scoring' in red team reports, providing overall risk scores (0-10), risk by category, trends, and visual heatmaps for remediation prioritization • Enables reusable custom policy libraries via CSV upload and reference in red team config • Adds new model support including Claude 4.5 Sonnet, GPT-5 series, Gemini 2.5 Flash, OpenAI Realtime API (audio I/O), web_fetch/web_search tools, and Nscale image generation provider • Enhances CSV exports to include latency, grader reason, and grader comment fields • Introduces `promptfoo export logs` command creating tar.gz archives for debugging • Adds auto-enabled cloud sharing when connected to Promptfoo Cloud Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#august-2025 https://www.promptfoo.dev/docs/releases/#august-2025 Fri, 01 Aug 2025 00:00:00 GMT Adds GPT-5 and Claude Opus 4.1 support, cloud model audit sharing, mTLS certificate storage, and MCP agent testing examples. • `model` provider now supports OpenAI's `gpt-5`, Anthropic's `claude-sonnet-4-20250514`, and xAI's `grok-code-fast` for evals and red teaming • Adds cloud-based model audit sharing to persist scan history, enable team collaboration via centralized storage, and manage recent scan paths • Adds `client_certificate` store capability for mTLS authentication and signature-based upload support in provider configs • Adds credential sanitization to prevent exposure of secrets in debug logs during evaluation runs • Adds auto-fill capabilities for HTTP provider setup to reduce configuration time • Improves HuggingFace dataset fetching speed and reliability with better retry suggestions • Adds `medical_off_label_use` plugin to detect inappropriate pharmaceutical recommendations • Adds `unverifiable_claims` plugin to test susceptibility to fabricated claims • Adds MCP agent testing example for red teaming tool call results Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#july-2025 https://www.promptfoo.dev/docs/releases/#july-2025 Tue, 01 Jul 2025 00:00:00 GMT Promptfoo July 2025 adds Docker Model Runner, MCP support, context transforms in assertions, and agentic red team strategies. • Adds `docker` model runner for isolated model execution via Docker containers • Adds `mcp` provider to connect to Model Context Protocol servers • Adds Google Imagen model support for multimodal testing • Adds AIMLAPI provider for unified access to various AI models • Adds Grok-4 and OpenAI Deep Research Models (o3-deep-research, o4-mini-deep-research) as new model providers • Enhances Azure provider with `system_prompt` support in configuration • Adds LiteLLM Embeddings provider for similarity testing and semantic search • Adds Google Vision provider for image understanding in multimodal evaluations • Enhances HTTP Provider to support JKS and PFX client certificates • Adds Browser Provider connecting to Chrome via Chrome DevTools Protocol (CDP) • Adds `context` transforms to extract data from provider responses for use in assertions • Adds `finish-reason` as an option in assertions to validate response termination • Adds tracing assertion fields: `trace-span-count`, `trace-span-duration`, and `trace-error-spans` • Allows `defaultTest` to load test cases from external files for external test configuration • Enhances search with metadata filtering using equals, contains, and not contains operators in Web UI • Adds first-class zooming support in the eval results page via Web UI • Adds re-grade functionality for existing red team scans after changing grading rules • Adds Identity Provider integration to map teams and roles for permission assignment • Adds MCP Proxy feature for access control and traffic monitoring of MCP servers • Adds `custom` agentic multi-turn red team strategy using natural language instructions • Adds `mischievous_user` agentic multi-turn red team strategy simulating playful user behavior Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo https://www.promptfoo.dev/docs/releases/#june-2025 https://www.promptfoo.dev/docs/releases/#june-2025 Sun, 01 Jun 2025 00:00:00 GMT Adds OpenTelemetry tracing, `PROMPTFOO_MAX_EVAL_TIME_MS` env var, ModelAudit `/model-audit` UI, and new red team plugins. • Adds OpenTelemetry tracing support to visualize execution flow and debug performance bottlenecks in complex RAG pipelines. • Adds `PROMPTFOO_MAX_EVAL_TIME_MS` environment variable to prevent runaway evaluations from consuming excessive resources. • Adds ModelAudit UI accessible at `/model-audit` when running `promptfoo view`, featuring drag-and-drop file selection and live scanning progress. • Adds support for scanning SafeTensors format and HuggingFace URLs directly in ModelAudit. • Adds System Prompt Override plugin to test LLM deployments for system instruction manipulation vulnerabilities. • Adds Intent plugin support for JSON file uploads with nested arrays to define multi-step attack sequences. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/promptfoo