Promptfoo

No entries match these filters. Clear filters

1. updated seen Sep 7, 2026 · issue 067

   Promptfoo adds Docker Model Runner, MCPMCPModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server support, Google Imagen, Grok-4, and JKSJKSJava KeyStore, a file format maintained by Oracle for storing cryptographic keys and certificates; tools use it to manage TLS identities without depending on the host OS certificate store./PFXPFXA binary file format (PKCS#12) that bundles a private key and its certificate chain into a single password-protected file, letting cyber tools load credentials without managing separate key and cert files. certificate support for HTTP providers.

   ⤢
   • Adds HTTP provider support for JKS and PFX client certificates, enabling mTLSmTLSMutual TLS, an encrypted connection where both client and server present certificates to prove their identity. It helps services verify who they are communicating with. connections to secured endpoints.
   • Adds Docker Model Runner provider to run models in isolated Docker containers for better security and reproducibility.
   • Adds MCP (Model Context ProtocolModel Context ProtocolModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface.) provider to connect to MCP servers, including a red-team agent testing example with tool call results.
   • Adds Google Imagen provider for image generation in multimodalmultimodalAble to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. testing scenarios.
   • Adds AIMLAPI provider for accessing various AI models through a unified interface.

   +5 more show less

   • Adds Grok-4 model support via xAI for advanced reasoning evaluations.
   • Adds o3-deep-research and o4-mini-deep-research OpenAI Deep Research models for complex problem-solving evaluations.
   • Adds LiteLLM EmbeddingsEmbeddingsLists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. support for similarity testing and semantic searchsemantic searchSearching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. evaluations.
   • Adds Google Vision provider for image understanding in multimodal evaluations.
   • Enhances Azure provider with system prompt support for better control over model behavior.
   • Adds Docker Model Runner provider to run models in isolated Docker containers for better security and reproducibility.
   • Adds MCP (Model Context ProtocolModel Context ProtocolModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface.) provider to connect to MCP servers, including a red-team agent testing example with tool call results.
   • Adds Google Imagen provider for image generation in multimodalmultimodalAble to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. testing scenarios.
   • Adds AIMLAPI provider for accessing various AI models through a unified interface.
   • Adds Grok-4 model support via xAI for advanced reasoning evaluations.

   +3 more show less

   • Adds o3-deep-research and o4-mini-deep-research OpenAI Deep Research models for complex problem-solving evaluations.
   • Adds LiteLLM EmbeddingsEmbeddingsLists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. support for similarity testing and semantic searchsemantic searchSearching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs. evaluations.
   • Adds Google Vision provider for image understanding in multimodal evaluations.

   2 more in Everything

2. updated seen Aug 19, 2026 · issue 050

   Promptfoo adds an Unverifiable Claims Detection plugin to test AI systems for susceptibility to unverifiable assertions.

   ⤢
3. updated seen Aug 5, 2026 · issue 036
   ⤢
   └──▷ TRY IT
   View Promptfoo runtime logs directly from the CLI without navigating to log files manually.
   copy

   $ promptfoo logs

   Add a description to a red team generation run for tracking scans across environments.
   copy

   $ promptfoo redteam generate -d "prod-telecom-scan-2026-08"

   Load multiple environment files so staging and shared secrets are both available during an eval.
   copy

   $ promptfoo eval --env-file .env.shared --env-file .env.staging

   Cap test case generation per strategy and load config with env variable path substitution to keep CI scans bounded.
   yaml copy

   # promptfooconfig.yaml snippet
   redteam:
     strategies:
       - id: jailbreak
         numTests: 20
   prompts:
     - file://$PROMPTS_DIR/system.txt

   • Adds test-level prompts filter and providers filter to scope test cases more precisely.
   • New Vercel AI Gateway and Cloudflare AI Gateway providers for request routing.
   • OpenAI Codex provider adds collaboration_mode support and integrated tracing.
   • Code scanning adds support for fork pull requests and comment-triggered scans.
4. updated seen Aug 4, 2026 · issue 035

   Promptfoo snapshot-20260804 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scanning.

   ⤢
   └──▷ TRY IT
   View recent promptfoo logs directly from the CLI without navigating log directories.
   copy

   $ promptfoo logs

   Load multiple environment files (e.g. base secrets plus environment-specific overrides) in a single eval run.
   copy

   $ promptfoo eval --env-file .env.base --env-file .env.staging

   • Adds promptfoo logs command for viewing log files directly.
   • Adds --share and --no-share flags to Model Audit for controlling cloud sharing.
   • Adds -d/--description flag to redteam generate commands.
   • Adds -j flag propagation to Python worker pools for concurrency control.
   • Adds numTests config key to cap test cases generated per red team strategy.

   +23 more show less

   • Adds $VAR syntax support for environment variables in file paths.
   • Supports multiple --env-file flags to load multiple environment files.
   • Adds word-count assertion type for validating response word counts.
   • Adds __count variable for use in derived metrics when computing averages.
   • Surfaces metadata.sessionId as a variable column in eval tables and exports.
   • Adds test-level prompts filter and providers filter for per-test-case scoping in configuration.
   • Supports per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the test case level.
   • New Transformers.js provider — run Hugging Face models locally in Node.js or the browser.
   • New Vercel AI Gateway provider for routing requests through Vercel's AI gateway.
   • New Cloudflare AI Gateway provider for routing requests through Cloudflare's AI gateway.
   • New AWS Bedrock Video provider supporting Nova Reel and Luma Ray 2 video generation.
   • New Azure AI Foundry Video provider for Sora video generation via Azure.
   • Adds native session endpoint support to the HTTP provider for stateful conversations.
   • Adds apiBaseUrl, websocketUrl override, and function call support to the xAI Voice provider.
   • Adds collaboration_mode support and integrated tracing to the OpenAI Codex provider.
   • New adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • New Telecom red team plugin suite for telecommunications AI systems.
   • New RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution red team plugin to test whether RAG systems properly attribute sources.
   • Red team scans now support multiple input variables for testing systems with complex input structures.
   • Adds early scan stop behavior when plugins fail to generate test cases.
   • Adds user-rated filter to show only manually rated eval results.
   • Adds provider config hover in eval results UI to view provider configuration details.
   • Adds Fork PR support and comment-triggered scans to Code Scanning.

   28 details in Everything

5. updated seen Jul 31, 2026 · issue 031

   Promptfoo January 2026 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers.

   └──▷ TRY IT
   View Promptfoo logs directly from the CLI after a long evaluation run to inspect rate limiting or provider errors.
   copy

   $ promptfoo logs

   Load multiple environment files (e.g., base config + environment-specific secrets) when running an evaluation.
   copy

   $ promptfoo eval --env-file .env.base --env-file .env.prod

   • Adds Transformers.js provider to run Hugging Face models locally in Node.js or browser without external API calls.
   • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests through managed gateways.
   • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation.
   • Adds native session endpoint support to the HTTP provider for stateful multi-turn conversations.
   • Adds apiBaseUrl, websocketUrl override, and function call support to the xAI Voice provider.

   +18 more show less

   • Adds word-count assertion type for validating response word counts.
   • Adds __count variable usable in derived metrics for computing averages.
   • Adds provider config hover in eval results UI to view provider configuration details inline.
   • Surfaces metadata.sessionId as a variable column in tables and exports.
   • Adds user-rated filter to show only manually rated results in the UI.
   • Adds promptfoo logs command for viewing log files directly.
   • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • Adds test-level prompts filter, providers filter, and per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the individual test case level.
   • Adds $VAR syntax support for environment variables in file paths.
   • Adds support for multiple --env-file flags to load multiple environment files.
   • Adds fork PR support and comment-triggered scan capability to Code Scanning.
   • Adds --share and --no-share auto-sharing flags to Model Audit.
   • Adds RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution plugin to test whether RAG systems properly attribute sources.
   • Adds multi-input variable support to red team scans for testing systems with complex input structures.
   • Adds numTests config to cap test cases generated per red team strategy.
   • Adds -d/--description flag to redteam generate commands.
   • Adds early scan stop behavior when plugins fail to generate test cases.
   • Adds automatic retries for transient 5xx errors in Enterprise.

   23 details in Everything

6. updated seen Jul 29, 2026 · issue 029
   └──▷ TRY IT
   View Promptfoo log output directly from the CLI for debugging provider or scan issues.
   copy

   $ promptfoo logs

   Generate a red team scan with a description and cap test cases per strategy, targeting a telecom AI system.
   copy

   $ promptfoo redteam generate -d "Telecom voice agent test" --plugins telecom --config redteam.yaml

   • Adds fork PR support for code scanning pull requests.
   • Adds comment-triggered scans to initiate code scans via PR comments.
   • Adds Telecom red team plugins for industry-specific testing of telecommunications AI systems.
   • Red team scans now support multiple input variables for testing systems with complex input structures.
   • Propagates -j flag to Python worker pools for improved Python concurrency.
7. updated seen Jul 27, 2026 · issue 027

   Promptfoo snapshot-20260727 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers.

   └──▷ TRY IT
   Load multiple environment files (e.g., base secrets plus environment-specific overrides) in a single eval run.
   copy

   $ promptfoo eval --env-file .env.base --env-file .env.staging

   View Promptfoo service logs directly from the CLI instead of hunting through log directories.
   copy

   $ promptfoo logs

   • Adds Transformers.js provider to run Hugging Face models locally in Node.js or browser.
   • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests.
   • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) for video generation.
   • Adds native session endpoint support to the HTTP provider for stateful conversations.
   • Adds xAI Voice provider overrides for apiBaseUrl, websocketUrl, and function call support.

   +22 more show less

   • Adds word-count assertion type for validating response word counts.
   • Adds __count variable for use in derived metrics to compute averages.
   • Adds provider config hover in eval results UI to view configuration details inline.
   • Surfaces metadata.sessionId as a variable column in tables and exports.
   • Adds user-rated filter to show only manually rated results in the UI.
   • Adds promptfoo logs command for viewing log files directly.
   • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • Adds test-level prompts filter to filter prompts at the individual test case level.
   • Adds providers filter at the test case level.
   • Adds per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the test case level.
   • Supports $VAR syntax for environment variables in file paths.
   • Adds support for multiple --env-file flags to load multiple environment files.
   • Adds fork PR support and comment-triggered scans to Code Scanning.
   • Adds --share and --no-share auto-sharing flags to Model Audit.
   • Adds Telecom industry-specific red team plugins for telecommunications AI systems.
   • Adds RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution plugin to test whether RAG systems properly attribute sources.
   • Adds support for multiple input variables in red team scans for complex input structures.
   • Adds numTests config to cap the number of test cases generated per red team strategy.
   • Adds -d/--description flag to redteam generate commands.
   • Adds early scan stop when plugins fail to generate test cases.
   • Adds automatic retries for transient 5xx errors.
   • Propagates -j concurrency flag to Python worker pools.

   27 details in Everything

8. updated seen Jul 24, 2026 · issue 024
   └──▷ TRY IT
   View promptfoo log output directly from the CLI for debugging rate limiting or provider issues.
   copy

   $ promptfoo logs

   • Adds xAI Voice provider overrides: apiBaseUrl, websocketUrl, and function call support.
   • Adds test-level prompts filter, providers filter, and per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the individual test case level.
   • Adds --share and --no-share flags to Model Audit for controlling cloud sharing.
   • Adds multi-input variable support to red team scans for testing systems with complex input structures.
   • Adds early scan stop when plugins fail to generate test cases.

   +1 more show less

   • Propagates -j flag to Python worker pools for Python concurrency in Enterprise.
9. updated seen Jul 22, 2026 · issue 022

   Promptfoo snapshot-20260722 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers.

   └──▷ TRY IT
   View Promptfoo service logs directly from the CLI without navigating the filesystem.
   copy

   $ promptfoo logs

   Load multiple environment files (e.g., shared secrets plus environment-specific overrides) in a single eval run.
   copy

   $ promptfoo eval --env-file .env.shared --env-file .env.staging

   Add a description to a red team generation run for traceability across scans.
   copy

   $ promptfoo redteam generate -d "Sprint 42 telecom compliance scan"

   • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser.
   • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing.
   • Adds AWS Bedrock Video (Nova Reel and Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation.
   • Adds native session endpoint support to the HTTP provider for stateful conversations.
   • Adds apiBaseUrl, websocketUrl override, and function call support to the xAI Voice provider.

   +20 more show less

   • New word-count assertion type for validating response word counts.
   • Adds __count variable for use in derived metrics to compute averages.
   • New provider config hover in eval results to view provider configuration details.
   • Surfaces metadata.sessionId as a variable column in tables and exports.
   • New user-rated filter to show only manually rated eval results.
   • New promptfoo logs command for viewing log files directly.
   • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • Enables test-level prompts and providers filtering per individual test case.
   • Supports per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the test case level.
   • Supports $VAR syntax for environment variables in file paths.
   • Supports multiple --env-file flags to load multiple environment files.
   • Adds fork PR support and comment-triggered scans to Code Scanning.
   • Adds --share and --no-share auto-sharing flags to Model Audit.
   • New Telecom red team plugin for telecommunications AI systems.
   • New RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution plugin to test whether RAG systems properly attribute sources.
   • Red team scans now support multiple input variables for complex input structures.
   • Adds numTests config to cap test cases generated per strategy.
   • Adds -d/--description flag to redteam generate commands.
   • Adds automatic retries for transient 5xx errors in Enterprise.
   • The -j flag now propagates to Python worker pools for improved concurrency.

   25 details in Everything

10. updated seen Jul 18, 2026 · issue 019
    └──▷ TRY IT
    Load multiple environment files in a single eval run — useful when secrets are split across team and personal dotenv files.
    copy

    $ promptfoo eval --env-file .env.shared --env-file .env.local

    View recent Promptfoo log output directly from the CLI for debugging provider or eval issues.
    copy

    $ promptfoo logs

    Add a description to a red team generation job so scan results are easy to identify in the UI.
    copy

    $ promptfoo redteam generate -d "Telecom Q1 regression scan"

11. updated seen Jul 17, 2026 · issue 018

    Promptfoo snapshot-20260717 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and video generation providers.

    └──▷ TRY IT
    Load multiple environment files (e.g., base secrets plus environment-specific overrides) in a single eval run.
    copy

    $ promptfoo eval --env-file .env.base --env-file .env.staging --prompts prompts.yaml --tests tests.yaml

    Cap test case generation per strategy to keep red team scans fast during development.
    copy

    $ promptfoo redteam generate -d 'telecom-smoke-test' --config redteam.yaml

    • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser.
    • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing.
    • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) for video generation.
    • Adds native session endpoint support on the HTTP provider for stateful conversations.
    • Adds xAI Voice provider overrides for apiBaseUrl, websocketUrl, and function call support.

    +22 more show less

    • Adds word-count assertion type for validating response word counts.
    • Adds __count variable for use in derived metrics to compute averages.
    • Adds provider config hover in eval results UI to view provider configuration details.
    • Surfaces metadata.sessionId as a variable column in eval tables and exports.
    • Adds user-rated filter to show only manually rated results in the UI.
    • Adds promptfoo logs command for viewing log files directly.
    • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
    • Adds test-level prompts filter to filter prompts at the individual test case level.
    • Adds providers filter at the test case level.
    • Adds per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the test case level.
    • Supports $VAR syntax for environment variables in file paths.
    • Supports multiple --env-file flags to load multiple environment files.
    • Adds fork PR support and comment-triggered scans for code scanning in CI/CD.
    • Adds --share and --no-share auto-sharing flags to Model Audit for controlling cloud sharing.
    • Adds Telecom industry-specific red team plugins for telecommunications AI systems.
    • Adds RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution plugin to test whether RAG systems properly attribute sources.
    • Red team scans now support multiple input variables for testing systems with complex input structures.
    • Adds numTests config to cap test cases generated per red team strategy.
    • Adds -d/--description flag to redteam generate commands.
    • Adds early scan stop when plugins fail to generate test cases.
    • Adds automatic retries for transient 5xx errors.
    • Propagates -j flag to Python worker pools for improved Python concurrency.

    27 details in Everything

12. updated seen Jul 15, 2026 · issue 015
    └──▷ TRY IT
    Run a red team scan with a description and capped test cases per strategy to keep CI runtimes predictable.
    copy

    $ promptfoo redteam generate -d 'PR #482 telecom safety check' --numTests 10

    Load secrets from multiple environment files (e.g., shared team vars plus local overrides) in a single eval run.
    copy

    $ promptfoo eval --env-file .env.shared --env-file .env.local

    Inspect recent Promptfoo service logs without leaving the terminal.
    copy

    $ promptfoo logs

    • New video generation providers: AWS Bedrock Nova Reel/Luma Ray 2 and Azure AI Foundry Sora.
    • xAI Voice provider adds apiBaseUrl, websocketUrl override, and function call support.
    • Provider config hover: view provider configuration details inline on hover in eval results.
    • Test-level prompts filter and providers filter: scope prompts and providers to individual test cases.
    • Support for $VAR syntax in file paths for environment variable interpolation.

    +6 more show less

    • Multiple --env-file flags now supported to load several environment files simultaneously.
    • Code scanning now supports fork PR scanning and comment-triggered scan workflows.
    • Model Audit adds --share and --no-share flags to control cloud sharing.
    • New Telecom red team plugin suite for industry-specific telecommunications AI security testing.
    • Red team scans now support multiple input variables for testing complex input structures.
    • The -j concurrency flag now propagates to Python worker pools.
13. updated seen Jul 13, 2026 · issue 014

    Promptfoo January 2026 snapshot adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scans.

    └──▷ TRY IT
    Run a red team scan against a system with multiple input variables to cover complex prompt structures.
    copy

    $ promptfoo redteam scan --env-file .env.prod --env-file .env.secrets

    Add a description to a generated red team test suite for traceability in CI.
    copy

    $ promptfoo redteam generate -d "Q3 telecom safety audit" --plugins telecom

    View recent Promptfoo log output to diagnose provider or scan issues.
    copy

    $ promptfoo logs

    • Adds Transformers.js provider to run Hugging Face models locally in Node.js or the browser.
    • Adds Vercel AI Gateway and Cloudflare AI Gateway providers for request routing.
    • Adds AWS Bedrock Video (Nova Reel, Luma Ray 2) and Azure AI Foundry Video (Sora) providers for video generation.
    • Adds native session endpoint support to the HTTP provider for stateful multi-turn conversations.
    • Adds apiBaseUrl, websocketUrl override, and function call support to the xAI Voice provider.

    +22 more show less

    • Adds word-count assertion type for validating response word counts.
    • Adds __count variable for use in derived metrics to compute averages.
    • Adds provider config hover in eval results to view provider configuration details inline.
    • Surfaces metadata.sessionId as a variable column in tables and exports.
    • Adds user-rated filter to show only manually rated results.
    • Adds promptfoo logs command for viewing log files directly.
    • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
    • Adds test-level prompts filter to filter prompts at the individual test case level.
    • Adds providers filter at the test case level.
    • Adds per-test structured outputstructured outputForcing a model's response to match a given format or JSON schema, so software can parse it reliably instead of hoping the text is well formed. configuration at the test case level.
    • Supports $VAR syntax for environment variables in file paths.
    • Supports multiple --env-file flags to load multiple environment files.
    • Adds fork PR support and comment-triggered scans to code scanning.
    • Adds --share and --no-share flags to Model Audit for controlling cloud sharing.
    • Adds Telecom industry-specific red team plugins for telecommunications AI systems.
    • Adds RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. Source Attribution plugin to test whether RAG systems properly attribute sources.
    • Enables multi-input variable support in red team scans for testing complex input structures.
    • Adds numTests config to cap test cases generated per strategy.
    • Adds -d/--description flag to redteam generate commands.
    • Adds early scan stop when plugins fail to generate test cases.
    • Adds automatic retries for transient 5xx errors in Enterprise.
    • Propagates -j flag to Python worker pools for improved Python concurrency.

    27 details in Everything

14. January 2026 Release Highlights Jan 1, 2026 · release notes

    Promptfoo January 2026 adds local inference, adaptive rate limiting, new providers, and per-test filtering.

    └──▷ USE IT
    Load environment variables from multiple files into the evaluation path context.
    shell copy

    promptfoo eval --env-file .env.local --env-file .env.prod

    Cap the number of generated test cases per strategy during redteam generation.
    shell copy

    promptfoo redteam generate --numTests 10 --description 'Basic safety check'

    • Adds Transformers.js provider for running models locally in Node.js or browser
    • Adds Vercel AI Gateway and Cloudflare AI Gateway as new eval providers
    • Adds AWS Bedrock Video and Azure AI Foundry Video (Sora) providers for video generation
    • Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider headers
    • Adds --share and --no-share flags to control cloud sharing in Model Audit

    +6 more show less

    • Adds Telecom red team plugins for telecommunications AI systems
    • Adds word-count assertion type for validating response word counts
    • Uses $VAR syntax to insert environment variables into file paths
    • Adds support for multiple --env-file flags to load multiple environment files
    • Adds -d/--description flag to redteam generate commands for adding descriptions
    • Adds numTests config option to cap the number of test cases generated per strategy

    11 details in Everything

15. December 2025 Release Highlights Dec 1, 2025 · release notes

    New video and voice providers, OTLPOTLPOpenTelemetry Protocol, a standard way to send traces, metrics and logs between applications, collectors and monitoring systems. It lets telemetry move across compatible tools. tracing, --extension flag, and GPT-5.2 support added to Promptfoo CLI.

    • Adds OpenAI Sora and Google Veo as new video generation providers
    • Adds xAI Voice Agent and ElevenLabs as new audio/voice providers
    • Adds support for the gpt-5.2 model in the GPT-5 series
    • Adds native OTLP tracing with GenAI semantic conventions via Protobuf support
    • Adds --extension CLI flag to load extensions via command line after initialization

    +11 more show less

    • Adds ability to generate shareable URLs from the Node.js evaluate() API
    • Adds Wildcard prompt filters allowing wildcards when filtering prompts in the UI
    • Displays total evaluation duration in the web UI during evaluations
    • Shows complete grading prompts directly in the UI for inspection
    • Adds OAuth authentication with proactive token refresh via MCP OAuthMCP OAuthAn authentication layer for the Model Context Protocol that lets MCP clients obtain delegated access tokens so cyber tools can call MCP servers on a user's behalf without handling raw credentials. provider
    • Adds interactive team selection during the login flow
    • Adds OpenTelemetryOpenTelemetryAn open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. tracing support in Hydra and IterativeMeta components
    • Adds Multi-modalMulti-modalAble to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. layer strategy to chain audio and image attacks in test strategies
    • Updates Browser provider to support multi-turn session persistence
    • Adds configurable numberOfResults for Bedrock Knowledge Base queries
    • Adds F1 score metric to evaluate tool callingtool callingA model's ability to answer with a structured request to run a named function or API, which the application executes and returns. It is how agents take actions. accuracy with Enterprise Assertions

    16 details in Everything

16. November 2025 Release Highlights Nov 1, 2025 · release notes

    Promptfoo November release adds Hydra multi-turn strategy, code scanning, XLSX config, and SharePoint integration.

    • Adds the hydra multi-turn red teaming strategy that adapts dynamically to target responses
    • Adds code scanning to analyze codebases for potential AI security issues
    • Introduces Microsoft SharePoint integration to load datasets directly from SharePoint
    • Enables loading test cases from Excel files using XLSX/XLS support in configuration
    • Adds the OpenAI ChatKit provider for conversational AI interactions

    +5 more show less

    • Supports Claude Opus 4.5, GPT-5.1, Gemini 3 Pro, and Groq reasoning models
    • Adds a web search assertion to assert on web search results
    • Introduces dot product and euclidean distance as new similarity metrics for embeddingsembeddingsLists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG.
    • Adds FERPA and Ecommerce plugins for education privacy and e-commerce vulnerability testing
    • Enables loading tool definitions from Python or JavaScript files via configuration

    10 details in Everything

17. October 2025 Release Highlights Oct 1, 2025 · release notes

    Promptfoo Oct 2025 adds jailbreak red-teaming, remediation reports, SARIFSARIFStatic Analysis Results Interchange Format, a standard for sharing code-analysis findings, including locations and rule details. It lets scanners report results to a common review interface. export, new providers, and MCPMCPModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. support.

    • jailbreak :meta strategy uses multiple AI agents to generate attacks, offering up to 50% more effectiveness in single-shot mode.
    • Remediation reports with executive summary, prioritized action items, and guardrail recommendations accessible via View Remediation Report from any vulnerability report.
    • SARIF export for vulnerability reports and expanded CSV exports including Strategy ID, Plugin ID, and Session IDs.
    • New Eval Providers: OpenAI Agents SDK, Claude Agent SDK, Azure AI Foundry Agents, Ruby scripts, Snowflake Cortex, and Slack bots.
    • metric filtering with operators allows filtering eval results using eq, neq, gt, gte, lt, lte, and is_defined operators.

    +6 more show less

    • New Model Support for Claude Haiku 4.5.
    • Provider Updates: Python provider gets persistent worker pools (10-100x performance improvement); WebSocketWebSocketA protocol that upgrades an HTTP connection into a persistent, full-duplex channel, letting a server push data to a client without repeated polling. Cyber tools use it for real-time event streaming and live command output. provider streams multiple responses; Ollama adds function callingfunction callingA model's ability to answer with a structured request to run a named function or API, which the application executes and returns. It is how agents take actions. and tool support.
    • MCP server configuration enables Model Context ProtocolModel Context ProtocolModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. server setup.
    • New Plugins: Wordplay, COPPA, and data protection preset for privacy testing.
    • Postman/cURL import auto-populates HTTP target connection details from curl commands or Postman files.
    • Scan template enhancements: display estimated probe count/runtime and preview generated test cases when selecting plugins.

    11 details in Everything

18. September 2025 Release Highlights Sep 1, 2025 · release notes

    Promptfoo Sept 2025: reusable policy libraries, risk scoring, new AI models/providers, eval pause/resume, and enterprise compliance features.

    • Adds pause/resume workflow with Ctrl+C to interrupt and promptfoo eval --resume to continue long-running evaluations
    • Introduces Risk Scoring in red team reports, providing overall risk scores (0-10), risk by category, trends, and visual heatmaps for remediation prioritization
    • Enables reusable custom policy libraries via CSV upload and reference in red team config
    • Adds new model support including Claude 4.5 Sonnet, GPT-5 series, Gemini 2.5 Flash, OpenAI Realtime API (audio I/O), web_fetch/web_search tools, and Nscale image generation provider
    • Enhances CSV exports to include latency, grader reason, and grader comment fields

    +2 more show less

    • Introduces promptfoo export logs command creating tar.gz archives for debugging
    • Adds auto-enabled cloud sharing when connected to Promptfoo Cloud

    7 details in Everything

19. August 2025 Release Highlights Aug 1, 2025 · release notes

    Adds GPT-5 and Claude Opus 4.1 support, cloud model audit sharing, mTLSmTLSMutual TLS, an encrypted connection where both client and server present certificates to prove their identity. It helps services verify who they are communicating with. certificate storage, and MCPMCPModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. agent testing examples.

    • model provider now supports OpenAI's gpt-5, Anthropic's claude-sonnet-4-20250514, and xAI's grok-code-fast for evals and red teaming
    • Adds cloud-based model audit sharing to persist scan history, enable team collaboration via centralized storage, and manage recent scan paths
    • Adds client_certificate store capability for mTLS authentication and signature-based upload support in provider configs
    • Adds credential sanitization to prevent exposure of secrets in debug logs during evaluation runs
    • Adds auto-fill capabilities for HTTP provider setup to reduce configuration time

    +4 more show less

    • Improves HuggingFace dataset fetching speed and reliability with better retry suggestions
    • Adds medical_off_label_use plugin to detect inappropriate pharmaceutical recommendations
    • Adds unverifiable_claims plugin to test susceptibility to fabricated claims
    • Adds MCP agent testing example for red teaming tool call results

    9 details in Everything

20. July 2025 Release Highlights Jul 1, 2025 · release notes

    Promptfoo July 2025 adds Docker Model Runner, MCPMCPModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. support, context transforms in assertions, and agentic red team strategies.

    • Adds docker model runner for isolated model execution via Docker containers
    • Adds mcp provider to connect to Model Context ProtocolModel Context ProtocolModel Context Protocol, a standard for connecting AI applications to external tools and data sources. A compatible server exposes capabilities an assistant can use through a common interface. servers
    • Adds Google Imagen model support for multimodalmultimodalAble to work with more than one type of input or output, such as text, images and audio. A multimodal model can combine information across those formats. testing
    • Adds AIMLAPI provider for unified access to various AI models
    • Adds Grok-4 and OpenAI Deep Research Models (o3-deep-research, o4-mini-deep-research) as new model providers

    +16 more show less

    • Enhances Azure provider with system_prompt support in configuration
    • Adds LiteLLM EmbeddingsEmbeddingsLists of numbers a model produces to represent the meaning of text, images or code, so similar items sit close together. They power semantic search and RAG. provider for similarity testing and semantic searchsemantic searchSearching by meaning rather than requiring the same words. It commonly compares vector representations to find related content, even when the wording differs.
    • Adds Google Vision provider for image understanding in multimodal evaluations
    • Enhances HTTP Provider to support JKSJKSJava KeyStore, a file format maintained by Oracle for storing cryptographic keys and certificates; tools use it to manage TLS identities without depending on the host OS certificate store. and PFXPFXA binary file format (PKCS#12) that bundles a private key and its certificate chain into a single password-protected file, letting cyber tools load credentials without managing separate key and cert files. client certificates
    • Adds Browser Provider connecting to Chrome via Chrome DevTools Protocol (CDPCDPCisco Discovery Protocol, a Cisco-proprietary Layer 2 protocol that advertises device identity, capabilities, and topology between directly connected Cisco gear, giving cyber tools a passive way to map network infrastructure.)
    • Adds context transforms to extract data from provider responses for use in assertions
    • Adds finish-reason as an option in assertions to validate response termination
    • Adds tracing assertion fields: trace-span-count, trace-span-duration, and trace-error-spans
    • Allows defaultTest to load test cases from external files for external test configuration
    • Enhances search with metadata filtering using equals, contains, and not contains operators in Web UI
    • Adds first-class zooming support in the eval results page via Web UI
    • Adds re-grade functionality for existing red team scans after changing grading rules
    • Adds Identity Provider integration to map teams and roles for permission assignment
    • Adds MCP Proxy feature for access control and traffic monitoring of MCP servers
    • Adds custom agentic multi-turn red team strategy using natural language instructions
    • Adds mischievous_user agentic multi-turn red team strategy simulating playful user behavior

    21 details in Everything

21. June 2025 Release Highlights Jun 1, 2025 · release notes

    Adds OpenTelemetryOpenTelemetryAn open standard and set of tools for collecting traces, metrics and logs from applications. It lets services export observability data without tying instrumentation to one monitoring vendor. tracing, PROMPTFOO_MAX_EVAL_TIME_MS env var, ModelAudit /model-audit UI, and new red team plugins.

    └──▷ USE IT
    Run evaluations with an automatic timeout to prevent CI/CD pipeline hangs during long-running checks.
    shell copy

    PROMPTFOO_MAX_EVAL_TIME_MS=30000 promptfoo eval

    • Adds OpenTelemetry tracing support to visualize execution flow and debug performance bottlenecks in complex RAGRAGRetrieval-augmented generation: looking up relevant documents and adding them to the prompt before the model answers. It grounds responses in private or current data without retraining. pipelines.
    • Adds PROMPTFOO_MAX_EVAL_TIME_MS environment variable to prevent runaway evaluations from consuming excessive resources.
    • Adds ModelAudit UI accessible at /model-audit when running promptfoo view, featuring drag-and-drop file selection and live scanning progress.
    • Adds support for scanning SafeTensors format and HuggingFace URLs directly in ModelAudit.
    • Adds System Prompt Override plugin to test LLMLLMA large language model, a neural network trained on large amounts of text to recognize patterns and generate language. Applications use it for tasks such as answering, summarizing and writing code. deployments for system instruction manipulation vulnerabilities.

    +1 more show less

    • Adds Intent plugin support for JSON file uploads with nested arrays to define multi-step attack sequences.

    6 details in Everything