https://cybertoolchain.io/tools/garak New releases and features in garak, tracked by The Cyber Toolchain. en Fri, 16 Oct 2026 20:14:17 GMT https://github.com/NVIDIA/garak/releases/tag/v0.17.1 https://github.com/NVIDIA/garak/releases/tag/v0.17.1 Fri, 16 Oct 2026 20:14:17 GMT garak v0.17.1 adds a hit@k attack success rate metric to probe/detector reporting and gains Python 3.14 support.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds `hit@k` attack success rate metric, exposed as terminal output and per probe/detector pair in the final digest entry of `report.jsonl`.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.17.0 https://github.com/NVIDIA/garak/releases/tag/v0.17.0 Wed, 09 Sep 2026 18:56:03 GMT garak v0.17.0 adds EU AI Act risk mapping for probe results and Ollama auth/custom client support.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds EU AI Act mapping that tags and groups probe results by EU AI Act risk categories.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠ • Adds authentication and custom client kwargs support for the Ollama generator. • Forwards generation parameters to the Ollama generator via an options dict. Breaking changes: • Python 3.10 is no longer supported; Python 3.13 is now supported. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.16.0 https://github.com/NVIDIA/garak/releases/tag/v0.16.0 Tue, 04 Aug 2026 19:03:13 GMT garak v0.16.0 adds Context Aware Scanning with technique/intent annotations, a native Anthropic generator, and a new adaptive attacks probe. • Introduces a unified `run.spec` selection grammar that consolidates `probe_spec` and `buff_spec` into the `run` config definition and adds support for selecting _intents_ alongside probes and buffs. • Adds `suppressed_params` support to `BedrockGenerator` for controlling which parameters are passed to the Bedrock API. • New Context Aware Scanning (CAS) framework: technique and intent annotations on probes, a `technique_intent_matrix` in reports enriched with names and descriptions, and a new `IntentProbe` base class for intent-driven evaluation. • Adds `probe_summary` as a new entry type in `report.jsonl` output, and expands the `digest` format with probe summary data. • New native Anthropic generator plugin for direct Anthropic API access. • New simple adaptive attacks probe (`probes` plugin). • Adds Unicode normalisation to `StringDetector` for more robust string matching across character representations. • Hardens `web_injection` markdown exfiltration detectors against domain, extension, and parameter evasion techniques. • New technique and intent tabs/lists in scan reports for browsing CAS results. Breaking changes: • `probe_spec` and `buff_spec` configuration keys move into the `run` definition as part of the new unified spec syntax — existing config files using top-level `probe_spec` or `buff_spec` will break. • CLI plugin selection options for probes and buffs are now deprecated in favour of the unified spec syntax (old options still accepted but marked deprecated). • The `report.jsonl` output structure changes: new `probe_summary` entry type is added, the `digest` format is expanded, and `digest` values tied to the old `*_spec` configuration keys are removed — any downstream parsers of `report.jsonl` must be updated. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.15.1 https://github.com/NVIDIA/garak/releases/tag/v0.15.1 Fri, 05 Jun 2026 17:24:50 GMT garak v0.15.1 adds ProPILE PII probes, `simonw/llm` generator support, and JSON/file output for garak.analyze.qual_review • `proPILE` probes now available in the `probes` module for PII leakage detection. • Generator plugin can now use the `simonw/llm` library. • `garak.analyze.qual_review` output mode now supports JSON and file writing. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.14.0 https://github.com/NVIDIA/garak/releases/tag/v0.14.0 Wed, 04 Feb 2026 17:31:04 GMT garak v0.14.0 adds JSON config support and new reporting features with breaking changes to report format • Supports JSON configuration files alongside existing YAML formats for the cyber tooling • Adds 'tier biased security' aggregate data to evaluation reports • Expands the content of `entry_type:eval` items in output reports • Removes `--generate_autodan` CLI option from garak Breaking changes: • The format of `eval` and `digest` entries in JSONL reports has been modified; compatibility expectations changed. • HTML reports are fully redesigned. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.13.3 https://github.com/NVIDIA/garak/releases/tag/v0.13.3 Fri, 12 Dec 2025 23:41:54 GMT Adds AWS Bedrock generator support and new Api key/probe plugins • `probe: Api key probe + detector` for testing API key exposure or validity • `probe: zero width bad char injection` using invisible characters to bypass filters • `generator: AWS bedrock generator` enables probing models via the AWS Bedrock service • `OpenAICompatible` classes do one generation at a time by default Breaking changes: • Support for init'ing `attempt.prompt` with `str` is removed • Support for `attempt.lang` value is removed Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.13.2 https://github.com/NVIDIA/garak/releases/tag/v0.13.2 Mon, 10 Nov 2025 16:44:13 GMT Adds `probe:base` class for iterative probing and FITD, plus a generic `detectors:any` detector. • `probe:base` - Base class for iterative probes and FITD as an example • `detectors:any` - Universal detector plugin Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.13.1 https://github.com/NVIDIA/garak/releases/tag/v0.13.1 Wed, 01 Oct 2025 20:20:30 GMT New probes (Atbash, DRA, token smuggling), CLI --list_* filtering, and detectors= in output • Supports `detectors=None` in output values to omit detector columns. • CLI adds filtering support for `--list_probes`, `--list_detectors`, etc. via `--list_*` options. • Adds the `probe.atbash` probe for Atbash Encoding attacks. • Adds the `probe.token_smuggling` probe module. • Adds the `probe.dra` probe (Disguise and Reconstruction Attack). • Adds the `dropbox.repeated_token_attack` plugin. • Adds support for hallucination probes in dart, perl, & raku packages. • Supports ansi escape codes in tokenizer probes. Breaking changes: • Config keys `model_*` are renamed to `target_*`. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.12.0 https://github.com/NVIDIA/garak/releases/tag/v0.12.0 Tue, 01 Jul 2025 17:46:00 GMT garak v0.12.0 adds audio NIM model support, Google Translate provider, and result summary object • `audio` NIM model and `audio` probes are now available for scanning audio inputs. • New `google` language provider enables translation via Google Translate API. • Adds a `result_summary` object to the reporting output structure. Breaking changes: • The `octo` generator has been removed. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.11.0 https://github.com/NVIDIA/garak/releases/tag/v0.11.0 Wed, 28 May 2025 01:05:08 GMT garak v0.11.0 adds SQL injection and Mistral probes, `-no-cnv` flag for GGML generators, and new reporting options including `--report-aggregation=min`. • Adds `mistral` generator support to integrate with Mistral AI models in garak scans • Adds `-no-cnv` CLI flag to disable CUDA neural network optimization in GGML-based generators • Adds `--report-aggregation=min` reporting option to aggregate probe scores using the minimum value instead of the previous default • Adds `--no-group-score` reporting flag to suppress group-level score aggregation in output reports • Updates `promptinject` detector to accept multiple triggers for broader injection pattern coverage Breaking changes: • The `--report-aggregation` default behavior changed from the previous logic (likely max/mean) to a new aggregation method, potentially altering reported scores in scripts relying on the old default. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.10.3 https://github.com/NVIDIA/garak/releases/tag/v0.10.3 Mon, 03 Mar 2025 20:09:09 GMT Adds watsonx.ai generator support and `module.classname` plugin config syntax • Adds `watsonx.ai` generator plugin for querying Watsonx models • Supports `module.classname` notation in configuration to specify plugins (e.g., `probes.xss_exfil`, `generators.watsonx_ai`) • Generators: adds option to skip output between given delimiters and introduces a hook for postprocessing Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.10.2 https://github.com/NVIDIA/garak/releases/tag/v0.10.2 Thu, 30 Jan 2025 19:43:47 GMT Adds configuration-based REST proxy and SSL suppression, `startswith` StringDetector matching, and a `StringDetector.case_sensitive` fix • Adds configuration-based REST proxy support via config settings for LLM probing • Adds configuration-based REST SSL suppression setting in the tool's configuration file • Adds `startswith` matching mode to `StringDetector` base class for string comparison Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.10.1 https://github.com/NVIDIA/garak/releases/tag/v0.10.1 Thu, 19 Dec 2024 18:25:35 GMT New probes (ANSI takeover, past tense), `RestGenerator` skip codes, config updates, hf chat support • Adds `probe: ansi_takeover` for ANSI terminal takeover testing • Adds `probe: past_tense_vulnerability` for past tense vulnerability probing • Promotes `generator: openai_compatible` to a first-class generator • Allows adjustable per-probe tags based on payload selection • Reinforces `trust_remote_code=False` for generators by default • Adds ability to specify HTTP codes to skip generation for `RestGenerator` Breaking changes: • Generators now default to `trust_remote_code=False`, which may change behavior for some Hugging Face models if they require remote code execution. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.10.0 https://github.com/NVIDIA/garak/releases/tag/v0.10.0 Thu, 31 Oct 2024 15:56:37 GMT garak v0.10.0 adds Ollama generator, Azure OpenAI support, `--parallel_attempts` flag, and payload/probe separation. • Adds `generator: azure openai` plugin to route probes through Azure OpenAI endpoints. • Adds `--parallel_attempts` CLI argument to control concurrent probing attempts. • Separates payloads from probes, allowing payload logic to be decoupled from the detection mechanism. • Updates `openai` generator to support o1 models via updated plugin configuration. Breaking changes: • The `replay` plugins have been renamed to `divergence`, so any references to the old plugin names in configs or scripts will break. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.16 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.16 Fri, 06 Sep 2024 15:28:03 GMT Garak v0.9.0.16 adds Groq generator, multiple new probes, and moves `generations` to probe config • Adds `generator: groq` support for probing with Groq-hosted models • Adds `probe: topic_pushing` detection plugin for evaluating topic consistency in LLM outputs • Adds `probe: ruby_package_hallucination` to detect hallucinated Ruby packages • Adds `probe: latent_prompt_injection` for detecting covert prompt injection attacks • Adds `probe: npm_…` to detect hallucinated NPM packages • Adds `probe: rust_crate_hallucination` to detect hallucinated Rust crates • Moves the `generations` parameter from generator config to probe config, allowing per-probe output count control • Enables tokenizer customization in `HFDetector` via a new configuration key • Renames the detector config key `knownbadsignatures` to `av_spam_scanning` Breaking changes: • The `knownbadsignatures` config key is renamed to `av_spam_scanning` • The `generations` parameter must now be set in the probe configuration instead of the generator configuration Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.15 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.15 Tue, 13 Aug 2024 15:51:08 GMT Adds new fileformats probe, configurable detector scores, HFDetector hf_args, and NIM seed/temperature flags • Add `fileformats` probe to detect potentially malicious files in repos • Report comparative info for detector scores • Enable HFDetector model configuration with the `hf_args` parameter • Add flags to vary NIM seed and temperature every call Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.14 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.14 Mon, 29 Jul 2024 13:24:01 GMT Adds BEAST/grandma probes, configurable plugins with nested settings, gpt-4o support, and JSON Path in RestGenerator. • Adds `BEAST` attack plugin for probing cyber tooling • Adds 'grandma' probe plugin for probing cyber tooling • Generates Lorem Ipsum content via the new lorem ipsum generator • Adds support for gpt-4o and gpt-4o-mini models • Enables nested setting of plugin configs • Allows configuration of the adversarial generator via atkgen configurability Breaking changes: • Removes support for openai api v0 Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.13 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.13 Fri, 10 May 2024 13:41:33 GMT Adds DoNotAnswer probe, LiteLLM/NIM/LangChain Serve generators, multimodal LLaVa/FigStep support, and parallel OpenAI calls • `generator: NIM` - Adds generator for NVIDIA Inference Microservices (NIM) via the `nim` generator name. • `probe: DoNotAnswer` - Adds a new probe to test prompts an LLM should decline, accessible as the `DoNotAnswer` probe plugin. • Adds `LangChain Serve` generator support, allowing garak to use 'langchain serve' endpoints via the `langserve` generator name. • Adds `LiteLLM` generator support for connecting to providers via LiteLLM, accessible as the `litellm` generator name. • Adds multimodal jailbreaking attacks using LLaVa with FigStep, enabling visual inputs in probes (e.g., `llava_figstep` probe/generator combinations). • Enables parallel OpenAI calls for faster scanning when using OpenAI-compatible generators. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.12 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.12 Tue, 20 Feb 2024 16:12:40 GMT New Rasa generator, Tree of Attacks probe, and multiple buffs support • Adds `Rasa` as a supported generator for probing models • Enables applying multiple buffs simultaneously to a scan via new core logic • Changes the default output directory from `runs/` to `garak_runs/` Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.11 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.11 Fri, 19 Jan 2024 09:52:55 GMT Adds new probes for replay nudging, invisible text injection, DAN jailbreaks, and data leakage, plus an NVIDIA cloud functions generator. • Adds `garak` CLI entry point to invoke the tool directly • Adds NVIDIA cloud functions (NVCF) generator support • Adds probe for repetition-based nudging into replay/spurious generation • Adds probe for invisible text prompt injections • Adds probe for the 'DAN in the wild' paper's library of jailbreak prompts • Adds probe for NYT & The Guardian content in training data leakage Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.10 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.10 Wed, 20 Dec 2023 13:35:47 GMT New AutoDAN probe, MISP tag probe selection, NeMo Guardrails generator support, and automatic HTML reporting with configurable grouping • Adds `NeMo-Guardrails` as a new generator for LLM interaction probes. • Adds `AutoDAN` probe for automated prompt injection attacks. • Enables probing selection via MISP tags using the `probe_tags` configuration key and corresponding CLI argument. • Configures HTML report grouping via the `report_groups` setting, supporting 'module' or 'probe tag category'. • Automates HTML report generation upon completion without requiring manual export steps. • Introduces `encoding.Base64` and `encoding.CharCode` as new buffs for prompt obfuscation. Breaking changes: • The subcommand previously named 'atk' has been renamed to 'atkgen'; scripts using the old name will fail. Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.9 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.9 Fri, 08 Dec 2023 13:40:07 GMT Adds GCG jailbreak probe, NVIDIA Optimum generator support, OWASP tags on probes, and a multi-layer config file system. • Added `probes.gcg.GCG_Probe` for the GCG jailbreak attack class • Added `generators.huggingface.OptimumPipeline` for NVIDIA Optimum inference • Added OWASP tags to probes via the new OWASP tag metadata field • Added `buffs.paraphrase.Fast` and `buffs.paraphrase.PegasusT5` paraphrase buffs • Support core, site, and CLI config files for system, run, and plugin parameters Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.8 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.8 Tue, 14 Nov 2023 16:17:32 GMT Garak v0.9.0.8 adds NeMo and OctoML generators, generic REST connector, parallelization options, AutoDAN probe, interactive mode, and LangChain interface support. • Adds `ne_mo` generator option for NVIDIA NeMo LLMs • Adds `octoml` generator option for OctoML models • Adds generic REST connector with configuration support • Adds `--parallelise_requests` option to parallelize API calls • Adds `--parallelise_attempts` option to parallelize attack attempts • Includes AutoDAN probe for jailbreak testing Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.6 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.6 Tue, 25 Jul 2023 03:57:54 GMT garak v0.9.0.6 adds AVID vulnerability reporting, package hallucination probe, and primary/extended detector selection. • Adds `--probe_options '{"encoding.options": ["default", "slurs", "xss"]}'` to select multiple payloads for the encoding module • Adds ability to designate a primary detector for a probe in probewise harness via config • Adds direct integration to report vulnerabilities found with garak to AVID • Adds `package_hallucination` probe for detecting package hallucination Breaking changes: • Removed shortnames from probes and detectors Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.5 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.5 Wed, 12 Jul 2023 21:50:27 GMT garak v0.9.0.5 adds AVID reporting, glitch token and data leakage probes, and narrow-format CLI output. • Enables `report_to_avid` config to send vulnerability reports to [AVID](https://avidml.org/) • Adds new `glitchtoken` probe for detecting glitch tokens in LLM outputs • Adds new `dataleakage` probe for detecting data leakage and replay attacks • Enables narrow-format CLI output via the `--narrow` flag Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak https://github.com/NVIDIA/garak/releases/tag/v0.9.0.4 https://github.com/NVIDIA/garak/releases/tag/v0.9.0.4 Tue, 04 Jul 2023 22:03:32 GMT Adds Hugging Face model loading without transformers pipeline and --report_prefix CLI flag for custom report paths in v0.9.0.4 • `probes.art` auto-red-team module now captures full exchange history and offers improved progress tracking • Adds the `--report_prefix` CLI argument to specify a custom prefix for generated report files • New generator capability allows loading Hugging Face models directly, removing the dependency on `transformers.pipeline` Breaking changes: • Default random seed is removed; generation results may now vary between runs unless a seed is explicitly set Full history on Cyber Toolchain: https://cybertoolchain.io/tools/garak