What is actually hiding inside this blob of base64 or XOR-obfuscated data?

pipes the sample through decoding and decompression units chained with the pipe operator, peeling one encoding layer per unit until the plaintext or embedded file falls out

Is there a Windows executable buried inside this sample?

carves the payload out by scanning for PE headers and structure, pulling the embedded binary out of the surrounding bytes without needing to know its offset in advance

What sections does this PE file have, and have I seen them before?

lists every section with its computed hash, so sections can be diffed or matched against known-good or known-bad samples

What indicators of compromise are scattered across this pile of samples?

recurses through a directory, running the same extraction pipeline against every file and collecting the indicators it finds along the way

What key or byte was used to obfuscate this payload?

brute-forces single-byte XOR (or tries a supplied key against AES-CBC with a PBKDF2-derived key) and reports which one produces recognizable output, such as a carved PE

Can I reconstruct the exact sequence of transforms that unpacked this sample?

expresses the whole decode as one chained command of single-purpose units, so the pipeline itself becomes the documented, rerunnable record of how the payload was unpacked

refinery release history — The Cyber Toolchain

0.11.3 Oct 26, 2026 · issue 122

refinery 0.11.3 adds six new units (dau3, xtwim, xtiex, dns, huffman, batev, batfs) and breaks several meta variable names and command names⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠

└──▷ GET THIS VERSION

$ git clone --branch 0.11.3 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.11.3

└──▷ TRY IT

Extract DNS hostnames queried in a PCAP to quickly triage C2 or data-exfil domains.

$ emit capture.pcap | pcap | udp | dns

Deobfuscate a suspicious AutoIt3 script dropped by malware.

$ emit suspicious.au3 | dau3

Unpack a WIM archive and list its contained files for triage.

$ emit payload.wim | xtwim

Adds dau3 script deobfuscator for AutoIt3 scripts, xtwim unit for unpacking WIM archives, xtiex unit for unpacking IExpress setup files, dns unit for parsing DNS packets from PCAPs (used with the pcap and udp units), huffman unit as a low-level option to directly handle huffman-compressed data, batev unit to extract environment variables set during a batch emulation, and batfs unit to extract file system artifacts from batch emulation.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠
Renames script deobfuscator commands (js becomes djs, ps1 becomes dps1, vba becomes dvba, php becomes dphp) to remove shadowing of real script interpreters, and improves deobfuscation quality and speed in both djs and dps1.
Every unit that reports a start meta variable now also reports the matching end offset.
The csv unit now outputs minified JSON.
The xtpdf unit metadata paths now end in .json.

+1 more show less

In struct, the format modifier !0 now reads a bit field and : was added as a prefix option for peeking a value.

Adds dau3 script deobfuscator for AutoIt3 scripts, xtwim unit for unpacking WIM archives, xtiex unit for unpacking IExpress setup files, dns unit for parsing DNS packets from PCAPs (used with the pcap and udp units), huffman unit as a low-level option to directly handle huffman-compressed data, batev unit to extract environment variables set during a batch emulation, and batfs unit to extract file system artifacts from batch emulation.⁠︀︄︇︀︅︉︀︅︀︀︃︀︀︅︅︀︅︊︀︄︆︀︃︄⁠

5 more in Everything

└──▷ BREAKING ON UPGRADE

!The minimum supported Python version was raised from 3.10 to 3.11.
!Script deobfuscator commands were renamed: js is now djs, ps1 is now dps1, vba is now dvba, and php is now dphp — any pipeline or script using the old names will break.
!The meta variable offset was renamed to start in carve, rex, xtp, xtw, all carve* units, ngrams, perc, vsect, vsnip, and xtasar — pipelines referencing offset will no longer find the value.
!The meta variable kind was renamed to type in xtw and xtpdf — pipelines referencing kind from those units will no longer find the value.
!The xt7z unit now reports the archive checksum as checksum rather than crc32 — pipelines referencing crc32 from xt7z will break.
!In struct, the format modifier !0 now reads a bit field and : was added as a prefix option for peeking — existing struct format strings using !0 for other purposes will behave differently.
!The pd and pb path handlers lost their separator argument and now always split on both slash and backslash — any invocation passing a separator argument will break.

0.11.2 Jul 14, 2026 · release notes

Refinery 0.11.2 adds scan, vcall, xtvhd units; modular pcap pipeline; JS improvements

└──▷ GET THIS VERSION

$ git clone --branch 0.11.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.11.2

Adds the scan unit for streaming accumulator processing with meta variables
Adds the vcall unit to emulate and call a single function inside an input executable
Adds the xtvhd unit to extract files from VHD/VHDX images, available via xt
Adds -x switch support for exclusion patterns in all path extraction units

4 details in Everything

└──▷ BREAKING ON UPGRADE

!The pcap unit no longer reassembles TCP streams automatically; the pipeline must now include [ | tcp ] to restore stream reassembly
!The previous pcap-http unit is renamed to http; scripts referencing pcap-http will fail

0.11.0 May 13, 2026 · release notes

Refinery 0.11.0 requires Python 3.10+, switches speakeasy backend to 2.0, and adds script deobfuscation capabilities.

└──▷ GET THIS VERSION

$ git clone --branch 0.11.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.11.0

Raises the minimum Python version requirement to 3.10
Switches the speakeasy emulator backend from a fork to speakeasy 2.0
Adds new capabilities to the script deobfuscation engine

3 details in Everything

└──▷ BREAKING ON UPGRADE

!Running refinery with a Python version lower than 3.10 will fail.

0.10.11 Apr 21, 2026 · release notes

Refinery 0.10.11 adds cmdarg, swf, bigint units and rewrites deobfuscation with new parsers

└──▷ GET THIS VERSION

$ git clone --branch 0.10.11 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.10.11

└──▷ USE IT

Extract unescaped command-line arguments from malicious scripts targeting common Windows utilities

shell

refinery analyze --units cmdarg <input_file>

Auto-detect and deobfuscate PowerShell, VBA, or JavaScript content in a sample

shell

refinery analyze --units defu <input_file>

Extract resources from a legacy Flash SWF file

shell

refinery analyze --units swf <input_file>

Adds cmdarg unit to extract and unescape arguments passed to wmic, cmd, start, or powershell commands
Adds swf unit for extracting resources from Flash SWF files
Adds bigint unit for big-integer encoding, split from base
Rewrites deobfuscation system with full parsers and emulators: new ps1, vba, js, and universal defu units replace old deob-ps1-*, deob-vba-*, deob-js-*
Adds script decompiler outputting human-readable NSIS source to the xtnsis unit

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The old deobfuscation units (deob-ps1-*, deob-vba-*, deob-js-*) have been replaced by new units (ps1, vba, js, defu)
!Big-integer encoding is now in the bigint unit instead of base

0.10.6 Mar 23, 2026 · release notes

Adds unq unit for restoring AV-quarantined files; simplifies modular dependency system options.

└──▷ GET THIS VERSION

$ git clone --branch 0.10.6 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.10.6

└──▷ TRY IT

Restore a file that was quarantined by antivirus software

$ refinery unq <path_to_quarantined_file>

Adds the unq unit to restore AV-quarantined files
Simplifies modular dependency system with new options: default, extended, and all

2 details in Everything

0.10.3 Mar 17, 2026 · release notes

Refinery 0.10.3 adds PNG chunk extraction, binary refinery overview mode, and AI agent guidance.

└──▷ GET THIS VERSION

$ git clone --branch 0.10.3 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.10.3

└──▷ TRY IT

Quickly list all available binary refinery units for documentation or discovery.

$ binref --brief

Extract specific chunks from a PNG file.

$ cat image.png | png

--brief (-b) flag added to binref command to list all available units for quick reference.
png unit introduced to extract data chunks from PNG images.
xtsf unit extended to support SetupFactory 10 archives.
py7zr dependency removed; 7zip unpacker now supports all archive types without external dependencies.
AI usage guidance provided via a new SKILL.md file for use with language model based agents.

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The hostname regex pattern was renamed to host, requiring updates to any configurations or scripts using the old name.

0.10.2 Mar 12, 2026 · release notes

Refinery 0.10.2 adds xtsf, cbor, asn1, and dncode units for new extraction and parsing capabilities

└──▷ GET THIS VERSION

$ git clone --branch 0.10.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.10.2

Adds the xtsf unit for extracting SetupFactory binaries.
Adds the cbor unit for parsing the CBOR format.
Adds the asn1 unit for generic ASN.1 structure parsing.
Adds the xtrar unit for extracting RAR archives.
Adds the xtdmg unit for extracting DMG volumes.

+6 more show less

Adds carve-elf and carve-pdf units.
Adds the a3xs unit for extracting only the script part of an AutoIt3 binary.
Adds the sm3, aria, twofish, scrypt, and simon cryptographic units.
Adds the plist, puny, and qp units for various common encodings.
Adds the dsstore unit for parsing MacOS' .DS_Store files.
Adds dncode unit for extracting the MSIL code from functions in a .NET assembly.

11 details in Everything

└──▷ BREAKING ON UPGRADE

!The qr unit can no longer parse arbitrary bar codes but is restricted to QR codes.

0.10.0 Feb 23, 2026 · release notes

Refinery 0.10.0 adds bat, idbmeta, carve-tar, carve-png units and --try global option.

└──▷ GET THIS VERSION

$ git clone --branch 0.10.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.10.0

└──▷ TRY IT

Parse or emulate a Batch file directly using the new unit.

$ refinery bat < script.bat

Adds --try as a new global option to forward input when a unit fails to process it
Adds the bat unit with a parser and emulator for Batch files
Adds the idbmeta unit to read metadata from older IDA databases
Adds the carve-tar unit for carving TAR files
Adds the carve-png unit for carving PNG files

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The -T switches in units alu and pemeta are removed

0.9.10 Nov 19, 2025 · release notes

Adds hlg/hls/hlb syntax highlighting units and integer interval args to bruteforce unit

└──▷ GET THIS VERSION

$ git clone --branch 0.9.10 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.9.10

Adds hlg, hls, and hlb units for accessing different style flavors of syntax highlighting
The bruteforce unit now accepts integer intervals as argument rather than Python slices

2 details in Everything

0.9.9 Nov 17, 2025 · release notes

Refinery 0.9.9 adds the hl terminal highlighting unit and obfuscated MSI support to xtmsi

└──▷ GET THIS VERSION

$ git clone --branch 0.9.9 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.9.9

0.9.5 Oct 17, 2025 · release notes

Refinery 0.9.5 adds strarray patterns to carve unit and slice-style args to vstack

└──▷ GET THIS VERSION

$ git clone --branch 0.9.5 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.9.5

Adds strarray pattern support in the carve unit for matching arrays of strings
Enables slice-style address arguments in the vstack unit to specify stop addresses more easily
Exposes the current binary-refinery version via the binref command

3 details in Everything

0.9.0 Sep 3, 2025 · release notes

Refinery 0.9.0 adds pdfcrypt, qr, z85, flz units and PDF image extraction

└──▷ GET THIS VERSION

$ git clone --branch 0.9.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.9.0

└──▷ TRY IT

Remove password protection from a PDF document using the new pdfcrypt unit.

$ refinery --input protected.pdf --unit pdfcrypt --remove-password

Decode QR codes from an image or video stream using the new qr unit.

$ refinery --input code.png --unit qr --decode

Adds pdfcrypt unit for removing & adding passwords from & to PDF documents.
Adds qr unit for decoding barcodes (especially QR codes).
Adds z85 unit implementing Base85 encoding scheme.
Adds flz unit implementing FastLZ, now included in the universal decompressor.
Extends pdf unit with support for extracting images based on the mupdf library.

+1 more show less

Allows xtzip unit to extract archives partially and supports --lenient option.

6 details in Everything

0.8.21 Jun 16, 2025 · release notes

Adds pecdb, pefix, and pkw units; enables zero-alignment struct peeking

└──▷ GET THIS VERSION

$ git clone --branch 0.8.21 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.21

└──▷ USE IT

Parse PE CDB debug info using the new pecdb unit.

shell

refinery pecdb <path_to_pe_file>

Fix issues in a PE binary using the new pefix unit.

shell

refinery pefix <path_to_pe_file>

Decompress PKWare archives using the new pkw unit.

shell

refinery pkw <path_to_pkw_archive>

Adds pecdb unit for parsing PE CDB (CodeView Debug) information
Adds pefix unit for fixing PE binary issues
Adds pkw unit for decompressing PKWare archives
In the struct unit, supports specifying an alignment value of zero to peek struct entries

4 details in Everything

0.8.19 Jun 6, 2025 · release notes

Refinery 0.8.19 adds MSIL disassembly (dnasm/dnopc), Access DB extraction (xtxs), image units (imgdb/imgtp), and filtering (iffc).

└──▷ GET THIS VERSION

$ git clone --branch 0.8.19 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.19

Adds dnasm unit for MSIL disassembly
Adds dnopc unit for MSIL disassembly
Adds iffc unit for filtering chunks by size constraints
Adds xtxs unit for extracting data from Microsoft Access databases
Adds b2f unit as a shortcut for pick ::-1

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The transposition option in stego has been removed.
!The HKDF key derivation unit is now lowercase (hkdf); using the old capitalization will fail.

0.8.15 Apr 26, 2025 · release notes

Refinery 0.8.15 adds configurable rounds to tea/xtea and input control to couple unit

└──▷ GET THIS VERSION

$ git clone --branch 0.8.15 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.15

└──▷ USE IT

Run a specific number of rounds on an XTEA encrypted file.

shell

refinery xtea --rounds 10 input.bin

Use the couple unit in silent mode where stderr is discarded and no input is required.

shell

refinery couple --noinput input.bin

Adds rounds option to tea and xtea units for controlling the number of cryptographic rounds.
Adds --noinput flag to the couple unit to explicitly toggle mode where stderr is discarded by default.
Changes standard path formatting in xtxml and xthtml units to simplify filtering elements by tag.

3 details in Everything

0.8.14 Apr 19, 2025 · release notes

Refinery 0.8.14 adds rtfc unit, date regex in xtp, csb/csd stripspace arg, and block separators.

└──▷ GET THIS VERSION

$ git clone --branch 0.8.14 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.14

└──▷ TRY IT

Strip whitespace from the output of a carve operation using the new argument.

$ refinery csb <input> --stripspace

Adds the rtfc unit for file carving operations.
Adds a named date regular expression pattern available inside xtp expressions.
Adds --stripspace argument support to csb and csd shortcuts for carve.

3 details in Everything

0.8.6 Mar 6, 2025 · release notes

Refinery 0.8.6 adds escps unit for PS1 string escaping, escvb for VB strings, and forward formatting to rex.

└──▷ GET THIS VERSION

$ git clone --branch 0.8.6 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.6

Adds escps unit (renamed from ps1str) to escape PowerShell strings
Adds escvb unit to escape and unescape Visual Basic strings
Adds input forward format character to rex for improved common use case support
Reworks dnfields unit to extract prettier paths based on method, type, and namespace

4 details in Everything

└──▷ BREAKING ON UPGRADE

!The ps1str unit has been renamed to escps

0.8.5 Feb 24, 2025 · release notes

Refinery 0.8.5 adds xtsim and lzx units for smart install makers and LZX decompression

└──▷ GET THIS VERSION

$ git clone --branch 0.8.5 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.5

└──▷ USE IT

Extract archives created by Smart Install Maker using the new unit.

shell

refinery xtsim archive.ism

Decompress data streams using the standalone LZX decompressor.

shell

refinery lzx input.lzx output.dat

Adds the xtsim unit to extract smart install maker archives
Adds the lzx unit for LZX decompression
Adds LZX support to the xtcab unit

3 details in Everything

0.8.3 Feb 3, 2025 · release notes

Refinery 0.8.3 adds extensive LZMA format support and embedded image extraction via xtinno unit.

└──▷ GET THIS VERSION

$ git clone --branch 0.8.3 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.3

The lzma unit now supports a lot more (especially custom) LZMA formats

1 detail in Everything

0.8.2 Jan 27, 2025 · release notes

Adds xtinno, speck, maru, dnarrays units and string encoding for ifps; -ne switch for iff.

└──▷ GET THIS VERSION

$ git clone --branch 0.8.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.2

└──▷ TRY IT

Extract files from an InnoSetup installer using the new xtinno unit.

$ refinery -u xtinno setup.exe

Process hard-coded arrays from a .NET binary with the new dnarrays unit.

$ refinery -u dnarrays assembly.dll

Scan an encrypted file using the new speck cipher unit.

$ refinery -u speck cipher.bin

Adds the xtinno unit for extracting files from InnoSetup installers
Adds the speck cipher unit
Adds the maru hash unit
Adds the dnarrays unit for extracting hard-coded arrays from compiled .NET code
Allows the ifps and ifpsstr units to accept a string encoding argument

+2 more show less

Extends the iff unit with an -ne switch
Enables the xjl unit to collect frame contents into a JSON list

7 details in Everything

0.8.0 Dec 13, 2024 · release notes

Refinery 0.8.0 adds unicorn/speakeasy/icicle support in vstack and updates multibin handler syntax

└──▷ GET THIS VERSION

$ git clone --branch 0.8.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.8.0

└──▷ TRY IT

Use speakeasy engine for vstack emulation instead of the default unicorn

$ refinery vstack --engine speakeasy <input_file>

vstack unit now supports three emulator engines: unicorn, speakeasy, and icicle (with unicorn as default)
copy: and cut: multibin handlers accept new argument format [offset]:[length]:[step]

2 details in Everything

└──▷ BREAKING ON UPGRADE

!copy: and cut: multibin handlers change argument syntax from [start]:[end]:[step] to [offset]:[length]:[step]
!binref command changes default search logic to conjunctive

0.7.8 Nov 20, 2024 · release notes

Refinery 0.7.8 adds --stream to snip, CSV export to xtmsi, relative meta-variable deduplication, and case-insensitive path matching.

└──▷ GET THIS VERSION

$ git clone --branch 0.7.8 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.8

└──▷ USE IT

Extract multiple chunks from a file where subsequent chunks start after the end of previous ones.

shell

refinery --unit snip --args "--stream" --input malware.bin

Deduplicate extracted fields based on a specific meta variable rather than the entire record.

shell

refinery --unit dedup --args "--meta_field id" --input logs.txt

Adds --stream argument to the snip unit to allow offsets relative to previously extracted data
Enables xtmsi unit to extract all MSI tables as CSV alongside the JSON blob
Allows the dedup unit to specify a meta variable for deduplication via a new optional argument
Makes path extraction units match paths case-insensitively when unambiguous

4 details in Everything

0.7.7 Nov 7, 2024 · release notes

Refinery 0.7.7 adds httprequest and b62 units, and aPLib header support

└──▷ GET THIS VERSION

$ git clone --branch 0.7.7 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.7

Adds the httprequest unit for parsing HTTP requests.
Adds the b62 unit (thanks to [@lukaskuzmiak][]).
Adds support for aPLib compressed data with headers.

3 details in Everything

0.7.6 Oct 30, 2024 · release notes

Adds brotli decompression, pym marshal interface, XChaCha/XSalsa units, and prng/rng argument handlers

└──▷ GET THIS VERSION

$ git clone --branch 0.7.6 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.6

Adds the brotli decompression unit.
The pym unit was added which provides an interface to Python's marshal serialization.
The units xsalsa, xchacha, and chacha20poly1305 were added. The latter only performs the decryption part of the scheme.
The argument handlers prng and rng were added for random number generation.

4 details in Everything

0.7.4 Oct 7, 2024 · release notes

Refinery 0.7.4 adds symlink control to ef, --join improvements, and EA05 support in a3x

└──▷ GET THIS VERSION

$ git clone --branch 0.7.4 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.4

Adds the -s/--symlinks option to the ef unit to control whether directory symlinks and junctions are followed during extraction.
autoxor key scaling method adjusted for larger keys to reduce false positives in scanning results.

2 details in Everything

0.7.2 Sep 8, 2024 · release notes

Refinery 0.7.2 adds p1/p3 chunk shortcuts, loop abort options, and regex handlers

└──▷ GET THIS VERSION

$ git clone --branch 0.7.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.2

└──▷ USE IT

Quickly grab the first frame chunk using the new p1 shortcut

shell

refinery input.raw | p1 output.bin

Use the new f: handler to parse a regex for abort conditions in a loop

shell

refinery input.raw | loop 'f:pattern' process

Toggle boolean negation on conditionals using the -R switch

shell

refinery input.raw | iff -R 'condition' true_branch false_branch

Adds p1, p2, and p3 units as shortcuts for picking the first 1, 2, or 3 chunks from a frame
Enhances loop unit with more options to abort execution based on regular expression patterns
Reworks conditional units (iff/iffp/iffx/iffs) -R switch to control boolean negation
Adds new handler f: for regular expression arguments to initialize from carve formats

4 details in Everything

└──▷ BREAKING ON UPGRADE

!The -s switch was removed from conditional units (iff/iffp/iffx/iffs)
!The cull unit was removed from refinery
!Conditional units now use a separate switch (not -s) to control whether chunks are hidden instead of being discarded

0.7.1 Sep 6, 2024 · release notes

Refinery 0.7.1 adds loop unit, multi-bin suffix support, Kramer deobfuscator, and more

└──▷ GET THIS VERSION

$ git clone --branch 0.7.1 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.7.1

Adds --iff option to units for format-specific application
Adds loop unit to apply multibin suffixes repeatedly
Allows piping a Chunk object directly to a pipeline
Adds csb and csd shortcuts for carve applications
Enables reduce unit to work with multibin suffix

+2 more show less

Adds deobfuscator for the kramer obfuscator
xtmsi unit now extracts embedded CAB files

7 details in Everything

0.6.28 Feb 5, 2024 · release notes

Refinery 0.6.28 removes init value in reduce and splits queue into qf/qb units

└──▷ GET THIS VERSION

$ git clone --branch 0.6.28 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.28

Adds optional argument to reduce to consume only a limited number of chunks instead of processing all available data
Replaces the queue unit with two new units: qf (queue front) and qb (queue back) for queuing chunks into the current frame

2 details in Everything

└──▷ BREAKING ON UPGRADE

!reduce no longer accepts an initialization value argument; the first chunk is always used as the base.

0.6.24 Nov 29, 2023 · release notes

Adds sosemanuk cipher unit and improves vbastr capability in refinery 0.6.24

└──▷ GET THIS VERSION

$ git clone --branch 0.6.24 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.24

└──▷ TRY IT

Use the new Sosemanuk cipher for lightweight encryption tasks.

$ refinery --cipher sosemanuk -i input.txt -o output.enc

Leverage the improved vbastr unit for better VBA string handling.

$ refinery --unit vbastr -i macro.vba -o result.txt

0.6.19 Nov 18, 2023 · release notes

Refinery 0.6.19 adds whitespace-resilient b85 unit and synthesized section extraction to vsect/vsnip.

└──▷ GET THIS VERSION

$ git clone --branch 0.6.19 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.19

b85 unit is now resilient against white space

1 detail in Everything

0.6.16 Oct 27, 2023 · release notes

Refinery 0.6.16 isolates path formatting in xhtml/xtxml units and adjusts vstack zero-byte patch logic.

└──▷ GET THIS VERSION

$ git clone --branch 0.6.16 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.16

0.6.15 Oct 25, 2023 · release notes

Adds refinery.shell string interface, vstack, lzw, hc256, and more units/options

└──▷ GET THIS VERSION

$ git clone --branch 0.6.15 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.15

└──▷ TRY IT

Extract remaining unparsed data after structure matching using the new --more flag.

$ refinery in -u struct:header --more payload.bin

Read interlaced integers from a binary stream using the enhanced btoi handler.

$ refinery in -u 'btoi: 32' interleaved.dat

Adds the ability to instantiate units via string arguments imported from refinery.shell
Adds the --more option to the struct unit for unparsed rest data
Adds the --length option to the snip unit
Adds the lzw decompression unit
Adds the xtmagtape unit for extracting files from SIMH tape files

+3 more show less

Adds the hc256 cipher unit
Adds a second argument to the btoi handler for interlaced integers
Adds the dnsfx unit for extracting .NET file bundles

8 details in Everything

0.6.14 Oct 13, 2023 · release notes

Refinery 0.6.14 adds ngrams/bruteforce units, shellcode execution in vstack, and Y: shortcut.

└──▷ GET THIS VERSION

$ git clone --branch 0.6.14 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.14

ngrams and bruteforce units added for simple brute forcing tasks
vstack unit can now execute shellcode blobs
vstack registers can be initialized using shell environment variables
trim unit performs case-insensitive trimming via new behavior
salsa and chacha units accept a 64-byte key for full initial state matrix initialization

+2 more show less

perc unit extracts resource languages
yara: handler supports the shorter Y: shortcut for regular expression arguments

7 details in Everything

0.6.11 Sep 12, 2023 · release notes

Refinery v0.6.11 adds machometa, pestrStrip/trim, and xtnuitka units for advanced binary metadata and overlay extraction

└──▷ GET THIS VERSION

$ git clone --branch 0.6.11 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.11

└──▷ USE IT

Extract metadata from a Mach-O executable (e.g., macOS binary) to identify libraries or build details.

shell

refinery machometa target.dylib

Clean a PE file by removing unwanted overlays, often used for static analysis or cleanup before further processing.

shell

refinery petrim target.exe

Extract files from a Nuitka-compiled Python executable to inspect the bundled archive content.

shell

refinery xtnuitka compiled_app.exe

Adds machometa unit to extract Mach-O binary metadata
Introduces petrim unit to remove overlays from PE files
Adds xtnuitka unit to extract Nuitka archives

3 details in Everything

0.6.8 May 31, 2023 · release notes

Adds --discard option to stream cipher units; removes vstack call tracing

└──▷ GET THIS VERSION

$ git clone --branch 0.6.8 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.8

All stream cipher units now support the --discard option to drop initial keystream bytes.

1 detail in Everything

0.6.4 May 4, 2023 · release notes

Adds xtmsi for MSI extraction and csv reverse operation for JSON-to-CSV

└──▷ GET THIS VERSION

$ git clone --branch 0.6.4 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.4

└──▷ TRY IT

Extract contents of an MSI installer and get its streams as JSON

$ refinery -i app.msi xtmsi

Convert a simple JSON array of objects into CSV format using the reverse csv operation

$ echo '[{"a":1}]' | refinery -i - csv --reverse

Adds the xtmsi unit to extract MSI files and stream metadata as a synthesized JSON document
Adds a reverse operation to the csv unit to convert simple JSON documents back to CSV format

2 details in Everything

0.6.3 Apr 22, 2023 · release notes

Refinery 0.6.3 adds xtnode for pkg/nexe extraction and AES support to xtzip

└──▷ GET THIS VERSION

$ git clone --branch 0.6.3 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.3

└──▷ TRY IT

Extract files from a Node.js binary packed by pkg or nexe

$ refinery xtnode payload.exe -o output_dir

Decrypt an AES-encrypted zip archive using pyzipper backend

$ refinery xtzip encrypted.zip --password mysecret

Adds xtnode unit for extracting contents of Node.js executables created with pkg or nexe
Adds AES-encrypted archive support to xtzip via the pyzipper module

2 details in Everything

0.6.0 Apr 6, 2023 · release notes

Refinery 0.6.0 adds a3x decompiler unit and path extractor fuzzy-matching controls

└──▷ GET THIS VERSION

$ git clone --branch 0.6.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.6.0

Adds the a3x AutoIt decompiler unit
Adds switches to control fuzzy matching behavior in path extractor units

2 details in Everything

0.5.10 Mar 30, 2023 · release notes

Adds --swap, --block-size, and --segment-size options to cipher units in refinery 0.5.10

└──▷ GET THIS VERSION

$ git clone --branch 0.5.10 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.5.10

└──▷ TRY IT

Process tea data with big endian byte order instead of the default little endian.

$ refinery tea --swap <input_file>

Configure xxtea to operate as a block cipher with a specific size (e.g., 128 bits).

$ refinery xxtea --block-size 128 <input_file>

Specify the segment size for rc5 or rc6 when using CFB mode.

$ refinery rc5 --segment-size 8 <input_file>

Adds --swap flag to tea and xtea units to switch endianness
Adds --block-size argument to xxtea unit to enable block cipher mode

2 details in Everything

0.5.2 Dec 1, 2022 · release notes

Refinery 0.5.2 adds lzg, ntlm, vmemref, vstack, min, max units and lzf chunked support

└──▷ GET THIS VERSION

$ git clone --branch 0.5.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.5.2

Adds lzg unit for LZG compression handling
Adds ntlm hash unit
Adds vmemref and vstack units (experimental)
Adds min and max units to replace sorted [| pick 0 ] patterns
lzf unit now supports the chunked format produced by the command-line lzf tool

5 details in Everything

0.5.0 Nov 22, 2022 · release notes

Refinery 0.5.0 adds variable scoping, mvg/mvc units, removes wm, adds vaddr.

└──▷ GET THIS VERSION

$ git clone --branch 0.5.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.5.0

pop unit now propagates variables to parent scope
mvg and mvc units introduced for variable scope management
vaddr unit added to convert virtual addresses to file offsets

3 details in Everything

0.4.49 Nov 12, 2022 · release notes

Adds pkcs7sig unit for PKCS#7 signature processing; improves pemeta and dedup units.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.49 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.49

dedup unit now uses MD5 instead of Python's built-in hash function because of the high risk of collisions.
The pemeta unit now also displays the module name stored in the export directory.
The pkcs7sig unit was added.

3 details in Everything

0.4.48 Oct 12, 2022 · release notes

refinery 0.4.48 adds [mod]:[exp] RSA input format and --pretty for perc.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.48 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.48

└──▷ USE IT

Fix missing headers in raw bitmap/icon resources via the perc unit.

shell

refinery perc --pretty -i resource_data.raw -o out.png

rsa and rsakey units now accept [modulus]:[exponent] hex format.

1 detail in Everything

0.4.47 Sep 22, 2022 · release notes

Adds queue unit, xtnsis unit for xt, --glob flags for ef on POSIX, and improved RICH header display in pemeta.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.47 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.47

The names of urlencode patterns for carve were shortened.
Adds the xtnsis unit to the units used in xt.
By default, the ef unit does no longer use glob-patterns on posix systems. The behavior can be explicitly adjusted using new command-line flags.

3 details in Everything

0.4.41 Jul 23, 2022 · release notes

Adds carve-lnk, carve-rtf, subfiles, xtnsis, ifps units; b64 unit now supports urlsafe variant

└──▷ GET THIS VERSION

$ git clone --branch 0.4.41 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.41

Adds carve-lnk unit to extract Windows Shortcut files from unstructured data
Adds carve-rtf unit to extract Rich Text Format documents from unstructured data
Adds subfiles unit which unifies all structured file format carvers into a single interface
The b64 unit now automatically detects and switches to the urlsafe encoding variant
Adds xtnsis unit to extract files from NSIS archives and disassemble setup scripts

+1 more show less

Adds ifps and ifpsstr units to disassemble and extract strings from compiled Pascal script files

6 details in Everything

0.4.39 Jun 24, 2022 · release notes

Refinery 0.4.39 adds vbapc, vbastr, mscf, b58, xkey, autoxor units and variable conversion syntax to pop.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.39 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.39

Adds vbapc and vbastr units to extract (decompiled) VBA p-code and VBA strings from Word documents.
Adds xkey and autoxor units to automatically decrypt XOR-encrypted files using frequency analysis.
Adds mscf unit which implements part of the Microsoft Compression API formats.
Adds b58 unit for base58 encoding.
Adds multibin handler prefix syntax to variables in the pop unit to convert input data before storage.

+1 more show less

Changes put unit behavior: storing current chunk in a variable when no second argument is provided.

6 details in Everything

0.4.31 May 15, 2022 · release notes

Refinery 0.4.31 adds xt, xtasar, lnk, urlfix, and xjl units; peek gains colorization.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.31 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.31

Adds the xtasar unit to extract data from ASAR files
Adds the lnk unit to extract metadata from Windows shortcut files
Adds the urlfix unit to strip URL fragments and query strings
Adds the xjl unit to convert JSON lists into a sequence of JSON chunks
Adds colorization to the peek hexdump preview, controlled via the -g switch

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The xvar: handler is renamed to eat:

0.4.27 Mar 11, 2022 · release notes

Refinery 0.4.27 adds deob-vba unit, camellia cipher, struct wide strings, and dnfields enhancements

└──▷ GET THIS VERSION

$ git clone --branch 0.4.27 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.27

└──▷ USE IT

Run the new deobfuscation pipeline on a suspicious script file.

shell

refinery deob-vba -i malware_sample.vbs

Decode a wide string using the new struct format.

shell

refinery struct -c 'w' -d <hex_data>

Adds deob-vba unit to apply multiple VBA/VBS deobfuscation units in one step.
Adds camellia cipher support via the camellia unit.
Adds w format character for struct unit to decode wide strings.
Extends dnfields unit to extract string fields assigned a unique value.

4 details in Everything

0.4.24 Mar 6, 2022 · release notes

Refinery 0.4.24 adds six new decompression and deobfuscation units (szdd, lzjb, isaac, bat, groupby) and meta-variable checks for iff

└──▷ GET THIS VERSION

$ git clone --branch 0.4.24 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.24

└──▷ TRY IT

Decompress SZDD archives using the new native support.

$ refinery szdd -i archive.szdd

Deobfuscate a batch script with the new bat unit.

$ refinery bat -i script.bat

Adds the szdd decompression unit.
Adds the lzjb decompression unit.
Adds the isaac cipher unit.
Adds the bat unit for deobfuscating batch scripts.
Adds the groupby unit.

+1 more show less

Adds an option to the iff unit to check for the existence of a certain meta variable.

6 details in Everything

0.4.22 Feb 19, 2022 · release notes

New hash units (RIPEMD), crypto wallet extraction (xtw), entropy heatmap (iemap), and PE header size calculation.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.22 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.22

└──▷ USE IT

Extract cryptocurrency wallet addresses from input data using the new xtw unit.

shell

cat data.txt | xtw

Calculate RIPEMD-256 hash of a file's contents using the new ripemd160 unit.

shell

cat data.bin | ripemd160

Visualize entropy distribution in a binary file as a heatmap.

shell

cat firmware.bin | iemap

ripemd160 and ripemd128 units added for RIPEMD hash calculations
xtw unit added to extract cryptocurrency wallet addresses from text streams
iemap unit added to display colored entropy heatmaps
struct unit gains new syntax for byte alignment handling
rsakey unit now supports an option to output the public key portion of a private key

+2 more show less

pemeta unit now computes PE file size based on header information
New comparison operator switches added to the iff unit

7 details in Everything

0.4.21 Feb 15, 2022 · release notes

Refinery 0.4.21 adds xlmdeobf for Excel macro deobfuscation and carve-7z for 7zip extraction

└──▷ GET THIS VERSION

$ git clone --branch 0.4.21 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.21

└──▷ TRY IT

Extract and deobfuscate Excel V4 macros to analyze potential malicious payloads in a suspicious spreadsheet.

$ refinery xlmdeobf <input_file.xlsx>

Extract 7zip archives from raw data blobs during forensic analysis or malware unpacking.

$ refinery carve-7z <blob_file>

Adds the xlmdeobf unit, wrapping XLMMacroDeobfuscator to extract and deobfuscate Excel V4 macros
Adds the carve-7z unit for carving 7zip archives from binary blobs

2 details in Everything

0.4.20 Feb 5, 2022 · release notes

Refinery 0.4.20 adds default whitespace stripping to resub and renames units.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.20 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.20

resub now strips whitespace from input by default
Renames the blockop unit to alu
Removes the shortcut unit carveb64z

3 details in Everything

└──▷ BREAKING ON UPGRADE

!The blockop unit is now renamed to alu
!The shortcut unit carveb64z is no longer available

0.4.14 Dec 22, 2021 · release notes

Refinery 0.4.14 adds .reg file extraction via winreg unit and improves password handling in key derivation units.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.14 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.14

Adds support for extracting data from Windows registry editor exports (.reg files) through the winreg unit
Updates pbkdf2 and pbkdf1 key derivation units to use a more forgiving decoder for Rfc2898DeriveBytes call signatures

2 details in Everything

0.4.13 Dec 6, 2021 · release notes

Adds dexstr and n40 units; index meta variable auto-population in frames

└──▷ GET THIS VERSION

$ git clone --branch 0.4.13 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.13

Adds dexstr unit for string decoding operations.
Adds n40 string decryption unit.
index meta variable is now automatically populated within frames.
xtpyi unit extracts Python disassembly when decompilation fails.

4 details in Everything

0.4.7 Nov 1, 2021 · release notes

Refinery 0.4.7 adds xtpdf unit and accu pre-configured FSMs

└──▷ GET THIS VERSION

$ git clone --branch 0.4.7 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.7

Adds the xtpdf unit for extracting embedded objects from PDF documents.

1 detail in Everything

0.4.6 Oct 17, 2021 · release notes

Refinery 0.4.6 adds xj0 and evtx units; improves hexload pattern matching and asm disassembly.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.6 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.6

└──▷ TRY IT

Scan files for Excel-based attacks using the new default password support.

$ refinery run 'hexload|xj0' < target.xlsx

Analyze Windows Event Log files using the new evtx unit.

$ refinery run 'evtx' < event.evtx

Adds xj0 and evtx units to detect new file formats
Improves hexload unit (formerly hexdmp) with better pattern matching
Redesigns asm unit using an Angr-based fallback for better disassembly
Enables pcap-http unit to extract source URLs from downloaded data

4 details in Everything

└──▷ BREAKING ON UPGRADE

!The ci property is no longer present in the output of peek --meta

0.4.5 Oct 1, 2021 · release notes

Refinery 0.4.5 adds cswap unit, new flags for swap, xtpyi, esc, and ef.

└──▷ GET THIS VERSION

$ git clone --branch 0.4.5 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.5

└──▷ TRY IT

Include dot-files and recurse into dot-directories when exploring a directory.

$ refinery ef --meta <path>

Adds an option to the swap unit to swap meta variable contents or rename a meta variable.
Adds an option to xtpyi to unpack but not decompile PYZ contents.
Adds the --bare option to esc and uses it in peek.
Adds the --meta option to ef enabling recursion into dot-directories.

4 details in Everything

0.4.4 Aug 30, 2021 · release notes

Refinery 0.4.4 adds xtvba, pcap, xthtml units and htmesc rename

└──▷ GET THIS VERSION

$ git clone --branch 0.4.4 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.4

Adds the xtvba unit to extract Office document macros.
Adds the pcap unit to extract TCP streams from packet capture files.
Adds the xthtml unit to extract components of HTML documents.

3 details in Everything

0.4.3 Aug 20, 2021 · release notes

Adds Murmur Hash and xtpyi units for new file extraction and hashing capabilities in Refinery v0.4.3

└──▷ GET THIS VERSION

$ git clone --branch 0.4.3 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.4.3

Adds xtpyi unit to extract PyInstaller-packed archives

1 detail in Everything

0.3.34 Jul 25, 2021 · release notes

Adds msgpack unit, cull unit for invisible filtering, and renames conditional units

└──▷ GET THIS VERSION

$ git clone --branch 0.3.34 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.34

Adds the msgpack unit to handle MessagePack data format processing.
Adds the cull unit to filter chunks invisibly without removing them from the stream.
Renames existing conditional units to iff, iffs, iffx, and iifp.

3 details in Everything

└──▷ BREAKING ON UPGRADE

!Existing conditional unit names have been renamed: use iff, iffs, iffx, or iifp instead of the previous names.

0.3.33 Jul 18, 2021 · release notes

Refinery 0.3.33 adds xfcc and arithmetically-based cm meta-var removal

└──▷ GET THIS VERSION

$ git clone --branch 0.3.33 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.33

Adds the xfcc unit to replace the deprecated intersection unit.
Introduces a list of default passwords for use in archive units.

2 details in Everything

└──▷ BREAKING ON UPGRADE

!The lprefix unit is removed; functionality must now be implemented with struct.
!Field names in dnfields have changed to match file names more closely.

0.3.30 Jun 22, 2021 · release notes

Refinery 0.3.30 adds xtjson/xtxml units, --tabular to ppjson, and pipeline callable support.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.30 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.30

└──▷ USE IT

Extract JSON keys using the new xtjson unit

shell

data | xtjson | str

Process XML data using the new xtxml unit

shell

data | xtxml | str

Get flattened tabular output from ppjson

shell

ppjson --tabular input.json

Adds the xtjson unit for extracting data from JSON files
Adds the xtxml unit for extracting data from XML files
Adds the --tabular option to ppjson to produce a flattened output
Allows pipelines ending with | callable to invoke a function with concatenated chunks

4 details in Everything

└──▷ BREAKING ON UPGRADE

!Connected pipelines (data | unit | ... | unit) can now be passed to str and bytes which may change behavior if previously relied on iteration over chunks without conversion.

0.3.29 May 5, 2021 · release notes

Refinery 0.3.29 adds rsakey unit, RICH header support in pemeta, and new archive extractors.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.29 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.29

Adds the rsakey unit for RSA key parsing.
Extends pemeta unit with RICH header data support.
Adds xt7z, xtace, xtiso, and xtcpio archive extractors.
Allows meta variables in struct formats.

4 details in Everything

0.3.28 Mar 24, 2021 · release notes

Adds cm, terminate, and struct units; pemeta gains table view

└──▷ GET THIS VERSION

$ git clone --branch 0.3.28 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.28

└──▷ USE IT

Switch legacy trivia unit references to the new name.

shell

refinery cm <input>

View PE header data in a tabular format for quick review.

shell

refinery pemeta --view table input.exe

pemeta supports a table view as an alternative to JSON output.
Python expressions in multibin arguments now allow any operator.
The trivia unit is renamed to cm (common meta).
Adds the new terminate unit.
Adds the new struct unit.

5 details in Everything

└──▷ BREAKING ON UPGRADE

!The trivia unit is renamed to cm; existing workflows using trivia will break.

0.3.24 Feb 6, 2021 · release notes

Refinery 0.3.24 adds xtrtf and officecrypt units for new document extraction and decoding capabilities

└──▷ GET THIS VERSION

$ git clone --branch 0.3.24 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.24

Adds the xtrtf unit to extract embedded objects from RTF documents
Adds the officecrypt unit to decode password-protected Office documents
Adds brieflz support to the universal decompress unit

3 details in Everything

0.3.23 Jan 9, 2021 · release notes

Refinery 0.3.23 adds accu/reduce/cycle/take handlers, --lenient flag, and renames unpack to btoi.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.23 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.23

└──▷ TRY IT

Process binary streams while allowing partial output if an error occurs.

$ refinery --lenient -h accu,input.bin

Use the new reduce handler to combine processed segments.

$ refinery -h reduce,processed.bin

Perform byte-to-integer conversion using the renamed handler.

$ refinery -h btoi,data.bin

Adds the global --lenient option to admit partial results as output.
Renames the unpack handler to btoi and adds the btoi handler which performs the inverse operation.

2 details in Everything

└──▷ BREAKING ON UPGRADE

!The unpack handler is renamed to btoi; existing configs using unpack must update the handler name.

0.3.21 Jan 2, 2021 · release notes

Adds xtdoc and trivia units for Office extraction and binary metadata; converts carve-pe to a path extractor

└──▷ GET THIS VERSION

$ git clone --branch 0.3.21 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.21

Adds the xtdoc unit which can extract more files from Office documents than xtzip.
Adds the trivia unit to attach integer variables size and index for binary chunk metadata.
Converts the carve-pe unit to a path extractor unit, adding new command-line options.

3 details in Everything

└──▷ BREAKING ON UPGRADE

!The eval: handler for numeric multibin values no longer accepts the special variable N to represent chunk size; use the trivia unit and its size variable instead.

0.3.17 Dec 14, 2020 · release notes

Adds ppjscript unit for JS beautification; expands resplit/resub with counts; enhances vsnip memory extraction

└──▷ GET THIS VERSION

$ git clone --branch 0.3.17 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.17

Adds the ppjscript unit (wrapper around jsbeautifier) to process JavaScript files.
Adds a count restriction argument to the resplit and resub units to limit match count.
Enables the vsnip unit to extract multiple memory regions in a single operation.

3 details in Everything

0.3.16 Nov 3, 2020 · release notes

Refinery 0.3.16 adds a conservative mode to peoverlay/pestrip and relaxes cipher interface requirements.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.16 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.16

└──▷ USE IT

Use conservative parsing when analyzing potentially malformed PE files.

shell

refinery peoverlay --conservative <target>

Adds the --conservative flag to the peoverlay command.
Adds the --conservative flag to the pestrip command.
Makes the encryption mode argument optional in the cipher units interface.

3 details in Everything

└──▷ BREAKING ON UPGRADE

!The encryption mode is no longer a mandatory argument for cipher units.

0.3.15 Oct 8, 2020 · release notes

refinery 0.3.15 adds HMAC unit for key derivation and round-configurable cipher units

└──▷ GET THIS VERSION

$ git clone --branch 0.3.15 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.15

Adds the HMAC unit for simple HMAC-based key derivation.
salsa and chacha cipher units now accept a configurable number of rounds in pure Python.
The dump unit's stream mode now supports writing consecutive data to a file inside a nested frame.

3 details in Everything

0.3.14 Oct 2, 2020 · release notes

Refinery 0.3.14 adds rabbit RABBIT cipher unit and fixes crc32/adler32 byte order.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.14 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.14

Adds the rabbit unit which implements the RABBIT stream cipher

1 detail in Everything

0.3.12 Sep 4, 2020 · release notes

Adds lprefix unit for length-prefixed parsing and renames binpng to stego.

└──▷ GET THIS VERSION

$ git clone --branch 0.3.12 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.12

lprefix unit added for parsing length-prefixed data.
binpng unit renamed to stego for more flexible image data extraction.

2 details in Everything

0.3.11 Aug 31, 2020 · release notes

New vsnip and vsect cross-format units for PE/ELF/Mach-O extraction replace removed units

└──▷ GET THIS VERSION

$ git clone --branch 0.3.11 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.11

Adds vsnip unit to extract data from virtual addresses in PE, ELF, and MachO files
Adds vsect unit to extract sections from PE, ELF, and MachO files

2 details in Everything

└──▷ BREAKING ON UPGRADE

!The peslice, elfslice, and pesect units are removed and no longer supported.

0.3.9 Aug 28, 2020 · release notes

Refinery 0.3.9 adds md2/md4 hashing support and updates CryptDeriveKey SHA2 behavior

└──▷ GET THIS VERSION

$ git clone --branch 0.3.9 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.9

md2 and md4 hashing algorithms are now available
the CryptDeriveKey unit now also mirrors the API call for SHA2 based hashing algorithms
message type attachments in Outlook email formats are now supported by xtmail

3 details in Everything

0.3.2 Aug 10, 2020 · release notes

Refinery 0.3.2 adds pesect, jvdasm units and LZ4 decompression, plus local unit loading for dynamic multibin prefixes

└──▷ GET THIS VERSION

$ git clone --branch 0.3.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.3.2

Adds pesect unit for parsing PE file sections.
Adds jvdasm unit for disassembling Java class files.
Adds lz4 unit with a pure Python implementation of LZ4 decompression.
Enables loading local refinery units (Python scripts in the current directory) for multibin prefixes and dynamic unit loading.

4 details in Everything

0.2.1 Jun 24, 2020 · release notes

Refinery 0.2.1 adds netbios/stretch/hc128 units, named match metadata, and xtzip/xtmail capabilities.

└──▷ GET THIS VERSION

$ git clone --branch 0.2.1 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.2.1

└──▷ TRY IT

Scan for NetBIOS names embedded in binary data.

$ refinery --unit netbios <input_file>

Process a password-protected ZIP archive with the new xtzip unit.

$ refinery --unit xtzip --password mysecretpass <archive.zip>

Extract email headers as JSON for further processing.

$ refinery --unit xtmail --format json <email.eml>

stretch unit was added
netbios unit was added
hc128 cipher unit was added
dnmr unit was added for unpacking managed .NET resources
Named match groups for the rex unit are now forwarded as metadata within frames

+2 more show less

xtzip unit gained an optional archive password parameter
xtmail unit can extract headers in text and json format

7 details in Everything

0.2.0 May 5, 2020 · release notes

Adds recode auto-detection, pestrip/peoverlay units and PowerShell deobfuscation improvements

└──▷ GET THIS VERSION

$ git clone --branch 0.2.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.2.0

Enables the recode unit to autodetect input encoding

1 detail in Everything

0.1.8 Apr 25, 2020 · release notes

Refinery 0.1.8 adds base32 unit, meta variable commands, --squeeze flag, and RSA blob support

└──▷ GET THIS VERSION

$ git clone --branch 0.1.8 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.8

Adds mpush, mpop, and mput units for preliminary support of meta variables
Adds b32 unit for base32 encoding and decoding
Adds --squeeze/-Z option to all units that produce multiple outputs to disable default line break separation

3 details in Everything

0.1.7 Apr 20, 2020 · release notes

Refinery 0.1.7 adds dnfields unit and -p/--plain flag for dump output control.

└──▷ GET THIS VERSION

$ git clone --branch 0.1.7 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.7

Adds -p and --plain flags to dump to prevent automatic file name string formatting

1 detail in Everything

0.1.5 Mar 30, 2020 · release notes

New couple unit for wrapping CLI tools, new carve-xml and dnstr units, Python __init__ deduction support.

└──▷ GET THIS VERSION

$ git clone --branch 0.1.5 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.5

└──▷ USE IT

Wrap an external tool like file into a refinery pipeline using the new couple unit.

shell

refinery -u couple,file input.txt

Use the new carve-xml unit to extract XML segments from a binary stream.

shell

refinery -u carve-xml input.bin

Adds the couple unit to wrap any stdin/stdout command line tool as a refinery unit
Adds the carve-xml unit for XML carving
Adds the dnstr unit for DNS-related transformations
Enables units to deduce CLI from a Python __init__ constructor with an enrichment decorator class

4 details in Everything

0.1.4 Mar 9, 2020 · release notes

Refinery 0.1.4 adds xtmail, carve-json, pemeta, b85 units and --meta flag for metadata support.

└──▷ GET THIS VERSION

$ git clone --branch 0.1.4 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.4

└──▷ TRY IT

Extract email content using the new xtmail unit piped to dump.

$ refinery -u 'xtmail' | refinery -u 'dump' --meta

Use the new base85 unit for encoding data.

$ echo 'data' | refinery -u 'b85'

Adds xtmail unit to extract email body and attachments (Outlook/MIME formats)
Adds carve-json unit
Adds pemeta unit
Adds b85 base85 encoding/decoding unit
Adds --meta option to dump unit to use file name metadata when present

+1 more show less

Implements sha256 and md5 as separate hashing units for multibin expressions

6 details in Everything

0.1.2 Feb 18, 2020 · release notes

Refinery 0.1.2 adds xtzip, carve-zip, rsa, rncrypt, dncfx units and TrendMicro URL guard support

└──▷ GET THIS VERSION

$ git clone --branch 0.1.2 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.2

Adds the xtzip unit to extract data from ZIP archives
Adds the carve-zip unit to carve ZIP files from buffers
Adds the rsa unit for RSA cryptographic operations
Adds the rncrypt unit for RNCryptor support
Adds the dncfx unit to extract strings from ConfuserEx obfuscated .NET binaries

+1 more show less

Adds TrendMicro Clicktime URL guard support in the urlguards unit

6 details in Everything

0.1.0 Jan 8, 2020 · release notes

Refinery 0.1.0 adds bytes support to xor, multi-output in rex, and ProofPoint V3 in urlguards

└──▷ GET THIS VERSION

$ git clone --branch 0.1.0 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.1.0

└──▷ USE IT

Instantiate an XOR unit using a raw byte string key directly in code

python

xor(B's3cr3t')

Adds bytes type support for unit arguments, enabling instantiation like xor(B's3cr3t') to decrypt with a byte string key
The rex unit now applies an arbitrary number of transformations to each match and returns results as separate outputs
The urlguards unit now supports ProofPoint V3 guarded URLs

3 details in Everything

0.0.5 Dec 22, 2019 · release notes

Refinery 0.0.5 adds ppxml unit, carve-pe for PE files, winreg with --nonce param on salsa/chacha

└──▷ GET THIS VERSION

$ git clone --branch 0.0.5 https://github.com/binref/refinery.git
# already have the repo? check out this version:
$ git checkout 0.0.5

└──▷ USE IT

Extract carved PE files from raw binary input.

shell

refinery carve-pe < input.bin

Read Windows registry hives from a file path.

shell

refinery winreg -i system.hiv

Adds the ppxml unit to pretty-print XML data.
Adds the carve-pe unit to extract carved PE files from binary streams.
Adds the winreg unit to read Windows registry hives, adding a dependency on python-registry.

3 details in Everything

└──▷ BREAKING ON UPGRADE

!The salsa and chacha units now require the --nonce parameter instead of --iv.

refinery

Summary

How refinery helps

Languages

Features

What refinery offers you

release 23 total

example

release 19 total

example

release 17 total

example

release 15 total

command line

example

release 14 total

example

release 8 total

example

release 10 total

example

release 7 total

example

release 11 total

release 7 total

example

release 9 total

example

release 8 total

example

release

example

release

example

release 7 total

example

release

example

release

release

example

release

release

release

example

release

command line

example

release

command line

example

release

example

release

release

example

release

release

release

example

release

release

example

release

example

release

command line

release

release

example

release

example

release

example

release

example

release

example

example

release