Callfuscated: Reverse Engineering a Stack-Based VM Protected by MBA, Opaque Predicates and Call-Based Jumps

A detailed technical write-up examines the HackTheBox challenge Callfuscated, which combines virtualization, mixed Boolean-arithmetic expressions, opaque predicates and junk instructions. The author first converts call-pop sequences into direct jumps by proving that register r8 is never read, then removes the resulting NOPs to obtain clean code. The resulting binary reveals a classic stack-based virtual machine whose handlers invoke heavily obfuscated functions. Triton is used to lift these functions into AST form, after which the CoBRA solver simplifies the MBA expressions into the original operations: addition, subtraction, multiplication, XOR, OR and AND. With semantics recovered, the author captures the bytecode at runtime, writes a C translator that emits readable operations, compiles the output and obtains a clean decompiled routine that checks the flag.

HabrTranslated from Russian

Read at source

The HackTheBox challenge Callfuscated presents one of the most layered crackmes published on the platform, combining virtualization, Mixed Boolean-Arithmetic (MBA), opaque predicates and junk instructions inside a single binary.

Initial analysis shows hundreds of functions that appear to receive arguments through mysterious arg_* variables. Disassembly reveals that every call is immediately followed by a pop r8 instruction; the return address is discarded and only r8 is clobbered. Unicorn emulation of the entire binary confirms that r8 is never subsequently read, proving that each call-pop pair is a hand-written jump.

A lightweight Python script using Capstone walks the code, replaces the call opcode (0xE8) with jmp (0xE9) and overwrites the pop r8 bytes with NOPs. After patching, Ghidra produces a readable main function containing the characteristic dispatch loop of a stack-based virtual machine.

Each handler invokes a separate function protected by MBA. The author employs Triton to symbolically execute these routines and extract an AST for the return value in RAX. The resulting expressions, sometimes exceeding 7 000 characters, are fed to the CoBRA MBA solver, which reduces them to the original operations: addition, subtraction, multiplication, bitwise AND, OR and XOR. Two handlers that receive random values as arguments simplify to expressions that ignore those arguments, confirming the presence of opaque predicates.

With opcode semantics recovered, the author uses Unicorn again to dump the bytecode at the point where it has been fully initialized. A small C emitter translates the bytecode into ordinary C statements that operate on a 256-element stack. The generated file is compiled with optimizations and loaded into a decompiler, yielding a clean routine whose final comparison against zero determines whether the flag is correct.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Malware & Botnets

Yoba Campaign Uses Signed ChatGPT and Claude Binaries for DLL Sideloading and XMRig Deployment
uCreator Spyware Infiltrates Modified Telegram, WeChat and GCash Builds on Android
Researchers Deploy Cowrie Honeypot Disguised as NVIDIA H100 Server to Study SSH Botnet Tactics