The HackTheBox challenge Callfuscated presents a stripped 64-bit ELF binary that requests a password and prints either Correct or Incorrect flag. Inside the binary lies a heavily obfuscated verification routine built from four complementary techniques that survive static analysis but collapse under concrete execution.
The Obfuscation Arsenal
The protection consists of a custom VM whose bytecode resides in a 586-element array called program[]. A dispatcher reads the current program counter, fetches an opcode, and dispatches to handlers. Every arithmetic primitive is further expanded with Mixed Boolean-Arithmetic expressions. Opaque predicates double the control-flow graph with branches whose outcome is known at compile time. Finally, genuine instructions are wrapped inside thousands of call gadgets of the form pop r8; <insn>; call next, producing the 187,064-instruction count mentioned in the title.
Dynamic Reconnaissance
Because all four layers are execution-dependent, the author recorded a full instruction trace using ptrace with PTRACE_SINGLESTEP on a known input. The resulting log contained 187,064 lines together with register values after each instruction inside the image range 0x401000–0x40d000. A separate memory dumper extracted the VM program array and stack frames directly from the running process.
Building a Faithful Emulator
The trace was replayed inside a custom x86-64 emulator written in Python that parsed objdump output and implemented handlers for the thirty opcodes actually encountered. Two notable bugs were corrected during validation:
- rand() is invoked from four distinct call sites (0x409235, 0x40ad5b, 0x40af37, 0x40b10b) for a total of 192 calls; each return address must be calculated as call-site + 5 rather than assuming a single fixed location.
- Operand-size detection used a naïve substring check that incorrectly treated “DWORD PTR” as containing “WORD”, truncating 32- and 64-bit memory accesses.
After these fixes the emulator matched the original trace line-by-line and reached the final instruction at address 0x40b537 with eax equal to 0xffffffff, exactly as the real binary behaves on an incorrect password.
Recovering the VM Semantics
With a working emulator the author dumped the data stack after every dispatch and decoded the 586 VM instructions. The bytecode implements a simple stack machine whose relevant operations are:
- PUSH imm – push constant
- H2 – addition (used to compute input buffer address 0x40f080)
- DEREF – read input byte
- H5 – multiply accumulator by 256
- H7 – add next byte
- G8 / G3 – XOR with per-group constants
Consequently every four input characters are accumulated into a 32-bit big-endian word. Eight such words are XORed with the following constant pairs:
- (0x0915033a, 0x41414141)
- (0x427d7872, 0x11111111)
- (0x30310a00, 0x55555555)
- (0x2a052e32, 0x5a5a5a5a)
- (0xcff5ecdf, 0xaaaaaaaa)
- (0x1914031e, 0x77777777)
- (0xf6f7c6ad, 0x99999999)
- (0x6c6a524e, 0x33333333)
The resulting 32-bit values must all be zero for the password to be accepted. Solving each equation yields the flag bytes directly: HTB{******_**_***_********_*_**}.
Conclusion
The exercise demonstrates that even an extreme combination of VM-based dispatch, MBA, opaque predicates and call obfuscation remains vulnerable once execution is recorded and replayed. All four techniques ultimately depend on concrete runtime values that a faithful emulator can capture and simplify.
