THM · Glyph — Reversing a Custom VM

Recovering the bytecode dispatcher of a hand-rolled stack VM and decrypting the flag with a 40-line Python lifter.

May 22, 2025· 7 min read

#thm#re#vm

Triage

The binary loads a blob from .rodata and dispatches on the low nibble of each byte. Twelve handlers — clean.

Each handler maps to a single stack operation. A trivial Python lifter produces readable pseudo-assembly:

PUSH 0x41
XOR
ROR 3
CMP imm

Treat the comparison sequence as a system of equations over GF(2^8). z3 finds the only satisfying input in under a second.