TryHackMe: x86 Architecture Overview

Author: Mario Martinez Jr. (ku5e / Gary7) | TryHackMe USA Rank #76 | Top 1%

Difficulty: Easy

Topics: CPU Architecture, x86 Registers, Memory Layout, Stack Analysis, Malware Analysis Fundamentals

Link: x86 Architecture Overview on TryHackMe

Answers are redacted within the narrative to allow you to complete the tasks on your own, but a full table of answers is available at the end of this walkthrough.

This room gives you the mental model that makes malware analysis readable. Before you open a binary in Ghidra or step through a sample in x64dbg, you need to know what the CPU is actually doing with its registers and memory. The room covers Von Neumann architecture, x86 registers from EAX down to the segment registers, the four-section memory layout, and the stack. It takes about an hour. If you plan to do any serious reverse engineering, that hour is not optional.

Task 1: Introduction

No questions. The learning objectives are worth reading because they set the scope: CPU components, register types, memory layout, and stack structure from a malware analysis point of view. The room is not trying to replace a full architecture course. It is giving you just enough to make sense of what a debugger is showing you.

Task 2: CPU Architecture Overview

The room bases everything on Von Neumann architecture, which is the foundation for every consumer processor you will encounter. The three CPU components are the Control Unit, the Arithmetic Logic Unit, and the Registers. Memory and I/O devices sit outside the CPU.

The Control Unit fetches instructions from memory using the Instruction Pointer. The ALU executes those instructions. The Registers hold the working data so the CPU does not have to reach out to RAM on every operation.

Technical Deep Dive: The Fetch-Decode-Execute Cycle

The room spends real time on the Fetch-Decode-Execute cycle, and that is the right call. Most people skip straight to assembly without understanding that the CPU is a physical machine doing three distinct steps on every instruction. The Fetch step pulls the instruction from memory at the address in the Instruction Pointer. The Decode step translates the raw bytes into an operation the CPU can act on. The Execute step carries it out.

The Decode step is where things get interesting for malware analysis. “Illegal Instruction” errors happen when the CPU hits bytes it cannot decode as a valid operation. Certain obfuscation techniques work by hiding code inside other instructions, exploiting how the decoder interprets byte sequences depending on where it starts reading. If you have ever seen malware that jumps into the middle of what looks like a normal instruction, that is exactly what is happening.

Answers:

• Where are the code and data for a program stored? [REDACTED]
• What part of the CPU stores small amounts of data? [REDACTED]
• Where are arithmetic operations performed? [REDACTED]

Task 3: Registers Overview

The Instruction Pointer (IP) holds the address of the next instruction. In 32-bit systems it is EIP. In 64-bit systems it is RIP. You will spend a lot of time watching this register in a debugger.

The general-purpose registers are the eight workhorses: EAX/RAX, EBX/RBX, ECX/RCX, EDX/RDX, ESP/RSP, EBP/RBP, ESI/RSI, and EDI/RDI. Each one has a documented purpose: RAX accumulates arithmetic results, ECX counts loops, ESP points to the top of the stack, and so on. The room covers the register hierarchy in detail, showing how EAX breaks down into AX, then AH and AL for 8-bit access. The R8 through R15 registers are 64-bit only.

Technical Deep Dive: Register Pressure in the Real World

The documented register purposes describe what the calling convention intends. The compiler does not always honor that.

In normal code, RAX holds the return value when a function exits, RDI and RSI carry the first two arguments on x86-64 Linux, and ECX counts loops. That is the convention. In practice, when a program is doing heavy computation, like encryption or compression, the compiler runs out of available registers and starts reusing them for whatever fits. You will see RAX used as a temporary counter in the middle of a loop, long before the function returns anything. You will see RDI hold something that has nothing to do with a function argument.

If you are stepping through a malware sample and expecting RAX to always hold a return value, you will misread the code. Read the assembly for what it is doing, not for what the convention says it should be doing.

Answers:

• Which register holds the address of the next instruction? [REDACTED]
• Which register in a 32-bit system is also called the Counter Register? [REDACTED]
• Which registers are not present in a 32-bit system? [REDACTED]

Task 4: Registers — Continued

The EFLAGS register is a 32-bit register where individual bits serve as status indicators. Four flags matter most for malware analysis.

The Zero Flag (ZF) sets to 1 when the result of the last instruction was zero. The Carry Flag (CF) sets when a result overflows the destination register. The Sign Flag (SF) sets when the result is negative or the most significant bit is 1. The Trap Flag (TF) puts the CPU into single-step mode, executing one instruction at a time and raising an exception after each one.

The six segment registers (CS, DS, SS, ES, FS, GS) divide the flat memory space into regions. CS points to the code section. DS, ES, FS, and GS point to data sections. SS points to the stack.

Technical Deep Dive: The Trap Flag in the Wild

The Trap Flag section mentions that malware uses it to detect debuggers. That is not theoretical. It is a standard technique you will hit when analyzing real samples, and if you are not expecting it, it will cost you hours.

GuLoader is the most consistent example. It is a downloader used to deliver ransomware and information stealers, and it is built specifically to survive analysis. The Trap Flag sequence it uses goes like this: push the EFLAGS register onto the stack with PUSHF, OR the value with 0x100 to flip the Trap Flag bit, pop the modified value back into EFLAGS with POPF. The CPU is now in single-step mode. The next instruction executes, and then the CPU immediately raises a SINGLE_STEP exception, which is interrupt 1.

What happens next depends on who catches the exception. In a clean run with no debugger attached, GuLoader’s own Structured Exception Handler catches it, the code continues normally, and the malware knows it is safe. With a debugger attached, the debugger intercepts the exception first because it sees a single-step event and assumes you are tracing through the code. The malware’s internal handler never runs. GuLoader checks whether its own handler fired. It did not, so it knows a debugger is present, and it terminates or redirects execution to junk code.

The Lampion banking malware family uses a variation of this that adds a timing component. It sets the Trap Flag before executing RDTSC, an instruction that reads the CPU timestamp counter. The idea is that a debugger introduces measurable delay. If the timing between the RDTSC and the exception looks too long, Lampion treats it as an automated analysis environment and exits.

Bypassing this takes one of two approaches. The first is patching: find the POPF instruction in memory and overwrite it with a NOP so the Trap Flag never gets set. The second is using a plugin like ScyllaHide in x64dbg, which intercepts the exception handling in a way that makes the malware’s own handler run as expected. ScyllaHide is the faster option when you are processing a lot of samples.

Answers:

• Which flag is used to identify if a program is being run in a debugger? [REDACTED]
• Which flag sets when the most significant bit in an operation is 1? [REDACTED]
• Which segment register contains the pointer to the code section? [REDACTED]

Task 5: Memory Overview

When a program runs on Windows, it does not see the full system memory. The operating system gives it an abstracted view of its own address space. That address space is divided into four sections: Code, Data, Heap, and Stack.

The Code section holds the executable instructions. It has execute permissions so the CPU can run it. The Data section holds initialized constants and global variables that do not change during execution. The Heap holds dynamic memory, created and freed at runtime. The Stack holds local variables, function arguments, and the return address.

The room notes that the section order can vary. It also notes that the Stack is the most relevant section for malware analysis because it controls program flow.

Technical Deep Dive: ASLR and Why Real Addresses Look Nothing Like This

The room uses short, clean memory addresses in its examples. A real 64-bit Windows or Linux system with ASLR enabled looks nothing like that.

Address Space Layout Randomization randomizes the base address of the stack, heap, and executable segments every time a program runs. An address that was 0x00401000 in one execution might be 0x7ff3a4210000 in the next. You cannot rely on a static address in a real exploit or during analysis. Instead, you look for offsets from a known base. If you find a leak that tells you where one module is loaded, you calculate the address of your target relative to that base. Every address in a live binary is an offset from something you had to work to find.

This is why modern binary exploitation is harder than the examples in most beginner rooms. The static addresses in training material are intentional, but they create an expectation that does not hold once ASLR is in the picture.

Answers:

• Does a program have a full view of system memory when loaded? Y or N? [REDACTED]
• Which section contains the code? [REDACTED]
• Which section contains information related to control flow? [REDACTED]

Task 6: Stack Layout

The stack is Last In First Out. The last thing pushed is the first thing popped. Two registers track it: the Stack Pointer (ESP or RSP), which points to the top of the stack and moves with every push and pop, and the Base Pointer (EBP or RBP), which stays fixed as the reference point for the current stack frame.

When a function is called, the Function Prologue sets up the frame: arguments are pushed, the return address is pushed by the CALL instruction, the old Base Pointer is saved, and the Base Pointer is updated to the current Stack Pointer. When the function exits, the Function Epilogue reverses this: the old Base Pointer is restored, the return address is popped into the Instruction Pointer, and control returns to the calling function.

The static site task asks you to arrange stack elements in the correct order. Three things to keep in mind:

First, the CALL instruction pushes the return address before your function does anything. The return address is already on the stack when the function body starts. A lot of people forget it is there.

Second, the stack grows down in memory but you read it top to bottom in the diagram. The last thing pushed is at the top.

Third, when the exercise shows you a PUSH RAX instruction, you are pushing the value in RAX, not the letters “RAX.” Check the specific values in the registers before placing them.

The flag is: [REDACTED]

Technical Deep Dive: Stack Buffer Overflow

The room mentions stack buffer overflow and points you to a separate TryHackMe room for hands-on work. Here is the core concept before you get there.

A stack frame has a predictable layout: local variables sit above the saved Base Pointer and return address. If a local variable is a buffer and the program writes to it without checking the length, you can overflow past the buffer, past the Base Pointer, and into the return address. When the function exits and pops that return address into the Instruction Pointer, the CPU jumps to wherever you wrote, not wherever the program intended.

The cheap routers and smart cameras in IoT devices are the most consistent real-world targets for this. The firmware is almost always written in C, often compiled without modern protections, and the code is full of strcpy and gets calls that do no length checking. A buffer you fill with 200 bytes when the developer expected 32 is frequently enough to control the Instruction Pointer.

Modern systems add three layers of protection. Stack canaries are a value the compiler places right before the return address. If you overflow the buffer, you hit the canary first. The program checks the canary before returning and crashes if it has been changed. NX or DEP marks the stack as non-executable, so even if you jump to shellcode you placed in the buffer, the CPU refuses to run it. ASLR randomizes where everything is loaded, making it harder to know what address to put in the overwritten return address.

ROP chains work around NX/DEP by not injecting new code at all. Instead, you chain together short sequences of existing executable code that already live in the program or its libraries, called gadgets, each one ending in a RET instruction. The RET pops the next address off the stack and jumps to it. You control what is on the stack, so you control the chain. No new code needed, just existing code in a new order.

When you get to the buffer overflow room, the first tool you will use is pattern_create. It generates a unique string where no 4-byte or 8-byte sequence repeats. Send it as input, watch where the program crashes, and the offset of the value in the Instruction Pointer tells you exactly how many bytes it takes to reach the return address. That is the moment it clicks for most people.

Answers:

• What is the flag from the static site? [REDACTED]

Task 7: Conclusion

The room covers Von Neumann architecture, the three CPU components, all eight general-purpose registers with their 64/32/16/8-bit breakdowns, the R8-R15 64-bit extensions, the four status flags most relevant to malware analysis, the six segment registers, the four memory sections, and the stack layout including function prologue and epilogue.

Who This Room Is For

If you have run exploits from Exploit-DB without knowing why they work, you need this room. If you plan to open a binary in Ghidra or IDA Pro, you need this room. If you write C or C++ and want to understand why a segmentation fault happens at the CPU level, this room will make that click.

If your work is web application pentesting, XSS, or SQL injection, you will almost never need to know what RDX is doing. GRC analysts writing SOC 2 reports do not need Little Endian format. Experienced binary exploiters will find this a review at best.

The verdict: if you plan to touch a debugger or write an exploit, take the hour. If you just want to find bugs in web applications or manage a security program, skip it.

Answer Table

Written by Mario Martinez Jr. (ku5e / Gary7) | TryHackMe Profile | ku5e.com/blog