The first assembly function that works feels like a magic trick. The second one — the one that segfaults three instructions in — teaches you something more useful: most of the difficulty isn't the instruction set. It's the set of unwritten rules about registers, flags, and memory that nobody mentions when they hand you a listing.
None of the mistakes below are exotic. They show up in nearly every first attempt at hand-written x86-64, and each one has a boring, reliable fix.
1. Clobbering Registers the Caller Still Needed
Every calling convention splits the register file in two: registers you may destroy freely, and registers you must restore exactly as you found them. On System V (Linux, macOS, BSD), rbx, rbp, r12 through r15 are callee-saved. Everything else — rax, rcx, rdx, rsi, rdi, r8 to r11 — is fair game.
Beginners grab whatever register looks free, and the bug shows up somewhere else entirely: a caller's loop counter quietly changes value, or a pointer you were holding turns into garbage after a call you thought was harmless. The fix is a habit, not a technique. Before you write a line, decide which registers hold inputs, which hold your results, and which are scratch. If you borrow a callee-saved register, save it on the stack at entry and restore it on every exit path, including error paths.
2. Violating the Calling Convention in Smaller Ways
The register list is only part of the contract. Argument order, return value, and stack setup all matter. On System V, the first six integer arguments arrive in rdi, rsi, rdx, rcx, r8, r9 and the return value leaves in rax. On Windows x64 the first four go in rcx, rdx, r8, r9, and the caller reserves 32 bytes of shadow space the callee is allowed to write into.
Two failures are especially common. First, writing a function that follows one convention and calling it as if it followed the other, which usually works right up until it doesn't. Second, assuming a register survives a call — the callee may legally destroy every caller-saved register, so anything you care about must live in memory or in a callee-saved register before the call instruction.
3. Misreading the Flags
Conditional jumps read flags, and flags are set by whatever ran last. That single fact accounts for a lot of silent wrong answers.
- cmp operand order. cmp a, b computes a minus b and discards the result, keeping only the flags. Swap the operands and every relative jump flips meaning. AT&T syntax puts the source first; Intel syntax puts the destination first. Pick one and stay consistent across a file.
- Signed versus unsigned jumps. ja and jb are unsigned comparisons; jg and jl are signed. Using the signed form on pointer arithmetic is a classic source of strange behavior.
- inc and dec do not touch CF. Code that expects a carry flag to survive an increment is broken.
- test reg, reg sets ZF and SF from the value and clears CF and OF. It is the idiom for "is this zero or negative," and it does not modify the register.
- mov, lea, push, and pop set no flags at all. A conditional jump placed after one of them reads whatever was left over from an earlier instruction.
4. Ignoring Alignment
Two alignment rules cause most of the grief. The stack must be 16-byte aligned at the moment of a call, and SIMD loads and stores want the alignment their instruction promises.
On System V, you enter a function with rsp equal to 8 modulo 16, because the call pushed a return address. Push an even number of registers and you are still off by eight; push three registers and subtract eight more bytes of locals, and your next call into library code may fault inside an aligned move you never wrote. The reliable recipe is to push rbp, set rbp to rsp, then subtract a multiple of 16 for locals. For data, use movups or vmovdqu when you are unsure of alignment, and reserve movaps and vmovdqa for addresses you have actually checked.
5. Operand Size and Sign-Extension Slips
Writing to a 32-bit register zeroes the upper 32 bits of its 64-bit parent, but writing to an 8-bit or 16-bit register does not. mov eax, 1 leaves rax equal to 1; mov ax, 1 leaves whatever was in the high half of rax still there. When you load a byte or a word and then use it in a 64-bit comparison or as part of an address, say what you mean: movzx for unsigned values, movsx or movsxd for signed ones.
Division deserves its own warning. The 64-bit div and idiv read a double-width dividend from rdx and rax. Zero rdx before an unsigned divide, and cqo before a signed one, or you get a divide error rather than the answer you expected.
6. Writing to Memory You Do Not Own
A stray offset is more destructive in assembly because nothing checks it. Local variables sit below rbp, saved registers sit beside it, and the return address sits just above it. A loop that writes one byte too far into a stack buffer can rewrite the address the function will eventually return to — which tends to produce a crash far away from the actual mistake.
The 128 bytes below rsp are the red zone, and a leaf function may use them without adjusting rsp. The moment you call another function, that space is no longer yours. Treat every pointer argument as untrusted until you have confirmed its size, and prefer a named constant for each stack offset over a bare number repeated in twenty places.
7. Debugging by Guesswork
The most expensive beginner habit is changing one instruction, rerunning the program, and hoping. Assembly hands you total visibility; use it. Assemble with debug information, break in gdb or lldb, and single-step while watching the registers and flags after each instruction. Write the same logic in C, compile it with optimization and ask for assembly output, then read what the compiler did. It is rarely the shortest solution, but it is almost always a correct one, and comparing the two teaches more than any tutorial.
A Short Pre-Flight Checklist
Before calling a function finished, walk through this list once. It takes two minutes and catches most of what is above.
- Write the intended prototype as a comment: argument registers, return register, and which registers the function preserves.
- Confirm the stack pointer is 16-byte aligned at every call site.
- Trace each conditional jump backwards to the instruction that actually set the flags you are testing.
- Check every byte and word load for an explicit zero- or sign-extension.
- Verify that every stack offset is inside the frame you reserved.
- Run the function under a debugger at least once, stepping, rather than only reading the output.
Get those right and the instruction set stops being the hard part. What remains is the same thing that makes any low-level code worth writing: knowing exactly what the machine is doing, one instruction at a time.
Photo: Daniil Komov / Pexels
