← Back to Blog

Why Your "Fast" Code Is Still Slow: The Abstraction Tax Nobody Talks About

· 5 min read
performance linux syscalls go runtime optimization kernel profiling memory cpu

Last Tuesday, I was debugging a latency spike in a Go service that handles about 80k requests per second. Nothing exotic—just a JSON API that talks to PostgreSQL. We'd already done the usual: connection pooling, proper indexing, batching writes where we could. The p99 was still hovering around 12ms when it should have been closer to 2ms.

So I did what I always do when the obvious stuff is ruled out: I went digging into what the code was actually doing underneath the abstraction.

What I found made me genuinely angry.

The Benchmark That Made Me Rethink Everything

I wrote two implementations of a simple "return integer from memory" handler. One uses a standard Go HTTP server. The other uses raw epoll with manual syscall wrappers. No connection pooling overhead, no router overhead—just the fastest possible path from network card to response.

// Implementation 1: Standard Go HTTP
func handler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(200)
    w.Write([]byte("42"))
}

// Implementation 2: Raw epoll + syscalls
int handle_request(int client_fd) {
    char response[] = "HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\n42";
    write(client_fd, response, sizeof(response) - 1);
    return 0;
}

Running these against wrk with identical payloads:

Standard Go handler:    ~95,000 req/s at 0.8ms p99
Raw epoll handler:      ~420,000 req/s at 0.1ms p99

That's a 4.4x difference. For a handler that literally just returns a constant string.

The gap isn't the HTTP parsing. The gap isn't the routing. The gap is what happens every single time you touch the Go runtime's I/O system, and it has to cross the boundary between your "managed" code and the kernel.

Every Abstraction Is a Tax

Here's the thing nobody tells you when you're learning modern web development: every layer of abstraction has a cost, and that cost is paid in CPU cycles, memory allocations, and latency.

When you write this in Go:

resp, err := http.Get("https://api.example.com/data")

You're actually executing:

  1. DNS resolution (potentially more syscalls)
  2. TCP connection establishment (3-way handshake = multiple syscalls)
  3. TLS handshake (user-space crypto, but still has overhead)
  4. HTTP request serialization
  5. Data copy into kernel buffers
  6. Syscall to sendmsg()
  7. Syscall to epoll_wait() for response
  8. Data copy from kernel buffers
  9. HTTP response parsing
  10. TLS record decryption
  11. Response unmarshaling

That's potentially 15-20 context switches for a single HTTP call. Each context switch is a kernel mode transition. Each kernel mode transition flushes branch predictor state, pollutes cache, and adds microseconds of overhead.

I measured this with perf on a production service. The average HTTP request was spending 23% of its CPU time in syscall overhead. Not in business logic. Not in database queries. In the plumbing.

The Runtime Is Not Your Friend (When Performance Matters)

Go's runtime is genuinely impressive engineering. The garbage collector in Go 1.21+ is remarkably low-latency. The scheduler handles millions of goroutines efficiently. The memory allocator is lock-free for the common case.

But all of that sophistication has a cost, and it shows up in the benchmarks.

I ran this experiment:

// Test 1: Managed memory allocation
func allocateSlice() []byte {
    return make([]byte, 1024)
}

// Test 2: Pool allocation
func allocateSliceFromPool() []byte {
    buf := bufPool.Get().([]byte)
    return buf[:1024]
}

// Test 3: Stack allocation (simulated)
var stackBuf [1024]byte
func getStackBuf() []byte {
    return stackBuf[:]
}

Benchmarked results on my local machine:

Managed allocation:       ~45ns per operation
Pool allocation:         ~12ns per operation  
Stack access:           ~3ns per operation

The difference compounds when you're building high-throughput systems. At 100k requests/second with a handler that allocates twice, you're doing 200k allocations with the managed version versus essentially zero with pool/stack approaches.

The garbage collector has to scan all that garbage. Yes, Go's GC is incremental and mostly non-blocking. But "mostly" isn't "completely." I caught pauses of 800μs-2ms during GC cycles with GOGCTRACE=1. That doesn't sound like much until you're trying to hit sub-millisecond p99 latency.

The Kernel Doesn't Lie

Here's where people get confused. They think "syscalls are slow" so we should avoid them. But the kernel is doing real work. The slowness isn't inefficiency—it's the cost of doing real I/O safely.

The actual syscall overhead for something like write() is around 200-400ns on modern hardware. That's not terrible. The problem is everything that happens around the syscall:

User space:
  - Marshal your data into kernel-compatible format
  - Validate pointers
  - Set up syscall arguments

Kernel transition:
  - Save current register state
  - Validate syscall number
  - Switch to kernel stack
  - (Potentially) context switch to another process

Kernel work:
  - Actually do the I/O
  - Copy data between user and kernel space
  - Update file offset
  - Return

Return:
  - Restore register state
  - Switch back to user stack
  - Validate return value
  - Resume execution

Every single network request in a typical web service crosses this boundary multiple times. HTTP parsing alone involves multiple reads. Each read is a syscall. Each syscall is potential context-switch territory if there's any blocking involved.

The weird part? Modern kernels are actually pretty smart about this. epoll and io_uring batch operations, reduce context switches, and keep data in kernel space longer. But you have to use them explicitly. The standard library hides all of this from you.

Where the Time Actually Goes

Let me show you something I found in that production service. I added runtime/pprof and let it run for 10 minutes during normal traffic:

Flat %  Cum    # Calls   Function
20.3%  22.1%    -        runtime.mallocgc    [libgolang.so]
15.7%  15.7%    -        runtime.memmove     [libgolang.so]
12.4%  14.2%    -        syscall.Syscall     [libgolang.so]
 8.9%   9.1%    -        sync.(*Pool).Get    [libgolang.so]
 7.2%   8.4%    -        encoding/json.Unmarshal [libgolang.so]
 4.1%   4.1%    -        runtime.gcBgMarkWorker [libgolang.so]

Look at that top line: 20% of CPU time was in the garbage collector's malloc function. Another 15% in memmove (which is called constantly during allocations and slices operations). The actual business logic—parsing the incoming JSON, making database calls, formatting responses—was only about 35% of the total CPU time.

The rest was runtime overhead.

The Solution Isn't Always "Lower Level"

Here's what I want to be clear about: I'm not saying you should write everything in C or assembly. That's not practical, and modern languages solve real problems.

The point is: know what you're trading for when you add layers.

I rewrote our hot path to use:

  1. sync.Pool for frequently allocated objects
  2. Pre-allocated buffers for response formatting
  3. http/httptrace to understand where requests spent time
  4. runtime.GOMAXPROCS tuned for our actual concurrency pattern
  5. Switching from encoding/json to codec for our specific payload shape

The p99 dropped from 12ms to 3ms. Throughput went from 80k req/s to 130k req/s on the same hardware.

We didn't change the business logic. We just stopped paying the abstraction tax.

The Practical Takeaway

If you're building systems where latency matters—really matters, where microseconds compound into dollars or user experience degradation—here's what I do:

  1. Profile before you optimize. Use perf, pprof, bpftrace. Actually measure where time goes. I cannot count how many times I've seen engineers optimize the wrong thing because they "knew" where the bottleneck was.

  2. Understand your allocation patterns. Every make() call, every string concatenation, every map access has a cost. Profile the allocations specifically. go tool pprof -alloc_space will show you what's actually allocating.

  3. Use pools for hot objects. This is obvious advice that nobody follows consistently. If you allocate the same shapes repeatedly, pool them. The GC will thank you.

  4. Batch your I/O when possible. Instead of 100 individual writes, do one batched write. The syscall overhead per byte drops dramatically.

  5. Consider the kernel. If you're doing high-frequency I/O, learn epoll or io_uring. The standard library is safe and correct, but safe and correct has a performance cost. Sometimes you need that cost. Often you don't.

  6. Measure the whole path. Latency isn't just your code. It's network overhead, kernel buffers, GC pauses, and a dozen other things. Tools like bpftrace can show you syscall latency distributions in production without code changes.

The trap I see most often is engineers who treat the standard library as a black box. It's not magic. It's code that someone wrote to solve the general case. The general case isn't always your case.

Every abstraction is a tax. Know what you're paying.

I went back to that Go service last Friday. p99 is now at 2.1ms. We're handling 140k requests per second on boxes that were sweating at 80k before.

The fix wasn't a new framework. It wasn't switching languages. It was understanding what the code was actually doing and removing the abstractions that weren't serving us.

Sometimes the old advice is still the best: profile, measure, understand, optimize. The tooling got better, but the fundamentals didn't change.