Skip to content

All writing

A simulator that gives the same answer every time

6 min read

Most system design practice ends at a diagram nobody runs. Systemlab runs yours: you draw the architecture, it simulates the traffic, and the grade comes from what actually happened.

Systemlab is a system design trainer. You get a mission (say, a profile service whose cache holds only 200 keys, so the database sits at 100% and p95 is over a second), you drag components onto a canvas (load balancers, caches, queues, databases) and you press Check. Behind that button is a discrete-event simulator I wrote. It pushes every request through your design one at a time and measures what happens: p95 latency, throughput, error rate, cost.

There are 70 lessons in 12 chapters, and 58 of them are graded by the simulator. So the simulator is the teacher. If it gives a different answer every time you press the button, it's a bad teacher.

Why a grader can't just roll dice

Real systems are noisy, and I wanted the model noisy too. Service times have a long tail. Keys follow a power law, so a few of them get most of the traffic. Retries back off with random jitter. Replicas fail. Take that out and you're teaching people that systems behave like spreadsheets.

But a learner needs two things that plain randomness breaks:

  1. If I change one thing and run again, the difference I see should come from my change, not from the dice.

  2. If I passed today, I should pass tomorrow. If I failed, anyone should be able to replay the exact run that failed.

So the randomness stays, but all of it is seeded. The engine prints this line in the assumptions of every run: "Identical graph order, settings, seed and engine version reproduce identical results, byte for byte."

Three random streams, not one

Every random number in the engine comes from a small xorshift32 generator. There's no Math.random anywhere in it.

typescript
// lib/simulation/distributions.ts
export function createRandom(seed: number): () => number {
  let state = (Math.trunc(seed) >>> 0) || 0x9e3779b9;
  return () => {
    state ^= state << 13;
    state ^= state >>> 17;
    state ^= state << 5;
    return (state >>> 0) / 4294967296;
  };
}

// lib/simulation/index.ts, one stream per kind of randomness
const workloadRandom = createRandom(plan.seed);
const serviceRandom = createRandom(plan.seed ^ 0x85ebca6b);
const chaosRandom = createRandom(plan.seed ^ 0xc2b2ae35);

The important part is that there are three streams. One for the workload: when each request arrives, whether it's a read or a write, which key it asks for, which region it comes from, its cache roll. One for service times: how long each piece of work takes. And one for "chaos": retry jitter and error bursts.

Why bother? With one shared stream, adding a single retry uses up one extra random number, and every request after it gets a different arrival time and key. A tiny change quietly reshuffles the whole workload, and you can't tell whether your change helped.

With separate streams, the traffic is the same whatever you build. The engine actually draws every request's arrival time, key and cache roll up front, schedules them all, and only then starts the clock. A test pins this down: it runs a cache-fronted design, then changes the API's capacity and latency and doubles the database replicas, and checks that the database processed exactly the same number of requests and that every traced request hit or missed the cache in exactly the same way.

The rule kept shaping later features. When I added object storage in engine 2.3, requests needed a payload size, which is one more random draw per request. If every request made that draw, every existing lesson would suddenly be measured against different traffic. So the draw only happens when the workload has objects in it, and the code says why: // Only object workloads draw a size, so a design without objects keeps the v2.1 random stream.

Ties have to break the same way

Seeding isn't enough on its own. A discrete-event simulator is basically a priority queue of "at time t, do this". When two events land on the same instant, which goes first? Leave that to chance and two runs can drift apart even with identical random numbers.

The engine schedules everything through SIM.JS 2.0.3, and its queue breaks ties by insertion order: each event gets an increasing counter, and equal timestamps come out in the order they went in. Since events go in in the same order every run, the whole timeline replays exactly. It's also why that assumptions line says identical graph order: list the same nodes in a different order and you can get a different run. Still reproducible, just a different one.

Running it in a Web Worker

A 30-second lesson at 500 requests per second is 15,000 requests, each hopping through several components, and Check runs that five times. The engine runs in a Web Worker so none of that blocks the editor while you're dragging boxes around.

Determinism made the worker easy. The engine is a pure function: architecture and workload in, result out, no shared state, and it doesn't touch its inputs (the determinism test checks that too). So the worker is 26 lines: one message in, one message out, every response stamped with the engine version.

The one snag was SIM.JS itself. Its npm bundle has a browser entry that writes a global window.Sim, and there's no window inside a worker. So I vendored the library with a small script that exports the scheduler directly and leaves that entry out, without changing any of the scheduling code.

One seed is not enough

Determinism creates its own trap. If grading used one fixed seed, you could tune a design that only passes that exact sample of traffic. Reproducible, and still luck.

So every graded lesson is checked on three seeds: the lesson's own seed, then 123 and 2026. You pass only if every objective holds on all three. Check also runs two extra seeds just for a variance view (min, median and max of p95, throughput and error rate), with the note "A design is a distribution, not a single number." If a seed fails, the results panel shows you that failing run, not the best one.

Here's what that catches, from lesson 14, Hot keys: 500 requests per second, 95% reads, heavily skewed keys over 20,000 profile ids, and a target of p95 at most 110 ms. The reference design (two API replicas at 400 req/s, a keyed cache, a database at 420 req/s) comes in at about 60, 75 and 62 ms on the three seeds.

Now cut every server and database capacity by 5%, so the APIs do 380 and the database does 399. On the lesson's own seed, p95 is 84.9 ms and every objective passes. On seed 123 it's 201.5 ms. On seed 2026 it's 140.8 ms. Same design. It's running right at the edge of the database's capacity, and on some traffic samples the queue builds and the tail blows up. A single seed would have called that a good design. Three seeds call it fragile, which is the actual lesson about hot keys.

And because the engine is deterministic, those aren't flaky numbers. Run it again and you get 84.88, 201.5 and 140.8 again.

The answer key has to pass too

The same rule applies to me. Every lesson ships a hidden reference solution, and the test suite runs each one on all three seeds and fails if any objective misses:

typescript
// lib/curriculum/curriculum.test.ts (trimmed)
it.each(graded)("%s: the reference passes every objective on all assessment seeds", (_, item) => {
  for (const seed of assessmentSeeds(item)) {   // [lesson seed, 123, 2026]
    const result = runSimulation(item.reference, { ...item.workload, seed });
    for (const objective of item.objectives) {
      expect(objectivePasses(result, objective)).toBe(true);
    }
  }
});

When I wrote the references I measured each one on all three seeds and set the targets about 25 to 40% away from the worst result, so that, in the words of the code comment, "no objective passes by a hair". There's a test for the opposite direction too: every starter design (except lesson 1 and the blank-canvas briefs) has to fail at least one objective, so there's always something to fix.

Keeping old grades honest

"Same answer every time" is a promise per engine version. When the model changes, the numbers change, and a design that passed before might not pass now. Every result carries engineVersion (2.3.0 today; v1's engine was 1.1.0), and every saved completion records an assessment version. When v2 rebuilt the engine and the curriculum, I bumped that to 3, and older completions now show "Earlier completion needs review after the v2 curriculum update. Your saved architecture is unchanged." A pass is also tied to the exact design revision it was checked on, so editing after Check clears the green ticks.

Takeaway

If a program grades people, it has to be reproducible before it can be fair, and reproducible on its own isn't fair yet. Seed everything. Give each kind of randomness its own stream so one change doesn't reshuffle the rest. Make tie-breaking explicit. Then grade on more than one seed, so passing means the design works and not that the dice were kind.

The suite is 956 tests now, including one per graded lesson that runs its reference on every seed. You can try it at https://system-design-playground-pi.vercel.app.