The Queue Is Not the Hard Part

I want to start with a confession. For some time now, I thought event driven architecture was something people put on a slide to sound smart. "We'll make it event driven." everyone nods. Nobody asks what happens when the event gets lost.

Then I had to build notifications. You know the thing. A little bell icon, a red badge, and users who will absolutely tell you about it if the badge says 3 and there are only 2 things in the list.

And I changed my mind. not completely. I still think a lot of people reach for queues way too early. But I now think the producer consumer pattern is one of the few "architecture" ideas that actually pays rent, as long as you respect a handful of rules that nobody tells you about until you break them.

This post is those rules. Or at least my version of them.

First, what are we even talking about

The producer consumer pattern is boring on purpose. Something produces work. something else consumes it. There's a buffer in the middle so the two dont have to be awake at the same time or run at the same speed.

Thats it. That's the whole idea. You've used it if you've ever written a for loop that pulls jobs off a channel in Go, or used a thread pool, or honestly if you've ever left a sticky note on the fridge for a roommate.

Event driven architecture is what happens when you take that idea and stretch it across services. Instead of service A calling service B and waiting, service A says "hey, this happened" and walks away. whoever cares about it picks it up later.

The difference sounds small. It isn't. One is a phone call, the other is leaving a voicemail. And voicemails get lost, get listened to twice, get listened to in the wrong order. you get the idea.

The setup

Picture any app where people do stuff and other people want to know about it. Someone posts. Someone comments. Someone mentions someone. Something gets scheduled, then moved, then cancelled. Every one of those should notify somebody.

The first version almost everyone writes looks like this:

And honestly? For a small app this is fine. Ship it.

The problems show up slowly, then all at once:

• The comment request gets slower every time you add a new kind of notification.
• If the notification insert fails, the comment fails too. Should it? (No. Absolutely not. nobody wants their comment to vanish because the bell icon had a bad day.)
• Once things get split into a few services, each one ends up with its own copy of "who should hear about this" logic. and they disagreed.

That last one is the real killer btw. It's never the performance. its the logic leaking everywhere.

The shape I ended up loving

Here's roughly where I landed. Lots of producers, one queue, one consumer that owns the hard question.

Read it left to right. Each box says what it does in plain words, and the arrows say what's actually moving between them.

The important bit is that the producers on the left never talk to each other and never talk to the notifications table. They just announce. Everything about who gets told lives in that one consumer box in the middle.

Notice the cron job in there. That one took me an embarrassingly long time to see. A scheduled reminder ("your thing starts in 30 minutes") is just another producer. Its trigger is the clock instead of a human. Once I saw it that way the reminder code shrank to almost nothing, because it just drops the same kind of message on the same queue and the consumer does the rest. I was about to write a whole seperate pipeline for it. glad I didn't.

Ok. Now the opinions.

Opinion 1: Producers should be dumb. Like, really dumb.

The producer knows three things: what happened, who did it, and what it touched. That's it. It does not know who should be notified.

This felt wrong at first. The posts service knows about the post! It has the post right there! Why not just look up the followers and send one message per person?

Because then the rule "who hears about a post" lives in the posts service. And the rule "who hears about a cancellation" lives in the scheduling service. And a month later someone asks for a mute button and now you're editing five places. Ask me how I know. (actually dont, I caught it before it got bad, which is why I'm smug about it now.)

So the message looks something like this:

{
  "event": "comment_added",
  "by": "user-12",
  "post": "post-42",
  "comment": "comment-7",
  "key": "comment-7",
  "preview": "looks good to me"
}
 

It's a fact, not a command. "A comment was added." Not "please notify these four people." The consumer turns facts into decisions. One place. one set of rules.

I'll go further: if your event has a recipients array in it, I think you've probably built a really complicated function call, not an event.

Opinion 2: Commit first, then announce

This is the one I'd tattoo somewhere if I were into that. (No, I actually won't. lol)

The producer writes its own data, waits for the commit, then puts the message on the queue. And it does the enqueue in the background, after the response is already on its way back to the user.\

Why this order? Think about the alternatives.

If you enqueue first and the db write fails, you've announced something that didn't happen. Now someone gets a notification for a comment that doesn't exist, clicks it, gets a 404. Lovely.

If you enqueue inside the request and make the user wait on it, a hiccup in the queue fails the user's comment. thats backwards. The comment is the thing they care about. The notification is a side effect.

"But what if step 3 fails, then the notification is lost?" Yes. It is. And I think that's fine for notifications. The source of truth (the comment) is safe, you can replay from it if you really need to. People who insist on exactly once delivery for a bell icon are, in my opinion, solving the wrong problem. If you need real guarantees (payments, say) go read about the transactional outbox pattern, it's the grown up version of this. For a lot of stuff though, commit then announce gets you 99% of the way for like 5% of the effort.

Opinion 3: Your consumer will see the same message twice. Plan for it on day one.

Every queue I've used is "at least once." Not "exactly once." At least. meaning sometimes twice. Sometimes, on a bad day, more.

So every write the consumer does has to be idempotent. Doing it twice must look exactly like doing it once.

The trick I like is giving every event a key. The producer builds it from something that uniquely identifies the real world thing that happened, like the comment's id. The consumer then makes "this person was already told about this key" impossible to write twice. Most databases give you this for free with a unique constraint.

-- example
create unique index one_per_person on alerts (person, key);
 
insert into alerts (person, key, message)
values ('user-3', 'comment-7', 'someone commented on your post')
on conflict do nothing;
 

The first delivery writes the rows. A redelivery writes nothing. And nothing is a correct answer, not an error. that took a mental shift for me. I kept wanting to log a warning when nothing got inserted. Don't. Nothing inserted can mean "already done," or "the only person who'd care is the one who posted," or "the post got deleted in the meantime." All fine.

Small aside: dont use the queue's own message id as your key unless you're really sure it stays the same across retries and across the producer resending. The id of the thing itself (the comment, the booking, whatever) is almost always the better choice. It's stable because the real world is stable. mostly.

Opinion 4: Not every failure deserves a retry

This is where I think most tutorials are just wrong, or at least lazy. They show you

try { 
    process()
} catch { 
    retry() 
} 

and move on.

But there are two very different kinds of failure:

If a message is missing its post, retrying it three times gives you three identical failures and a DLQ entry. The bug is in the producer. No amount of redelivery heals a producer bug. So validate up front, and if its malformed, log it with enough detail to find which producer sent it, then ack it and drop it.

Same goes for some database errors. A not-null violation or a check constraint failure is going to fail the exact same way on attempt two. Retrying just walks the message slowly into the DLQ and makes your DLQ noisy, which means nobody looks at it, which means the DLQ is useless.

The stuff thats actually transient (connection failures, timeouts) is what should throw and get retried.

One more thing. Ack and retry per message, not per batch, unless the failure really affects the whole batch. If you can't even connect to the database, sure, retry the whole batch, nothing got processed anyway. But if message 7 of 10 is broken, don't make 1 through 6 go around again. They're idempotent so it wouldn't break anything but it's wasteful and it makes the logs confusing to read at 11pm.

Opinion 5: The deploy order matters and nobody will warn you

Here's a fun one. On a lot of managed queues, if you deploy a producer and there is no consumer attached yet, the send succeeds. No error. The message just sits there. And after a few days, depending on the provider's retention, it expires. Quietly. Unread.

So the rule is: create the queue, deploy the consumer, then deploy producers. every time. I put this in a comment at the top of every producer's config file because I do not trust future me. (I actually dont write it in at the top of the file, but yes, I dont trust the future me.)

Related gotcha: queue names are often a flat namespace across your whole account. If two different apps both have a queue called notifications, congratulations, they might now be sharing a queue. Put the app name in the queue name. It's ugly. its fine.

Opinion 6: Keep the payload small, because fanout multiplies everything

When one event turns into one row per person, everything in the message gets copied N times. Someone writes a giant essay of a post and it goes out to forty people, and now you've got forty copies of that essay in your notifications table.

So the consumer should trim what it stores. Cut the preview short. Only keep the fields you actually show. The notification row is a pointer plus just enough text to show in a list. If someone wants the full post they click and you go fetch the real post. The notification is a voicemail, not the transcript.

Opinion 7: Read your own writes, not your cache

This one bit me, and its sort of next to the queue stuff, but the queue stuff is what caused it.

Once things are async, your reads are going to race your writes. Someone comments, gets bounced back to the page, the page reads from a cache, comment isn't there yet. Or the like count says 4 when they just made it 5. Everyone thinks the app is broken.

My rule of thumb now: anything a user just changed and immediately looks at, read it fresh. Cache the stuff that's genuinely fine being a minute stale. Going event driven makes this worse, not better, because you've made "eventually" part of the design on purpose. You have to decide where eventually is ok and where it really isn't.

When I would NOT do any of this

I promised opinions, so here's the other side.

Don't go event driven if:

• You need the answer right now. If the caller needs the result to continue, that's a function call or an HTTP request. Wrapping it in a queue and then polling for the result is just a slower function call with extra steps.
• You have one producer and one consumer and they live in the same codebase. Just call the function. seriously. You can always add the queue later, and the "commit then announce" habit makes that easy.
• You can't see what's happening. If you don't have logs that tell you which message failed, why, and who sent it, an async system is a haunted house. Things go bump and you have no idea where. Logging was easily a third of the time I spent on this and I don't regret a minute of it.
• Nobody is going to look at the DLQ. A dead letter queue nobody watches is just a slower trash can.

A rough checklist, for future me

writing this down because I will forget.

• Producer writes its data and commits before enqueueing
• Enqueue happens off the request path, a failure doesn't fail the user's action
• Events describe facts, not recipients
• every event carries a stable key built from your own data
• Consumer writes are idempotent (on conflict do nothing or similar)
• Malformed messages: log and drop, dont retry
• Transient errors: retry, then DLQ
• Ack per message
• Queue created, consumer deployed, then producers
• Queue name has the app in it
• Message trimmed before fanout
• Someone actually looks at the DLQ

Where I've landed

I came into this thinking the queue was the interesting part. Picking one, tuning batch sizes, all that. It's not. The queue is honestly the easy part, it's a managed service, you set a few numbers and forget about it.

The hard part is the agreement between the pieces. What goes in the message, who gets to decide what, what "done" means, what happens on the second delivery and what you do with the messages that will never work. Get those right and the whole thing almost feels boring. Which for infrastructure is the highest compliment I can give.

Anyway. The bell icon works now. the badge says 2 and there are 2 things in the list. I'll take it.