# Huddle: Real-Time Is Hard

> Build log · Month 6 of 12

self-hosted team chat on Matrix + LiveKit. Federated messaging, 3-level notifications, read receipts, screenshare, and the distributed-systems problems that don't exist until they do.

Day 151–175 · 25 posts · Building in public as @caelum0x4

**Day 151** — Huddle month. Self-hostable team chat: Matrix for messaging, LiveKit for voice/video/screenshare. "Self-host" is the whole pitch — your messages live on infra *you* control, not mine. For teams who won't put internal chat on someone else's box, that's the feature.

**Day 152** — Why Matrix, not my own chat protocol? Federated, E2E-encrypted, battle-tested messaging is a decade of work I'm not redoing. I stand on Matrix and spend my effort on the client. Don't reinvent the protocol; reinvent the product on top of it.

**Day 153** — Why LiveKit for calls: WebRTC is a swamp of NAT traversal, TURN servers, codec negotiation. LiveKit is an open-source SFU that handles it, self-hostable, so it fits the "your infra" promise. Build the 20% that's your product; adopt the 80% that's solved.

**Day 154** — Three-level notifications (all / mentions / nothing, per channel) is the feature I most underestimated. It's a matrix of preference × channel × presence × do-not-disturb. Getting "notify me for @mentions but mute this noisy channel" right is deceptively deep.

**Day 155** — Notification bug that taught me humility: a user muted a channel and still got pings, because mentions were evaluated *before* the mute check. Mute must be the outermost gate — check "is this muted" first, then everything else. Precedence is a design decision.

**Day 156** — Thread: what "just add read receipts" actually costs.
  - Every user's read position is per-room state that syncs across all their devices.
  - "Read" on phone must mark read on laptop — instantly, or the badge counts lie.
  - Privacy: some want to read without broadcasting it. Receipts need an off switch.
  - Scale: in a 500-person room, naive receipts are 500 writes per message. Batch and debounce.
  - One line in the UI, a distributed-consistency problem underneath.

**Day 157** — Read receipts shipped, and the hard part wasn't showing them — it was the unread *count*. Cross-device, the badge must reflect the latest read position from *any* device. I sync a read marker, not a per-device counter. The marker is truth; the badge is derived.

**Day 158** — Presence (online/away/offline) is a liar's game. Naive presence spams updates every few seconds and melts your server. I debounce, coalesce, and treat presence as best-effort. Nobody's workflow should depend on a green dot being real-time accurate.

**Day 159** — Screenshare via LiveKit works, and the gotcha was mobile permissions — iOS screen recording needs a broadcast extension, a separate process. Fiddly, platform-specific, no way around reading the docs carefully. Real-time media is where "cross-platform" leaks most.

**Day 160** — Halfway through the year. Huddle is the most technically ambitious app — federated, real-time, self-hostable — and, say it with me, almost no users. Building distributed systems is a flex. Convincing a team to switch chat tools is a completely different, harder sport.

**Day 161** — Offline-first for chat is subtle: messages sent offline must queue, send on reconnect, and *not* duplicate if the send half-succeeded before the connection dropped. Idempotency keys per message. "Did that actually send?" should never be a question the user has to ask.

**Day 162** — Message ordering across a federated network is genuinely hard: two servers, clock skew, a message that arrives "before" one sent earlier. Matrix uses a DAG of events for exactly this. I mostly trust the protocol — this is precisely the wheel I was right not to reinvent.

**Day 163** — Typing indicators: trivial to show, expensive to do well. High-frequency, ephemeral, pointless to persist. I send them as fire-and-forget events, never stored, auto-expiring. Not all data deserves durability. Some should be forgotten fast.

**Day 164** — Self-hosting UX is a paradox: the people who want it are technical, but setup still can't be a nightmare or nobody deploys it. I'm writing a one-command deploy (containerized Matrix + LiveKit + Huddle). "Own your data" dies if owning it takes a weekend of yaml.

**Day 165** — E2E encryption means key management, and key management means: what happens when a user loses their device? Cross-signing and key backup exist, but "verify your new session" is where encrypted chat apps lose normal people. Security that's unusable gets turned off.

**Day 166** — Thread: the "real-time" features that are secretly eventual-consistency.
  - Read receipts: sync a marker across devices. Not instant, and that's fine.
  - Presence (green dot): best-effort, debounced. Never build a workflow on it.
  - Typing indicators: fire-and-forget, never stored, auto-expiring.
  - Delivery: at the mercy of APNs/FCM and OS battery policy. Design for delay.
  - The art is choosing where to fake real-time and where you genuinely can't.

**Day 167** — Notification delivery on mobile is at the mercy of APNs/FCM and OS battery policy. A "real-time" chat can have its push delayed to save battery, and you can only design for it. I show a "last synced" signal rather than pretend delivery is instant.

**Day 168** — Rate limiting saved Huddle from itself. A misbehaving client (mine, in testing) hammered the sync endpoint in a loop and I watched the server sweat. Every endpoint now has a rate limit. In real-time systems the abusive caller is sometimes you.

**Day 169** — The "channel or DM" data model question ate two days. I unified them: a DM is just a room with two members and a flag. One code path, not two. Special-casing DMs would have doubled the surface area of every feature. Collapse the model when you can.

**Day 170** — Threading (replies to a message) is where chat UIs go to die. Flat is simple but noisy; threaded is organized but easy to miss. I went with lightweight threads that also surface in the main timeline, so a reply isn't a graveyard. Visibility over tidiness.

**Day 171** — Search across encrypted messages is a real puzzle: the server can't read them, so search happens client-side over decrypted local history. That means on-device indexing. Privacy and convenience are in tension, and I chose privacy. Server search would defeat E2E.

**Day 172** — Huddle's honest state: it works, it's self-hostable, it does the hard real-time stuff correctly, adoption is [N] teams. Team chat has a brutal cold-start problem — one person can't use it, you need the whole team. Network-effect products are the hardest to distribute.

**Day 173** — Month 6 lesson: real-time and federated are "adopt the protocol, build the product" problems. Matrix and LiveKit did the impossible parts; my value is the client, the notification model, the self-host UX. Knowing what NOT to build is the senior skill.

**Day 174** — Next month: knowledge graphs, the quiet superpower under CalDesk and Aether. The nutrition "healthier swaps" graph and the DeFi token graph are the same idea — model the *relationships*, not just the entities. Some of my favorite engineering in the fleet.

**Day 175** — Six months. Half a year in public. Build machine, wallet, verification spine, retention loop, App Store fluency, real-time system. The engineering half of the bet is basically proven. The distribution half is barely begun. That asymmetry IS the story of this year.

← [Month 5: The App Store Gauntlet](https://ereboslab.com/blog/month-5-the-app-store-gauntlet/) · [Month 7: Knowledge Graphs: Model the Relationships](https://ereboslab.com/blog/month-7-knowledge-graphs-model-the-relationships/) →

[← All build-log months](https://ereboslab.com/blog/)