Build log · Month 6 of 12
Huddle: Real-Time Is Hard
self-hosted team chat on Matrix + LiveKit. Federated messaging, 3-level notifications, read receipts, screenshare, and the distributed-systems problems that don't exist until they do.
- Day 151
Huddle month. Self-hostable team chat: Matrix for messaging, LiveKit for voice/video/screenshare. "Self-host" is the whole pitch — your messages live on infra you control, not mine. For teams who won't put internal chat on someone else's box, that's the feature.
- Day 152
Why Matrix, not my own chat protocol? Federated, E2E-encrypted, battle-tested messaging is a decade of work I'm not redoing. I stand on Matrix and spend my effort on the client. Don't reinvent the protocol; reinvent the product on top of it.
- Day 153
Why LiveKit for calls: WebRTC is a swamp of NAT traversal, TURN servers, codec negotiation. LiveKit is an open-source SFU that handles it, self-hostable, so it fits the "your infra" promise. Build the 20% that's your product; adopt the 80% that's solved.
- Day 154
Three-level notifications (all / mentions / nothing, per channel) is the feature I most underestimated. It's a matrix of preference × channel × presence × do-not-disturb. Getting "notify me for @mentions but mute this noisy channel" right is deceptively deep.
- Day 155
Notification bug that taught me humility: a user muted a channel and still got pings, because mentions were evaluated before the mute check. Mute must be the outermost gate — check "is this muted" first, then everything else. Precedence is a design decision.
- Day 156
🧵 Thread: what "just add read receipts" actually costs.
- Every user's read position is per-room state that syncs across all their devices.
- "Read" on phone must mark read on laptop — instantly, or the badge counts lie.
- Privacy: some want to read without broadcasting it. Receipts need an off switch.
- Scale: in a 500-person room, naive receipts are 500 writes per message. Batch and debounce.
- One line in the UI, a distributed-consistency problem underneath.
- Day 157
Read receipts shipped, and the hard part wasn't showing them — it was the unread count. Cross-device, the badge must reflect the latest read position from any device. I sync a read marker, not a per-device counter. The marker is truth; the badge is derived.
- Day 158
Presence (online/away/offline) is a liar's game. Naive presence spams updates every few seconds and melts your server. I debounce, coalesce, and treat presence as best-effort. Nobody's workflow should depend on a green dot being real-time accurate.
- Day 159
Screenshare via LiveKit works, and the gotcha was mobile permissions — iOS screen recording needs a broadcast extension, a separate process. Fiddly, platform-specific, no way around reading the docs carefully. Real-time media is where "cross-platform" leaks most.
- Day 160
Halfway through the year. Huddle is the most technically ambitious app — federated, real-time, self-hostable — and, say it with me, almost no users. Building distributed systems is a flex. Convincing a team to switch chat tools is a completely different, harder sport.
- Day 161
Offline-first for chat is subtle: messages sent offline must queue, send on reconnect, and not duplicate if the send half-succeeded before the connection dropped. Idempotency keys per message. "Did that actually send?" should never be a question the user has to ask.
- Day 162
Message ordering across a federated network is genuinely hard: two servers, clock skew, a message that arrives "before" one sent earlier. Matrix uses a DAG of events for exactly this. I mostly trust the protocol — this is precisely the wheel I was right not to reinvent.
- Day 163
Typing indicators: trivial to show, expensive to do well. High-frequency, ephemeral, pointless to persist. I send them as fire-and-forget events, never stored, auto-expiring. Not all data deserves durability. Some should be forgotten fast.
- Day 164
Self-hosting UX is a paradox: the people who want it are technical, but setup still can't be a nightmare or nobody deploys it. I'm writing a one-command deploy (containerized Matrix + LiveKit + Huddle). "Own your data" dies if owning it takes a weekend of yaml.
- Day 165
E2E encryption means key management, and key management means: what happens when a user loses their device? Cross-signing and key backup exist, but "verify your new session" is where encrypted chat apps lose normal people. Security that's unusable gets turned off.
- Day 166
🧵 Thread: the "real-time" features that are secretly eventual-consistency.
- Read receipts: sync a marker across devices. Not instant, and that's fine.
- Presence (green dot): best-effort, debounced. Never build a workflow on it.
- Typing indicators: fire-and-forget, never stored, auto-expiring.
- Delivery: at the mercy of APNs/FCM and OS battery policy. Design for delay.
- The art is choosing where to fake real-time and where you genuinely can't.
- Day 167
Notification delivery on mobile is at the mercy of APNs/FCM and OS battery policy. A "real-time" chat can have its push delayed to save battery, and you can only design for it. I show a "last synced" signal rather than pretend delivery is instant.
- Day 168
Rate limiting saved Huddle from itself. A misbehaving client (mine, in testing) hammered the sync endpoint in a loop and I watched the server sweat. Every endpoint now has a rate limit. In real-time systems the abusive caller is sometimes you.
- Day 169
The "channel or DM" data model question ate two days. I unified them: a DM is just a room with two members and a flag. One code path, not two. Special-casing DMs would have doubled the surface area of every feature. Collapse the model when you can.
- Day 170
Threading (replies to a message) is where chat UIs go to die. Flat is simple but noisy; threaded is organized but easy to miss. I went with lightweight threads that also surface in the main timeline, so a reply isn't a graveyard. Visibility over tidiness.
- Day 171
Search across encrypted messages is a real puzzle: the server can't read them, so search happens client-side over decrypted local history. That means on-device indexing. Privacy and convenience are in tension, and I chose privacy. Server search would defeat E2E.
- Day 172
Huddle's honest state: it works, it's self-hostable, it does the hard real-time stuff correctly, adoption is [N] teams. Team chat has a brutal cold-start problem — one person can't use it, you need the whole team. Network-effect products are the hardest to distribute.
- Day 173
Month 6 lesson: real-time and federated are "adopt the protocol, build the product" problems. Matrix and LiveKit did the impossible parts; my value is the client, the notification model, the self-host UX. Knowing what NOT to build is the senior skill.
- Day 174
Next month: knowledge graphs, the quiet superpower under CalDesk and Aether. The nutrition "healthier swaps" graph and the DeFi token graph are the same idea — model the relationships, not just the entities. Some of my favorite engineering in the fleet.
- Day 175
Six months. Half a year in public. Build machine, wallet, verification spine, retention loop, App Store fluency, real-time system. The engineering half of the bet is basically proven. The distribution half is barely begun. That asymmetry IS the story of this year.