slate: Turning a Phone into a Stream Deck for macOS - One Command, a Pluggable Backend, and the Trade-offs Behind It

Introduction

I wanted a phone-as-Stream-Deck for my Mac that worked on Android. Every option I found was either iOS-only, or paid and routed through a vendor cloud, or showed generic icons instead of my real Mac apps. None of them were the thing I actually wanted: a grid of buttons on the phone already in my hand, talking straight to my Mac over the same Wi-Fi, with no account in the middle.

So I built it. slate is two apps and a contract between them. A React Native app (one codebase, iOS and Android) renders the deck and builds commands. A macOS menu-bar helper receives them and performs the real side effect - launch an app, run a Shortcut, send a keystroke, switch Spaces, control media, run a macro. Between them is a WebSocket carrying JSON over your LAN.

That sounds simple, and the happy path is. The interesting part - the part worth writing down - is everything around the happy path: keeping three implementations of one protocol in lockstep, making a phone reliably find a Mac on a noisy network, synthesizing input on a macOS that actively resists it, and being honest about the security gap I have not closed yet. This is how it works and why it is shaped the way it is.

The One Decision Everything Else Follows From

Before any UI, I made one call that the rest of the system inherits: the app never touches OS logic. It constructs a semantic Command - a small, declarative description of intent - and gives it to a Transport. The transport's only job is to carry that command to a backend that knows how to run it. The app does not know, and does not care, whether the backend is a Swift process on the same machine or something else entirely.

This is not abstraction for its own sake. It pays off immediately in three ways. The protocol becomes the real product - a typed contract I can test in isolation. The backend becomes swappable: the Transport interface (setHandlers / connect / disconnect / send / teardown) is small enough that a future SshTransport could translate the same Command into a shell string over SSH, no app changes required. And it forces every platform-specific decision - keyboard layouts, focus stealing, process spawning - to live on the Mac, where it belongs, instead of leaking into the phone.

Everything below is a consequence of that one decision.

How a Tap Becomes an Action

Here is the full path of a single button press, traced end to end through the real code.

The tap handler is just onPress={() => sendCommand(button.action)}. The button's action is a Command that was validated by Zod when the deck loaded, so by the time it is sent it is already known-good. sendCommand gates on auth, not the socket - if the device is not paired, it fails fast with a clear error rather than dropping a command into a half-open connection.

If paired, the command gets wrapped in an envelope and sent:

function commandExecuteMessage(command: Command): Message {
  return {
    v: 1,
    id: Crypto.randomUUID(),
    reId: null,            // a fresh request has nothing to refer back to
    type: "command.execute",
    payload: command,
  };
}

The helper reads one WebSocket text frame, decodes it, checks the auth gate, and runs the command. It replies with command.result, setting reId to the request's id so the phone can correlate the response. The phone fires a success or error haptic and logs failures. One tap, one round trip, a real thing happens on the Mac.

The Wire Contract: One Zod Schema, Three Languages

The protocol is the single source of truth, and it is written once in Zod. Everything else - the JSON Schema, the Swift Codable types, the tests - is checked against it.

The heart of it is the Command, a discriminated union keyed on kind:

export const CommandSchema = z.discriminatedUnion("kind", [
  z.object({ kind: z.literal("launch_app"), app: z.string() }),
  z.object({ kind: z.literal("activate_app"), bundleId: z.string() }),
  z.object({ kind: z.literal("quit_app"), bundleId: z.string() }),
  z.object({ kind: z.literal("run_shortcut"), name: z.string(), input: z.string().optional() }),
  z.object({ kind: z.literal("run_applescript"), script: z.string() }),
  z.object({ kind: z.literal("run_shell"), script: z.string() }),
  z.object({ kind: z.literal("keystroke"), key: z.string(), modifiers: z.array(ModifierSchema) }),
  z.object({ kind: z.literal("media"), action: MediaActionSchema }),
  z.object({ kind: z.literal("space"), direction: DirectionSchema }),
  z.object({ kind: z.literal("app_switch"), direction: DirectionSchema }),
  z.object({ kind: z.literal("macro"), steps: z.array(MacroStepSchema) }),
]);

Eleven kinds, all shipped. There is a deliberate subtlety here: a macro is a list of steps, and each step is a base command, never another macro. I keep two unions for this - a BaseCommandSchema of the ten non-macro kinds, and CommandSchema which is the base plus macro. A macro cannot nest a macro, which keeps the type non-recursive and the executor a simple loop. The members are single-sourced so the two unions can never drift apart.

Every frame on the wire shares an envelope:

export const PROTOCOL_VERSION = 1;

const envelopeBase = {
  v: z.literal(PROTOCOL_VERSION),
  id: z.uuid(),
  reId: z.uuid().nullable(), // correlates a response to its request id; null on a fresh request
};

On top of that sit 21 message types (9 from app to helper, 12 back). Correlation is by id and reId rather than a strict RPC framework, which leaves room for unsolicited pushes - state.update and pair_pending arrive with reId: null because the helper, not the phone, initiated them.

Keeping three implementations honest

The phone speaks Zod-validated TypeScript. The Swift helper has a hand-written Codable port. A JSON Schema sits in the repo for tooling. Three representations of one protocol is three chances to drift, and drift is the kind of bug that passes every unit test and then silently drops a message in production. So I built a parity loop that fails CI the moment they disagree.

The generator runs the Zod schema through z.toJSONSchema and, in --check mode, fails the build if the committed JSON Schema has drifted. A shared fixtures/messages.json holds named frames bucketed into inbound, outbound, rejected, unknownType, and a couple of edge buckets. The TypeScript test asserts that safeParse accepts or rejects each bucket exactly as labeled. The Swift test loads the same fixture file, decodes the inbound frames, re-encodes the outbound ones, and diff-checks them against the fixtures by dotted key-path - comparing shape, not values, so the two languages must agree on the structure of every message.

There is even one deliberate asymmetry, encoded in a tsOnlyRejected bucket: a non-UUID id is rejected by Zod on the phone but accepted leniently by Swift. The phone is the boundary that enforces UUID format; the helper does not re-litigate it. Writing that asymmetry down as a test is the difference between a documented decision and a latent bug.

Transport: Plain WebSocket, and a Reconnect State Machine

The transport is plain ws://, not wss://, carrying JSON. JSON because it is human-readable and easy to debug on both ends, and because keeping the payload semantic lets the keyboard-layout dependency live on the Mac (more on that below) instead of being baked into the wire. I will come back to the security cost of plain ws:// - it is real, and it is the next thing I am fixing.

The harder engineering here is reconnection. A phone moves between networks, sleeps, drops Wi-Fi, and reconnects constantly, and React Native fires WebSocket callbacks asynchronously in ways that make naive reconnect logic spawn duplicate sockets. The transport is a hand-rolled state machine built around a generation counter as its core re-entrancy guard - every connect and disconnect bumps the generation, and any callback that fires for a stale generation simply no-ops.

const HEARTBEAT_MS = 5_000;   // app-level ping cadence
const STALE_MS = 10_000;      // no pong in this window -> drop
const BACKOFF_MIN_MS = 1_000;
const BACKOFF_MAX_MS = 30_000;

A few decisions worth calling out. The heartbeat is a JSON ping/pong at the application layer, not a WebSocket protocol ping - stock React Native cannot send protocol-level pings, so I send my own and drop the socket if no pong lands within ten seconds. Before closing a socket I nil out all its handlers (detachSocket), because otherwise RN's asynchronous onclose would re-enter the reconnect path against a newer generation and spawn a second socket. Backoff is exponential from one second to thirty. And a single app-lifetime network-state subscription reconnects immediately when connectivity returns, superseding any pending backoff timer. The throughline of the whole codebase is the zero-leak rule: every listener, timer, and socket has a matching teardown.

Finding the Mac: Three Layers of Discovery

Before any of that, the phone has to find the Mac at all. There is no cloud rendezvous, so discovery is a local-network problem, and it turned out to be the single most finicky part of the project. I ended up with three layers, tried in order.

The fast path probes the last-known host directly before doing anything else, because a single clean connection reliably answers where a wide sweep often starves. Next is mDNS - the phone browses for _slate._tcp, which the helper advertises. If both of those come up empty, it falls back to scanning the whole /24, opening a WebSocket to each of the 254 hosts and waiting for a hello_ack. And manual host:port entry is always there as the floor.

Two war stories live in this section. First, the subnet scan originally ran at concurrency 24 and it was worse than running at 10. The reason was not the subnet or the IP math - it was Wi-Fi radio saturation. Roughly 250 dead hosts, each holding a connection slot open for the full timeout, starved the one real helper's handshake. Trimming concurrency to 10 and leaning on the last-known-host fast path made the common case instant and the cold case reliable.

Second, on the helper side I advertise over Bonjour using the low-level dns-sd C API (DNSServiceRegister) rather than the modern NWListener.Service. The modern API is simply broken for this on macOS 15.4+ (Apple bug FB14321888); the service never reliably appears. The C API works, at the cost of having to remember that the port goes out in network byte order. A NWPathMonitor re-advertises whenever the Mac's own IP changes, so a phone can rediscover a helper that moved networks.

The Helper: A Node Proof-of-Loop, Then a Swift Menu-Bar App

There are two helpers in the repo, and that is on purpose. I wrote a Node helper first, purely to prove the loop - WebSocket server, Bonjour, decode a command.execute, run open -a, send a result. It implements exactly one command (launch_app) and advertises all other capabilities as false. Its entire job was to validate the architecture - connect, hello, ack, heartbeat, reconnect, execute - cheaply, before I committed to any Swift. Once the loop was proven, the Swift menu-bar app became the real backend, and the Node helper stayed as a minimal reference. The Swift launch_app path is even written to match the Node version's proven open -a behavior.

The Swift helper is a MenuBarExtra app (no Dock icon), running an NWListener with NWProtocolWebSocket. It accepts a single active connection at a time - a ConnectionRegistry actor enforces newest-wins, closing the previous connection when a new one arrives. Each connection re-arms its receive only after the current message is fully handled, so per-connection state stays strictly ordered.

The interesting part is how each command kind becomes a real macOS action, because macOS does not make this easy.

A few of these earned their complexity the hard way:

• activate_app retries because macOS fights it. On macOS 26 (Tahoe), the anti-focus-stealing behavior makes a single activate() call unreliable, so the executor activates, checks whether the app actually became frontmost, and retries up to three times before reporting "focus denied."
• keystroke keeps the layout dependency on the Mac. The wire carries a human-readable key token like "a" or "left"; a KeyTokens map turns it into the platform's virtual key code. Arrow keys additionally carry the secondary-fn and numeric-pad device flags, without which WindowServer shortcuts like Mission Control will not fire.
• app_switch is not Cmd+Tab. Synthesizing Cmd+Tab is timing-flaky, so instead it enumerates the regular running apps, sorts them deterministically by bundle id, finds the frontmost app's neighbour, and activates it directly.
• media splits by permission. Volume changes go through osascript and need no special permission; play/pause/next/previous are synthesized as HID media keys via NSEvent, which does require Accessibility.

App icons are their own small pipeline. The helper asks NSWorkspace for the icon the user actually sees - which composites the asset catalog, masks, and .icns fallback exactly as Finder would - draws it into a 256px bitmap, and base64-encodes the PNG. Crucially, it sends one icon per WebSocket frame, because a single large batched frame OOMs Android's WebSocket implementation. Each icon carries an iconVersion equal to the app bundle's modification time, so the phone's cache key only changes when the app is actually updated.

The App: Decks, Buttons, Gestures, and Live Icons

The phone app is Expo SDK 56 on React Native 0.85, with the New Architecture mandatory (Expo Go cannot load it). State is a single Zustand store, sliced by concern - connection, pairing, deck, apps, discovery, logs. It persists to MMKV with an AsyncStorage fallback, and re-validates the persisted blob with Zod on rehydrate, so a corrupt deck array falls back to a seeded default instead of crashing the app. The auth token deliberately never enters the store - it lives in expo-secure-store (Keychain on iOS, Keystore on Android), with a process-lifetime cache so the reconnect path stays synchronous.

The data model is a simple hierarchy, all Zod-typed:

const IconRefSchema = z.discriminatedUnion("kind", [
  z.object({ kind: z.literal("appIcon"), bundleId: z.string() }), // real Mac icon
  z.object({ kind: z.literal("emoji"), value: z.string() }),
  z.object({ kind: z.literal("symbol"), name: z.string() }),      // MaterialCommunityIcons glyph
  z.object({ kind: z.literal("glyph"), name: z.string() }),       // lettermark fallback
]);

const DeckButtonSchema = z.object({
  id: z.string(),
  position: z.object({ col: z.number(), row: z.number() }),
  label: z.string().optional(),
  icon: IconRefSchema,
  action: CommandSchema,
  gestures: GestureMapSchema.optional(),
  color: z.string().optional(),
});

A Deck has pages, a page is a grid of buttons (default 4 columns by 6 rows), and a button binds an icon and a primary command. The four icon sources mean a button can show a real Mac app icon, an emoji, a vector glyph, or a generated lettermark.

Gestures are where the touch experience lives, and they are tuned to stay fast. The grid uses a one-finger horizontal pan to change pages and a two-finger pan to change decks - two fingers for deck nav specifically because a one-finger long-press is already reserved for entering edit mode, and the two would fight. Per button, a tap fires the primary action, with optional long-press, double-tap, and four-way swipe slots each bound to their own command. The key optimization: a button with no double-tap and no swipe skips React Native Gesture Handler entirely and takes a snappy direct-press path. Only buttons that actually need the richer gestures pay for them.

The feature I am happiest with is auto-switching decks to follow the frontmost app. The phone subscribes to a foregroundApp state topic (gated on the helper advertising the liveState capability), the helper pushes a state.update whenever you switch apps - which needs no special permission, since foreground tracking is public - and the app finds the deck whose autoProfile.matchBundleId matches and switches to it. Open your editor, your editor deck appears; switch to your browser, the browser deck slides in. You bind it from a simple app picker in the deck settings.

Pairing and Auth, Brute-Force Resistant by Design

A device on your network should not be able to drive your Mac just by connecting. Pairing is a one-time 6-digit code shown in the helper's menu bar, with a 120-second TTL. Confirm the code once and the helper issues a per-device bearer token (32 random bytes); after that the phone authenticates silently with the token on every reconnect.

The pairing flow is built to resist brute force, and the details matter:

• The failure counter and lockout live at actor scope, not per-connection, so an attacker cannot reset their progress by opening a new socket or spamming pair_request. A fresh request reuses the still-active code rather than minting a new one.
• After 5 wrong confirmations it locks out, with an exponentially growing window - 30 seconds, doubling, capped at 300. The lockout is global across all connections.
• The helper never sends the code over the wire. The pair_pending message carries only a countdown; the code is shown on the Mac's screen and typed on the phone.

There is a standalone regression test (brute-force-pairing.test.mjs) that proves the lockout is reached, cannot be reset by pair_request, and holds globally. Every privileged message - command.execute, apps.list, apps.icon, subscribe.state - sits behind a requireAuth gate; only the handshake and the heartbeat pass unauthenticated. Revoking a device from the menu deletes its token and drops its live socket immediately, and the phone's next reconnect cleanly falls back to needing a new pairing.

One asymmetry I like: the phone stores its token in the OS secure store, but the helper stores the issued tokens in a plain 0600 JSON file, not the Keychain. The token is a bearer secret the helper hands out and checks against LAN peers - it is not a user credential to hide from the machine's own owner, and a flat JSON map is simpler to enumerate and revoke than Keychain items.

What's Next: Closing the Encryption Gap

I want to be precise about the current security posture, because it has a real gap. Today slate is local-network only, paired, and token-authenticated - but it is not yet end-to-end encrypted. The transport is plain ws://, which means the bearer token travels in the clear. On a trusted home network that is fine. On a hostile Wi-Fi it is open to man-in-the-middle and replay. The honest rule for now is: keep the helper on networks you trust.

The obvious question is "why not just use WSS?" I looked hard at it and rejected it. Stock React Native's WebSocket exposes no TLS configuration hooks, so getting WSS with a pinned self-signed certificate would mean writing and maintaining bespoke native WebSocket modules on both iOS and Android - a large amount of platform-specific native code for the same end guarantee I can get at the application layer with portable cryptography.

So the fix is an application-layer encrypted channel over the same ws://, modeled on a Noise-style handshake (XX with a pre-shared key):

The design: ephemeral X25519 keys for forward secrecy, HKDF-SHA256 to derive session keys, and ChaCha20-Poly1305 for the authenticated encryption. The 6-digit pairing code is mixed into the key derivation as a pre-shared key, so a man-in-the-middle who never saw the code cannot derive the session key, and the very first authenticated frame fails to decrypt. It fails closed, with no human fingerprint comparison required. The phone side uses the audited pure-JS noble libraries plus expo-crypto for its CSPRNG - zero native modules - and the helper uses CryptoKit. The helper's long-term identity key gets pinned at pairing time as the trust root. And of course the new sec_* and enc messages go through the same three-language parity loop, with a shared crypto-vectors.json so the JS and Swift AEAD implementations are proven to interoperate byte for byte.

Two more things on the roadmap. Discovery will get faster with UDP broadcast on Android (a small custom Expo Module) and NWBrowser on iOS - iOS raw broadcast is blocked without a paid Apple Developer account and the multicast entitlement, which is the real constraint there. And the Mac helper will get notarized once a Developer ID is in place; until then it ships self-signed, and the README walks through the one-time quarantine removal.

Conclusion

slate works because of one decision made before any code: the app builds a semantic command and hands it to a pluggable transport, and the backend translates that command into a real action. Everything good about the system falls out of that. The protocol became a testable product I keep honest across three languages. The platform's worst behaviors - focus stealing, flaky Cmd+Tab, Android's fragile WebSocket - all got contained on the side that owns them. And because nothing is platform-locked at the core, one React Native codebase drives the same Mac from both iOS and Android, over your own network, with no cloud and no account.

It is not finished. The encryption gap is real and it is the next milestone, designed and waiting to be built. But the shape is right, and the shape was the hard part. If you want to try it or read the source, it is all there.

References

• slate - source on GitHub
• slate - releases (DMG + Android APK)
• Zod - TypeScript-first schema validation
• The Noise Protocol Framework
• noble - audited JavaScript cryptography
• RFC 8439 - ChaCha20 and Poly1305
• RFC 7748 - Elliptic Curves for Security (X25519)
• Apple CryptoKit
• expo-secure-store
• React Native - the New Architecture
• Elgato Stream Deck - the inspiration