Control Protocol

Verbs: Other machines

Verbs: Other machines: part of the Control Protocol of TUIOS.

This page is one part of docs/protocol.md of the tuios repository. The Control Protocol page lists every part.

stream-pane

Stream one pane as bytes. A client that is not tuios uses it to show a pane in its own terminal emulator, such as a phone app.

Params:

  • session (optional).
  • window (required): the window id or name of the pane.
  • from_seq (optional int): the seq that the client got to on an earlier stream of this pane.
  • boot_id (optional): the boot_id of the stream that from_seq came from.
  • lease_cols, lease_rows (optional ints): hold a size lease from the start. See The size lease.
  • session_id (optional): the session by its id, which a rename does not change. It wins over session. The reply carries it.
  • fit (optional bool): make the lease a viewer's. See The size lease.
  • viewer (optional object): the terminal the stream shows the pane in: bg, fg (#rrggbb), palette (sixteen comma-separated #rrggbb), kitty and sixel (bool). While no TUI client shows the session, the daemon answers the pane's OSC 10, 11 and 4, DA1 and kitty graphics queries from the newest viewer's report. A client that attaches pushes its own.

Request:

{"id": 1, "verb": "stream-pane", "params": {"session": "work", "window": "3f2a9c1e", "from_seq": 48211, "boot_id": "9f2c41d07a3e8b65"}}

Response, one line:

{"id": 1, "result": {"type": "pane_stream", "mode": "resume", "seq": 48211, "cols": 80, "rows": 24, "boot_id": "9f2c41d07a3e8b65", "session": "work", "window": "3f2a9c1e-...", "title": "zsh"}}
  • mode is resume or snapshot. It is resume only when boot_id is this daemon's and the daemon still holds the output after from_seq. Then the first frames are output from from_seq. Otherwise it is snapshot, and the first frame is S.
  • seq is the stream position where the first frame starts.
  • cols and rows are the size of the pane at seq. In resume mode, resize your emulator to them when they are different.
  • window is the window id.
  • input_offsets is true when the daemon takes J frames. An older daemon leaves it out.

After the response line, the connection carries binary frames in both directions. A frame is one type byte, a 4-byte payload length (big-endian), and the payload. All numbers are big-endian. The connection does not take JSON again.

Skip a frame type that you do not know. Read its length, then read and drop that many bytes. The daemon can add a frame type in a later release, and a client that stops on an unknown type breaks then. The daemon sends a new frame type only when the client can ignore it safely.

The daemon charges each snapshot to its memory budget for client reads, until the S frame is sent. The charge is 160 bytes for each cell of the screen and of 500 rows of history, and at most the whole budget. When the budget has no room for 2 seconds, stream-pane fails with busy. When the stream needs a new snapshot and the budget has no room, the stream ends with an X frame that starts with busy:. Connect again with from_seq.

Frames from the daemon:

TypePayloadMeaning
S (0x53)u64 seq, u16 cols, u16 rows, bytesA snapshot. Reset the emulator to cols by rows and write the bytes into it.
O (0x4F)u64 seq, bytesOutput. Write the bytes into the emulator. seq is the stream position after the last byte.
R (0x52)u64 seq, u16 cols, u16 rowsThe pane changed size at seq. Resize the emulator.
E (0x45)JSON {"code", "message"}, and accepted on busyThe daemon refused an I, J or L frame, or released a lease that the client did not renew (code lease_expired). The other codes are the verb error codes. The stream continues.
X (0x58)UTF-8 textThe stream ended: exited N when the pane's program exited with status N, killed N when signal N ended it, closed when its window closed first, the daemon stopped, or refused: ... when the caller may no longer read the pane. The daemon then closes the connection.

Frames from the client:

TypePayloadMeaning
I (0x49)bytesInput for the pane, as typed. The client encodes the keys.
J (0x4A)u64 offset, bytesInput, as I, that starts at this stream's input byte offset. Typed only when offset is the number of input bytes the stream took.
L (0x4C)u16 cols, u16 rowsHold a size lease. 0 0 releases it.

The snapshot bytes paint the pane on a fresh xterm-compatible emulator: up to 500 rows of history above the screen, every cell with its colours and attributes, soft wraps, and the cursor with its position, shape and visibility. A palette colour stays a palette index, and a truecolour stays a truecolour. When a program uses the alternate screen, the bytes paint the shell's screen first and then switch to the alternate screen. The bytes also set the scroll region, the character sets, the pen, the kitty keyboard flags, modifyOtherKeys, and the DEC modes that change the input a client must send: application cursor keys, the keypad mode, bracketed paste, focus reports, mouse tracking and its encodings, and alternate scroll. They do not set left and right margins, origin mode, insert mode (IRM), newline mode (LNM), the saved cursor or protected cells.

The daemon takes the snapshot from its own emulator under the lock that reads the emulator's position. So the snapshot shows exactly the stream up to its seq, and the output frames continue from that seq.

Every O frame starts where the one before it ended. A client keeps the last seq it wrote into its emulator and sends it as from_seq when it connects again. The daemon keeps the last 64 KiB of each pane's output. When the client is too slow and the daemon cannot keep its output, the daemon continues from those 64 KiB, or sends a new S frame when they do not reach back far enough. The seq also moves past some bytes that the daemon does not send: a kitty graphics frame that a newer frame of the same image replaced while the client was behind.

An I frame passes the checks that send-text passes: the link policy needs write, the pane grants of the caller, and a restricted connection. An L frame passes the checks that resize passes. A refused frame gets an E frame, and the stream continues. A frame from the client larger than 64 KiB ends the stream. Send a longer paste as more than one I frame.

The daemon writes I frames into the pane in order, on its own goroutine. A pane that does not read its input does not delay L frames. Up to 256 KiB of input, in up to 4096 frames, waits for the pane, so a client may send one frame per key and a burst is never refused while the pane reads. When the pane has not read that much, the next I frame gets an E frame with code busy and accepted, the number of input bytes this stream took so far, all typed or waiting, in order. Every I frame after it is refused the same way until the pane has read what waits, so nothing is typed out of order. Send again from byte accepted. The daemon logs each run of refusals.

A frame the client sent before it saw the refusal can reach the daemon after the pane has read what waited, and an I frame then is typed, after the bytes that were refused. A client that resends uses J frames instead, when the reply has input_offsets: true: each one names the input byte it starts at, and the daemon types it only at the offset it expects, accepted. A J frame at any other offset gets E with busy and accepted, nothing of it is typed, and the refusals of a full queue are not changed by it. So a resend from accepted is typed once, and a frame from before the resend never is. I and J frames count toward the same offset, and a stream may send both. A daemon without input_offsets refuses a J frame as an unknown type: send I frames to it.

The daemon checks the caller again each second, as it checks a new stream-pane. When a pane loses its read grant, or the link policy no longer allows list, the stream ends with an X frame.

Link capability: list for the verb. write for I and L frames.

Errors: invalid_params when window is missing or a lease size is not between 1 and 4096. window_not_found and session_not_found as for every verb.

The size lease

The stream does not change the size of the pane or of the session. The client shows the pane at the size that the pane has.

A lease holds one pane at a size, while the stream is open. Each dimension is the smaller of the lease and the size that the session's clients ask for. So a lease never makes a pane larger. While the lease is held, a resize from any client is limited to the lease. When the stream releases the lease with L 0 0, or the connection closes, the pane goes back to the size that the clients asked for last. Other panes and the session size do not change. An attached tuios client shows the smaller pane in the same rectangle.

A stream that came over a link is the exception. When the link drops, its lease stays for 15 seconds after the connection closes. A viewer that opens the stream again in that time holds the pane at the same size, so the program in the pane does not redraw twice. When the far side closes the stream on purpose, tuios stdio-proxy tells the daemon first (see closes under What a linked machine may do here), and the lease ends at once.

When two streams hold a lease on one pane, the smaller lease applies in each dimension.

A stream opened with fit holds a viewer's lease. While no TUI client shows the session, the pane takes the size of the largest viewer's lease, which can make it larger than the size its clients asked for. When the last viewer's lease ends, the pane goes back to the size its clients asked for. While a client shows the session, a viewer's lease only holds the pane smaller, as any lease does. The daemon decides under the pane's lock, and decides again on every attach, detach and disconnect.

Leases resize a pane at most once every 100 ms. A lease that comes sooner applies when the 100 ms end, and the pane takes the newest lease.

A lease lasts 30 seconds. To keep it, send the same L frame again every 10 seconds. When the client sends no L frame for 30 seconds, the daemon releases the lease within one more second. It then sends an E frame with code lease_expired, and the pane goes back to the size that the clients asked for. A new L frame sets the lease again.

The renewal bounds a lease that a lost client holds. A phone that drops off the network sends nothing, and ssh and TCP can keep its connection open for hours. A ping from the daemon does not find that sooner: its write goes into the buffers of the proxy, ssh and TCP, and does not fail until they are full. The stream itself ends when the connection closes, or when a write to the client blocks for 10 seconds. Until then, the stream holds only its place in the pane output. It never holds back the pane.

attach-presence

Give this connection the person's human_nonce without an attach. A phone that only answers the Inbox uses it.

Params: session (optional). Without session, the nonce is for all sessions.

Request:

{"id": 1, "verb": "attach-presence"}

Response:

{"id": 1, "result": {"type": "presence", "human_nonce": "294e0a278581acd0f07ff40561c52bf7", "client_id": "client-1791531791506586672"}}

respond, reply-approval, answer-ask, dismiss-attention and every other verb that takes human_nonce accept this nonce. A presence made with session acts only in that session, and a presence made with no session acts in every session. See Nonce scope. The other rules are the same as for the nonce of an attach:

  • The caller must be outside every pane of this daemon. Over a link, the stream must come on the link-human socket, so the hub must open it with {"human":true}. Otherwise the call fails with forbidden.
  • A verb that uses the nonce must come over the same kind of connection: a local socket, a plain link or a link-human stream.
  • When the kernel gives the process ids, the verb must come from the same process. All streams of one tuios stdio-proxy come from the same process, so a phone can send the nonce on any stream of that link.

The presence is not an attach. It does not change the session size, it does not count as a client that shows or focuses a pane, and it gets no broadcasts. So request-approval holds a prompt for the Inbox and does not answer viewed. The connection continues to take verbs. The presence ends when the connection closes, or when the connection calls restrict-connection. A second call replaces the nonce.

Link capability: list. The verbs that use the nonce need their own capability, for example respond for reply-approval.

register-push

Register a phone's Web Push subscription. From then on, the daemon pushes Inbox items to the phone. The phone needs no connection to tuios. Its push service, or its UnifiedPush distributor, wakes it.

Params:

  • endpoint (required): the push service's address for the phone. It must be https. It can be http only when its host is localhost or a loopback or private IP address, and [notify.webpush] allow_insecure is true. A loopback or private host, given as localhost or an IP address, needs allow_insecure also with https. A link-local, multicast or unspecified IP address is refused. The daemon checks a host name when it connects: it does not connect to an address that the endpoint could not name. A push does not follow a redirect.
  • p256dh (required): the phone's public key, the uncompressed P-256 point (65 bytes, first byte 0x04), base64url. Padding is optional.
  • auth (required): the phone's authentication secret, 16 bytes, base64url.
  • device (required): a name for the phone, 1 to 64 bytes of printable text. A second call with the same name replaces the phone. Over a link, a call with the name of a registered phone fails with forbidden.
  • kinds (optional): the Inbox kinds to push. The default is approval, plan, ask and question.
  • human_nonce (required): the nonce of an attach, or of an attach-presence made with no session, from the same process.

Request:

{"id": 1, "verb": "register-push", "params": {"endpoint": "https://push.example.net/s/abc", "p256dh": "BKZ91bcH6n7KanE3PpyzUohdcLmK8Nze7jWdSbKRc9I4ZnFUdH6u_p5rNRnpzq_6CCKbzDwOprt1CjQPXjNbY04", "auth": "GT7vKYlZkqz3Qw7cTswShg", "device": "pixel", "human_nonce": "294e0a278581acd0f07ff40561c52bf7"}}

Response:

{"id": 1, "result": {"type": "push_registered", "device": "pixel", "kinds": ["approval", "plan", "ask", "question"], "replaced": false, "vapid_public_key": "BCM4cML1lw6X1BHn3-nOS6-QtjYUql5FDx-etpdO8RHCTTipUMWpGY46YBB4bLwbc31yGx_O9LQVABj-B0Dqvp0", "machine": "studio"}}

vapid_public_key is the daemon's VAPID public key (RFC 8292), the uncompressed P-256 point, base64url. A phone that subscribes through a push service that takes an application server key gives it this key. Get it first with list-push. The daemon makes the key once and keeps it in the state directory, at push/vapid.pem, with mode 600. The phones are in push/subscriptions.json, with mode 600.

Who may call it:

  • Only the person. A missing nonce, a nonce that does not verify, and any call from a process inside a pane fail with not_human.
  • A phone gets the Inbox of every session, so it is an act in no one session (see Nonce scope). A presence made for one session covers only that session. With such a nonce, register-push, list-push and remove-push fail with not_human, and the message says the nonce is for another session.
  • Over a link, the link needs the respond capability. Without it the call fails with forbidden. A link can not replace a registered phone.
  • A restricted connection cannot call it.

Each registration opens an Inbox item of kind errored, with no session, named phone NAME. Its summary says who registered the phone (this machine, or the link from a named machine) and whether it replaced a phone of that name. A remove-push over a link opens the same item. A remove-push on this machine closes it.

At most 16 phones. A bad endpoint or key fails with invalid_params.

What the phone gets

Each push is one HTTP POST to endpoint with these headers:

HeaderValue
Content-Encodingaes128gcm
Content-Typeapplication/octet-stream
TTL3600 for the open and the close of approval, plan, ask and question, else 120
Urgencyhigh for an open of approval, plan, ask or question, else normal
Topic32 base64url characters, the same for the open and the close of one item. A push service that holds an open replaces it with the close.
Authorizationvapid t=JWT, k=VAPID_PUBLIC_KEY

The JWT is ES256, with the claims aud (the endpoint's origin), exp (12 hours from the send) and sub ([notify.webpush] subject, by default https://tuios.dev/push). The push service reads the JWT, so the default sub names no machine.

The TTL of an item that waits for the person is one hour. The item stays answerable while it is open, often after the hook's hold ends, and a phone that sleeps can get a push some minutes late. A stale open does not outlive its item: the close has the same Topic and the same TTL, and a push service that still holds the open replaces it with the close.

The body is encrypted for the phone by RFC 8291, as one record of the aes128gcm content coding (RFC 8188). The body is:

salt (16 bytes) | rs (4 bytes, big endian, 4096) | idlen (1 byte, 65) | keyid (65 bytes) | ciphertext

keyid is the sender's one-time public key, as_public, an uncompressed P-256 point. The phone keeps ua_private (its private key), ua_public (the p256dh it registered) and auth_secret. To decrypt, with HMAC-SHA-256 as HMAC(key, data):

  1. ecdh_secret = ECDH(ua_private, as_public), the 32-byte x coordinate.
  2. PRK_key = HMAC(auth_secret, ecdh_secret).
  3. IKM = HMAC(PRK_key, "WebPush: info" || 0x00 || ua_public || as_public || 0x01).
  4. PRK = HMAC(salt, IKM).
  5. CEK = HMAC(PRK, "Content-Encoding: aes128gcm" || 0x00 || 0x01), first 16 bytes.
  6. NONCE = HMAC(PRK, "Content-Encoding: nonce" || 0x00 || 0x01), first 12 bytes.
  7. Decrypt ciphertext with AES-128-GCM, key CEK, nonce NONCE, no associated data. The 16-byte tag is at the end.
  8. Remove the zero bytes at the end, then the 0x02 before them, which marks the last record. tuios pads the JSON with zero bytes up to 256, 512, 1024, 2048 or 3072 bytes, the smallest that holds it, so the length of a push tells little about its text.

Steps 2 and 3 are HKDF (RFC 5869) with auth_secret as the salt. Steps 4 to 6 are HKDF with salt as the salt. The ntfy UnifiedPush distributor gives the body to the app base64-encoded.

The plain text is JSON, at most 3 KiB, with no HTML escapes:

{"v":1,"type":"open","id":"1","kind":"approval","session":"demo","window":"f9ab67c1-6226-4335-8c7e-ea784574a3ef","harness":"claude-code","name":"Terminal f9ab67c1","summary":"approve Bash: npm test","request_id":"69a569a6ab65115e","options":["once","deny"],"machine":"studio"}
FieldWhat it is
v1. A change that breaks a reader gets a new number.
typeopen or close.
idThe Inbox item's id. A close has the id of its open.
kindThe Inbox kind.
session, windowThe pane the item is about.
hostThe linked machine the item is on, empty for this one.
harness, name, summaryWhat the Inbox row shows.
riskThe risk rules the call matched, as the Inbox item has them.
request_id, optionsA held prompt's request and its answers, for reply-approval.
machineThe name of the machine that sent the push.
truncatedtrue when text was cut to fit 3 KiB.

An empty field is left out. A payload that does not fit 3 KiB is cut, not dropped: first the summary, then the risk, then each text field. The id, request_id and options are never cut, because an answer gives them back. A close carries only v, type, id, kind, session, window, host and machine.

The daemon sends an open when an item opens. It sends the open again, with the same id, when the item gets a held prompt, or loses one: then request_id, options or risk change. A phone replaces the notification with the same id. It sends a close when the item closes, also when you answer it at the desk. A phone that does not know the id ignores the close.

Delivery never blocks the daemon. Each phone has its own queue. A push that gets no answer, 429 or a 5xx is tried again after 1, 4 and 15 seconds. When the push service answers 404 or 410, the daemon removes the phone, logs it, and opens the Inbox item phone NAME to say so. [notify] enabled = false stops the opens. A close still goes, so a phone can remove what it shows. max_per_hour limits the opens to each phone. An item from another machine is not pushed: that machine pushes to its own phones.

Link capability: respond.

tuios lives only at tuios.dev and github.com/Gaurav-Gosain/tuios. tuios has no X or Twitter account. An account that offers a tuios token or presale is not this project.

On this page