# Verbs: Other machines

URL: https://tuios.dev/docs/control-protocol/verbs-other-machines

> Verbs: Other machines: part of the Control Protocol of TUIOS.

This page is one part of `docs/protocol.md` of the [tuios repository](https://github.com/Gaurav-Gosain/tuios/blob/f32fbfd7faca14e5260a1d711262cb81c49f68a6/docs/protocol.md). The [Control Protocol](https://tuios.dev/docs/control-protocol) page lists every part.

## stream-pane

Stream one pane as bytes. A client that is not tuios uses it to show a pane in
its own terminal emulator, such as a phone app.

Params:

- `session` (optional).
- `window` (required): the window id or name of the pane.
- `from_seq` (optional int): the `seq` that the client got to on an earlier
  stream of this pane.
- `boot_id` (optional): the `boot_id` of the stream that `from_seq` came from.
- `lease_cols`, `lease_rows` (optional ints): hold a size lease from the
  start. See [The size lease](https://tuios.dev/docs/control-protocol/verbs-other-machines#the-size-lease).
- `session_id` (optional): the session by its id, which a rename does not
  change. It wins over `session`. The reply carries it.
- `fit` (optional bool): make the lease a viewer's. See
  [The size lease](https://tuios.dev/docs/control-protocol/verbs-other-machines#the-size-lease).
- `viewer` (optional object): the terminal the stream shows the pane in:
  `bg`, `fg` (`#rrggbb`), `palette` (sixteen comma-separated `#rrggbb`),
  `kitty` and `sixel` (bool). While no TUI client shows the session, the
  daemon answers the pane's OSC 10, 11 and 4, DA1 and kitty graphics queries
  from the newest viewer's report. A client that attaches pushes its own.

Request:

```json
{"id": 1, "verb": "stream-pane", "params": {"session": "work", "window": "3f2a9c1e", "from_seq": 48211, "boot_id": "9f2c41d07a3e8b65"}}
```

Response, one line:

```json
{"id": 1, "result": {"type": "pane_stream", "mode": "resume", "seq": 48211, "cols": 80, "rows": 24, "boot_id": "9f2c41d07a3e8b65", "session": "work", "window": "3f2a9c1e-...", "title": "zsh"}}
```

- `mode` is `resume` or `snapshot`. It is `resume` only when `boot_id` is this
  daemon's and the daemon still holds the output after `from_seq`. Then the
  first frames are output from `from_seq`. Otherwise it is `snapshot`, and
  the first frame is `S`.
- `seq` is the stream position where the first frame starts.
- `cols` and `rows` are the size of the pane at `seq`. In `resume` mode,
  resize your emulator to them when they are different.
- `window` is the window id.
- `input_offsets` is true when the daemon takes `J` frames. An older daemon
  leaves it out.

After the response line, the connection carries binary frames in both
directions. A frame is one type byte, a 4-byte payload length (big-endian),
and the payload. All numbers are big-endian. The connection does not take
JSON again.

Skip a frame type that you do not know. Read its length, then read and drop
that many bytes. The daemon can add a frame type in a later release, and a
client that stops on an unknown type breaks then. The daemon sends a new frame
type only when the client can ignore it safely.

The daemon charges each snapshot to its memory budget for client reads, until
the `S` frame is sent. The charge is 160 bytes for each cell of the screen and
of 500 rows of history, and at most the whole budget. When the budget has no
room for 2 seconds, `stream-pane` fails with `busy`. When the stream needs a
new snapshot and the budget has no room, the stream ends with an `X` frame
that starts with `busy:`. Connect again with `from_seq`.

Frames from the daemon:

| Type       | Payload                                              | Meaning                                                                                                                                                                                                                                                                          |
| ---------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `S` (0x53) | u64 seq, u16 cols, u16 rows, bytes                   | A snapshot. Reset the emulator to cols by rows and write the bytes into it.                                                                                                                                                                                                      |
| `O` (0x4F) | u64 seq, bytes                                       | Output. Write the bytes into the emulator. `seq` is the stream position after the last byte.                                                                                                                                                                                     |
| `R` (0x52) | u64 seq, u16 cols, u16 rows                          | The pane changed size at `seq`. Resize the emulator.                                                                                                                                                                                                                             |
| `E` (0x45) | JSON `{"code", "message"}`, and `accepted` on `busy` | The daemon refused an `I`, `J` or `L` frame, or released a lease that the client did not renew (code `lease_expired`). The other codes are the verb error codes. The stream continues.                                                                                           |
| `X` (0x58) | UTF-8 text                                           | The stream ended: `exited N` when the pane's program exited with status N, `killed N` when signal N ended it, `closed` when its window closed first, `the daemon stopped`, or `refused: ...` when the caller may no longer read the pane. The daemon then closes the connection. |

Frames from the client:

| Type       | Payload            | Meaning                                                                                                                                 |
| ---------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `I` (0x49) | bytes              | Input for the pane, as typed. The client encodes the keys.                                                                              |
| `J` (0x4A) | u64 offset, bytes  | Input, as `I`, that starts at this stream's input byte `offset`. Typed only when `offset` is the number of input bytes the stream took. |
| `L` (0x4C) | u16 cols, u16 rows | Hold a size lease. `0 0` releases it.                                                                                                   |

The snapshot bytes paint the pane on a fresh xterm-compatible emulator: up to
500 rows of history above the screen, every cell with its colours and
attributes, soft wraps, and the cursor with its position, shape and
visibility. A palette colour stays a palette index, and a truecolour stays a
truecolour. When a program uses the alternate screen, the bytes paint the
shell's screen first and then switch to the alternate screen. The bytes also
set the scroll region, the character sets, the pen, the kitty keyboard flags,
modifyOtherKeys, and the DEC modes that change the input a client must send:
application cursor keys, the keypad mode, bracketed paste, focus reports,
mouse tracking and its encodings, and alternate scroll. They do not set left
and right margins, origin mode, insert mode (IRM), newline mode (LNM), the
saved cursor or protected cells.

The daemon takes the snapshot from its own emulator under the lock that reads
the emulator's position. So the snapshot shows exactly the stream up to its
`seq`, and the output frames continue from that `seq`.

Every `O` frame starts where the one before it ended. A client keeps the last
`seq` it wrote into its emulator and sends it as `from_seq` when it connects
again. The daemon keeps the last 64 KiB of each pane's output. When the client
is too slow and the daemon cannot keep its output, the daemon continues from
those 64 KiB, or sends a new `S` frame when they do not reach back far enough.
The `seq` also moves past some bytes that the daemon does not send: a kitty
graphics frame that a newer frame of the same image replaced while the client
was behind.

An `I` frame passes the checks that `send-text` passes: the link policy needs
`write`, the pane grants of the caller, and a restricted connection. An `L`
frame passes the checks that `resize` passes. A refused frame gets an `E`
frame, and the stream continues. A frame from the client larger than 64 KiB
ends the stream. Send a longer paste as more than one `I` frame.

The daemon writes `I` frames into the pane in order, on its own goroutine. A
pane that does not read its input does not delay `L` frames. Up to 256 KiB of
input, in up to 4096 frames, waits for the pane, so a client may send one
frame per key and a burst is never refused while the pane reads. When the
pane has not read that much, the next `I` frame gets an `E` frame with code
`busy` and `accepted`, the number of input bytes this stream took so far, all
typed or waiting, in order. Every `I` frame after it is refused the same way
until the pane has read what waits, so nothing is typed out of order. Send
again from byte `accepted`. The daemon logs each run of refusals.

A frame the client sent before it saw the refusal can reach the daemon after
the pane has read what waited, and an `I` frame then is typed, after the bytes
that were refused. A client that resends uses `J` frames instead, when the
reply has `input_offsets: true`: each one names the input byte it starts at,
and the daemon types it only at the offset it expects, `accepted`. A `J`
frame at any other offset gets `E` with `busy` and `accepted`, nothing of it
is typed, and the refusals of a full queue are not changed by it. So a resend
from `accepted` is typed once, and a frame from before the resend never is.
`I` and `J` frames count toward the same offset, and a stream may send both.
A daemon without `input_offsets` refuses a `J` frame as an unknown type: send
`I` frames to it.

The daemon checks the caller again each second, as it checks a new
`stream-pane`. When a pane loses its read grant, or the link policy no longer
allows `list`, the stream ends with an `X` frame.

Link capability: `list` for the verb. `write` for `I` and `L` frames.

Errors: `invalid_params` when `window` is missing or a lease size is not
between 1 and 4096. `window_not_found` and `session_not_found` as for every
verb.

### The size lease

The stream does not change the size of the pane or of the session. The client
shows the pane at the size that the pane has.

A lease holds one pane at a size, while the stream is open. Each dimension is
the smaller of the lease and the size that the session's clients ask for. So
a lease never makes a pane larger. While the lease is held, a resize from any
client is limited to the lease. When the stream releases the lease with
`L 0 0`, or the connection closes, the pane goes back to the size that the
clients asked for last. Other panes and the session size do not change. An
attached tuios client shows the smaller pane in the same rectangle.

A stream that came over a link is the exception. When the link drops, its
lease stays for 15 seconds after the connection closes. A viewer that opens
the stream again in that time holds the pane at the same size, so the program
in the pane does not redraw twice. When the far side closes the stream on
purpose, `tuios stdio-proxy` tells the daemon first (see `closes` under
[What a linked machine may do here](https://tuios.dev/docs/control-protocol/verbs-other-verbs#what-a-linked-machine-may-do-here)), and the lease ends at once.

When two streams hold a lease on one pane, the smaller lease applies in each
dimension.

A stream opened with `fit` holds a viewer's lease. While no TUI client shows
the session, the pane takes the size of the largest viewer's lease, which can
make it larger than the size its clients asked for. When the last viewer's
lease ends, the pane goes back to the size its clients asked for. While a
client shows the session, a viewer's lease only holds the pane smaller, as any
lease does. The daemon decides under the pane's lock, and decides again on
every attach, detach and disconnect.

Leases resize a pane at most once every 100 ms. A lease that comes sooner
applies when the 100 ms end, and the pane takes the newest lease.

A lease lasts 30 seconds. To keep it, send the same `L` frame again every 10
seconds. When the client sends no `L` frame for 30 seconds, the daemon
releases the lease within one more second. It then sends an `E` frame with
code `lease_expired`, and the pane goes back to the size that the clients
asked for. A new `L` frame sets the lease again.

The renewal bounds a lease that a lost client holds. A phone that drops off
the network sends nothing, and ssh and TCP can keep its connection open for
hours. A ping from the daemon does not find that sooner: its write goes into
the buffers of the proxy, ssh and TCP, and does not fail until they are full.
The stream itself ends when the connection closes, or when a write to the
client blocks for 10 seconds. Until then, the stream holds only its place in
the pane output. It never holds back the pane.

## attach-presence

Give this connection the person's `human_nonce` without an attach. A phone
that only answers the Inbox uses it.

Params: `session` (optional). Without `session`, the nonce is for all
sessions.

Request:

```json
{"id": 1, "verb": "attach-presence"}
```

Response:

```json
{"id": 1, "result": {"type": "presence", "human_nonce": "294e0a278581acd0f07ff40561c52bf7", "client_id": "client-1791531791506586672"}}
```

`respond`, `reply-approval`, `answer-ask`, `dismiss-attention` and every other
verb that takes `human_nonce` accept this nonce. A presence made with
`session` acts only in that session, and a presence made with no session acts
in every session. See [Nonce scope](https://tuios.dev/docs/control-protocol/versioning#nonce-scope). The other rules are the
same as for the nonce of an attach:

- The caller must be outside every pane of this daemon. Over a link, the
  stream must come on the link-human socket, so the hub must open it with
  `{"human":true}`. Otherwise the call fails with `forbidden`.
- A verb that uses the nonce must come over the same kind of connection: a
  local socket, a plain link or a link-human stream.
- When the kernel gives the process ids, the verb must come from the same
  process. All streams of one `tuios stdio-proxy` come from the same process,
  so a phone can send the nonce on any stream of that link.

The presence is not an attach. It does not change the session size, it does
not count as a client that shows or focuses a pane, and it gets no
broadcasts. So `request-approval` holds a prompt for the Inbox and does not
answer `viewed`. The connection continues to take verbs. The presence ends
when the connection closes, or when the connection calls
`restrict-connection`. A second call replaces the nonce.

Link capability: `list`. The verbs that use the nonce need their own
capability, for example `respond` for `reply-approval`.

## register-push

Register a phone's Web Push subscription. From then on, the daemon pushes
Inbox items to the phone. The phone needs no connection to tuios. Its push
service, or its UnifiedPush distributor, wakes it.

Params:

- `endpoint` (required): the push service's address for the phone. It must
  be `https`. It can be `http` only when its host is `localhost` or a
  loopback or private IP address, and `[notify.webpush] allow_insecure` is
  `true`. A loopback or private host, given as `localhost` or an IP address,
  needs `allow_insecure` also with `https`. A link-local, multicast or
  unspecified IP address is refused. The daemon checks a host name when it
  connects: it does not connect to an address that the endpoint could not
  name. A push does not follow a redirect.
- `p256dh` (required): the phone's public key, the uncompressed P-256 point
  (65 bytes, first byte `0x04`), base64url. Padding is optional.
- `auth` (required): the phone's authentication secret, 16 bytes, base64url.
- `device` (required): a name for the phone, 1 to 64 bytes of printable
  text. A second call with the same name replaces the phone. Over a link,
  a call with the name of a registered phone fails with `forbidden`.
- `kinds` (optional): the Inbox kinds to push. The default is `approval`,
  `plan`, `ask` and `question`.
- `human_nonce` (required): the nonce of an attach, or of an
  [attach-presence](https://tuios.dev/docs/control-protocol/verbs-other-machines#attach-presence) made with no session, from the same
  process.

Request:

```json
{"id": 1, "verb": "register-push", "params": {"endpoint": "https://push.example.net/s/abc", "p256dh": "BKZ91bcH6n7KanE3PpyzUohdcLmK8Nze7jWdSbKRc9I4ZnFUdH6u_p5rNRnpzq_6CCKbzDwOprt1CjQPXjNbY04", "auth": "GT7vKYlZkqz3Qw7cTswShg", "device": "pixel", "human_nonce": "294e0a278581acd0f07ff40561c52bf7"}}
```

Response:

```json
{"id": 1, "result": {"type": "push_registered", "device": "pixel", "kinds": ["approval", "plan", "ask", "question"], "replaced": false, "vapid_public_key": "BCM4cML1lw6X1BHn3-nOS6-QtjYUql5FDx-etpdO8RHCTTipUMWpGY46YBB4bLwbc31yGx_O9LQVABj-B0Dqvp0", "machine": "studio"}}
```

`vapid_public_key` is the daemon's VAPID public key (RFC 8292), the
uncompressed P-256 point, base64url. A phone that subscribes through a push
service that takes an application server key gives it this key. Get it first
with [list-push](https://tuios.dev/docs/control-protocol/verbs-other-verbs#list-push). The daemon makes the key once and keeps it in
the state directory, at `push/vapid.pem`, with mode 600. The phones are in
`push/subscriptions.json`, with mode 600.

Who may call it:

- Only the person. A missing nonce, a nonce that does not verify, and any
  call from a process inside a pane fail with `not_human`.
- A phone gets the Inbox of every session, so it is an act in no one
  session (see [Nonce scope](https://tuios.dev/docs/control-protocol/versioning#nonce-scope)). A presence made for one
  session covers only that session. With such a nonce, `register-push`,
  `list-push` and `remove-push` fail with `not_human`, and the message says
  the nonce is for another session.
- Over a link, the link needs the `respond` capability. Without it the call
  fails with `forbidden`. A link can not replace a registered phone.
- A restricted connection cannot call it.

Each registration opens an Inbox item of kind `errored`, with no session,
named `phone NAME`. Its summary says who registered the phone (this machine,
or the link from a named machine) and whether it replaced a phone of that
name. A `remove-push` over a link opens the same item. A `remove-push` on
this machine closes it.

At most 16 phones. A bad endpoint or key fails with `invalid_params`.

### What the phone gets

Each push is one HTTP `POST` to `endpoint` with these headers:

| Header             | Value                                                                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `Content-Encoding` | `aes128gcm`                                                                                                                             |
| `Content-Type`     | `application/octet-stream`                                                                                                              |
| `TTL`              | `3600` for the open and the close of `approval`, `plan`, `ask` and `question`, else `120`                                               |
| `Urgency`          | `high` for an open of `approval`, `plan`, `ask` or `question`, else `normal`                                                            |
| `Topic`            | 32 base64url characters, the same for the open and the close of one item. A push service that holds an open replaces it with the close. |
| `Authorization`    | `vapid t=JWT, k=VAPID_PUBLIC_KEY`                                                                                                       |

The JWT is ES256, with the claims `aud` (the endpoint's origin), `exp` (12
hours from the send) and `sub` (`[notify.webpush] subject`, by default
`https://tuios.dev/push`). The push service reads the JWT, so the default
`sub` names no machine.

The TTL of an item that waits for the person is one hour. The item stays
answerable while it is open, often after the hook's hold ends, and a phone
that sleeps can get a push some minutes late. A stale open does not outlive
its item: the close has the same `Topic` and the same TTL, and a push
service that still holds the open replaces it with the close.

The body is encrypted for the phone by RFC 8291, as one record of the
`aes128gcm` content coding (RFC 8188). The body is:

```
salt (16 bytes) | rs (4 bytes, big endian, 4096) | idlen (1 byte, 65) | keyid (65 bytes) | ciphertext
```

`keyid` is the sender's one-time public key, `as_public`, an uncompressed
P-256 point. The phone keeps `ua_private` (its private key), `ua_public` (the
`p256dh` it registered) and `auth_secret`. To decrypt, with HMAC-SHA-256 as
`HMAC(key, data)`:

1. `ecdh_secret = ECDH(ua_private, as_public)`, the 32-byte x coordinate.
2. `PRK_key = HMAC(auth_secret, ecdh_secret)`.
3. `IKM = HMAC(PRK_key, "WebPush: info" || 0x00 || ua_public || as_public || 0x01)`.
4. `PRK = HMAC(salt, IKM)`.
5. `CEK = HMAC(PRK, "Content-Encoding: aes128gcm" || 0x00 || 0x01)`, first 16 bytes.
6. `NONCE = HMAC(PRK, "Content-Encoding: nonce" || 0x00 || 0x01)`, first 12 bytes.
7. Decrypt `ciphertext` with AES-128-GCM, key `CEK`, nonce `NONCE`, no
   associated data. The 16-byte tag is at the end.
8. Remove the zero bytes at the end, then the `0x02` before them, which
   marks the last record. tuios pads the JSON with zero bytes up to 256,
   512, 1024, 2048 or 3072 bytes, the smallest that holds it, so the length
   of a push tells little about its text.

Steps 2 and 3 are HKDF (RFC 5869) with `auth_secret` as the salt. Steps 4 to
6 are HKDF with `salt` as the salt. The ntfy UnifiedPush distributor gives
the body to the app base64-encoded.

The plain text is JSON, at most 3 KiB, with no HTML escapes:

```json
{"v":1,"type":"open","id":"1","kind":"approval","session":"demo","window":"f9ab67c1-6226-4335-8c7e-ea784574a3ef","harness":"claude-code","name":"Terminal f9ab67c1","summary":"approve Bash: npm test","request_id":"69a569a6ab65115e","options":["once","deny"],"machine":"studio"}
```

| Field                        | What it is                                                     |
| ---------------------------- | -------------------------------------------------------------- |
| `v`                          | `1`. A change that breaks a reader gets a new number.          |
| `type`                       | `open` or `close`.                                             |
| `id`                         | The Inbox item's id. A `close` has the id of its `open`.       |
| `kind`                       | The Inbox kind.                                                |
| `session`, `window`          | The pane the item is about.                                    |
| `host`                       | The linked machine the item is on, empty for this one.         |
| `harness`, `name`, `summary` | What the Inbox row shows.                                      |
| `risk`                       | The risk rules the call matched, as the Inbox item has them.   |
| `request_id`, `options`      | A held prompt's request and its answers, for `reply-approval`. |
| `machine`                    | The name of the machine that sent the push.                    |
| `truncated`                  | `true` when text was cut to fit 3 KiB.                         |

An empty field is left out. A payload that does not fit 3 KiB is cut, not
dropped: first the `summary`, then the `risk`, then each text field. The
`id`, `request_id` and `options` are never cut, because an answer gives them
back. A `close` carries only `v`, `type`, `id`, `kind`,
`session`, `window`, `host` and `machine`.

The daemon sends an `open` when an item opens. It sends the `open` again,
with the same `id`, when the item gets a held prompt, or loses one: then
`request_id`, `options` or `risk` change. A phone replaces the notification
with the same `id`. It sends a `close` when the item closes, also when you
answer it at the desk. A phone that does not know the `id` ignores the close.

Delivery never blocks the daemon. Each phone has its own queue. A push that
gets no answer, `429` or a `5xx` is tried again after 1, 4 and 15 seconds.
When the push service answers `404` or `410`, the daemon removes the phone,
logs it, and opens the Inbox item `phone NAME` to say so.
`[notify] enabled = false` stops the opens. A close still goes, so a phone
can remove what it shows. `max_per_hour` limits the opens to each phone. An
item from another machine is not pushed: that machine pushes to its own
phones.

Link capability: `respond`.
