mcp-interact

mcp-interact is a native macOS MCP server that gives AI agents full accessibility, input, and system control. Built in Swift on the Model Context Protocol SDK, it exposes 51 tools across 11 modules. It is the first Bus Commons open-source release.

mcp-interact implements AXI-INTERACT — the Agent eXperience Interface applied to desktop interaction. AXI is a set of design principles for building agent-native tools that minimize token waste, maximize action density, and give agents reliable feedback about their effects on the world.

Architecture

Sources/
  MCPInteract/          Entry point (main.swift)
  MCPInteractKit/       MCP server, tool registry, dispatch
  Accessibility/        AX tree inspection, search, caching, observer
  Input/                Mouse, keyboard, scroll via CGEvent
  Capture/              Screenshot, screen recording (ScreenCaptureKit)
  Audio/                Device listing, text-to-speech
  Hotkeys/              Hotkey manifest index and navigation
  Session/              Single-agent session mutex, audit log
  System/               App listing, display info, clipboard
  Policy/               Element and app denylist
  Windows/              CGWindowList queries
  TOON/                 Token-Oriented Object Notation encoder

MCPInteractKit is a library product. It can run standalone as a stdio MCP server or be embedded in another application (CUBEdesk) via Swift Package Manager. When embedded, the server listens on TCP port 9781 instead of stdio.

The dispatch layer routes all 51 tools through a single dispatch() function that wraps each response with TOON encoding, contextual help lines, and generation tracking for mutating operations.

Transport

mcp-interact supports two transports:

• stdio — default, for MCP clients like Claude Code. JSON-RPC 2.0 over stdin/stdout.
• TCP — for library embedding. MCPInteractServer.startOnPort(9781) creates an NWListener and wraps each accepted connection in NetworkTransport.

Tool Inventory

51 tools organized into 11 categories. 24 require an active session. All responses are TOON-encoded with contextual help lines.

Perception (no session)

Tool	Description
screenshot	Full screen or region capture, JPEG scaled to 1280px max
screenshot_window	Capture a specific window by name or ID
get_screen_info	Display dimensions, Retina scale, cursor position
list_windows_tool	All visible windows with positions, sizes, PIDs
inspect_ui	AX element tree for a running app (compact or full detail)
find_element	Search AX tree for first element matching role/title/description
find_elements	Search for all matching elements (max 100)
ax_focused	Currently focused element across all apps
ax_menu_bar	Full menu bar structure for an app

System observation

Tool	Description
app_list	Running applications with bundle IDs and PIDs
app_frontmost	Currently frontmost application
app_activate	Bring an app to front or launch by bundle ID (session required)
clipboard_read	Clipboard contents: text, file URLs, image presence
system_state	Battery, thermal, displays, memory, OS version

AX observers

Tool	Description
ax_watch	Start observing an app for focus, value, and window events
ax_unwatch	Stop observing an app
ax_events	Drain buffered AX events from watched apps

AX actions (session required)

Tool	Description
ax_press	Press a button or activate a control via AX
ax_set_value	Set value on a text field, slider, or control
ax_focus	Set keyboard focus to an element
ax_raise	Bring an app and its main window to front via AX
ax_actions	List available actions and attributes for an element
ax_perform	Perform any AX action on an element
ax_cache	Find an element and cache it; returns a handle_id for reuse

CGEvent input (session required)

Tool	Description
click_at	Click at coordinates or on a named AX element
double_click_at	Double-click at coordinates
type_text_tool	Type text at current focus (handles Unicode)
key	Press a key combination (e.g. cmd+shift+n)
scroll_at	Scroll at a coordinate position
run_applescript_tool	Execute AppleScript with denylist filtering

Session management

Tool	Description
acquire_session	Acquire exclusive screen control (blocks until granted or timeout)
release_session	Release session, wake next agent in queue
extend_session	Extend TTL of active session
session_status	Current holder, TTL remaining, queue state

Hotkey index

Tool	Description
list_hotkeys	Query the hotkey index by app, kind, or search query
list_hotkey_apps	List apps with registered hotkey manifests
hotkey_jump	Navigate to a surface by firing its hotkey chord (session required)
hotkey_rescan	Re-scan filesystem for hotkey manifests
hotkey_register	Register a hotkey manifest programmatically

Audio

Tool	Description
audio_devices	List audio input and output devices
speak	Text-to-speech via macOS TTS (session required)
voice_list	Available TTS voices with language and gender
now_playing	Current media: track, artist, album, elapsed time

Screen recording (session required, macOS 15+)

Tool	Description
record_start	Start recording screen or window to MP4
record_stop	Stop recording, return file path and duration
record_list	List active recordings

Server info

Tool	Description
capabilities	Server version, tool count, session state, hotkey index stats

Compound AXI tools (session required)

Tool	Description
navigate	Hotkey jump + inspect UI in one call, with settle delay
interact	Find element + perform action + return updated state
fill	Find text field + focus + type + return confirmed value
activate_and_inspect	Bring app forward + return its AX tree

Compound tools collapse multi-step agent patterns into single MCP calls. Each includes a configurable settle_ms parameter so the UI can stabilize between the action and the perception step. Default settle times: navigate 300ms, interact 200ms, fill 100ms, activate_and_inspect 500ms.

AXI Design Principles

AXI-INTERACT applies ten design principles that make every tool response optimized for agent consumption.

1. TOON output

All responses are encoded in Token-Oriented Object Notation instead of JSON. TOON achieves 30-60% token savings through:

• Unquoted keys and safe string values
• Indentation-based nesting instead of braces
• Tabular array encoding with field names in a header row
• Inline primitive arrays with count prefix

JSON response:

{"server":"mcp-interact","version":"0.1.0","platform":"macOS",
"tool_count":51,"session":{"current_session":null,"queue_length":0,
"queue":[]},"hotkey_index":{"apps":0,"entries":0}}

TOON response:

server: mcp-interact
version: 0.1.0
platform: macOS
tool_count: 51
session:
  current_session: null
  queue_length: 0
  queue: []
hotkey_index:
  apps: 0
  entries: 0

For tabular data (lists of windows, apps, hotkeys), TOON uses a header row with field names and compact inline rows:

windows[3]{name,pid,x,y,width,height}:
  Finder,412,0,25,1728,1092
  Safari,891,100,50,1200,800
  Terminal,1023,200,100,800,600

2. Compact default schemas

inspect_ui defaults to compact mode, returning only role, title, value, and bounds per element. Full mode adds subrole, description, identifier, enabled, focused, and selected. In compact mode, position data is collapsed into a single bounds array [x, y, width, height].

Compact mode reduces per-element token count by approximately 60% in tree views. Compound tools (navigate, activate_and_inspect) use compact mode for embedded trees.

3. Contextual help lines

Every successful response includes a help[] line with 2-3 suggested next actions:

server: mcp-interact
tool_count: 51
...
help[acquire_session to start acting | list_hotkeys to see navigation shortcuts]

30 tools have help hints. The mapping is defined in ToolDispatch.swift and appended by the dispatch wrapper. Error responses do not include help lines.

4. Generation-tracked element references

Element handles use the format g{generation}:{counter} (e.g. g1:42). When a mutating action succeeds, the element cache generation advances. If an agent tries to use a handle from a previous generation, it gets a STALE_REF error with the current generation number and a recovery hint to re-inspect.

16 mutating tools advance the generation: ax_press, ax_set_value, ax_focus, ax_raise, ax_perform, click_at, double_click_at, type_text_tool, key, scroll_at, run_applescript_tool, app_activate, navigate, interact, fill, activate_and_inspect.

Every mutation response includes _gen: N so agents can track freshness.

5. Structured errors with recovery hints

Every error response includes a recovery: line with actionable guidance:

error: Missing required parameter: app_name
recovery: check the tool schema and provide the missing parameter

error: STALE_REF: handle g1:42 is from a previous generation (current: g2).
recovery: call inspect_ui or find_element to get fresh handles for the current UI state

Error categories and their recovery patterns:

Category	Recovery
Missing parameter	Check the tool schema and provide it
Policy denied	Choose a different app or element
Stale reference	Call inspect_ui or find_element for fresh handles
Permission denied	Grant Accessibility access in System Settings
App not found	Call app_list to see running apps
Handle not found	Call find_element or ax_cache for a fresh handle
Session required	Call acquire_session first
Unknown tool	Call tools/list to see available tools

6. Definitive empty states

Perception tools return descriptive messages instead of bare false values:

found: false
empty: No element matching [button, Save] in Finder

found: false
empty: No element currently has keyboard focus

7. Compound operations

The four compound tools (navigate, interact, fill, activate_and_inspect) combine perception and action into single calls. Each performs a UI-settling wait between the action and the return read, eliminating the agent’s need to guess timing. This is the highest-value AXI optimization: it turns a 3-4 tool call sequence with sleep-and-retry into a single deterministic call.

8. Return state after mutation

AX action tools (ax_press, ax_set_value, ax_focus, ax_perform) return the updated element state after performing the action. The agent does not need to call find_element or ax_focused to verify its effect.

9. Tool annotations

Every tool carries MCP annotations that inform the client about its behavior:

• readOnlyHint — perception tools that do not change system state
• destructiveHint — action tools that modify UI or input state
• idempotentHint — tools whose output is stable across repeated calls

TOON Format Specification

TOON (Token-Oriented Object Notation) is a text serialization format designed for LLM consumption. It optimizes for token count rather than parse speed.

Rules

Objects use key: value pairs with indentation-based nesting:

name: mcp-interact
version: 0.1.0
config:
  max_clicks: 10
  timeout: 60

Keys are unquoted if they match [A-Za-z_][A-Za-z0-9_.]*. Otherwise they are double-quoted.

String values are unquoted unless they:

1. Are empty
2. Have leading or trailing whitespace
3. Are a reserved word (true, false, null)
4. Look like a number
5. Contain :, ", \, [, ], {, }, ,, or control characters
6. Equal - or start with -

Quoted strings use JSON-style escaping (\\, \", \n, \r, \t, \uXXXX).

Numbers use shortest round-trippable representation. Trailing zeros are stripped. Integer-valued doubles render without a decimal point. NaN and Infinity become null. Negative zero becomes 0.

Booleans render as true / false. Correctly distinguished from NSNumber via CFBooleanGetTypeID().

Null renders as null.

Arrays have five representations:

1. Empty — []
2. Inline primitive — items[3]: apple,banana,cherry
3. Tabular (all dicts with identical keys, all-primitive values) — header row with field names, data rows with comma-separated values:

   users[2]{name,age,active}:
     alice,30,true
     bob,25,false

4. Array of arrays — - [count]: values per row
5. Mixed — - key: value list items with indented sub-fields

Token savings

TOON saves tokens through six mechanisms:

Mechanism	JSON	TOON	Saving
No braces/brackets for objects	{ } ,	indentation	3+ tokens per object
Unquoted keys	"key"	key	2 tokens per key
Unquoted safe strings	"value"	value	2 tokens per value
Tabular arrays	repeat field names per row	single header	N-1 repetitions saved
Inline primitives	[1, 2, 3]	items[3]: 1,2,3	brackets + spaces
Count prefix	implicit from parsing	[3]	no overhead

Real-world measurements on mcp-interact tool responses show 30-60% token reduction depending on response structure, with tabular arrays achieving the highest savings.

Hotkey Manifests

Applications publish hotkey manifests so agents can navigate by semantic name instead of memorizing key chords.

{
  "app": "Finder",
  "app_version": "14.0",
  "entries": [
    {
      "key": "finder.go.home",
      "chord": "cmd+shift+h",
      "label": "Home Folder",
      "kind": "panel"
    },
    {
      "key": "finder.file.new_folder",
      "chord": "cmd+shift+n",
      "label": "New Folder",
      "kind": "action"
    }
  ]
}

Each entry has four fields:

Field	Description
key	Unique dotted identifier (e.g. finder.go.home)
chord	Key combination string (e.g. cmd+shift+h)
label	Human-readable display name
kind	Category: action, view, panel, modal, toggle

Manifests can be registered via the hotkey_register tool or placed as JSON files in ~/Library/Application Support/{AppName}/hotkeys.json for auto-scanning via hotkey_rescan.

Session Model

mcp-interact uses a single-agent session mutex. Only one agent can hold screen control at a time.

Agent A → acquire_session(agent_id, purpose, ttl=120)
  → granted: true, session_id: "abc123", expires_in: 120

Agent B → acquire_session(agent_id, purpose, timeout=60)
  → blocks until A releases or timeout

Agent A → release_session(agent_id, session_id)
  → Agent B unblocked, granted session

TTL — sessions expire after ttl seconds (default 120) without renewal. extend_session resets the timer. An expiry watchdog runs on the session manager and automatically releases expired sessions.

Queue — when a session is held, new requests queue with a configurable timeout. If the timeout expires before the session becomes available, the request returns granted: false with a reason.

Action recording — every session-required tool call records an action timestamp via recordAction(), which is used for audit logging and activity tracking.

Session-scoped element cache — cached AX element handles are associated with a session. When a session ends, all its cached handles are cleared via clearSession().

Security Model

Five layers protect the system from misuse:

Single-agent mutex — only one agent controls input at a time. No concurrent screen manipulation.

Rate limiting — configurable clicks-per-second cap (default 10). The rate limiter is checked before every click and double-click operation.

Audit logging — all actions are logged with agent ID, session ID, and detail metadata. Denied operations log the denial reason.

Policy denylist — element-level and app-level blocking. checkElementPolicy() inspects the serialized element info. checkFocusedAppPolicy() checks the frontmost app’s bundle ID. checkAppleScript() filters script content. All denylist checks happen before the action executes.

AppleScript filtering — AppleScript execution (run_applescript_tool) passes through the denylist before execution, blocking scripts that target restricted applications.

Requirements

• macOS 14+ (Sonoma)
• Swift 5.10+
• Accessibility permission (System Settings > Privacy & Security > Accessibility)
• Screen Recording permission (for screenshot and recording tools)

Installation

swift build -c release
cp .build/release/mcp-interact /usr/local/bin/

Claude Code configuration:

claude mcp add mcp-interact -- /path/to/mcp-interact

Or in .mcp.json:

{
  "mcpServers": {
    "mcp-interact": {
      "command": "/path/to/mcp-interact",
      "args": []
    }
  }
}

Library embedding

Add to Package.swift:

dependencies: [
    .package(path: "../mcp-interact"),
],
targets: [
    .target(
        name: "YourApp",
        dependencies: [
            .product(name: "MCPInteractKit", package: "mcp-interact"),
        ]
    ),
]

Start the server on a TCP port:

import MCPInteractKit

let server = MCPInteractServer(name: "my-app-mcp", version: "0.1.0")
await server.configure()
try await server.startOnPort(9781)

The @_exported import MCP in MCPInteractServer.swift makes all MCP SDK types available transitively through MCPInteractKit.