⚡ Running this agent 24/7? tiyuvta inference — hosted LLM inference built for always-on agents, OpenAI/Anthropic-compatible APIs.

computer-use-linux reads accessibility trees, takes screenshots, and drives clicks, scrolls, and keystrokes across GNOME, KDE/KWin, Hyprland, i3, and COSMIC — Wayland-first, X11 best-effort.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

The Rust crate is published as computer-use-linux and the npm wrapper as @agent-sh/computer-use-linux. Prebuilt binaries ship with the latest release.

computer-use-linux is a Rust MCP server and CLI for Linux desktop control. The crate ships the main computer-use-linux binary plus a small computer-use-linux-cosmic helper used only for COSMIC Wayland window management. Any MCP host — Codex Desktop's Linux build, Claude Desktop, Hermes Agent, or your own client — can spawn it and gain full control of the local Linux desktop: read accessibility trees, list and focus windows, take screenshots, click, drag, scroll, type, and invoke semantic accessibility actions.

Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive xdotool against an X11 root window or shell out to OCR over screenshots. Four things set this one apart:

• Wayland actually works. Pointer actions can use the org.freedesktop.portal.RemoteDesktop interface on Wayland, with ydotool / ydotoold (uinput) as the deterministic fallback. Literal text prefers wtype on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, org.freedesktop.portal.Screenshot otherwise, and fall back to spawning gnome-screenshot for background/systemd contexts where both DBus paths are denied.
• Window targeting is compositor-aware. The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, i3 IPC, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed.
• Semantic selectors, not pixel coordinates. Tools like click, perform_action, and set_value accept role / name / text / states selectors backed by AT-SPI. Pixel coordinates remain available as a fallback for rendering-only surfaces (canvas, games, X clients without ATK).
• One JSON readiness report. computer-use-linux doctor returns a structured document covering platform, portals, AT-SPI, windowing, input, and a readiness summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.

The crate was extracted from codex-desktop-linux (the Linux distribution of Codex Desktop), which still bundles this binary as a built-in plugin. This standalone repo is the upstream.

MCP tools exposed by the server:

Diagnostics

• doctor — single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and a capability map of available backends)
• setup_accessibility — enables GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees
• setup_window_targeting — installs and enables the bundled GNOME Shell extension when org.gnome.Shell.Introspect is locked down

Discovery

• list_apps — running desktop apps visible to the AT-SPI registry
• list_windows — compositor windows with title, app id, wm_class, focus state, client type (Wayland/X11), and bounds
• focused_window — the window currently holding keyboard focus
• get_app_state — combined screenshot + accessibility tree for a chosen app, with element indices that the input tools accept
• screenshot — capture the screen as a bounded PNG or JPEG image; can target a window, which is raised to the front and cropped to just that window

Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass max_width, max_height, max_bytes, scale, format: "jpeg", or quality, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes coordinate_width, coordinate_height, scale, format, and quality so callers can convert from a downscaled preview to desktop coordinate pixels.

Input

• click — by element index, semantic selector, or desktop coordinate pixels
• drag — desktop coordinate drag (start / end)
• scroll — page-based scroll on an element or at a pixel location
• press_key — keys / chords; can focus a window or terminal first
• type_text — literal text input, optionally targeted at a window or terminal

Targeted press_key/type_text results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. get_app_state returns a compact readiness block by default; pass verbose: true for the full diagnostics report.

Semantic actions

• perform_action — invoke any AT-SPI action exposed by an element (Press, Activate, Toggle, …); defaults to the primary action
• set_value — write to a settable accessibility element (text fields, sliders, spinners)

Navigation

• activate_window — focus a window by window_id, pid, app_id, wm_class, title, or terminal selectors
• move_window / resize_window — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen

Conditional host execution

• run_shell — same-user /bin/sh -c execution without login-profile loading, registered only when the server operator starts the MCP process with COMPUTER_USE_LINUX_ENABLE_SHELL=1. It is deliberately absent by default and is not a sandbox.

computer-use-linux is not a read-only data source. It can observe the local desktop and, when a mutating tool is called, can change real application state. The tools/list response includes MCP ToolAnnotations so hosts can surface this distinction before invocation:

Annotations are safety hints, not an authorization system. MCP hosts should still ask the user before calls that could submit, delete, send, purchase, overwrite, or otherwise commit state.

run_shell is an explicit trust-boundary opt-in, not a restricted command runner. Enabling it grants an approved MCP call the same file and network authority as the user running the server. The tool clears the ambient environment and inherits only a small desktop/runtime allowlist (PATH, home/user/locale fields, display/session-bus fields); additional variables must be supplied in the visible call payload. Commands use a fixed non-login /bin/sh, an existing canonical working directory, a 30-second default / 120-second hard timeout, process-group cleanup, and stderr audit records keyed by the command SHA-256 rather than command text. Collected streams up to 8 MiB are returned with a 512 KiB per-stream response cap and truncation flag; exceeding 8 MiB on either stream fails the call without partial output. These controls bound accidental leakage and runaway work; they do not make arbitrary shell code safe.

The binary also exposes the same capabilities from the CLI for scripting and debugging:

computer-use-linux mcp                                  # stdio MCP server
computer-use-linux doctor                               # JSON readiness report
computer-use-linux setup                                # enable AT-SPI
computer-use-linux setup-window-targeting               # install GNOME Shell extension
computer-use-linux apps
computer-use-linux state [APP_NAME]
computer-use-linux screenshot                           # JSON screenshot summary
computer-use-linux windows

Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). Other compositor backends are implemented and covered by parser / contract tests, but real desktop behavior still depends on each session exposing its expected control API.

If you run on a desktop not covered above, or a covered backend does not come up cleanly, please open an issue with the output of computer-use-linux doctor so we can extend the matrix honestly.

COSMIC users do not need a second package or a separate helper install when using ./install.sh, cargo install, or the npm wrapper. Those paths install computer-use-linux-cosmic alongside the main binary automatically. Only manual prebuilt-binary installs need you to copy both release assets.

Installs system packages on Debian/Ubuntu, Fedora/RHEL-like, Arch-like, or Artix systems; installs Rust if needed; builds both release binaries; installs them to ~/.local/bin; configures ydotoold as a systemd user service when available; enables GNOME AT-SPI settings when running under GNOME; and installs the bundled GNOME Shell extension on GNOME Wayland.

git clone https://github.com/agent-sh/computer-use-linux
cd computer-use-linux
./install.sh
# log out and back in if the GNOME extension was newly installed
computer-use-linux doctor | jq .readiness

ydotool is an optional fallback. On X11, the installer includes xdotool as the required keyboard backend. If ydotool is unavailable from the configured repositories, the installer continues, but doctor still requires a keyboard-capable RemoteDesktop portal on Wayland or xdotool on X11; direct uinput provides absolute pointer input only. On non-systemd hosts, automatic ydotoold service setup is skipped and the installer prints a command suitable for a per-user supervisor. For an unrecognized distro, pass --package-manager apt|dnf|pacman; --force-unknown-distro auto-selects only when exactly one of those managers is available.

Installs the Rust binaries from crates.io. You still handle the system-level pieces yourself: AT-SPI, desktop portals, the optional ydotoold fallback, and the GNOME extension if you need the GNOME Wayland exact-focus backend.

cargo install computer-use-linux
computer-use-linux doctor

For unreleased changes from main, install directly from Git:

cargo install --git https://github.com/agent-sh/computer-use-linux

Then, as needed:

sudo apt install ydotool at-spi2-core         # ydotool 1.0.3+ when using this fallback
sudo apt install wtype                        # optional Unicode typing on wlroots/Hyprland Wayland
systemctl --user enable --now ydotoold         # only when doctor selects ydotool
computer-use-linux setup                      # gsettings AT-SPI bridge
computer-use-linux setup-window-targeting     # GNOME Shell extension

Good for users who already have Node.js and want a no-Rust install. The npm package downloads and verifies the matching main and COSMIC helper binaries during install, then the wrapper sets COMPUTER_USE_LINUX_COSMIC_HELPER to the bundled helper automatically.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor

You will still need AT-SPI enabled (computer-use-linux setup) and one input backend reported ready by doctor. Start ydotoold only when using the ydotool fallback.

Linux x86_64 / aarch64 builds are published with each tag. Each binary ships a .sha256 next to it.

• Latest release: https://github.com/agent-sh/computer-use-linux/releases/latest

target=x86_64-unknown-linux-gnu
base=https://github.com/agent-sh/computer-use-linux/releases/latest/download
for binary in computer-use-linux computer-use-linux-cosmic; do
  asset="$binary-$target"
  curl -L -O "$base/$asset"
  curl -L -O "$base/$asset.sha256"
  sha256sum -c "$asset.sha256"
  install -m 0755 "$asset" "$HOME/.local/bin/$binary"
done

You will still need ydotoold running and AT-SPI enabled (run computer-use-linux setup and the systemd commands above).

The binary speaks the rmcp 2024-11-05 stdio protocol. Pass mcp as the only argument; everything else is configured through MCP tool calls.

The Linux build of Codex Desktop already bundles this binary as a plugin. You don't need to wire it up manually — the plugin definition lives in codex-desktop-linux under its plugins/ directory and is enabled by default. To upgrade the plugin in place, replace the binary it ships with the one from this repo's release assets.

Use the claude mcp add command to register the binary as a stdio MCP server. Pick a scope:

• --scope user — available across all projects for your user.
• --scope project — written to .mcp.json at the project root for team sharing.
• --scope local (default) — only the current project, stored in ~/.claude.json.

# User-wide install (recommended for desktop control)
claude mcp add --scope user computer-use-linux -- computer-use-linux mcp

# Verify the server is registered and reachable
claude mcp list

If computer-use-linux is not on PATH, pass the absolute path (e.g. ~/.local/bin/computer-use-linux). Inside a Claude Code session, run /mcp to confirm the tools are loaded.

Edit ~/.config/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "computer-use-linux": {
      "command": "computer-use-linux",
      "args": ["mcp"]
    }
  }
}

Restart Claude Desktop. The tools should appear in the tools list.

pi install npm:pi-mcp-adapter
pi install npm:@agent-sh/computer-use-linux

Restart pi or run /reload. The MCP proxy tool mcp() will have the desktop tools available:

mcp({ server: "computer-use-linux" })             # list all tools
mcp({ search: "windows" })                         # search for window tools
mcp({ tool: "computer_use_linux_doctor" })         # run readiness check
mcp({ tool: "computer_use_linux_list_windows" })   # list desktop windows

The extension auto-registers the computer-use-linux MCP server into pi-mcp-adapter's config. If the binary is not found, check the Pi setup guide.

Install the companion Hermes skill so Hermes has the desktop-specific runbook:

hermes skills tap add agent-sh/computer-use-linux
hermes skills install agent-sh/computer-use-linux/computer-use-linux

The skill is optional but recommended for Hermes users. It teaches Hermes how to install, configure, verify, and call the Linux desktop MCP safely. It follows the same skills/<name>/SKILL.md tap layout used by Hermes community skills.

Then add the stdio MCP server:

hermes mcp add computer-use-linux --command computer-use-linux --args mcp
hermes mcp test computer-use-linux
hermes mcp configure computer-use-linux

configure opens Hermes' tool-selection UI for the server. The generated config should look like this:

mcp_servers:
  computer-use-linux:
    command: computer-use-linux
    args: ["mcp"]
    timeout: 120
    connect_timeout: 30

# Optional: expose the tools to subagents as well.
inherit_mcp_toolsets: true

If you installed the binary somewhere that is not on PATH, pass the absolute path as --command.

Restart Hermes after editing the config. Hermes registers the tools as mcp_computer_use_linux_<tool> and creates the mcp-computer-use-linux runtime toolset.

You can verify both sides before asking Hermes to use the desktop:

computer-use-linux doctor | jq .readiness
hermes skills inspect agent-sh/computer-use-linux/computer-use-linux
hermes chat --toolsets mcp-computer-use-linux -q "List the current desktop windows."

For one-off installs without adding the tap first, Hermes also accepts hermes skills install agent-sh/computer-use-linux/skills/computer-use-linux.

Spawn the binary with ["mcp"] as the argv tail. It speaks JSON-RPC over stdio per the rmcp 2024-11-05 protocol; capability discovery happens through tools/list and the doctor tool. The server normally needs no MCP-specific configuration, but desktop runtime environment still matters (DBUS_SESSION_BUS_ADDRESS, XDG_RUNTIME_DIR, portals, AT-SPI, ydotoold, and optionally COMPUTER_USE_LINUX_COSMIC_HELPER).

1. Run doctor.

   computer-use-linux doctor | jq .readiness

   Aim for can_register_mcp_tools, can_build_accessibility_tree, can_send_development_input, and can_query_windows all true. The blockers array should be empty.

2. If accessibility.at_spi_bus.ok = false — run computer-use-linux setup (or call the setup_accessibility MCP tool). This sets:

   • org.gnome.desktop.interface toolkit-accessibility true

   You may need to restart toolkit-using apps for the change to take effect.

3. If windowing.can_list_windows = false — inspect doctor.windowing.backends. On GNOME Wayland, run computer-use-linux setup-window-targeting (or call setup_window_targeting) to install the bundled computer-use-linux@avifenesh.dev Shell extension, then log out and back in so GNOME Shell loads it. On KDE, Hyprland, i3, COSMIC, or generic X11, install or expose the matching compositor tool/helper shown in the backend details.

4. Grant the screencast portal on first screenshot. The first time get_app_state or any screenshot subcommand runs, GNOME will pop a portal dialog asking to share the screen. Accept once and tick "remember" to make it sticky for the session.

5. Confirm compatible ydotool 1.0.3+ and ydotoold are available.

   systemctl --user status ydotoold

   Its socket should appear at /run/user/$UID/.ydotool_socket.

Most setups need none of these — doctor and the installers pick sensible defaults. They exist for overriding auto-detected paths and input backends.

Server runtime (set in the MCP host's environment):

Build-time identity overrides (set while compiling a downstream embedded bundle): CUL_GNOME_EXTENSION_UUID, CUL_DBUS_SERVICE, and CUL_DBUS_OBJECT_PATH replace the default standalone GNOME Shell extension UUID and DBus endpoint in both the Rust probes and the generated extension files.

npm wrapper (set during npm install, or before running):

• Accessibility tree — atspi crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to (role, name, text, states, bounds) tuples and indexed; element indices are stable for the duration of a get_app_state snapshot.
• DBus where desktops expose it — zbus for portal calls (org.freedesktop.portal.Screenshot, …RemoteDesktop, …ScreenCast), GNOME Shell screenshots (org.gnome.Shell.Screenshot), the bundled GNOME extension's dev.avifenesh.ComputerUseLinux.WindowControl service, and temporary KWin scripting.
• MCP transport — rmcp with the transport-io feature; stdio framing, no network.
• Input fallback — on X11, keyboard input prefers xdotool/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses wtype when installed and the remote-desktop portal is unavailable; wtype supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and ydotoold socket. A launched wtype failure is returned without replaying the text. install.sh can configure ydotoold; the setup command only enables the GNOME AT-SPI bridge.
• Native X11 coordinate clicks — eligible native X11 sessions use one supervised xdotool mousemove -- X Y click --repeat N BUTTON command for left, middle, and right clicks; ydotool is used only when xdotool cannot launch, while a launched nonzero xdotool command is reported as an error without replay. COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1 skips this xdotool path.
• Window registry — list_windows, focused_window, activate_window, press_key, and type_text share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland hyprctl, i3 IPC, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
• GNOME extension fallback — recent GNOME builds deny org.gnome.Shell.Introspect.GetWindows to non-blessed clients. The bundled Shell extension exposes window data and exact activation under dev.avifenesh.ComputerUseLinux.WindowControl.
• COSMIC helper — computer-use-linux-cosmic talks to COSMIC toplevel protocols and is resolved from COMPUTER_USE_LINUX_COSMIC_HELPER, next to the running binary, or from PATH.
• Terminal enrichment — list_windows cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so type_text / press_key can target "the terminal where pytest is running" without the host ever knowing the window id.

Computer-use tooling is, by definition, a privilege-escalation surface. The threat model:

• ydotoold runs as a per-user service with read/write access to /dev/uinput. install.sh automates this for systemd user sessions and prints manual supervisor guidance elsewhere. Any process that can connect to its socket (/run/user/$UID/.ydotool_socket, mode 0600 by default) can synthesize arbitrary input — keypresses, clicks, anything. Keep the socket in the user runtime dir (the default), not in /tmp or any world-readable location. Do not run ydotoold as root or as a system service.
• The screencast portal asks for permission once per session. Granting it lets the calling MCP host capture the screen for the rest of the session. If you don't want that, decline the portal dialog and use get_app_state with include_screenshot: false.
• AT-SPI exposes window contents to any client on your session bus. Enabling the AT-SPI bridge (setup_accessibility) is a prerequisite for this binary; it's also what screen readers use, and it shares the same trust boundary.
• The GNOME Shell extension is loaded only into your user's GNOME Shell, runs in the Shell's JS sandbox, and exposes a single DBus interface on the user session bus. It does not request any extra permissions.
• No network. This binary opens no TCP/UDP listener, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user ydotoold Unix socket.
• Mutating tools are explicit. The MCP tool list annotates read-only versus mutating tools, and CI fails if the published tool annotations drift from the table above. Treat those annotations as hints; the host is still responsible for user approval and policy.

If you're running this on a shared workstation, set ydotoold's socket permissions to 0600 (the default) and audit which processes on your user can connect() to it.

computer-use-linux doctor is the source of truth. Common failure modes and fixes:

• accessibility.at_spi_bus.ok = false — AT-SPI registry isn't running or the toolkit bridge is off. Fix: computer-use-linux setup (or call the setup_accessibility MCP tool). Restart the apps you want to drive.
• windowing.gnome_shell_introspect.ok = false and gnome_shell_extension_dbus.ok = false — GNOME blocks introspection and the extension isn't installed. Fix: computer-use-linux setup-window-targeting, then log out and log back in.
• input.ydotool_socket.ok = false while ydotool is the selected fallback — daemon isn't running. On systemd, run systemctl --user enable --now ydotoold. On other init systems, rerun ./install.sh and configure your per-user supervisor with the command it prints. If ydotool is not packaged for your distro, use another input backend or install a compatible ydotool release manually.
• input.ydotool.ok = false with an unsupported CLI message — install ydotool 1.0.3 or newer. A running daemon or socket alone is not enough; doctor verifies the raw key, wheel, stdin typing, and absolute-movement command family before advertising the backend.
• input.uinput.ok = false — /dev/uinput isn't accessible to your user. Fix: add yourself to the input group (sudo usermod -aG input $USER) and re-login. On distros that ship uinput as a kernel module without auto-loading it, add uinput to /etc/modules-load.d/. Direct uinput supplies absolute pointer input only, so doctor also requires a keyboard-capable portal, xdotool, or ydotool backend.
• Portal calls hang or time out — xdg-desktop-portal or its backend (-gnome, -gtk, -kde, -wlr) crashed. Fix: check journalctl --user -u xdg-desktop-portal -u xdg-desktop-portal-gnome --since '5 min ago' and restart the relevant unit.
• KWin / Hyprland / i3 / COSMIC / X11 windowing is unavailable — check doctor.windowing.backends. KWin needs session-bus scripting; Hyprland needs hyprctl; i3 needs i3-msg and its IPC socket; generic X11 needs wmctrl and xprop. COSMIC needs computer-use-linux-cosmic, which the standard installers provide automatically; if you copied binaries by hand, copy the helper too or set COMPUTER_USE_LINUX_COSMIC_HELPER.
• Screenshots return black frames on multi-monitor setups — known portal / compositor edge case. Use get_app_state with include_screenshot: false and rely on AT-SPI until the portal backend is healthy.
• type_text types into the wrong window — pass an explicit target (window_id, pid, wm_class, title, or for terminals tty / terminal_pid / terminal_command / terminal_cwd). Without a target, input goes to whatever window currently has compositor focus.

If doctor is green and a specific tool still misbehaves, file an issue with the JSON output of doctor and the failing tool's request payload.

• agent-workspace-linux — the sibling MCP that gives an agent its own isolated Linux desktop (a hidden Xvfb display with its own apps and browser) instead of driving yours. It is the inverse of this project: computer-use-linux automates the desktop you are already on; agent-workspace-linux sandboxes the agent in a separate one. Use them together.

Contributions are welcome. See CONTRIBUTING.md for the local development workflow, CI gates, and PR expectations. Report security vulnerabilities through SECURITY.md, not public issues.

Extracted from codex-desktop-linux, the Linux distribution of Codex Desktop, which continues to ship this same binary as a bundled plugin. Maintained by Avi Fenesh.

Built on top of:

• atspi — AT-SPI bindings
• zbus — async DBus
• rmcp — MCP runtime
• ydotool — Wayland-friendly uinput driver
• cosmic-protocols — COSMIC Wayland toplevel protocol bindings

Publishing is tag-driven from GitHub Actions. The repository needs these Actions secrets:

gh secret set CARGO_REGISTRY_TOKEN -R agent-sh/computer-use-linux
gh secret set NPM_TOKEN -R agent-sh/computer-use-linux

Then bump Cargo.toml and package.json together, update CHANGELOG.md, and push a vX.Y.Z tag. CI runs the full Rust and MCP safety gates, builds release assets for both architectures, publishes computer-use-linux to crates.io, and publishes the npm wrapper after the GitHub release binaries are available.

MIT — see LICENSE.