How the bot sees and acts

The agent has no special access to the programs in the tank. It works the way a person does: it looks at the screen, decides, and uses the mouse and keyboard. This page explains that loop and the few tools around it, so its behavior makes sense when you watch it.

The loop#

Every task is a repetition of the same four moves:

1. Look. Deskfish takes a screenshot of the tank's screen and sends it to the model.
2. Think. The model, given the task, the conversation so far and the screenshot, decides what to do next. It may say something to you first.
3. Act. The chosen actions are carried out on the tank: move the mouse, click, type, press keys, scroll, wait.
4. Settle. After actions that could change the screen, Deskfish waits a moment (deskfish.settleMs, 800 ms by default) so pages can react, and then looks again.

One trip round the loop is a step. The model is told to check each new screenshot against what it intended, and to correct itself if a click landed in the wrong place.

What the agent can do#

The model is given a small set of tools, the same on every provider:

The agent is also told, in its standing instructions, exactly which tools these are and what its computer contains, so it does not have to discover by trial that there is no git but there is curl. Beyond those tools it has no other channel into the tank. The terminal is a normal shell with bash, python3, curl, jq, zip/unzip and nano, but there is no root, no sudo and no package installation, and the only network is the tank's own.

Seeing where the pointer is#

A screenshot of a Linux desktop does not include the mouse pointer. So before each look, Deskfish asks the tank where the pointer is and draws a small red crosshair at that spot on the screenshot. The model is told the crosshair is an overlay, not part of the screen. After a click, the crosshair shows exactly where it landed, which is how the model notices and fixes a miss.

You do not see the crosshair in the Desktop tab; it exists only in the agent's copy.

Zooming instead of guessing#

Small targets, such as toolbar icons, checkboxes and links in dense text, are where vision models make mistakes. The agent is told not to guess at anything small but to zoom first. A zoom returns a three-times magnified crop of the real screen around the point it asked for, overlaid with rulers and a dotted grid labeled in ordinary screenshot coordinates. The model reads its target's exact position off the grid, then clicks with those numbers. There is only one coordinate system, so nothing can get lost in translation.

Reading the page instead of the picture#

Most tasks happen in Firefox, and a web page knows more about itself than a picture of it shows. The tank's Firefox carries a small extension, the Deskfish page bridge, that lets the agent ask the page directly:

• find takes a few words, such as Sign in, search box, Add to cart button or Order total, and returns the best-matching elements with their role, their name, their current state (a field's value, whether a box is checked, which option is selected) and the exact coordinates to click, in the same screenshot pixels the agent uses for everything else. A match outside the visible part of the page is reported as off-screen, with how far to scroll, and one hidden behind a dialog or menu is reported as covered.
• read_page lists every link, button, field, checkbox, menu item and heading visible in the viewport, in page order, with the same details, and says how many more lie below or above. With scope text it returns the page's text instead, so the agent can read an article, a results list or a confirmation without scrolling through it.

Both are passive: nothing on screen changes. The agent is told to use find before guessing where something is and to zoom only for what the bridge cannot see: native dialogs, the terminal, the panel, PDFs, drawings, and anything that is not an http(s) page. Frames inside a page (a payment form, an embedded editor) are read too.

The bridge talks only to the tank's own control daemon; it has no network access of its own, and the page's contents go to the model only when the agent asks for them, the way a screenshot does. If Firefox is closed, the tools say so and the agent opens it.

Standing by#

Some work is mostly waiting: an ad set that takes ten minutes to process, an upload, a reply, or a rule like "send the next one in five minutes". A plain wait is capped at thirty seconds and every wait is followed by a model turn, so waiting out minutes that way costs money and, because nothing on screen changes, looks to Deskfish like the agent is stuck.

wait_for is waiting that costs nothing. The agent names what it is waiting for and how long at most, and Deskfish takes over: it takes small local screenshots every few seconds and compares them, without involving the model. It wakes the agent, with a fresh screenshot, either when the screen has changed and then held still for one more look (a spinner keeps moving and does not count; a finished page is a new picture that stays), or when the time is up. With until set to time it simply waits the whole period, which is how the agent spaces actions out. It can also watch just one area of the screen, so a clock or an animation elsewhere does not wake it.

While it stands by, the chat shows a line with the reason and a live countdown, which turns into the outcome when the wait ends, and the status row reads Standing by with the time left. Stop ends it at once. When waiting is part of the plan, the agent announces it in a sentence first. A thirty-minute wait costs one model turn instead of sixty. It is for things the page or the world does; the agent's own actions, typing included, are complete when they return, and it is told not to stand by for them.

Screenshots and coordinates#

Screenshots are scaled down to deskfish.screenshotWidth (1280 pixels by default) before they go to the model, and the model's coordinates are scaled back to real pixels when actions run. With the default 1280 × 800 screen that is exactly one to one. A larger virtual screen still costs the same tokens per screenshot, just at a smaller scale, which is why very large screens make small targets harder rather than easier.

Images are pruned in batches, keeping the most recent three after each prune; older ones are replaced by the note earlier screenshot omitted. The text of the conversation is kept in full.

Keys and shortcuts#

The model presses keys by name, in the style of the xdotool tool used inside the tank: Return, Escape, ctrl+l, alt+Tab, Page_Down. The tank also understands common aliases such as enter and pageup. It is encouraged to prefer keyboard shortcuts where they are reliable, for instance Ctrl+L for the address bar and Ctrl+T for a new tab, and to type URLs rather than search for them.

Batches#

Obviously sequential actions, such as click the field, type the text, press Return, are sent as one batch and executed in order, with a single screenshot afterwards. Anything less certain is done one action at a time, with a look in between. When a batch contains only passive actions (a zoom, a find or read_page, a standby, a pointer check, a documentation lookup), Deskfish skips the settle delay, because nothing on screen could have changed.

What it is told#

The agent's instructions are short. In summary: you are Deskfish; this computer is yours, and everything in it is yours to use; you operate it with screenshots, mouse and keyboard; check every screenshot against what you intended; on a web page ask the page with find or read_page instead of hunting in the picture, and zoom before clicking anything small; files you are given are in Uploads and anything for the user goes in Downloads; when you need something only the user has (a code, a card, a confirmation, a CAPTCHA you cannot pass) or you are stuck, hand over; do what the user asks all the way through; be economical with steps; and when the task is done, write a short summary. Plus what it has: its tools by name and what is installed in the tank; the size of the screenshots; the list of documentation pages it can read; and its memories. In guided mode (deskfish.autonomy) it is also told to ask before anything irreversible and never to use credentials it was not given.

It is told that steps cost money and, if you set a step cap, how many it has, with a warning when three remain and a request for a summary at the cap, so a capped task ends with a report rather than nothing. Deskfish also watches for stalls on its behalf: an unchanged screen and repeated actions earn a nudge to change approach, then a hand-over to you.

It does not know which AI provider it runs on, that it is inside a container, or that you are watching. It knows the desktop is its own.