Running tasks

Everything you say to the agent goes through the chat sidebar, and everything it says comes back there. This page is a tour of that sidebar and of the habits that make tasks go well.

The sidebar, top to bottom#

The status rows. Three lines at the top show the state of things, each with buttons that say what they do:

The conversation. Your messages, the agent's replies, and a few kinds of cards: the hand-over card when the agent needs you, and a file card whenever a new file lands in its Downloads folder. Hover any message and a small copy icon appears in its corner; click it to put the whole message on your clipboard. Your own messages also get a send again icon next to it: one click sends the same text once more, handy when a run failed for a reason that had nothing to do with the task (no API key, the desktop was off) and you do not want to retype it. Attached files are not re-sent. The conversation stays until you start a new chat, so you can scroll back through the day's work.

The status row. One line saying Ready, Working, Paused, Done, Stopped or Error, often with a short message, and a running count for the conversation: tokens in, tokens out, how much came from the prompt cache, and for Claude models an estimated cost. "In" counts everything the model processed; because each step re-sends the conversation so far, that number grows quickly on long tasks, and the cached share is what keeps it cheap.

The composer. A text box with a paperclip for attaching files, a Run button (Send while a task is running), and, while the agent works, Pause, Resume and Stop buttons. Enter sends; Shift+Enter makes a new line.

Phrasing a task#

The agent is a capable reader of instructions, so write the way you would write to a colleague:

• Say where to go. "On booking.com" or "at https://…" saves a search. The agent types URLs straight into the address bar.
• Say what "done" looks like. "…and tell me the three cheapest with free cancellation" gives it a stopping point and a shape for the answer.
• Give the constraints up front. Dates, budgets, must-haves. It cannot ask a website what you meant.
• Multi-step is fine. "Open a terminal, run df -h, and tell me which disk is fullest." The agent batches obvious sequences by itself.
• Do not put secrets in the task. Anything you type goes to the model provider. When a login is needed, let the agent knock on the glass and type it yourself. See Knocking on the glass.

What happens after you press Run#

1. The Desktop tab opens beside the chat (if it is not open already), so you can watch. Your cursor stays in the chat box. Turn this off with deskfish.desktop.openOnRun.
2. If the tank is off, Deskfish turns it on first; the Desktop tab shows the progress.
3. The agent takes a screenshot, thinks, and decides on one or more actions.
4. The actions run on the tank, the screen settles for a moment, and a fresh screenshot is taken. That is one step.
5. Repeat until the agent decides the task is finished, at which point it writes a summary and stops.

In the chat, what the agent says appears as ordinary messages, with basic markdown (bold, inline code, headings). What it does is folded into a small chip that reads N actions; click it to expand the list. The chip turns red and says how many failed if any action did not work, for example an unknown key name. There are no screenshots in the chat on purpose: the Desktop tab is the screen.

The complete, unfolded log of every step is always in the Deskfish output channel (Deskfish: Show Log).

Talking to it while it works#

You can type while a task is running. The button says Send, and your message reaches the agent together with its next screenshot, so it can change course, take a hint, or answer a question mid-task.

Follow-ups keep their context#

When a task ends with Done, the next thing you type continues the same conversation. The agent remembers what it did, what it saw, and what it told you, so "how much was the second one?" or "now book it" work without repeating yourself.

The conversation starts fresh when:

• you click + (New chat) in the title bar of the sidebar, or
• you press Stop, or the task ends in an error, or
• you change the provider, the model, the base URL or the API key, or
• you reload VS Code (the conversation does not yet survive a reload).

Starting fresh#

The + button in the title bar of the Deskfish sidebar (or Deskfish: New Chat in the command palette) clears the conversation and the agent's memory of it. If a task is running, it is stopped first. The desktop is not touched: Firefox, its tabs, its logins and the files in the tank stay exactly as they were, so the new conversation starts by looking at whatever is on screen. The chat itself is not lost: every chat is saved as a transcript, and the clock icon next to + lists them, to reopen or to continue in a new chat. See Memory.

What the agent remembers#

Within one conversation the agent keeps the full text of everything: your messages, its replies, and the result of every action. What it does not keep is every picture. Only the three most recent screenshots stay in its context; older ones are replaced by a short note. That is what keeps long tasks affordable, and it is why the agent sometimes takes a fresh screenshot rather than reasoning from an old one.

Across chats, the agent keeps more than a list. Its Memory has facts about you and the sites it uses, a journal with one line per finished task, how-to playbooks it writes after learning a site, the transcripts of past chats, and a page about who it is. During a task it can search all of that with its recall tool, so "what did we do about the placements last week" is one call, not a re-investigation. Every fact, note and playbook it writes shows in the chat as a small Memory updated chip with a count; open it to read what was saved.

When the agent is waiting for something to finish, or spacing actions out, the chat shows a line of its own, Standing by, what for, and a countdown, and the status row says the same. It is not stuck: Deskfish is watching the screen for it without spending steps, and wakes it when the screen changes or the time is up. When the wait ends, the line says what happened. If waiting is part of the plan for a task, the agent says so in a sentence before it starts. Stop ends a wait at once. See Standing by.

A task can also start itself at a set time or on a repeat; see Schedules.

After every few finished tasks the status row says Reflecting… for a moment. The agent is alone with its notes: it saves what is worth keeping, and may make a small change to the page about itself. In the chat that whole moment is one folded card whose line says what changed; open it if you are curious. It costs a few cents and never runs while a task is in progress; deskfish.reflectEvery sets how often, 0 for only when you ask.

Pause, Resume, Stop#

• Pause hands you the desktop: the agent stops before its next action and the Desktop tab accepts your mouse and keyboard. The same happens when you click Take over in the Desktop tab.
• Resume gives control back. The agent is told that you had the desktop for a while and re-reads the screen before continuing.
• Stop ends the task now. The agent lets go of any key or mouse button it was holding.

Stop takes effect at once: a model call in flight is cancelled, a wait is cut short, and the status row says Stopping… until the loop has let go of the mouse and keyboard, usually well under a second.

Limits worth knowing#

• There is no step limit by default. Like a coding agent, the agent works until the task is done or you press Stop; the status row counts steps (Working · step 42) and shows the running cost, and that is your control. On Anthropic, the conversation is compacted server-side when it grows very long, so a task can run for hundreds of steps. If you want a hard cap, set deskfish.maxSteps: the agent is warned as it nears it, writes a summary at the cap, and continue resumes from there.
• It cannot wander for long. Deskfish compares each screenshot with the last one. If the agent repeats the same actions three times and nothing on screen changes, it is told to stop repeating and change approach. If that happens again, it knocks on the glass with "I seem to be stuck" so you decide what to do. Scrolling a long page or working through a form is not a stall: the screen changes and the actions differ.
• A cost budget, if you want one. deskfish.maxCostUsd caps what a task may spend (Claude models, list prices). At 80% the agent is told to wrap up; at the budget it writes a summary and stops, and continue resumes.
• After each batch prune, only the three most recent screenshots stay in the agent's memory; older ones are replaced by a note. It keeps the text of the whole conversation.
• One task at a time, on one screen. There is no audio and no webcam in the tank.

Some tasks that work well#

Find the three highest-rated Italian restaurants near Plaza Mayor in Madrid that are
open on Sunday evening, and give me their names, prices and links.

Go to the Debian download page, download the latest netinst ISO for amd64, and hand it
to me when it is done.

Open a terminal and show me how much free space the disk has.

On my account at example-shop.com (you will need me to log in), find my last order and
tell me its tracking number.