Driving the screen directly
Computer use is the capability of an AI to look at a computer screen and perform actions — clicking, typing, scrolling. It lets a model work with websites and desktop applications that offer no API, in much the way a person operates them.
Typical uses include finding information on a website, entering data into an admin interface, and moving between several services to complete a process. The difference from ordinary chat is that the model acts rather than only describing.
How it works
The model reads a screenshot or the structure of the screen, decides the next action, then examines the resulting screen and repeats look, decide, act until the goal is reached.
Implementations differ: some drive only a browser, some the whole computer, and some combine screen actions with code or API calls. OpenAI and Anthropic both offer models and mechanisms for this.
How it differs from tool calling
With tool calling and MCP, the model selects from prepared functions such as fetch my schedule or draft an email. Inputs and outputs have defined shapes, which is generally faster than clicking and makes errors easier to detect.
Computer use works where no integration exists, but it can get lost when a button moves or the layout changes. Doing what APIs and MCP can do reliably through those, and reserving computer use for what only exists on screen, is the practical arrangement.
Using it safely
- Require human confirmation before deleting, purchasing, sending, or publishing
- Restrict which sites, applications, and folders can be operated
- Do not hand over more credentials or permissions than the task needs
- Do not let instructions embedded in a page be treated as commands
- Keep a record of actions so results can be reviewed afterward
Web pages can carry prompt injection aimed at the model reading them. Computer use is powerful and correspondingly wide in blast radius, so run it in an isolated environment and keep human approval on consequential actions.