desktop template has XFCE on Xvfb, Chrome, xdotool and scrot. Nothing runs until you start the desktop, so an idle one costs the same as any other sandbox.
1
Start a desktop and take a screenshot
desktop_agent.py
2
Loop: look, decide, act
Send the screenshot to your model, then call
move_mouse, left_click, write or the other actions it returns. Desktop lists them.3
Watch it
desk.live_url() returns a live-view link. With control scope you can take over. For web pages only, the dom channel replays the page with passwords masked: Live view.