Agent S — Open Computer-Use Agent (S1 to S4, Real GUI Control on macOS, Windows and Linux)

Open computer-use agent: give it a task in plain English and it looks at the screen, then clicks, types and scrolls through real desktop and web apps on macOS, Windows or Linux. Ships the S1, S2, S2.5, S3 and S4 agent generations with grounding models, behaviour best-of-N, per-agent memory, OSWorld and WindowsAgentArena harnesses, and Simular integrations.

Uploaded by TalentDelta Team Member (Scout) · via TalentDelta

Computer-Use Automation Browser Automation Task Decomposition Tool Use Planning Long-Horizon Reasoning Agent Evaluation AI Agents Automation Robotic Process Automation (RPA)

build 1.0.0 · 3962.5 KB · published 2026-10-24 · 356 files

Licence: Apache-2.0

Provenance

Scouted from github.com/simular-ai/Agent-S at commit 0fff0e634869dab72027435b4586e62300185d41 (2026-10-21). Licence Apache-2.0. Upstream credit: Simular (simular-ai). Packaged byte-identical to the pin apart from the provenance record appended to README.md. Upstream code and licence stay upstream's.

Brought into the catalogue by TalentDelta Team Member (Scout).

Our review

What we like

Five generations of a computer-use agent — S1, S2, S2.5, S3 and S4 — with the OSWorld and WindowsAgentArena harnesses they were measured on. 259 Python files, evaluation code, and the vendored runtime's own MIT licence kept intact.

Potential improvements

The eval harnesses hard-code the sandbox user path, so they only run where that user exists. No identity or config file ships, so nothing tells an installed agent who it is.

Minimum requirements

Python 3, a virtual machine to host the desktop being driven, and a vision-capable model key.

Security risks

This agent controls a real graphical desktop: it can click, type and read whatever is on screen. Run it against an isolated virtual machine, never a daily-use machine, and treat any credential visible there as exposed.

Tested: packaging, counts, licences and credential sweep. Not tested: a live graphical run.

356 files · 6033.3 KB · previews are generated at approval and never executed

agent-s-computer-use/.github/workflows/lint.ymlcode · 0.8 KB

name: lint
on:
  pull_request:
    types: [opened, reopened, synchronize]
    paths:
      - "gui_agents/**"
      - "tests/**"
      - ".github/workflows/lint.yml"
  push:
    branches:
      - main
    paths:
      - "gui_agents/**"
      - "tests/**"
      - ".github/workflows/lint.yml"

env:
  SUPPORTED_PYTHON_VERSIONS: "3.11"

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11"]
    steps:
    - uses: actions/checkout@v3

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -e .[dev]

    - name: Run Linter
      run: |
        black --check gui_agents agent_s4

agent-s-computer-use/.gitignoretext · 3.1 KB

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
#   However, in case of collaboration, if having platform-specific dependencies or dependencies
#   having no cross-platform support, pipenv may install dependencies that don't work, or not
#   install all needed dependencies.
#Pipfile.lock

# poetry
#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
#   This is especially recommended for binary packages to ensure reproducibility, and is more
#   commonly ignored for libraries.
#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
#   in version control.
#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#  and can be added to the global gitignore or merged into this file.  For a more nuclear
#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
logs/
.DS_Store

agent-s-computer-use/LICENSEtext · 11.1 KB

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

agent-s-computer-use/README.mdmarkdown · 11.4 KB

Agent S is an open-source computer use agent framework from Simular. Give it a task in plain English. It looks at the screen and completes the task by clicking, typing, and scrolling in ordinary desktop and web apps. It runs on macOS, Windows, and Linux.

🥳 Updates

2026/10/20: Released Agent S4 and its paper "Evolving Agent Interfaces": an autonomous computer-use scientist that evolves the interfaces its agents work through from task experience. With Qwen3.8-27B, Agent S4 sets a new open-source state of the art on OSWorld V2 at 21.3%, 3.8 times the same model with fixed interfaces and above GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%) at a fraction of their cost. (OSWorld V2 is scored as a strict binary pass/fail with no partial credit, so 21.3% reflects benchmark difficulty, not low reliability — see the Agent S4 README for details.)
2026/10/01: The Simular API is live. Build your always-on computer agents in a minute. Simular Platform is an autonomous computer API for building always-on agents that fit your needs. World-leading agent infrastructure, 90% token saving. Get an API key →

🚀 Quick Start

No setup: run Sai, our production agent, on a cloud computer through the Simular API. See the API docs and the workflow template gallery.
On your own computer: follow the Agent S3 README.

📚 Agent S Versions

Version	What it is	Docs	Paper
Agent S4	Evolves the tool interfaces a computer use agent works through, from its own task experience. Open-source state of the art on OSWorld V2 (21.3% with Qwen3.8-27B).	README	arXiv 2026
Agent S3	The agent in `gui-agents` today. First to surpass human performance on OSWorld (72.60%).	README	TMLR 2026
Agent S2.5	A simpler, faster Agent S2.	README	—
Agent S2	A compositional generalist-specialist framework.	README	COLM 2025
Agent S	The first version: uses computers like a human, through an agent-computer interface.	README	ICLR 2025

💬 Citations

If you find this codebase useful, please cite:

@misc{Agent-S4,
      title={Evolving Agent Interfaces}, 
      author={Saaket Agashe and Yuzhe Yang and Gonzalo Gonzalez-Pumariega and Bhuvan Chandra Koduru and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2026},
      eprint={2610.28601},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.28601}, 
}

@misc{Agent-S3,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Agent-S2,
      title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.00906}, 
}

@inproceedings{Agent-S,
    title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2025},
    url={https://arxiv.org/abs/2410.08164}
}

Star History

TalentDelta provenance record (added at packaging)

Packaged for: TalentDelta catalogue (draft; gated by review)
Upstream: simular-ai/Agent-S
Pinned commit: 0fff0e634869dab72027435b4586e62300185d41 (2026-10-21)
Licence: Apache-2.0 — Apache License 2.0 at the repository root, shipped byte-identical to the pin. Permissive: use, modification, distribution and commercial use allowed with attribution, licence text and change notices. No NOTICE file exists upstream; no per-file headers are added and no file is modified. Attribution to Simular is recorded in the pack README's provenance section and in this upload record.
Upstream author credit: Simular (simular-ai)
Runtime: Python 3.11+ with platform extras (pip install -r requirements.txt; pyobjc on macOS, pywinauto/pywin32 on Windows); GUI control libraries pyautogui, pytesseract and PaddleOCR; model backends through OpenAI, Anthropic, Google GenAI or Together keys; optional OSWorld / WindowsAgentArena VMs for evaluation
Changes made at packaging: this section was appended to README.md. No upstream path is excluded. Apart from this provenance section appended to README.md, no file was added, removed, renamed or edited; every upstream file is byte-identical to the pinned commit.
Safety read of the packaged archive (packaging time, 2026-10-24): Static safety read of the shipped archive (356 files): 75 pattern hits, none a live credential. 40 are api-key patterns, all of them environment-variable reads or client-construction arguments with no literal value; 31 are home-directory placeholders inside the VM provisioning scripts; 2 are addresses - the maintainer's in setup.py metadata and an example address in a test; 2 are absolute user paths from upstream, one a default benchmark example directory. No .env file ships.

Packaging notes (TalentDelta)

A working agent, not a prompt set: each generation ships an agent class, grounding model wiring, action loop, memory and worker code under gui_agents/ and agent_s4/.
Computer use is powerful by construction: the agent clicks and types in real applications with the operator's own session. The shipped README and agent READMEs describe the sandbox/VM route for evaluation; the pack adds no elevation and no credential handling of its own.
No API keys or model weights ship. Keys are read from the operator's environment; requirements.txt names the vendor SDKs only.
Upstream benchmark claims (OSWorld, WindowsAgentArena, ScienceBoard) are quoted from its README and papers; this pack re-runs none of them and asserts no score.
Screenshots in images/ are upstream product and benchmark figures, not captures of any user's desktop.

agent-s-computer-use/WAA_setup.mdmarkdown · 10.0 KB
Introduction

This is the WindowsAgentArena (WAA) setup with Agent S2.5 (and beyond). Why do we need a setup guide? Despite the thorough README.md, we have to include our code into their repository and fix up a number of setup issues from the WAA environment. Sadly, this isn’t the most straightforward.

Initial WAA Setup

The initial WAA setup is straightforward. Follow the README.md on their repository. After you’ve finished this, try running run-local.sh. This will start up an experiment with their default Navi agent. At this point, the environment is sufficient to run evaluation, but it’s incomplete and thus the evaluation won’t be exactly correct due to environment issues.

agent-s-computer-use/images/waa_setup/fig1.png

Figure 1: Bash script chain of execution.

While we’re at it, look to understand the following things:
- the entire README.md (especially the Bring Your Own Agent guide)
- the long chain of bash scripts that start the run (Figure 1)
- the run.py to see how the agent/environment are instantiated and used together
- the folder structure of the repository and the purpose of each folder
Fixing Setup Issues

By now, your WAA environment should be set up to run locally. There are two major problems:
- setup issues
- the VM persists across examples (it won’t reset after every example is completed which may make evaluation unfair)
Let’s tackle the first one: setup issues.

Office Apps Aren’t Installed

The first issue I ran into was the office apps aren’t installed. Why is that? Turns out all apps installed in the VM during the initial setup stage install via the links from this file (tools_config.json). At the time of writing this, only the office links do not work. Try out all the links to make sure they work. If the links do not lead to a download (and some error occurs), then that app was not installed in the VM. What do we do? Two options:
- redo the entire initial setup stage (time consuming; ~4 hours for me and even then, it would just not work a lot of the times; ideally, WAA is setup on Linux as I’ve had no issues so far with it)
- Enter the VM and install the apps manually (easier and faster)
We’ll do the second approach.

You can access the VM via https://localhost:8006. You can turn the VM on by run-local.sh. There’s probably a better/faster way to do it, but this doesn’t take too much time anyways (~1-2 mins). After the VM has started, enter the VM (the agent may be trying to take actions, but you can either just override the action in run.py with import time; time.sleep(10000) here or fight the agent for control of the VM!).

Inside the VM, navigate to their download page and download the latest LibreOffice version. After it’s downloaded, complete the setup wizard and make sure to delete the downloaded *.msi file in the VM. Finally, test the download by opening up LibreOffice Writer and Calc.

Google Chrome Pop-ups

In Google Chrome, there a couple unexpected pop-ups.

agent-s-computer-use/images/waa_setup/fig2.png

Figure 2: Pop-ups on Chrome.

Close all these pop-ups and make Google Chrome your default web browser.

VSCode Pop-ups

This isn’t as important, but there are a couple initial pop-ups in VSCode that you can close.

Note: set_cell_values

Important if you’re using set_cell_values

Agent S2.5 uses a special grounding function called set_cell_values that takes advantage of the soffice CLI and unotools Python library. TL; DR, this function lets the agent set the cell values for a given spreadsheet and sheet.

For this function to work on WAA, the set up is a bit messy…
1. Connect into the VM
2. Open up a terminal and run python --version, you should see you’re using the GIMP Python which is 2.x. This won’t let you use the soffice CLI or import uno in Python code.
3. In the Desktop directory within a terminal, do pip freeze > requirements.txt to save all the PYPI libraries from the GIMP Python to a requirements.txt.
4. Configuring Python path to LibreOffice’s Python
  1. In the File Explorer, locate the python.exe file from LibreOffice. You can do this with where python. Copy this path.
  2. In the Search bar in the bottom task bar inside the VM, search for “environment variables”.
  3. Click on “Environment Variables” and click on “Path” under “System variables”. Paste the copied path from step (a) into there and ensure this path is above the GIMP Python path so it takes precedence.
  4. Reopen a terminal and run soffice to ensure it is now working. Create a temporary python file and ensure import uno works.
5. LibreOffice’s Python should be 3.10 or above. However, it does not come with pip. To install pip, download this file and execute python get-pip.py to install it. Ensure the python here is LibreOffice’s Python. Next, install pip install -r requirements.txt using the requirements.txt from step 3. This is to ensure LibreOffice’s Python has all the dependencies needed for evaluation (pyautogui, etc).
6. Clean up all installer files. Then, inside the WAA repository code, change this line
command_list = ["python", "-c", self.pkgs_prefix.format(command=command)]

to:

command_list = ["absolute/path/to/libreoffice/python", "-c", self.pkgs_prefix.format(command=command)]

This ensures that the subprocess running in the flask server inside the VM will use that specific Python version.

Double Checking…

Double check all apps can be used and no unexpected pop-ups or issues are in the way. Any apps you open make sure to close them upon finishing your clean-up. Make sure any installation files you have in Downloads are deleted (and removed from Recycle Bin) to keep the environment clean. At the end, this is our golden image. You may want to save a copy of this VM somewhere safe so that you can always copy it back into the WAA repository to be reused (refer to this).

Set up Agent S2.5 with WAA Locally

Take the time to understand the Agent-S repository.
1. Instead of following the README.md for Agent S2.5, you need to clone the repository then pip install -r requirements.txt
2. Move the S2.5 folder to the mm_agents folder in WAA. Follow the Bring Your Own Agent guide.
  1. You will need to move the agent_s.py file out to the S2.5 folder and update all the relevant import statements
3. Make the necessary changes in run.py and lib_run_single.py to accommodate Agent S2.5 (replace the Navi Agent with Agent S2.5).
4. Test it by running the experiments! Don’t forget when you do run-local.sh, now you need to specify Agent S2.5 instead of the navi agent agent="agent_s".
5. You may have some import errors and these libraries need to be installed inside the winarena container (I think). You can just add the pip install commands to the bash script where the error stems from (hacky).
Agent S2.5 with WAA on Azure
1. Ensure you have:
  1. a clean copy of the golden image
  2. the correct Azure subscription (so you’re not using your own payment method)
2. Follow the Azure deployment in the README.md.
3. Test it! If this works, then we have a resettable golden image and WAA can be ran in parallel, making evaluation much much faster! Good luck!

agent-s-computer-use/agent_s4/LICENSEtext · 11.1 KB

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

agent-s-computer-use/agent_s4/README.mdmarkdown · 9.9 KB

agent_s4

The Agent S4 package, from the paper Evolving Agent Interfaces. It evolves a shadow interface: a small Python tool library that a computer-use agent imports to operate a desktop. Each step, a Builder agent edits an interface on the host. A task agent then runs benchmark tasks on fresh AWS VMs with that interface installed, and its score becomes the variant's fitness.

Computer-use agents are limited by the interfaces humans give them: an agent may type a spreadsheet cell by cell when one line of code would do. Agent S4 builds those interfaces itself. The Builder studies the task agent's trajectories on synthetic training tasks, without success labels, and writes and revises the interfaces and the knowledge of when to use them. Population-based evolution then keeps the variants that score best on validation tasks the Builder never sees, which favors changes that generalize.

Results

On OSWorld V2, Agent S4 with Qwen3.8-27B and its evolved interfaces reaches a 21.3% binary score, a new state of the art for open-source computer-use agents and 3.8 times the same model with fixed interfaces. That is above GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs.

Note: OSWorld V2 scores a task as a strict binary pass/fail — 1 only if every part of the task's checker succeeds, 0 otherwise, with no partial credit for getting most of the way there. That's why these numbers look low next to benchmarks that award partial credit: OSWorld V2 is widely regarded as the hardest computer-use benchmark, and these are state-of-the-art results under that strict scoring, not an indication of low real-world reliability.

On ScienceBoard, the evolved interfaces raise the success rate from 41.4% to 58.6% with the same model.

Setup guides

This file describes the code. For setup, credentials and the launch scripts behind the published runs, see the workspace READMEs in the Agent-S repository:

osworld_setup/s4/README.md: OSWorld-V2, which is the default benchmark.
scienceboard_setup/s4/README.md: ScienceBoard.

Install

agent-s4 is its own package; installing gui-agents does not include it.

The package contains only the code. A run also reads setup files from the Agent-S repository: the Builder prompts, the seed interface, the task splits and the pinned Prime Agent runtime, all under osworld_setup/s4/. So you need a clone of the repository either way. Install the package from the clone:

git clone https://github.com/simular-ai/Agent-S.git
cd Agent-S
pip install -e ./agent_s4

or install it from PyPI and point it at the clone:

pip install agent-s4
export SHADOW_EVOL_SHARED_ROOT=/path/to/Agent-S/osworld_setup/s4

Without those files, python -m agent_s4.cli run stops and says what to set. The benchmark itself (the OSWorld-V2 checkout, AWS and model credentials) is set up as described in the workspace READMEs above.

Entry points

Command	What it does
`python -m agent_s4.cli run ...`	Starts or resumes an experiment (`--help` lists the options).
`python -m agent_s4.cli rerun --config-path runs/<timestamp>/config.json`	Runs again from a saved config.
`python -m agent_s4.osworld_v2_preflight`	Runs the offline OSWorld-V2 workspace checks on their own.
`python -m agent_s4.eval.osworld_v2 ...`, `python -m agent_s4.eval.scienceboard_prime ...`	Eval workers. The evaluator harness normally starts these; they can also be run by hand on chosen tasks.

run defaults to --benchmark osworld_v2, which uses the splits pinned in config/osworld_v2_release_pin.json: osworld_v2_train_50.json, osworld_v2_val_25.json and osworld_v2_test.json as the held-out set. No train or val splits ship for --benchmark osworld (the original OSWorld) or --benchmark scienceboard, so both require --train-set and --val-set.

How a run works

1. Bootstrap (orchestrator.run):

If Prime Agent is used, checks its pinned runtime. For OSWorld-V2 it also runs the workspace preflight and checks the configured splits against the release pin.
Makes one clone of the benchmark's Evol checkout per island, named <template>_<island> (for example OSWorldV2_Evol_<island>).
Copies the seed interface into the clone's shadow_interface/ directory, which has its own git repository, and records it as the seed variant.
Evaluates the seed on every island in parallel, writes the resolved config to runs/<timestamp>/config.json, and hands over to the driver.

2. Driver loop (driver.Driver):

Keeps one global step counter.
Each step, the scheduler (config/scheduler.py) picks one operator, which runs on every island in parallel. The step ends only when all islands have finished.
Every dispatch is logged in the database, so a step interrupted by a crash is resumed rather than run twice.
Validation reevaluation runs every --reevaluation-interval steps.
The loop stops at --max-steps or --max-cost.

3. Operators (operators/): each step runs one of these.

Operator	What it does
`mutation`	The usual step; see the sequence below.
`crossover`	The Builder combines two parents from the same island into one child, which is then evaluated like a mutation.
`migration`	Two phases: each island sends a tournament pick to another island, and the receiving island evaluates the migrant with its own task agent before adding it. `--migration-mode` and `--migration-topology` control the details.
`elimination`	Removes the lowest-ranked variants from each island.
`reevaluation`	Evaluates each island's current leaders on validation again and averages the results with their earlier scores. It creates no new variants.

A mutation runs these steps:

A tournament picks the parent (tournament.py).
The parent is checked out, and its task trajectories are staged for the Builder to read.
Optionally, a scratch VM is started (scratch_vm.py) so the Builder can try things in a live app.
The Builder harness edits shadow_interface/. For OSWorld-V2 and ScienceBoard it runs inside a Landlock sandbox, because the Base checkout next to the clone holds the benchmark's gated evaluators.
Edits outside the allowed scope are reverted, and the child is tagged evol/<island>/if-NNN.
The evaluator harness runs the child on the train and val splits. The per-task results, scores and costs go into the database.

Module map

Area	Modules
Configuration	`cli.py` (argument parsing), `config/experiment.py` (`ExperimentConfig`, default splits per benchmark), `config/island.py`, `config/scheduler.py`
Control loop	`orchestrator.py` (bootstrap and seed eval), `driver.py`, `state.py` (`LoopState`, passed to each operator), `operators/`, `tournament.py`, `scoring.py`
State	`db.py` (SQLite tables: variants, per-task eval results, islands, history, cost events, dispatch log, reevaluation samples), `costs.py`, `git_utils.py` (variant tags and checkouts in each island's `shadow_interface/` repo)
Builder side	`harness/builder_common.py` (shared Builder subprocess and log parsing), `builder_sandbox.py` (Landlock), `builder_scratch_vm.py`, `scratch_vm.py`
Agent harnesses	`harness/prime/`, `harness/opencode/`: each has the Builder (`builder.py`), the task-agent install and VM command (`evaluator.py`), a runner, and pin checks (`source.py`); see the README in each directory. `harness/__init__.py` is the registry.
Evaluation	`harness/ubuntu_vm.py` (`UbuntuVMEvaluator`: starts an eval worker as a subprocess), `eval/osworld.py` (worker code shared by every benchmark), `eval/osworld_v2.py`, `eval/scienceboard_prime.py`, `eval/grader_image_limit.py`
Benchmarks	`benchmark_spec.py` (template and base checkout names, how run artefacts are named), `osworld_v2_preflight.py`, `synthetic_tasks.py`, `scienceboard_synthetic_runtime.py`
VMs	`vm/utils.py` (shared VM helpers), `vm/owner_tag.py` (adds an `Owner` tag to every EC2 instance)
Paths	`paths.py`, `constants.py`, `utils.py`

Where files go

paths.py sets two roots:

Shared root (SHADOW_EVOL_SHARED_ROOT; defaults to osworld_setup/s4/): the vendored Prime runtime, vm_assets/, the seed interfaces and the generic Builder prompts.
Project root (SHADOW_EVOL_PROJECT_ROOT; defaults to the shared root): the run workspace. It holds the benchmark's prompts, splits, data, release pin, checkouts and every run artefact. The ScienceBoard launchers set it to scienceboard_setup/s4/.

Run artefacts are named per benchmark (and per SHADOW_EVOL_RUN_NAMESPACE) by benchmark_spec.ns(). For example, OSWorld-V2 uses shadow_evolution_osworld_v2_<ns>.db, eval_cache_osworld_v2_<ns>/ and builder_logs_osworld_v2_<ns>/.

Extending

Add an agent harness:

Subclass BuilderHarness (harness/base.py) in harness/<name>/builder.py.
Put the VM install and task-agent command in harness/<name>/evaluator.py.
Register the key in both BUILDER_HARNESSES and EVALUATOR_HARNESSES.
Teach UbuntuVMEvaluator and eval/osworld.py to dispatch on the key. Both currently accept only prime_agent and opencode.
Pin and verify the upstream release in a source.py, and record it in a README, as the existing two harnesses do.

Add a benchmark:

Add an entry to benchmark_spec.BENCHMARKS, and to _DEFAULT_SPLITS in config/experiment.py if it ships default splits.
Write an eval worker that plugs into eval.osworld.main(impl=...), as eval/scienceboard_prime.py does.
The benchmark-specific branches in harness/ubuntu_vm.py and orchestrator.py (search for config.benchmark ==) cover how the base checkout is chosen, which prompt addenda are added and how splits are checked.

agent-s-computer-use/agent_s4/__init__.pycode · 1.1 KB

"""Shadow interface evolution framework.

Side-effect on import: set PROXY_CONFIG_FILE to an OSWorld template copy
if the env var isn't already set and one of the templates carries it.

This has to happen BEFORE `desktop_env` is imported anywhere under
agent_s4/, because desktop_env.controllers.setup initializes its
global ProxyPool from PROXY_CONFIG_FILE at module-import time. The
default is a cwd-relative path that only resolves when run from inside
a clone — the orchestrator runs from the repo root, one level up.

Both OSWorld templates ship the same dataimpulse.json, so the first one
present wins; a checkout with only one of them still resolves.
"""

import os

from agent_s4.paths import PROJECT_ROOT as _REPO_ROOT

if "PROXY_CONFIG_FILE" not in os.environ:
    for _template in ("OSWorld_Evol", "OSWorldV2_Evol"):
        _proxy = (
            _REPO_ROOT
            / _template
            / "evaluation_examples"
            / "settings"
            / "proxy"
            / "dataimpulse.json"
        )
        if _proxy.is_file():
            os.environ["PROXY_CONFIG_FILE"] = str(_proxy)
            break

agent-s-computer-use/agent_s4/benchmark_spec.pycode · 2.3 KB

"""Benchmark descriptor + workspace-artifact namespacing.

Pure data + one pure function. Imports nothing from the package, so it can be
imported anywhere without risking an import cycle. The benchmark->evaluator-class
mapping deliberately lives in agent_s4/harness/__init__.py, not here.
"""

from __future__ import annotations

import os
import re

# template / base are distinct benchmark checkout directory names taken
# verbatim.  OSWorld v1 currently uses registered git submodules; OSWorld V2
# and ScienceBoard use release-pinned local clones that are intentionally
# ignored by the parent repository.  Callers must not infer git ownership from
# these names.
# tag is the suffix ns() applies to the four bare run-workspace constants in
# constants.py (db, eval_cache, builder_logs, migrate_transit). osworld == ""
# so every osworld path is the current one (back-compat).
BENCHMARKS = {
    "osworld": {"template": "OSWorld_Evol", "base": "OSWorld_Base", "tag": ""},
    "scienceboard": {
        "template": "ScienceBoard_Evol",
        "base": "ScienceBoard_Base",
        "tag": "_scienceboard",
    },
    # OSWorld V2 (xlang-ai/OSWorld-V2). Its DesktopEnv kwargs are a superset of
    # the v1 fork's, so eval/osworld.py drives it unchanged; only the template /
    # base directory names and the artifact tag differ.
    "osworld_v2": {
        "template": "OSWorldV2_Evol",
        "base": "OSWorldV2_Base",
        "tag": "_osworld_v2",
    },
}


def ns(name: str, benchmark: str) -> str:
    """Namespace a run-workspace artifact name by benchmark so OSWorld and
    ScienceBoard never share workspace state. osworld -> bare name; others ->
    tag inserted before any extension ('shadow_evolution.db' ->
    'shadow_evolution_scienceboard.db', 'eval_cache' -> 'eval_cache_scienceboard')."""
    tag = BENCHMARKS[benchmark]["tag"]
    run_namespace = os.environ.get("SHADOW_EVOL_RUN_NAMESPACE", "").strip()
    if run_namespace:
        if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_-]*", run_namespace):
            raise ValueError(
                "SHADOW_EVOL_RUN_NAMESPACE must contain only letters, digits, "
                f"underscores, or hyphens, got {run_namespace!r}"
            )
        tag += f"_{run_namespace}"
    if not tag:
        return name
    stem, dot, ext = name.partition(".")
    return f"{stem}{tag}{dot}{ext}"

agent-s-computer-use/agent_s4/builder_sandbox.pycode · 13.2 KB · preview truncated

"""Unprivileged Landlock filesystem sandbox for OSWorld V2 Builders."""

from __future__ import annotations

import argparse
import ctypes
import errno
import hashlib
import os
import shutil
import stat
import sys
import tempfile
from pathlib import Path

_CREATE_RULESET = 444
_ADD_RULE = 445
_RESTRICT_SELF = 446
_CREATE_RULESET_VERSION = 1
_RULE_PATH_BENEATH = 1
_PR_SET_NO_NEW_PRIVS = 38

_EXECUTE = 1 << 0
_WRITE_FILE = 1 << 1
_READ_FILE = 1 << 2
_READ_DIR = 1 << 3
_REMOVE_DIR = 1 << 4
_REMOVE_FILE = 1 << 5
_MAKE_CHAR = 1 << 6
_MAKE_DIR = 1 << 7
_MAKE_REG = 1 << 8
_MAKE_SOCK = 1 << 9
_MAKE_FIFO = 1 << 10
_MAKE_BLOCK = 1 << 11
_MAKE_SYM = 1 << 12
_REFER = 1 << 13
_TRUNCATE = 1 << 14


class _RulesetAttr(ctypes.Structure):
    _fields_ = [("handled_access_fs", ctypes.c_uint64)]


class _PathBeneathAttr(ctypes.Structure):
    _fields_ = [
        ("allowed_access", ctypes.c_uint64),
        ("parent_fd", ctypes.c_int32),
    ]


def _syscall(number: int, *args) -> int:
    libc = ctypes.CDLL(None, use_errno=True)
    result = libc.syscall(number, *args)
    if result < 0:
        code = ctypes.get_errno()
        raise OSError(code, os.strerror(code))
    return int(result)


def landlock_abi() -> int:
    """Return the host Landlock ABI, or zero when unsupported/disabled."""
    if sys.platform != "linux":
        return 0
    try:
        return _syscall(
            _CREATE_RULESET,
            ctypes.c_void_p(),
            ctypes.c_size_t(0),
            ctypes.c_uint32(_CREATE_RULESET_VERSION),
        )
    except OSError as exc:
        if exc.errno in {errno.ENOSYS, errno.EOPNOTSUPP, errno.EINVAL}:
            return 0
        raise


def _handled_access(abi: int) -> int:
    access = (1 << 13) - 1
    if abi >= 2:
        access |= _REFER
    if abi >= 3:
        access |= _TRUNCATE
    return access


def _add_path_rule(ruleset_fd: int, path: Path, access: int, handled: int) -> None:
    if not path.exists():
        return
    path = path.resolve()
    is_dir = path.is_dir()
    if is_dir:
        allowed = access
    else:
        # Directory mutation and READ_DIR rights cannot be granted to files.
        directory_only = (
            _READ_DIR
            | _REMOVE_DIR
            | _REMOVE_FILE
            | _MAKE_CHAR
            | _MAKE_DIR
            | _MAKE_REG
            | _MAKE_SOCK
            | _MAKE_FIFO
            | _MAKE_BLOCK
            | _MAKE_SYM
            | _REFER
        )
        allowed = access & ~directory_only
    fd = os.open(path, os.O_PATH | os.O_CLOEXEC)
    try:
        attr = _PathBeneathAttr(allowed & handled, fd)
        _syscall(
            _ADD_RULE,
            ctypes.c_int(ruleset_fd),
            ctypes.c_int(_RULE_PATH_BENEATH),
            ctypes.byref(attr),
            ctypes.c_uint32(0),
        )
    finally:
        os.close(fd)


def apply_builder_landlock(
    *,
    workspace: Path,
    executable_paths: list[Path],
    read_paths: list[Path] | None = None,
) -> None:
    """Allow only the Builder workspace plus minimal runtime/config paths.

    ``read_paths`` grants read+execute on additional roots an agent runtime
    needs but cannot reach from its own install dir (e.g. a pinned source
    tree with its own venv's site-packages).
    """
    abi = landlock_abi()
    if abi == 0:
        raise RuntimeError("Landlock is unavailable; refusing unsandboxed V2 Builder")
    handled = _handled_access(abi)
    attr = _RulesetAttr(handled)
    ruleset_fd = _syscall(
        _CREATE_RULESET,
        ctypes.byref(attr),
        ctypes.sizeof(attr),
        ctypes.c_uint32(0),
    )
    read_only = _EXECUTE | _READ_FILE | _READ_DIR
    read_write = handled
    try:
        for path in (
            Path("/usr"),
            Path("/bin"),
            Path("/sbin"),
            Path("/lib"),
            Path("/lib64"),
            Path("/etc"),
            Path("/proc"),
            Path("/sys"),
        ):
            _add_path_rule(ruleset_fd, path, read_only, handled)
        # /etc/resolv.conf commonly points into /run/systemd/resolve. Landlock
        # follows inode rules rather than treating /etc's symlink as authority
        # over its external target; without this file, Builders have network
        # access but every model request fails at DNS resolution.
        _add_path_rule(
            ruleset_fd,
            Path("/etc/resolv.conf").resolve(),
            read_only,
            handled,
        )
        # The run's locked interpreter/dependencies are read-only. The
        # standalone scratch bridge is refreshed inside ``workspace`` before
        # launch, so no parent-project source tree needs to be exposed.
        _add_path_rule(ruleset_fd, Path(sys.prefix), read_only, handled)
        for path in (Path(f"/run/user/{os.getuid()}"),):
            _add_path_rule(ruleset_fd, path, read_write, handled)
        for path in (
            Path("/dev/null"),
            Path("/dev/zero"),
            Path("/dev/random"),
            Path("/dev/urandom"),
            Path("/dev/tty"),
        ):
            _add_path_rule(ruleset_fd, path, _READ_FILE | _WRITE_FILE, handled)

        home = Path.home()
        for relative in (
            ".claude",
            ".claude.json",
            ".qwen",
            ".gitconfig",
            ".config/claude",
            ".config/qwen",
            ".cache/claude",
            ".cache/qwen",
            ".local/state/claude",
            ".local/state/qwen",
        ):
            _add_path_rule(ruleset_fd, home / relative, read_write, handled)
        for extra in read_paths or []:
            _add_path_rule(ruleset_fd, extra, read_only, handled)
        for executable in executable_paths:
            _add_path_rule(ruleset_fd, executable, read_only, handled)
            _add_path_rule(ruleset_fd, executable.resolve(), read_only, handled)
            install_dir = executable.resolve().parent
            # JS/native CLIs often load sibling runtime files. Grant only the
            # installation directory, never a parent that contains the V2
            # workspace or its gated sibling repositories.
            workspace_path = workspace.resolve()
            if install_dir not in {workspace_path, *workspace_path.parents}:
                _add_path_rule(ruleset_fd, install_dir, read_only, handled)
        workspace = workspace.resolve()
        # The agent may inspect the whole island, but only the evolved inner
        # repository, notebook, and private temp directory are mutable.
        _add_path_rule(ruleset_fd, workspace, read_only, handled)
        _add_path_rule(ruleset_fd, workspace / "shadow_interface", read_write, handled)
        _add_path_rule(ruleset_fd, workspace / "notebook.jsonl", read_write, handled)
        _add_path_rule(ruleset_fd, workspace / ".builder_tmp", read_write, handled)

        libc = ctypes.CDLL(None, use_errno=True)
        if libc.prctl(_PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0:
            code = ctypes.get_errno()
            raise OSError(code, os.strerror(code))
        _syscall(_RESTRICT_SELF, ctypes.c_int(ruleset_fd), ctypes.c_uint32(0))
    finally:
        os.close(ruleset_fd)


def sandboxed_builder_argv(
    argv: list[str], clone_dir: Path, read_paths: list[Path] | None = None
) -> list[str]:
    """Wrap one command with this module while preserving the original argv."""
    resolved = shutil.which(argv[0])
    if resolved is None and Path(argv[0]).is_file():
        resolved = str(Path(argv[0]).resolve())
    if resolved is None:
        # Preserve the harness's normal not-found result rather than failing
        # with a misleading sandbox diagnostic.
        return argv
    extra: list[str] = []
    for path in read_paths or []:
        extra += ["--read-path", str(Path(path).resolve())]
    return [
        sys.executable,
        str(Path(__file__).resolve()),
        "--workspace",
        str(Path(clone_dir).resolve()),
        "--executable",
        resolved,
        *extra,
        "--",
        *argv,
    ]


# sockaddr_un.sun_path holds 108 bytes including the terminator, so a socket
# pathname may be at most 107 characters. Prime Agent appends its own names to
# TMPDIR, the longest observed being
# "/prime-agent-<uid>/worker-<12 hex>-<12 hex>.sock" (55 characters for a
# four-digit uid). Past this budget bind() truncates silently and Prime then
# lstat()s the untruncated name and dies with ENOENT, which surfaces as a
# Builder exit far from its cause.
_SOCKET_PATHNAME_MAX = 107
_PRIME_SOCKET_SUFFIX_BUDGET = 55
MAX_TMPDIR_LEN = _SOCKET_PATHNAME_MAX - _PRIME_SOCKET_SUFFIX_BUDGET


def _alias_parent_candidates() -> list[tuple[Path, bool]]:
    """Directories that could hold the alias, most preferred first.

    The second element says whether the directory is shared with other users
    and therefore needs a private, ownership-checked parent of our own before
    a symlink inside it can be trusted.
    """
    candidates: list[tuple[Path, bool]] = []
    for value in (os.environ.get("XDG_RUNTIME_DIR"), f"/run/user/{os.getuid()}"):
        if value:
            candidates.append((Path(value), False))
    # Hosts without a systemd user session have no /run/user/<uid> at all, and
    # the fallback has to be short or it defeats the purpose.
    candidates.append(
        (Path(tempfile.gettempdir()) / f"shadow-builder-{os.getuid()}", True)
    )
    return candidates


def _usable_alias_parent(directory: Path, needs_private_parent: bool) -> bool:
    """Whether we own *directory* outright, creating it when it is ours to make."""
    try:
        if needs_private_parent:
            directory.mkdir(mode=0o700, exist_ok=True)
        info = directory.lstat()
    except OSError:
        return False
    # A symlink or another user's directory could redirect the Builder's temp
    # files, so require a real directory owned by us.
    return (
        stat.S_ISDIR(info.st_mode)
        and info.st_uid == os.getuid()
        and not (needs_private_parent and info.st_mode & 0o077)
    )


def _short_tmp_alias(workspace: Path, private_tmp: Path) -> Path:
    """Return a short pathname for the private temp inode when possible.

    Prime Agent appends daemon/worker socket names to TMPDIR. Deep evolution
    checkout paths can exceed Linux's Unix-socket pathname limit even though
    the underlying directory is valid. A symlink in a short directory we own
    keeps the socket pathname inside that limit while resolving to the same
    Landlock-confined ``.builder_tmp`` directory. The alias is created before
    the ruleset is applied, and Landlock authorizes the resolved target rather
    than the symlink, so the alias needs no rule of its own.
    """
    digest = hashlib.sha256(str(workspace.resolve()).encode()).hexdigest()[:12]
    for parent, needs_private_parent in _alias_parent_candidates():
        if not _usable_alias_parent(parent, needs_private_parent):
            continue
        # A parent we created already carries the name, so the alias there is
        # just the digest: every character counts against the socket budget.
        alias = parent / (
            digest if needs_private_parent else f"shadow-builder-{digest}"
        )
        if len(str(alias)) > MAX_TMPDIR_LEN:
            continue
        try:
            if alias.is_symlink():
                if alias.resolve(strict=False) != private_tmp.resolve():
                    alias.unlink()
                    alias.symlink_to(private_tmp, target_is_directory=True)
            elif not alias.exists():
                alias.symlink_to(private_tmp, target_is_directory=True)
            else:
                continue
        except OSError:
            continue
        return alias
    return private_tmp


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--workspace", type=Path, required=True)
    parser.add_argument("--executable", type=Path, action="append", default=[])
    parser.add_a

agent-s-computer-use/agent_s4/builder_scratch_vm.pycode · 4.3 KB

"""Builder-side bridge to the scratch VM provisioned by the orchestrator.

This module is imported by the host-side Builder subprocess. It never creates
or destroys a VM: the orchestrator owns that lifecycle and injects the exact
connection parameters through ``SCRATCH_VM_*`` environment variables.
"""

from __future__ import annotations

import os
from pathlib import Path


class ScratchVmUnavailable(RuntimeError):
    """Raised when a Builder is launched without a complete scratch handle."""


def _required(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise ScratchVmUnavailable(
            f"{name} is missing; this Builder has no usable scratch VM"
        )
    return value


def _required_port(name: str) -> int:
    value = _required(name)
    try:
        port = int(value)
    except ValueError as exc:
        raise ScratchVmUnavailable(f"{name} is not an integer: {value!r}") from exc
    if not 1 <= port <= 65535:
        raise ScratchVmUnavailable(f"{name} is outside the TCP port range: {port}")
    return port


_VM_IP = _required("SCRATCH_VM_IP")
_SERVER_PORT = _required_port("SCRATCH_VM_SERVER_PORT")
_INSTANCE_ID = _required("SCRATCH_VM_INSTANCE_ID")
_CLIENT_PASSWORD = _required("SCRATCH_VM_CLIENT_PASSWORD")
_CLONE_DIR = Path(_required("SCRATCH_VM_CLONE_DIR")).resolve()
_controller = None
_setup_controller = None


def get_controller():
    """Return a cached PythonController connected to the scratch desktop."""
    global _controller
    if _controller is None:
        from desktop_env.controllers.python import PythonController

        _controller = PythonController(vm_ip=_VM_IP, server_port=_SERVER_PORT)
    return _controller


def _import_setup_controller():
    """Import SetupController without its import-time ``load_dotenv()``.

    desktop_env's controllers call ``dotenv.load_dotenv()`` at import, which
    walks up past the workspace to the host project's ``.env``. That file holds
    host secrets and Landlock denies it, so the import would fail with
    PermissionError; skip the load instead of widening the sandbox.
    """
    import dotenv

    original = dotenv.load_dotenv
    dotenv.load_dotenv = lambda *args, **kwargs: False
    try:
        from desktop_env.controllers.setup import SetupController
    finally:
        dotenv.load_dotenv = original
    return SetupController


def get_setup_controller():
    """Return OSWorld's cached SetupController.

    ScienceBoard scratch VMs intentionally omit browser/media ports; calling
    this function on such a handle fails loudly instead of constructing a
    partially configured controller.
    """
    global _setup_controller
    if _setup_controller is None:
        chromium_port = _required_port("SCRATCH_VM_CHROMIUM_PORT")
        vlc_port = _required_port("SCRATCH_VM_VLC_PORT")
        SetupController = _import_setup_controller()
        _setup_controller = SetupController(
            vm_ip=_VM_IP,
            server_port=_SERVER_PORT,
            chromium_port=chromium_port,
            vlc_port=vlc_port,
            cache_dir=str(_CLONE_DIR / ".scratch_vm_cache"),
            client_password=_CLIENT_PASSWORD,
            screen_width=1920,
            screen_height=1080,
        )
    return _setup_controller


def redeploy_interface() -> None:
    """Upload the Builder's current ``shadow_interface/`` and run setup.sh."""
    shadow_dir = _CLONE_DIR / "shadow_interface"
    if not shadow_dir.is_dir():
        raise ScratchVmUnavailable(
            f"shadow_interface directory is missing from {_CLONE_DIR}"
        )
    # Staged beside this file by the orchestrator: the parent project's
    # agent_s4 package is outside the Builder sandbox.
    from scratch_vm_utils import run_setup_script, upload_shadow_interface

    controller = get_controller()
    if not upload_shadow_interface(controller, str(shadow_dir)):
        raise RuntimeError("scratch VM interface upload failed")
    if not run_setup_script(controller, client_password=_CLIENT_PASSWORD):
        raise RuntimeError("scratch VM interface setup failed")


def scratch_vm_identity() -> dict[str, str | int]:
    """Return non-secret connection identity for Builder notes/debugging."""
    return {
        "instance_id": _INSTANCE_ID,
        "vm_ip": _VM_IP,
        "server_port": _SERVER_PORT,
        "clone_dir": str(_CLONE_DIR),
    }

agent-s-computer-use/agent_s4/cli.pycode · 13.9 KB · preview truncated

"""Command-line entry for shadow-evol.

Shape:
    shadow-evol run --island name=X,evaluator_harness=Y,eval_model=Z \
                    [--island ...] \
                    [--scheduler deterministic[:m,c,e] | probabilistic:k=w,...] \
                    [--seed-dir DIR --prompt-template FILE] \
                    [--builder-model X --builder-harness Y ...]

    shadow-evol rerun --config-path runs/<ts>/config.json
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

from agent_s4.benchmark_spec import BENCHMARKS
from agent_s4.config.experiment import ExperimentConfig
from agent_s4.config.island import IslandConfig
from agent_s4.config.scheduler import (
    DeterministicScheduler,
    ProbabilisticScheduler,
    Scheduler,
)


def _parse_kv_list(arg: str) -> dict[str, str]:
    """Parse 'k1=v1,k2=v2' into {'k1': 'v1', 'k2': 'v2'}."""
    out: dict[str, str] = {}
    for part in arg.split(","):
        part = part.strip()
        if not part:
            continue
        if "=" not in part:
            raise argparse.ArgumentTypeError(
                f"expected k=v pairs separated by commas, got {part!r}"
            )
        k, v = part.split("=", 1)
        out[k.strip()] = v.strip()
    return out


def _parse_island_arg(arg: str) -> dict[str, str]:
    """Parse + validate an --island argument at argparse time so missing
    keys fail with a clean usage message.

    v2 required keys: name, evaluator_harness, eval_model. Seed, prompt,
    builder model/harness are global on ExperimentConfig.
    """
    kv = _parse_kv_list(arg)
    required = {"name", "evaluator_harness", "eval_model"}
    missing = required - kv.keys()
    if missing:
        raise argparse.ArgumentTypeError(
            f"--island missing keys {sorted(missing)}; got {kv!r}"
        )
    return kv


def _parse_scheduler(spec: str) -> Scheduler:
    """Parse --scheduler argument.

    Accepted forms:
        "deterministic"
            → DeterministicScheduler() with defaults (5, 6, 10).
        "deterministic:M,C,E"
            → DeterministicScheduler(migration_cadence=M,
                                     crossover_cadence=C,
                                     elimination_cadence=E).
        "probabilistic:mutation=W,migration=W,crossover=W,elimination=W"
            → ProbabilisticScheduler({...}).
    """
    name, _, rest = spec.partition(":")
    name = name.strip()
    if name == "deterministic":
        if not rest:
            return DeterministicScheduler()
        try:
            cadences = [int(x.strip()) for x in rest.split(",") if x.strip()]
        except ValueError as e:
            raise argparse.ArgumentTypeError(
                f"--scheduler deterministic cadences must be ints, got {rest!r}"
            ) from e
        if len(cadences) != 3:
            raise argparse.ArgumentTypeError(
                f"--scheduler deterministic expects 3 cadences "
                f"(migration,crossover,elimination), got {cadences!r}"
            )
        return DeterministicScheduler(
            migration_cadence=cadences[0],
            crossover_cadence=cadences[1],
            elimination_cadence=cadences[2],
        )
    if name == "probabilistic":
        if not rest:
            raise argparse.ArgumentTypeError(
                "--scheduler probabilistic requires kind=weight pairs"
            )
        kv = _parse_kv_list(rest)
        try:
            weights = {k: float(v) for k, v in kv.items()}
        except ValueError as e:
            raise argparse.ArgumentTypeError(
                f"--scheduler probabilistic weights must be floats, got {kv!r}"
            ) from e
        return ProbabilisticScheduler(weights)
    raise argparse.ArgumentTypeError(
        f"unknown scheduler {spec!r}; expected 'deterministic[:M,C,E]' "
        f"or 'probabilistic:k=w,...'"
    )


def _build_island(kv: dict[str, str]) -> IslandConfig:
    required = {"name", "evaluator_harness", "eval_model"}
    missing = required - kv.keys()
    if missing:
        raise argparse.ArgumentTypeError(
            f"--island missing keys {sorted(missing)}; got {kv!r}"
        )
    return IslandConfig(
        name=kv["name"],
        evaluator_harness=kv["evaluator_harness"],
        eval_model=kv["eval_model"],
    )


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    p = argparse.ArgumentParser(prog="shadow-evol")
    sub = p.add_subparsers(dest="command", required=True)

    run = sub.add_parser("run", help="Launch an experiment")
    run.add_argument(
        "--island",
        action="append",
        type=_parse_island_arg,
        required=True,
        help="Repeatable. Example: "
        "--island name=main,evaluator_harness=prime_agent,eval_model=openai/<deployment>",
    )
    run.add_argument(
        "--scheduler",
        type=_parse_scheduler,
        default=None,
        help="Scheduler spec. Examples: 'deterministic', "
        "'deterministic:5,6,10', "
        "'probabilistic:mutation=0.7,migration=0.15,"
        "crossover=0.1,elimination=0.05'. "
        "Default: DeterministicScheduler(5, 6, 10).",
    )
    run.add_argument(
        "--seed-dir", type=Path, help="Seed module directory shared by all islands."
    )
    run.add_argument(
        "--prompt-template", type=Path, help="Builder prompt template path."
    )
    run.add_argument("--builder-model")
    run.add_argument("--builder-harness")
    run.add_argument("--builder-timeout", type=int)
    run.add_argument(
        "--builder-no-time-limit",
        action="store_true",
        default=None,
        help="prime_agent Builder: disable the Builder wall-clock deadline.",
    )
    run.add_argument("--eval-timeout", type=int)
    run.add_argument(
        "--eval-max-steps",
        type=int,
        help="Maximum agent steps per task.",
    )
    run.add_argument(
        "--eval-no-time-limit",
        action="store_true",
        default=None,
        help="Disable both the per-task wall-clock "
        "limit and the outer evolve batch watchdog.",
    )
    run.add_argument("--max-steps", type=int)
    run.add_argument("--max-cost", type=float)
    run.add_argument("--region")
    run.add_argument("--seed", type=int)
    run.add_argument(
        "--elimination-count",
        type=int,
        help="How many bottom variants Elimination removes per fire.",
    )
    run.add_argument(
        "--reevaluation-interval",
        type=int,
        help="Reevaluate the top interfaces on validation after every N completed "
        "evolution steps (0 disables; default: 0). Does not consume steps.",
    )
    run.add_argument(
        "--reevaluation-top-k",
        type=int,
        help="Number of interfaces per island to reevaluate in parallel (default: 3).",
    )
    run.add_argument(
        "--no-scratch-vm",
        dest="scratch_vm_enabled",
        action="store_false",
        default=None,
        help="Ablation: run the builder without a per-mutation "
        "AWS scratch VM. Builder learns only from "
        "eval_results/ trajectories (no live app probing).",
    )
    run.add_argument(
        "--train-set",
        dest="train_set_file",
        help="Filename under eval_sets/ for the training task list. "
        "Default for osworld_v2: osworld_v2_train_50.json; required for "
        "the other benchmarks.",
    )
    run.add_argument(
        "--val-set",
        dest="val_set_file",
        help="Filename under eval_sets/ for the validation task list. "
        "Default for osworld_v2: osworld_v2_val_25.json; required for "
        "the other benchmarks. Pass 'none' to disable val "
        "entirely (requires --selection-signal train).",
    )
    run.add_argument(
        "--selection-signal",
        choices=("val", "train"),
        help="Which score drives tournament selection. "
        "Default: val. Set to 'train' for ablation runs that "
        "still record val_score but rank parents by train.",
    )
    run.add_argument(
        "--parent-selection",
        choices=("tournament", "latest"),
        help="How mutation picks its parent. Default: tournament "
        "(binary tournament over the scored population, "
        "ranked by --selection-signal). 'latest' always "
        "mutates the most recent scored variant, giving a "
        "strict linear chain of depth --max-steps and taking "
        "the selection signal out of the search.",
    )
    run.add_argument(
        "--migration-mode",
        choices=("move", "copy"),
        help="Migration semantics. Default: move (ship the picked "
        "variant off its source island). 'copy' keeps the "
        "variant on its source and only seeds the neighbor.",
    )
    run.add_argument(
        "--migration-topology",
        choices=("ring", "complete"),
        help="Migration topology. Default: ring (receive from the "
        "single predecessor). 'complete' is all-to-all — each "
        "island receives from every other island (= bidirectional "
        "ring at N=3).",
    )
    run.add_argument(
        "--benchmark",
        choices=sorted(BENCHMARKS),
        default="osworld_v2",
        help="Benchmark suite to drive (default: osworld_v2).",
    )
    run.add_argument(
        "--allow-v2-noncanonical-splits",
        action="store_true",
        default=None,
        help="Unsafe diagnostic escape hatch: permit non-canonical OSWorld V2 "
        "train/val subsets. Test IDs remain forbidden.",
    )

    rerun = sub.add_parser("rerun", help="Re-execute from a saved config.json")
    rerun.add_argument("--config-path", type=Path, required=True)

    return p.parse_args(argv)


def build_config_from_args(args: argparse.Namespace) -> ExperimentConfig:
    if args.command == "rerun":
        with open(args.config_path) as f:
            payload = json.load(f)
        return _config_from_dict(payload)

    islands = [_build_island(kv) for kv in args.island]

    kwargs: dict = {"islands": islands}
    for field in (
        "builder_model",
        "builder_harness",
        "builder_claude_bin",
        "builder_timeout",
        "builder_no_time_limit",
        "eval_timeout",
        "eval_max_steps",
        "eval_no_time_limit",
        "max_steps",
        "max_cost",
        "region",
        "seed",
        "scratch_vm_enabled",
        "train_set_file",
        "val_set_file",
        "selection_signal",
        "parent_selection",
        "migration_mode",
        "migration_topology",
        "elimination_count",
        "benchmark",
        "reevaluation_interval",
        "reevaluation_top_k",
        "allow_v2_noncanonical_splits",
    ):
        v = getattr(args, field, None)
        if v is not None:
            kwargs[field] = v
    for path_field in ("seed_dir", "prompt_template"):
        v = getattr(args, path_field, None)
        if v is not None:
            kwargs[path_field] = v
    if args.scheduler is not None:
        kwargs["scheduler"] = args.scheduler
    return ExperimentConfig(**kwargs)


def _scheduler_from_payload(payload: dict) -> Scheduler:
    """Reconstruct a Scheduler from its serialized form.

    Serialization shape (see agent_s4.orchestrator._config_to_dict):
        {"type": "DeterministicScheduler" | "ProbabilisticScheduler",
         "args": {...}}
    """
    sched_type = payload["type"]
    args = payload["args"]
    if sched_type == "DeterministicScheduler":
        return DeterministicScheduler(**args)
    if sched_type == "ProbabilisticScheduler":
        # ProbabilisticScheduler.__init__ takes a weights dict, but stores
        # kinds + probs. Reconstruct weights from those.
        weights = dict(zip(args["kinds"], args["probs"]))
        return ProbabilisticScheduler(weights)
    raise ValueError(f"unknown scheduler type {sched_type!r}")


def _config_from_dict(payload: dict) -> ExperimentConfig:
    """Round-trip deserialization for `shadow-evol rerun`. Must agree with
    agent_s4.orchestrator._

agent-s-computer-use/agent_s4/config/__init__.pycode · 0.0 KB

agent-s-computer-use/agent_s4/config/experiment.pycode · 8.8 KB

"""ExperimentConfig — the single object the CLI produces and driver consumes."""

from __future__ import annotations

from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal

from agent_s4.benchmark_spec import BENCHMARKS
from agent_s4.config.island import IslandConfig
from agent_s4.config.scheduler import DeterministicScheduler, Scheduler

# (train, val, held-out) defaults per benchmark. Only the files that ship in
# eval_sets/ are named; the others must be passed explicitly.
_DEFAULT_SPLITS: dict[str, tuple[str | None, str | None, str | None]] = {
    "osworld_v2": (
        "osworld_v2_train_50.json",
        "osworld_v2_val_25.json",
        "eval_sets/osworld_v2_test.json",
    ),
    "scienceboard": (None, None, "eval_sets/scienceboard_test.json"),
    "osworld": (None, None, None),
}


@dataclass
class ExperimentConfig:
    islands: list[IslandConfig]

    # Shared across all islands in v2
    seed_dir: Path = Path("seeds/combined")
    prompt_template: Path = Path("prompts/combined.md")
    builder_model: str = "opus"
    builder_harness: str = "prime_agent"
    # Kept only so run configs written by older versions still round-trip
    # through `shadow-evol rerun`; the Prime builder ignores it.
    builder_claude_bin: str | None = None

    # Scheduler controls when each op fires
    scheduler: Scheduler = field(default_factory=DeterministicScheduler)
    elimination_count: int = 5
    # Validation maintenance runs after completed evolution steps. It does
    # not advance the scheduler, allocate an interface ID, or use a step.
    # Zero keeps existing experiments unchanged; opt in with an interval.
    reevaluation_interval: int = 0
    reevaluation_top_k: int = 3

    builder_timeout: int = 600
    # Prime Builder can run without a wall-clock deadline. Kept separate from
    # builder_timeout so saved configs state the policy explicitly.
    builder_no_time_limit: bool = False
    eval_timeout: int = 600
    # Per-episode agent budget. None preserves the evaluator's harness default.
    # no_time_limit disables both the in-episode wall clock and the evolve
    # subprocess's outer batch watchdog. Infrastructure setup/upload calls
    # retain their own bounded timeouts.
    eval_max_steps: int | None = None
    eval_no_time_limit: bool = False
    region: str = "us-east-1"

    scratch_vm_enabled: bool = True

    max_steps: int | None = None
    max_cost: float | None = None
    seed: int | None = None

    # Canonical task sets — no tiers
    # None is an internal sentinel so benchmark-aware defaults do not erase a
    # caller's explicit choice of the legacy filenames for a V2 diagnostic.
    train_set_file: str | None = None
    val_set_file: str | None = None
    held_out_tasks_path: Path | None = None

    selection_signal: Literal["val", "train"] = "val"

    # How mutation picks its parent.
    #
    # "tournament" (default): a binary tournament over the island's scored
    # population, ranked by selection_signal. This is a branching search --
    # with a population of 5 and k=2 the current best is not drawn 60% of the
    # time, so the lineage that comes out is a shallow tree rooted at the
    # seed, not a chain. The opus5xover25v15 run made the cost concrete:
    # seed was a parent in 7 of 10 steps and the deepest lineage was 3.
    #
    # "latest": always mutate the most recently produced scored variant, so
    # step N builds on step N-1's output whatever it scored. The lineage is a
    # strict chain of depth max_steps, which is what a linear-route baseline
    # needs; the trade is that one bad mutation carries forward instead of
    # being stepped around. Deliberate -- it removes the selection signal
    # from the search entirely, so the run measures what accumulating
    # mutations does without a val_score too noisy to rank on getting a vote.
    parent_selection: Literal["tournament", "latest"] = "tournament"

    # Migration semantics. "move" (default): phase 1 removes the picked
    # variant from its source island (today's behavior). "copy": phase 1
    # seeds the ring successor without removing the variant from its source,
    # so a known-good variant stays home. See migration.apply_phase1.
    migration_mode: Literal["move", "copy"] = "move"

    # Migration topology. "ring" (default): each island receives from its
    # single predecessor (today's behavior). "complete": all-to-all — each
    # island receives a migrant from EVERY other island, so its intra-island
    # crossover pool holds material from all islands at once. At N=3 this is
    # the bidirectional ring. See migration.sources_for / apply_phase2.
    migration_topology: Literal["ring", "complete"] = "ring"

    # Which benchmark suite the loop drives. Persisted in config.json so rerun
    # is benchmark-aware. See agent_s4/benchmark_spec.py.
    benchmark: Literal["osworld", "osworld_v2", "scienceboard"] = "osworld_v2"

    # Diagnostic escape hatch for deliberately smaller V2 train/val subsets.
    # Even when enabled, the subsets must be unique, disjoint, and contain no
    # canonical test IDs. Formal runs leave this false.
    allow_v2_noncanonical_splits: bool = False

    def __post_init__(self):
        if (
            type(self.reevaluation_interval) is not int
            or self.reevaluation_interval < 0
        ):
            raise ValueError("reevaluation_interval must be an integer >= 0")
        if type(self.reevaluation_top_k) is not int or self.reevaluation_top_k < 1:
            raise ValueError("reevaluation_top_k must be a positive integer")
        if self.reevaluation_interval and self.val_set_file == "none":
            raise ValueError("validation reevaluation requires a validation set")
        if self.builder_no_time_limit and self.builder_harness not in {
            "prime_agent",
            "opencode",
        }:
            raise ValueError(
                "builder_no_time_limit currently requires builder_harness="
                "'prime_agent' or 'opencode'"
            )
        if self.selection_signal not in ("val", "train"):
            raise ValueError(
                f"selection_signal must be 'val' or 'train', got "
                f"{self.selection_signal!r}"
            )
        if self.parent_selection not in ("tournament", "latest"):
            raise ValueError(
                f"parent_selection must be 'tournament' or 'latest', got "
                f"{self.parent_selection!r}"
            )
        if self.migration_mode not in ("move", "copy"):
            raise ValueError(
                f"migration_mode must be 'move' or 'copy', got "
                f"{self.migration_mode!r}"
            )
        if self.migration_topology not in ("ring", "complete"):
            raise ValueError(
                f"migration_topology must be 'ring' or 'complete', got "
                f"{self.migration_topology!r}"
            )
        if self.benchmark not in BENCHMARKS:
            raise ValueError(
                f"benchmark must be one of {sorted(BENCHMARKS)}, got "
                f"{self.benchmark!r}"
            )
        train_default, val_default, held_out_default = _DEFAULT_SPLITS[self.benchmark]
        if self.train_set_file is None:
            if train_default is None:
                raise ValueError(
                    f"benchmark {self.benchmark!r} ships no default train split; "
                    "pass train_set_file (--train-set)"
                )
            self.train_set_file = train_default
        if self.val_set_file is None:
            if val_default is None:
                raise ValueError(
                    f"benchmark {self.benchmark!r} ships no default val split; "
                    "pass val_set_file (--val-set), or 'none'"
                )
            self.val_set_file = val_default
        if self.held_out_tasks_path is None and held_out_default is not None:
            self.held_out_tasks_path = Path(held_out_default)
        if self.val_set_file == "none" and self.selection_signal == "val":
            raise ValueError("val_set_file=none requires selection_signal=train")

    def island_by_name(self, name: str) -> IslandConfig:
        for i in self.islands:
            if i.name == name:
                return i
        raise KeyError(
            f"no island named {name!r}; " f"known: {[i.name for i in self.islands]}"
        )

    def resolve_builder_model(self, island: IslandConfig) -> str:
        return self.builder_model  # uniform in v2

    def resolve_builder_bin(self, island: IslandConfig) -> str | None:
        return self.builder_claude_bin  # uniform in v2; unused by the Prime builder

    def resolve_eval_model(self, island: IslandConfig) -> str:
        return island.eval_model

    def resolve_builder_harness_key(self, island: IslandConfig) -> str:
        return self.builder_harness  # uniform in v2

    def resolve_evaluator_harness_key(self, island: IslandConfig) -> str:
        return island.evaluator_harness

agent-s-computer-use/agent_s4/config/island.pycode · 0.3 KB

"""Per-island configuration. Pure data — no runtime logic.

In v2, the only per-island variation is the evaluator. Builder, seed,
and prompt are shared globally via ExperimentConfig.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass
class IslandConfig:
    name: str
    evaluator_harness: str
    eval_model: str

agent-s-computer-use/agent_s4/config/scheduler.pycode · 3.5 KB

"""Scheduler — picks one operator per global step.

The Scheduler is the only point of pluggability in the driver's dispatch
loop. DeterministicScheduler (cadence-based with rarest-wins priority) is
the default. ProbabilisticScheduler samples per step from a fixed
weight distribution. Custom subclasses can implement any other scheme
(adaptive, threshold-based, etc.).

A cadence of 0 means "never fire" — useful for ablations.
"""

from __future__ import annotations

from abc import ABC, abstractmethod
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from agent_s4.state import LoopState


class Scheduler(ABC):
    @abstractmethod
    def choose_op(self, step: int, state: "LoopState | None") -> str:
        """Return one of: 'mutation', 'migration', 'crossover', 'elimination'."""
        ...


class DeterministicScheduler(Scheduler):
    """Cadence-based with rarest-wins priority on collisions.

    Default cadences match today's intent: migration=5, crossover=6,
    elimination=10. Mutation is the fallback when no other cadence fires.
    Cadence of 0 means "never fire".

    When more than one cadence divides the same step (a collision), the
    *rarest* operator wins — the one with the largest cadence, since it
    fires least often and is the harder slot to give up. Equal cadences
    fall back to a fixed tie-break order: elimination > crossover >
    migration. (For the default 10/6/5 cadences this is identical to a
    fixed elim>crossover>migration priority, because crossover and
    migration only ever collide on a step that elimination also claims;
    the rarity sort only diverges once you flip the usual ordering, e.g.
    crossover fires more often than migration.)
    """

    def __init__(
        self,
        migration_cadence: int = 5,
        crossover_cadence: int = 6,
        elimination_cadence: int = 10,
    ):
        self.migration_cadence = migration_cadence
        self.crossover_cadence = crossover_cadence
        self.elimination_cadence = elimination_cadence

    def choose_op(self, step, state):
        if step <= 0:
            return "mutation"
        # (cadence, tie_break_rank, op) for every operator firing this step.
        # Lower tie_break_rank wins an equal-cadence tie.
        firing = [
            (cadence, rank, op)
            for cadence, rank, op in (
                (self.elimination_cadence, 0, "elimination"),
                (self.crossover_cadence, 1, "crossover"),
                (self.migration_cadence, 2, "migration"),
            )
            if cadence > 0 and step % cadence == 0
        ]
        if not firing:
            return "mutation"
        # Rarest wins: largest cadence first, then the tie-break rank.
        firing.sort(key=lambda c: (-c[0], c[1]))
        return firing[0][2]


class ProbabilisticScheduler(Scheduler):
    """Weighted random sample per step. Weights are normalized internally;
    any positive-valued weights are accepted. State.rng provides the
    randomness so seeded runs are reproducible.
    """

    def __init__(self, weights: dict[str, float]):
        total = sum(weights.values())
        if total <= 0:
            raise ValueError(f"weights must sum to > 0, got {total}")
        self.kinds = list(weights)
        self.probs = [weights[k] / total for k in self.kinds]

    def choose_op(self, step, state):
        if state is None or not hasattr(state, "rng"):
            raise ValueError("ProbabilisticScheduler requires a LoopState with .rng")
        return state.rng.choices(self.kinds, weights=self.probs)[0]

agent-s-computer-use/agent_s4/constants.pycode · 1.9 KB

"""Project-wide constants.

Moved out of island_common.py during the framework refactor. These
describe experiment geometry (cache paths, eval set layout) rather than
per-experiment configuration — ExperimentConfig fields take precedence
when they overlap.
"""

from __future__ import annotations

# SQLite file (under repo root) that holds the run's evolution state.
# All modules that open the DB (driver, orchestrator, db) reference this
# name so the path is defined in one place.
DB_FILENAME = "shadow_evolution.db"

# Subdirectory (under repo root) that holds eval task-list JSON files
# (eval_island_*.json + eval_heldout_*.json). Centralized here so callers
# never hardcode the directory name.
EVAL_SETS_DIR = "eval_sets"

# Historical note: there used to be an EVAL_SETS dict mapping tier-level
# → eval_island_*.json filename, plus a TIER_ORDER constant used by
# operators' promotion cascade. Both were retired in v2 — variants now
# run on the single canonical train_set_file (see
# ExperimentConfig.train_set_file). The `variants.eval_level` DB column
# is retained for legacy-row compatibility but is not read or written by
# v2 code.

# Directory (under repo root) where per-variant eval trajectories are cached.
EVAL_CACHE_DIR = "eval_cache"

# Directory (under repo root) where per-mutation builder logs are written.
BUILDER_LOGS_DIR = "builder_logs"

# Directory (under repo root) where migration phase 1 stages outbound
# variants for phase 2 destinations to pick up. Each source island writes
# to MIGRATE_TRANSIT_DIR/<source_island>/ via tmp+rename for atomicity.
# The presence of MIGRATE_TRANSIT_DIR/<src>/ marks "phase 1 done for src"
# during resume; phase 2 reads and then deletes the dir.
MIGRATE_TRANSIT_DIR = "_migrate_transit"

# Note: the promote-threshold (fraction of tier-N tasks a variant must
# pass to earn tier N+1 eval) was retired in v2 along with the tier
# cascade. Variants now evaluate on the single canonical train_set_file.

agent-s-computer-use/agent_s4/costs.pycode · 7.6 KB

"""Cost event recording helpers.

Parses stream-json logs (builder subprocess) and eval task trajectory files,
inserts rows into the cost_events table. Sync-path hooks in run_island.py,
island_orchestrator.py, and island_tournament.py call these helpers right
after each subprocess completes.
"""

import json
import logging
import os

from agent_s4.benchmark_spec import ns
from agent_s4.constants import EVAL_CACHE_DIR
from agent_s4.db import insert_cost_event

logger = logging.getLogger("island")


def _parse_stream_json(path):
    """Read a stream-json log, return (init_model, result_event) pair.

    init_model is the model from the first `{"type": "system", "subtype": "init"}`
    line, or None if no such line.
    result_event is the last `{"type": "result"}` line as a dict, or None if
    no result event was found (indicates a crash before completion).
    """
    init_model = None
    result_event = None
    if not os.path.exists(path):
        return init_model, result_event
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                d = json.loads(line)
            except json.JSONDecodeError:
                continue
            t = d.get("type")
            if t == "system" and d.get("subtype") == "init" and init_model is None:
                init_model = d.get("model")
            elif t == "result":
                result_event = d
    return init_model, result_event


def _parse_opencode_trajectory(path):
    """Aggregate cost and token usage across an OpenCode trajectory.

    OpenCode's --format json emits one `{"type":"step_finish"}` event per
    agent step with per-step `cost` and `tokens.{input,output,reasoning,
    cache.{read,write}}` payload. Unlike Prime's stream-json, which has
    a single terminal `result` event with cumulative usage, OpenCode emits
    no cumulative summary — the caller must sum across step_finish events to get
    per-task totals.

    Returns a dict shaped like {cost_usd, input_tokens, output_tokens,
    cache_read_tokens, cache_creation_tokens} on success, or None if the
    trajectory has no step_finish events (which is indistinguishable from
    a non-OpenCode trajectory).

    Reasoning tokens are folded into output_tokens because cost_events
    has no dedicated reasoning column and OpenRouter bills reasoning as
    output for most providers anyway.
    """
    if not os.path.exists(path):
        return None

    saw_step_finish = False
    total_cost = 0.0
    sum_input = 0
    sum_output_plus_reasoning = 0
    sum_cache_read = 0
    sum_cache_write = 0

    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                d = json.loads(line)
            except json.JSONDecodeError:
                continue
            if d.get("type") != "step_finish":
                continue
            saw_step_finish = True
            part = d.get("part") or {}
            total_cost += float(part.get("cost") or 0)
            tokens = part.get("tokens") or {}
            sum_input += int(tokens.get("input") or 0)
            sum_output_plus_reasoning += int(tokens.get("output") or 0)
            sum_output_plus_reasoning += int(tokens.get("reasoning") or 0)
            cache = tokens.get("cache") or {}
            sum_cache_read += int(cache.get("read") or 0)
            sum_cache_write += int(cache.get("write") or 0)

    if not saw_step_finish:
        return None

    return {
        "cost_usd": total_cost,
        "input_tokens": sum_input,
        "output_tokens": sum_output_plus_reasoning,
        "cache_read_tokens": sum_cache_read,
        "cache_creation_tokens": sum_cache_write,
    }


def _extract_usage(result_event):
    """Pull token fields out of the result event's `usage` dict."""
    usage = (result_event or {}).get("usage") or {}
    return {
        "input_tokens": usage.get("input_tokens"),
        "output_tokens": usage.get("output_tokens"),
        "cache_read_tokens": usage.get("cache_read_input_tokens"),
        "cache_creation_tokens": usage.get("cache_creation_input_tokens"),
    }


def record_builder_cost(conn, variant_id, island, log_path, model_fallback, timestamp):
    """Parse a builder stream-json log and insert one `build` cost_event row.

    If the log has no result event (crash), nothing is inserted.
    """
    init_model, result = _parse_stream_json(log_path)
    if result is None:
        logger.warning("record_builder_cost: no result event in %s", log_path)
        return
    usage = _extract_usage(result)
    insert_cost_event(
        conn,
        event_type="build",
        island=island,
        variant_id=variant_id,
        task_id=None,
        model=init_model or model_fallback,
        cost_usd=result.get("total_cost_usd") or 0,
        timestamp=timestamp,
        **usage,
    )


def record_eval_costs(
    conn,
    variant_id,
    island,
    clone_dir,
    task_ids,
    event_type,
    model_fallback,
    timestamp,
    *,
    benchmark="osworld",
    result_dir=None
):
    """Walk each task's claude_trajectory.jsonl and insert one cost_event
    row per task that has a result event.

    Tasks without a trajectory, or whose trajectory has no result event,
    are skipped silently — the task's pass/fail is recorded separately
    in eval_results.

    model_fallback is used both as the cost_event.model when the trajectory
    has no init event AND as the path segment for locating trajectories
    (the eval subprocess writes under .../shadow_eval/screenshot/<model>/...
    — previously hardcoded to 'haiku' here, silently dropping cost events
    for any non-haiku eval model).
    """
    base_dir = os.path.dirname(os.path.abspath(clone_dir))
    if result_dir is None:
        result_dir = os.path.join(base_dir, ns(EVAL_CACHE_DIR, benchmark), variant_id)
    traj_root = os.path.join(result_dir, "shadow_eval", "screenshot", model_fallback)
    for tid in task_ids:
        traj_path = os.path.join(traj_root, tid, "claude_trajectory.jsonl")

        # Prime Agent writes stream-json with a single terminal `result`
        # event containing cumulative usage and total_cost_usd.
        init_model, result = _parse_stream_json(traj_path)
        if result is not None:
            usage = _extract_usage(result)
            cost_usd = result.get("total_cost_usd") or 0
            model = init_model or model_fallback
        else:
            # OpenCode path: aggregate per-step `step_finish` events. The
            # trajectory has no init event identifying the model, so use
            # model_fallback (the path-segment slug).
            opencode_summary = _parse_opencode_trajectory(traj_path)
            if opencode_summary is None:
                # Neither schema matched: skip silently (the task's pass/fail
                # is still recorded separately in eval_results; we just have
                # no cost data to attribute to it).
                continue
            usage = {
                k: opencode_summary[k]
                for k in (
                    "input_tokens",
                    "output_tokens",
                    "cache_read_tokens",
                    "cache_creation_tokens",
                )
            }
            cost_usd = opencode_summary["cost_usd"]
            model = model_fallback

        insert_cost_event(
            conn,
            event_type=event_type,
            island=island,
            variant_id=variant_id,
            task_id=tid,
            model=model,
            cost_usd=cost_usd,
            timestamp=timestamp,
            **usage,
        )

agent-s-computer-use/agent_s4/db.pycode · 21.5 KB · preview truncated

"""SQLite DB layer for island evolution state.

Schema:
    variants — one row per variant in any island's population
    eval_results — per-variant, per-task pass/fail
    eval_sets — task_id → eval level membership (nested subsets)
    islands — per-island counters (mutation_count, last_X_at)
    history — append-only log of mutations, migrations, crossovers, eliminations
"""

import json
import math
import os
import sqlite3

SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS variants (
    id TEXT PRIMARY KEY,
    island TEXT NOT NULL,
    tag TEXT NOT NULL,
    parent_id TEXT,
    parent_b_id TEXT,
    origin TEXT NOT NULL,
    generation INTEGER DEFAULT 0,
    score REAL,
    val_score REAL,
    eval_level INTEGER DEFAULT 25,
    builder_cost REAL DEFAULT 0,
    created_at TEXT
);

CREATE TABLE IF NOT EXISTS val_set (
    task_id TEXT PRIMARY KEY
);

CREATE TABLE IF NOT EXISTS eval_results (
    variant_id TEXT NOT NULL,
    task_id TEXT NOT NULL,
    passed INTEGER NOT NULL,
    raw_score REAL,
    fitness_score REAL,
    details_json TEXT,
    failure_type TEXT,
    evaluated_at TEXT,
    PRIMARY KEY (variant_id, task_id),
    FOREIGN KEY (variant_id) REFERENCES variants(id) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS eval_sets (
    task_id TEXT NOT NULL,
    level INTEGER NOT NULL,
    PRIMARY KEY (task_id, level)
);

CREATE TABLE IF NOT EXISTS islands (
    name TEXT PRIMARY KEY,
    mutation_count INTEGER DEFAULT 0,
    last_migration_at INTEGER DEFAULT 0,
    last_crossover_at INTEGER DEFAULT 0,
    last_elimination_at INTEGER DEFAULT 0,
    last_updated TEXT
);

CREATE TABLE IF NOT EXISTS history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    type TEXT NOT NULL,
    island TEXT,
    parent TEXT,
    child TEXT,
    score REAL,
    cost REAL,
    details TEXT,
    timestamp TEXT
);

CREATE INDEX IF NOT EXISTS idx_variants_island ON variants(island);
CREATE INDEX IF NOT EXISTS idx_eval_results_variant ON eval_results(variant_id);
CREATE INDEX IF NOT EXISTS idx_eval_sets_level ON eval_sets(level);

CREATE TABLE IF NOT EXISTS cost_events (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    event_type TEXT NOT NULL,
    island TEXT NOT NULL,
    variant_id TEXT,
    task_id TEXT,
    model TEXT,
    cost_usd REAL,
    input_tokens INTEGER,
    output_tokens INTEGER,
    cache_read_tokens INTEGER,
    cache_creation_tokens INTEGER,
    timestamp TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_cost_events_island ON cost_events(island);
CREATE INDEX IF NOT EXISTS idx_cost_events_variant ON cost_events(variant_id);
CREATE INDEX IF NOT EXISTS idx_cost_events_type ON cost_events(event_type);
-- COALESCE the nullable columns so SQLite's "each NULL is distinct in a UNIQUE
-- index" rule doesn't defeat idempotency for build rows (where task_id is NULL).
CREATE UNIQUE INDEX IF NOT EXISTS uniq_cost_events ON cost_events(
    event_type, COALESCE(variant_id, ''), COALESCE(task_id, ''), timestamp
);

CREATE TABLE IF NOT EXISTS driver_state (
    current_step INTEGER NOT NULL DEFAULT 0
);

CREATE TABLE IF NOT EXISTS dispatch_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    step INTEGER NOT NULL,
    op_kind TEXT NOT NULL,
    started_at TEXT NOT NULL,
    completed_at TEXT,
    status TEXT NOT NULL CHECK(status IN ('pending', 'complete'))
);

CREATE INDEX IF NOT EXISTS idx_dispatch_log_step ON dispatch_log(step);
CREATE INDEX IF NOT EXISTS idx_dispatch_log_status ON dispatch_log(status);

-- Maintenance has its own journal: an unfinished reevaluation must never
-- make the normal dispatch recovery delete the interfaces it evaluated.
CREATE TABLE IF NOT EXISTS validation_reevaluation_rounds (
    island TEXT NOT NULL,
    after_step INTEGER NOT NULL,
    candidates_json TEXT NOT NULL,
    context_json TEXT NOT NULL,
    status TEXT NOT NULL CHECK(status IN ('pending', 'complete')),
    started_at TEXT NOT NULL,
    completed_at TEXT,
    PRIMARY KEY (island, after_step)
);

-- after_step=0 is the original evaluation. Later rows are independent
-- validation repeats. Keep the original per-task eval_results untouched.
CREATE TABLE IF NOT EXISTS validation_score_samples (
    variant_id TEXT NOT NULL,
    after_step INTEGER NOT NULL,
    score REAL NOT NULL,
    task_results_json TEXT,
    evaluated_at TEXT NOT NULL,
    PRIMARY KEY (variant_id, after_step),
    FOREIGN KEY (variant_id) REFERENCES variants(id) ON DELETE CASCADE
);
"""


def connect(db_path):
    """Open a SQLite connection with WAL mode and FK enforcement enabled."""
    conn = sqlite3.connect(db_path, isolation_level=None)  # autocommit
    conn.row_factory = sqlite3.Row
    conn.execute("PRAGMA journal_mode = WAL")
    conn.execute("PRAGMA foreign_keys = ON")
    return conn


def init_schema(conn):
    """Apply the full schema. Idempotent (CREATE IF NOT EXISTS)."""
    conn.executescript(SCHEMA_SQL)
    conn.execute("PRAGMA journal_mode = WAL")
    conn.execute("PRAGMA foreign_keys = ON")
    # Backfill: add step column to variants if missing (SQLite has no
    # ADD COLUMN IF NOT EXISTS, so check first)
    cols = {r[1] for r in conn.execute("PRAGMA table_info(variants)")}
    if "step" not in cols:
        conn.execute("ALTER TABLE variants ADD COLUMN step INTEGER NOT NULL DEFAULT 0")
    eval_cols = {r[1] for r in conn.execute("PRAGMA table_info(eval_results)")}
    for name, sql_type in (
        ("raw_score", "REAL"),
        ("fitness_score", "REAL"),
        ("details_json", "TEXT"),
        ("failure_type", "TEXT"),
    ):
        if name not in eval_cols:
            conn.execute(f"ALTER TABLE eval_results ADD COLUMN {name} {sql_type}")
    # Old rows predate raw_score; binary passed is their lossless fallback.
    conn.execute("UPDATE eval_results SET raw_score = passed WHERE raw_score IS NULL")
    conn.execute(
        "UPDATE eval_results SET fitness_score = passed WHERE fitness_score IS NULL"
    )
    # Ensure driver_state has exactly one row
    row = conn.execute("SELECT COUNT(*) FROM driver_state").fetchone()
    if row[0] == 0:
        conn.execute("INSERT INTO driver_state (current_step) VALUES (0)")


VARIANT_COLUMNS = [
    "id",
    "island",
    "tag",
    "parent_id",
    "parent_b_id",
    "origin",
    "generation",
    "score",
    "val_score",
    "eval_level",
    "step",
    "builder_cost",
    "created_at",
]


def insert_variant(conn, variant):
    """Insert a variant dict.

    Only columns present in `variant` are written; absent columns fall
    back to the schema's DEFAULT (or stay NULL for nullable columns).
    This matters for NOT NULL columns with a DEFAULT — e.g. `step` is
    `NOT NULL DEFAULT 0`, so callers like the seed-row inserter that
    pre-date the column can omit it without violating the constraint.
    Unknown keys raise ValueError to surface typos early.
    """
    unknown = set(variant) - set(VARIANT_COLUMNS)
    if unknown:
        raise ValueError(f"Unknown variant columns: {sorted(unknown)}")
    cols = [c for c in VARIANT_COLUMNS if c in variant]
    placeholders = ", ".join("?" for _ in cols)
    values = [variant[c] for c in cols]
    conn.execute(
        f"INSERT INTO variants ({', '.join(cols)}) VALUES ({placeholders})",
        values,
    )


def get_variant(conn, variant_id):
    """Return the variant row as a dict, or None if not found."""
    row = conn.execute("SELECT * FROM variants WHERE id = ?", (variant_id,)).fetchone()
    return dict(row) if row else None


def get_population(conn, island_name):
    """Return all variants for an island as a list of dicts."""
    rows = conn.execute(
        "SELECT * FROM variants WHERE island = ? ORDER BY id",
        (island_name,),
    ).fetchall()
    return [dict(r) for r in rows]


def update_variant(conn, variant_id, **fields):
    """Update arbitrary fields on a variant. Only known columns allowed."""
    allowed = set(VARIANT_COLUMNS) - {"id"}
    for k in fields:
        if k not in allowed:
            raise ValueError(f"Not a variant column: {k!r}")
    if not fields:
        return
    set_clause = ", ".join(f"{k} = ?" for k in fields)
    values = list(fields.values()) + [variant_id]
    conn.execute(f"UPDATE variants SET {set_clause} WHERE id = ?", values)


def delete_variant(conn, variant_id):
    """Remove a variant. Cascades to eval_results."""
    conn.execute("DELETE FROM variants WHERE id = ?", (variant_id,))


def insert_eval_result(
    conn,
    variant_id,
    task_id,
    passed,
    evaluated_at,
    *,
    raw_score=None,
    fitness_score=None,
    details_json=None,
    failure_type=None,
):
    """Insert or replace one task result for a variant."""
    if raw_score is None:
        raw_score = float(passed)
    if fitness_score is None:
        fitness_score = raw_score
    conn.execute(
        "INSERT OR REPLACE INTO eval_results "
        "(variant_id, task_id, passed, raw_score, fitness_score, details_json, failure_type, evaluated_at) "
        "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
        (
            variant_id,
            task_id,
            passed,
            raw_score,
            fitness_score,
            details_json,
            failure_type,
            evaluated_at,
        ),
    )


def load_eval_sets(conn, base_dir, eval_sets_map):
    """Populate eval_sets table from JSON files.

    eval_sets_map: dict of {level: relative_path}, e.g. in production
    {25: 'eval_sets/eval_island_25.json', ...} or, in tests with their
    own layout, {25: 'eval_island_25.json'}. The path is joined with
    base_dir verbatim — the caller is responsible for the convention.
    Files contain JSON arrays of task_ids.
    """
    rows = []
    for level, filename in eval_sets_map.items():
        path = os.path.join(base_dir, filename)
        with open(path) as f:
            task_ids = json.load(f)
        for tid in task_ids:
            rows.append((tid, level))
    conn.executemany(
        "INSERT OR IGNORE INTO eval_sets (task_id, level) VALUES (?, ?)",
        rows,
    )


def recompute_score_from_train_set(conn, variant_id, train_task_ids):
    """Count this variant's passes restricted to an explicit train task set.

    v2 replacement for the retired tier-joined `recompute_score`: rather
    than joining eval_results against the eval_sets/level row (which
    encoded the old tier-cascade promotion model), the caller passes the
    canonical train task IDs directly. Returns 0 when train_task_ids is
    empty.
    """
    if not train_task_ids:
        return 0
    placeholders = ",".join("?" * len(train_task_ids))
    row = conn.execute(
        # COALESCE preserves compatibility with legacy/direct SQL writers
        # that only populate the historical binary ``passed`` column.
        f"SELECT COALESCE(SUM(COALESCE(fitness_score, passed)), 0) "
        f"FROM eval_results "
        f"WHERE variant_id = ? AND task_id IN ({placeholders})",
        (variant_id, *train_task_ids),
    ).fetchone()
    return float(row[0])


ISLAND_UPDATABLE_COLUMNS = {
    "mutation_count",
    "last_migration_at",
    "last_crossover_at",
    "last_elimination_at",
    "last_updated",
}

HISTORY_COLUMNS = [
    "type",
    "island",
    "parent",
    "child",
    "score",
    "cost",
    "details",
    "timestamp",
]


def init_island(conn, name):
    """Ensure an island row exists. Idempotent."""
    conn.execute(
        "INSERT OR IGNORE INTO islands (name) VALUES (?)",
        (name,),
    )


def get_island(conn, name):
    row = conn.execute("SELECT * FROM islands WHERE name = ?", (name,)).fetchone()
    return dict(row) if row else None


def update_island(conn, name, **fields):
    for k in fields:
        if k not in ISLAND_UPDATABLE_COLUMNS:
            raise ValueError(f"Not an island column: {k!r}")
    if not fields:
        return
    set_clause = ", ".join(f"{k} = ?" for k in fields)
    values = list(fields.values()) + [name]
    conn.execute(f"UPDATE islands SET {set_clause} WHERE name = ?", values)


def append_history(conn, entry):
    cols = ", ".join(HISTORY_COL

agent-s-computer-use/agent_s4/driver.pycode · 11.6 KB

"""Global lockstep driver.

One process replaces N tmux panes. The driver maintains a single global
step counter, dispatches one operator per step via a pluggable Scheduler,
and fans the operator out to all islands in parallel with a barrier.
"""

from __future__ import annotations

import concurrent.futures
import logging
import os
import random
import shutil
import subprocess
from datetime import datetime, timezone
from hashlib import sha256
from pathlib import Path

from agent_s4.config.experiment import ExperimentConfig
from agent_s4.benchmark_spec import ns
from agent_s4.constants import (
    DB_FILENAME,
    EVAL_CACHE_DIR,
    MIGRATE_TRANSIT_DIR,
)
from agent_s4.db import (
    connect,
    delete_orphan_variants_at_step,
    find_pending_dispatch,
    get_current_step,
    init_schema,
    log_dispatch_complete,
    log_dispatch_start,
    set_current_step,
)
from agent_s4.operators.crossover import Crossover
from agent_s4.operators.elimination import Elimination
from agent_s4.operators.migration import Migration
from agent_s4.operators.mutation import Mutation
from agent_s4.operators.reevaluation import Reevaluation
from agent_s4.state import LoopState

logger = logging.getLogger("agent_s4.driver")


def derive_seed(global_seed: int | None, island_name: str) -> int:
    """Per-island rng seed derived deterministically from the global seed +
    island name. `None` global seed → fresh randomness."""
    if global_seed is None:
        return int.from_bytes(os.urandom(8), "big")
    h = sha256(f"{global_seed}:{island_name}".encode()).digest()
    return int.from_bytes(h[:8], "big")


def _build_loopstate(
    island_name: str, step: int, config: ExperimentConfig, base_dir: Path
):
    """Construct a LoopState inside a worker subprocess.

    Returns (state, conn) so the caller can close conn in a finally block.
    Picklable inputs only — conn is opened locally, never passed in."""
    from agent_s4.benchmark_spec import BENCHMARKS

    template = BENCHMARKS[config.benchmark]["template"]
    worker_conn = connect(str(base_dir / ns(DB_FILENAME, config.benchmark)))
    rng = random.Random(derive_seed(config.seed, island_name))
    state = LoopState(
        conn=worker_conn,
        current_island=island_name,
        step=step,
        rng=rng,
        config=config,
        clone_dir=base_dir / f"{template}_{island_name}",
        base_dir=base_dir,
        eval_cache_dir=base_dir / ns(EVAL_CACHE_DIR, config.benchmark),
    )
    return state, worker_conn


# Default operator registry used by _worker_entry when no op_classes
# injection is provided. Tests pass a custom op_classes dict to substitute
# marker operators that the production registry doesn't know about.
_DEFAULT_OP_CLASSES = {
    "mutation": Mutation,
    "reevaluation": Reevaluation,
    "crossover": Crossover,
    "elimination": Elimination,
    "migration_phase1": Migration,
    "migration_phase2": Migration,
}


def _worker_entry(
    island_name: str,
    step: int,
    op_kind: str,
    config: ExperimentConfig,
    base_dir: Path,
    op_classes: dict | None = None,
) -> None:
    """ProcessPoolExecutor worker entry. Picklable; runs in subprocess.

    Builds own LoopState (with own DB conn), dispatches the operator
    named by op_kind, closes conn in finally. op_classes lets tests
    substitute marker operators across the process boundary; production
    callers pass None to use the default registry."""
    registry = op_classes if op_classes is not None else _DEFAULT_OP_CLASSES
    state, conn = _build_loopstate(island_name, step, config, base_dir)
    try:
        if op_kind == "elimination":
            cls = registry.get("elimination", Elimination)
            cls(count=config.elimination_count).apply(state)
        elif op_kind == "migration_phase1":
            cls = registry.get("migration_phase1", Migration)
            cls().apply_phase1(state)
        elif op_kind == "migration_phase2":
            cls = registry.get("migration_phase2", Migration)
            cls().apply_phase2(state)
        elif op_kind in ("mutation", "crossover", "reevaluation"):
            cls = registry.get(op_kind)
            if cls is None:
                cls = _DEFAULT_OP_CLASSES[op_kind]
            cls().apply(state)
        else:
            raise ValueError(f"unknown op_kind: {op_kind!r}")
    finally:
        conn.close()


def _total_cost_for_run(conn) -> float:
    row = conn.execute("SELECT COALESCE(SUM(cost_usd), 0) FROM cost_events").fetchone()
    return float(row[0] if row else 0)


def _now() -> str:
    return datetime.now(timezone.utc).isoformat()


class Driver:
    def __init__(
        self, config: ExperimentConfig, base_dir: Path, op_classes: dict | None = None
    ):
        self.config = config
        self.base_dir = Path(base_dir)
        self.conn = connect(str(self.base_dir / ns(DB_FILENAME, self.config.benchmark)))
        init_schema(self.conn)
        # Scheduler RNG: shared across choose_op calls so probabilistic
        # schedulers produce reproducible sequences for a given seed.
        # Coerce non-seed values (e.g. MagicMock in tests that don't care)
        # to None so random.Random falls through to fresh entropy.
        seed = getattr(config, "seed", None)
        if not isinstance(seed, (int, float, str, bytes, bytearray, type(None))):
            seed = None
        self.rng = random.Random(seed)
        # Optional operator class registry; passed to _worker_entry across
        # the process boundary. None → workers use _DEFAULT_OP_CLASSES.
        self.op_classes = op_classes
        # One warm ProcessPoolExecutor reused across all dispatches.
        # ProcessPoolExecutor (not ThreadPoolExecutor) because OSWorld's
        # DesktopEnv.__init__ calls signal.signal() which is main-thread-only;
        # each worker needs its own main thread.
        self._pool = concurrent.futures.ProcessPoolExecutor(
            max_workers=len(config.islands)
        )

    def stop_condition(self, step: int) -> bool:
        if self.config.max_steps is not None and step > self.config.max_steps:
            return True
        if self.config.max_cost is not None:
            if _total_cost_for_run(self.conn) >= self.config.max_cost:
                return True
        return False

    def _fan_out(self, op_kind: str, step: int) -> None:
        """Submit one _worker_entry per island to the warm pool, barrier on
        all futures. Exceptions from worker subprocesses are re-raised in the
        driver via f.result() (pickled across IPC with _RemoteTraceback)."""
        futures = [
            self._pool.submit(
                _worker_entry,
                i.name,
                step,
                op_kind,
                self.config,
                self.base_dir,
                self.op_classes,
            )
            for i in self.config.islands
        ]
        for f in futures:
            f.result()

    def run_op_on_all_islands(self, op: str, step: int) -> None:
        """Fan an operator dispatch out across islands. For migration the
        driver runs phase 1 to completion (barrier) before phase 2 starts;
        other ops are a single fan-out."""
        if op == "migration":
            self._fan_out("migration_phase1", step)
            self._fan_out("migration_phase2", step)
            # Purge transit dirs only AFTER the phase-2 barrier. Under a
            # complete topology a source transit is read by several
            # destinations in parallel, so cleanup can't live in phase 2; by
            # here every destination has consumed every source. A mid-phase-2
            # crash skips this line, leaving transits for idempotent re-dispatch.
            transit_root = self.base_dir / ns(
                MIGRATE_TRANSIT_DIR, self.config.benchmark
            )
            if transit_root.exists():
                shutil.rmtree(transit_root, ignore_errors=True)
            return
        self._fan_out(op, step)

    def _resume_check(self) -> bool:
        """If a previous run crashed mid-dispatch, clean up partial state.

        Returns True if cleanup happened, False if no pending dispatch.
        """
        pending = find_pending_dispatch(self.conn)
        if pending is None:
            return False

        pending_id = pending["id"]
        step = pending["step"]
        op_kind = pending["op_kind"]
        started_at = pending["started_at"]
        logger.warning(
            "Resume: found pending dispatch step=%d op=%s started_at=%s — cleaning up",
            step,
            op_kind,
            started_at,
        )

        # Delete orphan variants from the partial step
        from agent_s4.benchmark_spec import BENCHMARKS

        template = BENCHMARKS[self.config.benchmark]["template"]
        orphans = delete_orphan_variants_at_step(self.conn, step)
        for orphan in orphans:
            # Best-effort git tag deletion on that island's clone
            clone_dir = self.base_dir / f"{template}_{orphan['island']}"
            if clone_dir.exists():
                subprocess.run(
                    ["git", "tag", "-d", orphan["tag"]],
                    cwd=str(clone_dir),
                    capture_output=True,
                    check=False,
                )
            logger.info(
                "Resume: deleted orphan variant %s (tag %s)",
                orphan["id"],
                orphan["tag"],
            )

        # For migration: PRESERVE transit dirs. Phase 2 will read them on
        # re-dispatch and complete the receive. Only purge transit dirs
        # if we're rolling back a non-migration op (defensive — shouldn't happen).
        if op_kind != "migration":
            transit_root = self.base_dir / ns(
                MIGRATE_TRANSIT_DIR, self.config.benchmark
            )
            if transit_root.exists():
                logger.warning("Resume: non-migration crash had transit dirs; purging")
                shutil.rmtree(transit_root, ignore_errors=True)

        # Delete the pending dispatch log row
        self.conn.execute("DELETE FROM dispatch_log WHERE id = ?", (pending_id,))
        self.conn.commit()
        return True

    def _reevaluate_if_due(self, completed_step: int) -> None:
        interval = getattr(self.config, "reevaluation_interval", 0)
        if (
            isinstance(interval, int)
            and interval > 0
            and completed_step > 0
            and completed_step % interval == 0
        ):
            # Separate from dispatch_log/current_step: a failed maintenance
            # round must not roll back the completed evolutionary operator.
            self.run_op_on_all_islands("reevaluation", completed_step)

    def run(self) -> None:
        self._resume_check()
        step = get_current_step(self.conn) + 1
        try:
            # Includes resuming an experiment paused after step 5, and a
            # retry after a maintenance failure at the final budgeted step.
            self._reevaluate_if_due(step - 1)
            while not self.stop_condition(step):
                # Pass self as state — Scheduler.choose_op only needs `.rng`,
                # which Driver provides. DeterministicScheduler ignores state;
                # ProbabilisticScheduler uses state.rng.choices().
                op = self.config.scheduler.choose_op(step, state=self)
                logger.info("[step=%d] dispatching op=%s", step, op)
                dispatch_id = log_dispatch_start(self.conn, step, op, _now())
                self.run_op_on_all_islands(op, step)
                log_dispatch_complete(self.conn, dispatch_id, _now())
                set_current_step(self.conn, step)
                self._reevaluate_if_due(step)
                step += 1
        finally:
            self._pool.shutdown(wait=True)
        logger.info("Driver stopping at step=%d", step)

agent-s-computer-use/agent_s4/eval/__init__.pycode · 0.0 KB

agent-s-computer-use/agent_s4/eval/grader_image_limit.pycode · 2.3 KB

"""Keep Claude judge images under Anthropic's 5 MB limit on the OpenAI-compatible path.

OSWorld V2's native ``anthropic`` backend downsizes every judge image through
``desktop_env.image_utils.anthropic_image_file_to_mime_and_base64``. The
``openai_compatible`` backend -- what the grader uses since it moved to
OpenRouter (``OSWORLD_EVAL_MODEL_PROVIDER=openai_compatible`` in .env) -- sends
the raw file. Tasks 075 and 078 ship reference images of 13 MB and 7.7 MB, so
their LLM-judge step fails with HTTP 400 "image exceeds 5 MB maximum", the
harness counts that as infrastructure, and after three attempts the task
becomes an imputed zero.

The fix belongs in OSWorldV2_Evol, but the preflight demands that checkout be
byte-identical to the pinned release, so it is applied here at import time
instead. Only a Claude model is affected, and an image already under the limit
is sent byte-for-byte, so every other judge call is unchanged.
"""

import contextvars
import functools

_claude_call = contextvars.ContextVar("osworld_claude_judge_call", default=False)


def apply() -> bool:
    try:
        from desktop_env.evaluators.backends import openai_backend
        from desktop_env.image_utils import anthropic_image_file_to_mime_and_base64
    except ImportError:
        return False
    backend = openai_backend.OpenAIBackend
    if getattr(backend._generate_once, "_shadow_image_limit", False):
        return True

    plain_mime, plain_encode = openai_backend.guess_mime, openai_backend.encode_image

    @functools.lru_cache(maxsize=32)
    def _limited(path):
        return anthropic_image_file_to_mime_and_base64(path)

    def guess_mime(path):
        return _limited(path)[0] if _claude_call.get() else plain_mime(path)

    def encode_image(path):
        return _limited(path)[1] if _claude_call.get() else plain_encode(path)

    original = backend._generate_once

    @functools.wraps(original)
    def _generate_once(self, *args, **kwargs):
        token = _claude_call.set("claude" in str(self.config.model).lower())
        try:
            return original(self, *args, **kwargs)
        finally:
            _claude_call.reset(token)

    _generate_once._shadow_image_limit = True
    openai_backend.guess_mime = guess_mime
    openai_backend.encode_image = encode_image
    backend._generate_once = _generate_once
    return True

agent-s-computer-use/agent_s4/eval/osworld.pytext · 68.9 KB
No inline preview for this entry.

agent-s-computer-use/agent_s4/eval/osworld_v2.pycode · 26.2 KB · preview truncated

"""
Shadow Interface evaluation runner — OSWorld V2 (xlang-ai/OSWorld-V2).

Run as `python -m agent_s4.eval.osworld_v2 ...` with an OSWorld V2 clone as
cwd, so `desktop_env` resolves to V2's rather than the editable-installed v1
fork. The task definitions come from OSWorldV2_Base/evaluation_examples, which
is the only checkout holding the gated task_class files.

Everything generation-neutral — prompts, agent-command and episode plumbing,
result persistence, the worker/main fan-out, the shared CLI surface — is
imported from agent_s4.eval.osworld. This module holds what V2 actually
changes:

  - tasks are Python BaseTask subclasses under task_class/, loaded through
    V2's own task_loader rather than json.load
  - a task declares its own VM: platform, image, instance_type, volume_size,
    proxy, and the vnc/recording toggles
  - evaluate() may return a dict of partial-credit and safety detail, not
    just a float
  - a task may be multi-phase, with per-phase setup, instruction and
    evaluation, and gates that stop the run early
  - a task may carry a user simulator the agent can question through an
    ASK_USER protocol

Usage:
    python -m agent_s4.eval.osworld_v2 \
        --task_ids 017 --num_envs 1 --harness prime_agent \
        --model openai/<deployment> \
        --test_config_base_dir <OSWorldV2_Base>/evaluation_examples
"""

import argparse
import json
import logging
import os
import re
import sys
import time

# Bound here so osworld.worker(impl=this module) constructs V2's DesktopEnv,
# and so tests can substitute it on this module rather than on osworld.
from desktop_env.desktop_env import DesktopEnv

from agent_s4.eval import grader_image_limit
from agent_s4.eval import osworld
from agent_s4.eval.osworld import (
    EnvSpec,
    HarnessFailure,
    add_common_args,
    build_env_kwargs as _base_env_kwargs,
    finalize_task_result,
    persist_evaluation_result,
    prepare_task_vm,
    resolve_cache_dir,
    resolve_model_config,
    _run_agent_episode,
    _task_get,
)

logger = logging.getLogger("desktopenv.shadow_eval")

grader_image_limit.apply()

DEFAULT_MAX_USER_TURNS = 5

# The V2 AMI ships a 29 GB root of which snapd and flatpak alone occupy ~13 GB,
# leaving under 3 GB free before we add anything. That is not enough for the
# agent runtime, its dependencies and a screen recording, and a full disk
# corrupts the agent's exit-code marker rather than failing loudly. Only 18 of
# the 108 tasks declare a volume_size, so the remaining 90 would otherwise boot
# onto a root that is already full.
DEFAULT_VOLUME_SIZE_GB = 60
# Below this the runtime fits but leaves little room for task artefacts.
MIN_COMFORTABLE_VOLUME_SIZE_GB = 40

HITL_SYSTEM_PROMPT_SUFFIX = """\
## Asking the simulated user

This task provides a benchmark user simulator. If a required fact is genuinely
ambiguous, finish your current episode with exactly one line in this form:

    ASK_USER: <one concise question>

The harness will return the simulated user's answer and start a continuation
episode on the same VM. Do not search the filesystem or task configuration for
hidden user knowledge.
"""


# ---------------------------------------------------------------------------
# Task-declared VM specification
# ---------------------------------------------------------------------------


def _platform_to_os_type(platform: str | None) -> str:
    return (
        "Windows" if str(platform or "").strip().lower().startswith("win") else "Ubuntu"
    )


def _resolve_volume_size(example, args) -> int | None:
    """Root volume GB for this task: task declaration, then CLI, then default.

    A task's own declaration stays authoritative -- upstream sized it -- so a
    small value is warned about rather than silently raised.
    """
    task_volume = _task_get(example, "volume_size")
    if task_volume is not None:
        source, volume_size = "task", int(task_volume)
    else:
        cli_volume = getattr(args, "volume_size", None)
        if cli_volume is not None:
            source, volume_size = "--volume_size", int(cli_volume)
        else:
            source, volume_size = "harness default", DEFAULT_VOLUME_SIZE_GB
    if volume_size < MIN_COMFORTABLE_VOLUME_SIZE_GB:
        logger.warning(
            "Root volume is %dGB (from %s); the V2 AMI uses ~26GB before the "
            "agent runtime is installed, so this may run out of disk",
            volume_size,
            source,
        )
    return volume_size


def build_env_spec(example, args) -> EnvSpec:
    """Resolve the VM spec *after* loading a V2 task.

    A V2 task class may override the platform, the image, the instance type
    and the root volume, and declares whether it needs the proxy. The VM must
    therefore be built per task rather than once per worker.
    """
    screen_size = (args.screen_width, args.screen_height)
    os_type = _platform_to_os_type(_task_get(example, "platform"))
    snapshot_name = getattr(args, "snapshot_name", "init_state")
    if args.provider_name == "aws":
        image = _task_get(example, "image")
        if image:
            snapshot_name = image
        else:
            from desktop_env.providers.aws.manager import get_default_ami_id

            snapshot_name = get_default_ami_id(
                region=args.region,
                screen_size=screen_size,
                os_type=os_type,
            )

    # The official runner enables the proxy and lets each task's own `proxy`
    # field decide whether to route through it, so default to on.
    proxy_override = getattr(args, "enable_proxy", None)
    enable_proxy = True if proxy_override is None else bool(proxy_override)
    volume_size = _resolve_volume_size(example, args)
    instance_type = getattr(args, "instance_type_override", None) or _task_get(
        example, "instance_type"
    )
    return EnvSpec(
        os_type=os_type,
        snapshot_name=snapshot_name,
        instance_type=instance_type,
        volume_size=volume_size,
        enable_proxy=enable_proxy,
        force_disable_vnc=bool(_task_get(example, "disable_vnc", False)),
        force_disable_recording=bool(_task_get(example, "disable_recording", False)),
        cache_dir=resolve_cache_dir(example, args),
        is_v2=True,
    )


def build_env_kwargs(spec: EnvSpec, args) -> dict:
    kwargs = _base_env_kwargs(spec, args)
    # These kwargs exist only in OSWorld V2's DesktopEnv.
    kwargs.update(
        instance_type=spec.instance_type,
        volume_size=spec.volume_size,
        force_disable_vnc=spec.force_disable_vnc,
        force_disable_recording=spec.force_disable_recording,
        # The provider returns the private address unless asked otherwise, so
        # an orchestrator outside the VPC has to opt in or it never reaches a
        # single VM: the connections sit in SYN-SENT, setup_controller never
        # reports ready, and every task fails in vm_allocation after burning
        # its retries. Default stays private for in-VPC runs.
        use_public_ip=bool(getattr(args, "use_public_ip", False)),
    )
    return kwargs


# ---------------------------------------------------------------------------
# Task discovery and loading
# ---------------------------------------------------------------------------


def build_task_index(test_config_base_dir):
    """Build task_id -> definition-file path over both V2 task corpora.

    Two corpora live under one evaluation_examples/ tree:
      - examples/<domain>/<uuid>.json — the v1-style corpus, which V2 also
        ships in a larger, patched form.
      - task_class/task_<NNN>.py      — V2-native tasks, one BaseTask
        subclass per file. The id is the filename suffix ('task_001.py' ->
        '001'), which matches the ids listed in test_v2.json.

    IDs never collide (UUIDs vs. zero-padded ints), so one flat index serves
    both; load_task() dispatches on the file extension.
    """
    index = osworld.build_task_index(test_config_base_dir)

    task_class_dir = os.path.join(test_config_base_dir, "task_class")
    if os.path.isdir(task_class_dir):
        for fname in sorted(os.listdir(task_class_dir)):
            # task_<id>.py only — skips __init__.py and generated_task_utils.py
            if not (fname.startswith("task_") and fname.endswith(".py")):
                continue
            task_id = fname[len("task_") : -len(".py")]
            index[task_id] = os.path.join(task_class_dir, fname)
    from agent_s4.synthetic_tasks import extend_task_index

    extend_task_index(index)
    return index


_TASK_LOADER = None


def _task_loader(test_config_base_dir):
    """Import OSWorld V2's task_loader module, cached.

    It sits at the benchmark repo root (the parent of evaluation_examples),
    and we load it by path rather than plain `import task_loader` because the
    eval subprocess's cwd differs by caller — the orchestrator runs it inside
    an island clone, the held-out scripts from the repo root.
    """
    global _TASK_LOADER
    if _TASK_LOADER is None:
        import importlib.util

        repo_root = os.path.dirname(os.path.abspath(test_config_base_dir))
        loader_path = os.path.join(repo_root, "task_loader.py")
        if not os.path.isfile(loader_path):
            raise RuntimeError(
                f"{loader_path} not found — V2-native (.py) tasks need "
                f"OSWorld V2's task_loader at the benchmark repo root."
            )
        spec = importlib.util.spec_from_file_location(
            "osworld_task_loader", loader_path
        )
        mod = importlib.util.module_from_spec(spec)
        sys.modules["osworld_task_loader"] = mod
        try:
            spec.loader.exec_module(mod)
        except ModuleNotFoundError as e:
            if "task_base" not in str(e):
                raise
            # `desktop_env` resolved to the v1 fork, which has no task_base.
            # The eval imports desktop_env by name, so its cwd picks the
            # generation — both evaluator harnesses run it with cwd=clone_dir.
            import desktop_env

            raise RuntimeError(
                f"V2-native tasks need OSWorld V2's desktop_env, but "
                f"desktop_env resolved to {desktop_env.__file__} (no "
                f"task_base module). Run the eval with an OSWorld V2 clone as "
                f"cwd — its clones are OSWorldV2_Evol_<island>/."
            ) from e
        _TASK_LOADER = mod
    return _TASK_LOADER


def load_task(task_path, test_config_base_dir):
    """Load one task definition: a V2 BaseTask instance, or a v1 JSON dict.

    BaseTask subclasses dict, so callers treat both the same — `task["id"]`,
    `task.get(...)`, and `DesktopEnv.reset(task_config=...)` all work
    unchanged. V2 task modules read env vars at import time
    (WEBSITE_HOST_SUFFIX for website tasks, GITLAB_* for the GitLab ones), so
    a missing one surfaces here as an ImportError-ish failure.
    """
    if task_path.endswith(".py"):
        return _task_loader(test_config_base_dir).load_task_from_file(task_path)
    from agent_s4.synthetic_tasks import prepare_loaded_task

    return prepare_loaded_task(
        osworld.load_task(task_path, test_config_base_dir), task_path
    )


def load_all_task_ids(test_all_meta_path):
    """Load all task IDs from a manifest.

    Handles both shapes: v1's domain -> [ids] mapping (test_all.json), and
    V2's {"tasks": [ids]} (test_v2.json).
    """
    with open(test_all_meta_path) as f:
        data = json.load(f)
    if isinstance(data, list):
        return list(data)
    if set(data) == {"tasks"}:
        return list(data["tasks"])
    all_ids = []
    for task_ids in data.values():
        all_ids.extend(task_ids)
    return all_ids


# ---------------------------------------------------------------------------
# Instructions and the human-in-the-loop protocol
# ---------------------------------------------------------------------------


def _instruction_for_task(example, instruction: str) -

agent-s-computer-use/agent_s4/eval/scienceboard_prime.pycode · 23.4 KB · preview truncated

"""ScienceBoard evaluation through the shared OSWorld worker.

Run as ``python -m agent_s4.eval.scienceboard_prime ...``. This module
plugs ScienceBoard into ``osworld.main(impl=...)`` the same way ``osworld_v2``
does, so Prime Agent, the per-task wall-clock budget, infra
retries, artifact collection and summaries are the code V2 already runs.

What is ScienceBoard-specific lives here:

  - the task index: official tasks under ScienceBoard_Base/tasks/VM (ids
    ``<App>:<stem>``) plus the ungraded synthetic training bank named by
    ``SCIENCEBOARD_SYNTHETIC_ROOT`` (ids ``sb_<app>_<NNN>``)
  - ``ScienceBoardEnv``: one fresh EC2 per task from the ScienceBoard AMI,
    exposing the ``controller`` / ``reset`` / ``close`` surface the shared
    worker drives, with the per-app ``sci`` VMManager wired onto it
  - scoring: official tasks run ScienceBoard's own ``Task.eval`` after
    bridging the agent's /tmp/AGENT_ANSWER and /tmp/INFEASIBLE sentinels into
    a stop primitive; synthetic tasks have no evaluator and record the same
    ``dummy_constant_zero`` placeholder the V2 synthetic bank uses
"""

from __future__ import annotations

import argparse
import hashlib
import json
import logging
import os
import shlex
import sys
import tempfile
import time
import traceback
from dataclasses import dataclass
from pathlib import Path, PurePosixPath

from agent_s4.eval import osworld
from agent_s4.eval.osworld import (
    INSTRUCTION_CONTEXT_SUFFIX,
    add_common_args,
    finalize_task_result,
    persist_evaluation_result,
    prepare_task_vm,
    resolve_model_config,
    _run_agent_episode,
)

logger = logging.getLogger("desktopenv.shadow_eval.scienceboard")

from agent_s4.paths import PROJECT_ROOT

SYNTHETIC_ROOT_ENV = "SCIENCEBOARD_SYNTHETIC_ROOT"

# A ScienceBoard AMI in your own account (see osworld_setup/s4/README.md).
SB_AMI_ID = os.environ.get("SB_AMI_ID", "")
# t3.medium (4 GB) runs out of memory once VS Code starts Lean over Mathlib.
SB_INSTANCE_TYPE = os.environ.get("SB_INSTANCE_TYPE", "t3.large")
# sci.Prompts.VM_PASSWORD -- the AMI's sudo password for user "user".
SB_CLIENT_PASSWORD = os.environ.get("SB_CLIENT_PASSWORD", "osworld-public-evaluation")
# The scientific apps bind their REST ports well after the :5000 daemon is up;
# ScienceBoard's own task waits (2.5-5s) assume a warm desktop. Below this the
# next init step races the app and fails with a JSON decode error.
LAUNCH_WAIT_FLOOR_SECONDS = 60
DAEMON_BOOT_DEADLINE_SECONDS = 420

SCIENCEBOARD_PROMPT_ADDENDUM = """

## ScienceBoard task semantics

- If the task asks for a specific answer (e.g. a number, a name, a value), write your final answer to /tmp/AGENT_ANSWER as plain text on a single line, then exit.
- If you determine the task is impossible to complete, create /tmp/INFEASIBLE and exit.
- Most tasks have no answer expectation; for those, just complete the operations and exit normally.
- The scientific application the task concerns is already open. Keep it running: the benchmark inspects its live state after you exit.
"""

UNGRADED = {
    "verification": "dummy_constant_zero",
    "score_semantics": "UNGRADED: constant zero; use trajectories, not scores",
}


# ---------------------------------------------------------------------------
# Task index
# ---------------------------------------------------------------------------


def _validate_synthetic_package(root: Path) -> list[dict]:
    """Check the input bundle against its own manifest before any VM exists."""
    manifest = json.loads((root / "PACKAGE_MANIFEST.json").read_text())
    for relative, expected in manifest.items():
        path = (root / relative).resolve()
        if not path.is_relative_to(root) or not path.is_file():
            raise ValueError(f"ScienceBoard synthetic package is missing {relative}")
        if hashlib.sha256(path.read_bytes()).hexdigest() != expected:
            raise ValueError(
                f"ScienceBoard synthetic package hash mismatch: {relative}"
            )
    entries = json.loads((root / "index.json").read_text())
    if len(entries) != 50 or any(e.get("verification") != "none" for e in entries):
        raise ValueError("Expected the 50-task ungraded ScienceBoard synthetic bank")
    return entries


def build_task_index(test_config_base_dir):
    """Map task id -> definition path for official and synthetic tasks."""
    root = Path(test_config_base_dir)
    if not root.is_dir():
        raise SystemExit(f"ScienceBoard tasks root not found: {root}")
    index = {}
    for app_dir in sorted(p for p in root.iterdir() if p.is_dir()):
        for path in sorted(app_dir.glob("*.json")):
            index[f"{app_dir.name}:{path.stem}"] = str(path)
    configured = os.environ.get(SYNTHETIC_ROOT_ENV)
    if configured:
        synthetic_root = Path(configured).resolve()
        for entry in _validate_synthetic_package(synthetic_root):
            if entry["id"] in index:
                raise ValueError(f"Synthetic task id collides: {entry['id']}")
            index[entry["id"]] = str(synthetic_root / entry["path"] / "task.json")
    return index


# The guest recorder is ffmpeg x11grab -> libx264 at 1080p30. On the 2-vCPU
# ScienceBoard instance it saturates the CPU for the whole episode, competing
# with the agent and the scientific application, and after SIGINT it spends
# tens of minutes draining its backlog -- /end_recording has no timeout, so the
# task's VM and worker stay pinned that long. Trajectories are what the Builder
# reads; recordings are not worth either cost.
DISABLE_RECORDING = True


def load_task(task_path, test_config_base_dir):
    del test_config_base_dir
    path = Path(task_path)
    spec = json.loads(path.read_text())
    if "primary_application" in spec:
        return {
            "id": spec["id"],
            "instruction": spec["instruction"],
            "type": spec["primary_application"],
            "steps": spec["steps"],
            "sb_kind": "synthetic",
            "sb_dir": str(path.parent),
            "disable_recording": DISABLE_RECORDING,
        }
    task = dict(spec)
    task["id"] = f"{path.parent.name}:{path.stem}"
    task["sb_kind"] = "official"
    task["sb_path"] = str(path)
    task["disable_recording"] = DISABLE_RECORDING
    return task


def load_all_task_ids(test_all_meta_path):
    return osworld.load_all_task_ids(test_all_meta_path)


# ---------------------------------------------------------------------------
# One EC2 per task
# ---------------------------------------------------------------------------


@dataclass(frozen=True)
class EnvSpec:
    # The task id makes every spec distinct, so the shared worker never reuses
    # a VM: ScienceBoard has no in-place snapshot revert on AWS.
    task_id: str
    app: str
    ami_id: str
    instance_type: str


def build_env_spec(example, args) -> EnvSpec:
    del args
    if not SB_AMI_ID:
        raise RuntimeError("SB_AMI_ID is not set: point it at a ScienceBoard AMI")
    return EnvSpec(
        task_id=example["id"],
        app=example["type"],
        ami_id=SB_AMI_ID,
        instance_type=SB_INSTANCE_TYPE,
    )


def build_env_kwargs(spec: EnvSpec, args) -> dict:
    return {
        "spec": spec,
        "region": args.region,
        "use_public_ip": bool(getattr(args, "use_public_ip", False)),
    }


class _SciEnvShim:
    """What sci's VManager expects of OSWorld's DesktopEnv, on our instance.

    sci's AWS revert terminates and relaunches through OSWorld's provider (no
    Owner tag, private addressing). The instance here is already fresh, so
    revert reduces to waiting for the daemon, which sci then does itself.
    """

    def __init__(self, env: "ScienceBoardEnv"):
        self.controller = env.controller
        self.vm_ip = env.vm_ip
        self.snapshot_name = env.spec.ami_id

    def _revert_to_snapshot(self):
        return None

    def _start_emulator(self):
        return None

    def close(self):
        return None


def _ensure_sci_importable() -> None:
    root = str(
        Path(
            os.environ.get(
                "SCIENCEBOARD_PATH",
                PROJECT_ROOT / "ScienceBoard_Base",
            )
        ).resolve()
    )
    if root not in sys.path:
        sys.path.insert(0, root)


def _floor_launch_waits(initialize: list[dict]) -> list[dict]:
    out = []
    for item in initialize:
        item = dict(item)
        if item.get("func") == "launch":
            item["wait"] = max(item.get("wait", 0), LAUNCH_WAIT_FLOOR_SECONDS)
        out.append(item)
    return out


class ScienceBoardEnv:
    def __init__(
        self,
        *,
        spec: EnvSpec,
        region: str,
        use_public_ip: bool,
        volume_size: int | None = None,
    ):
        import boto3
        import requests
        from desktop_env.controllers.python import PythonController

        self.spec = spec
        self.region = region
        self.client_password = SB_CLIENT_PASSWORD
        self.action_history = []
        self.is_environment_used = False
        self.manager = None
        self.task = None
        self._tmp_dir = None
        self._closed = False

        subnet = os.environ.get("AWS_SUBNET_ID")
        group = os.environ.get("AWS_SECURITY_GROUP_ID")
        if not subnet or not group:
            raise RuntimeError("AWS_SUBNET_ID and AWS_SECURITY_GROUP_ID are required")
        tags = [
            {
                "Key": "Name",
                "Value": os.environ.get(
                    "AWS_INSTANCE_NAME", f"shadow-evol-sb-{spec.task_id}"
                ),
            },
            {"Key": "sb_task", "Value": spec.task_id},
        ]
        self._ec2 = boto3.client("ec2", region_name=region)
        volume_options = {}
        self.root_volume_gib = None
        if volume_size is not None:
            if volume_size < 1:
                raise ValueError("volume_size must be positive")
            image = self._ec2.describe_images(ImageIds=[spec.ami_id])["Images"][0]
            root_name = image["RootDeviceName"]
            root_ebs = next(
                mapping["Ebs"]
                for mapping in image["BlockDeviceMappings"]
                if mapping["DeviceName"] == root_name
            )
            # An EBS snapshot cannot be restored onto a smaller volume.
            self.root_volume_gib = max(volume_size, root_ebs["VolumeSize"])
            volume_options["BlockDeviceMappings"] = [
                {
                    "DeviceName": root_name,
                    "Ebs": {
                        "VolumeSize": self.root_volume_gib,
                        "VolumeType": "gp3",
                        "DeleteOnTermination": True,
                    },
                }
            ]
        response = self._ec2.run_instances(
            ImageId=spec.ami_id,
            InstanceType=spec.instance_type,
            MinCount=1,
            MaxCount=1,
            NetworkInterfaces=[
                {
                    "SubnetId": subnet,
                    "Groups": [group],
                    "AssociatePublicIpAddress": True,
                    "DeviceIndex": 0,
                }
            ],
            TagSpecifications=[{"ResourceType": "instance", "Tags": tags}],
            **volume_options,
        )
        self.instance_id = response["Instances"][0]["InstanceId"]
        try:
            self._ec2.get_waiter("instance_running").wait(
                InstanceIds=[self.instance_id],
                WaiterConfig={"Delay": 5, "MaxAttempts": 60},
            )
            instance = self._ec2.describe_instances(
                InstanceIds=[self.instance_id],
            )[
                "Reservations"
            ][0]["Instances"][0]
            self.vm_ip = (
                instance.get("PublicIpAddress")
                if use_public_ip
                else instance["PrivateIpAddress"]
            )
            if not self.vm_ip:
                raise RuntimeError(f"{self.instance_id} has no reachable address")

agent-s-computer-use/agent_s4/git_utils.pycode · 6.3 KB

"""Git helpers for inner shadow_interface/ repos and outer island clones.

Each per-island clone (OSWorld_Evol_<name>) has TWO git repos:
- the outer clone repo (tracks the whole fork), and
- the inner shadow_interface/ repo (tracks just the evolved code,
  tagged per-variant as evol/<island>/if-NNN).

These helpers operate on the inner repo unless noted otherwise.
"""

from __future__ import annotations

import logging
import os
import re
import subprocess

logger = logging.getLogger("agent_s4.git_utils")


def git(args_list: list[str], cwd: str | None = None) -> tuple[str, int]:
    """Run `git <args>` in cwd. Returns (stdout, returncode). Logs non-
    zero returncodes except the 'nothing to commit' case."""
    result = subprocess.run(
        ["git"] + args_list,
        cwd=cwd,
        capture_output=True,
        text=True,
        timeout=60,
    )
    if result.returncode != 0 and "nothing to commit" not in result.stdout:
        logger.warning(
            "git %s failed in %s: %s",
            " ".join(args_list),
            cwd,
            result.stderr.strip()[:200],
        )
    return result.stdout.strip(), result.returncode


def shadow_path(clone_dir: str) -> str:
    """Path to the inner shadow_interface git repo for a clone."""
    return os.path.join(clone_dir, "shadow_interface")


def git_tag_variant(clone_dir: str, tag_name: str, message: str) -> None:
    """Commit only evolved state and tag it in the island repository.

    Some V2 checkouts do not have a nested ``shadow_interface/.git``. Running
    ``git add -A`` from that subdirectory still scans/stages the entire outer
    repository, including multi-GB ``eval_results`` artifacts. Use explicit
    root-relative pathspecs so evidence caches can never enter a variant tag.

    Every step raises rather than warning. ``git`` itself is tolerant by
    contract -- ``status`` and ``tag`` callers read a non-zero return as an
    answer -- but here a failure is silent data loss: if the commit does not
    land, ``tag -f`` still succeeds and pins the child's name onto the
    parent's commit. Nothing downstream can tell. The held-out runner
    materializes variants from their tags, so it would score the parent's
    interface and label it the child, and the next step's checkout would
    reset the Builder's work out of the tree. Failing here instead leaves
    that work staged and recoverable, and costs only the step.
    """
    for args in (
        ["add", "-A", "--", "shadow_interface", "notebook.jsonl"],
        ["commit", "-m", message, "--allow-empty"],
        ["tag", "-f", tag_name],
    ):
        _, returncode = git(args, cwd=clone_dir)
        if returncode != 0:
            raise RuntimeError(
                f"git {args[0]} failed in {clone_dir} while tagging "
                f"{tag_name}; refusing to tag a variant whose commit did not "
                "land. The Builder's output is still staged."
            )


def git_checkout_variant(clone_dir: str, tag: str) -> None:
    """Checkout a variant tag in the inner shadow_interface git repo.

    Force-cleans dirty state first. Every call site (mutation start,
    crossover start, fill_eval) expects a clean tree — any uncommitted
    modifications or untracked files at this point are stale artifacts
    from a killed prior run and should be discarded.
    """
    inner = shadow_path(clone_dir)

    status_out, _ = git(["status", "--porcelain"], cwd=inner)
    if status_out.strip():
        dirty_lines = status_out.strip().splitlines()
        logger.warning(
            "Discarding stale state in %s before checkout of %s: %s",
            inner,
            tag,
            dirty_lines[:5],
        )

    _, rc1 = git(["reset", "--hard", tag], cwd=inner)
    _, rc2 = git(["clean", "-fdx"], cwd=inner)
    if rc1 != 0 or rc2 != 0:
        raise RuntimeError(f"Failed to checkout {tag} in {inner}")


def git_archive_variant(source_clone: str, tag: str, dest_dir: str) -> None:
    """Extract a tag's shadow_interface content into dest_dir/shadow_interface/.

    Reads from the source's inner git (.git inside shadow_interface/) — safe
    even if the source's working tree is mid-mutation.
    """
    dest_shadow = os.path.join(dest_dir, "shadow_interface")
    os.makedirs(dest_shadow, exist_ok=True)
    inner = shadow_path(source_clone)
    proc = subprocess.run(
        f"git -C {inner} archive {tag} | tar -x -C {dest_shadow}",
        shell=True,
        capture_output=True,
        text=True,
        timeout=30,
    )
    if proc.returncode != 0:
        raise RuntimeError(f"git archive failed: {proc.stderr}")


def revert_out_of_scope(clone_dir: str) -> list[str]:
    """Revert any builder modifications to files OUTSIDE shadow_interface/
    and notebook.jsonl. Uses the outer clone's git to reset tracked files.
    Returns list of reverted paths for logging."""
    protected = [
        "scripts/",
        "mm_agents/",
        "desktop_env/",
        "evaluation_examples/",
        "CLAUDE.md",
        "lib_run_single.py",
    ]
    out, _ = git(["status", "--porcelain"] + protected, cwd=clone_dir)
    modified = [line.split()[-1] for line in out.splitlines() if line.strip()]
    if modified:
        logger.warning(
            "Builder modified out-of-scope files in %s: %s",
            os.path.basename(clone_dir),
            modified,
        )
        git(["checkout", "HEAD", "--"] + protected, cwd=clone_dir)
    return modified


def next_variant_num_from_population_and_tags(
    population: list[dict],
    clone_dir: str | None = None,
) -> int:
    """Return the next variant number, avoiding collisions with existing
    inner git tags.

    Checks BOTH the population AND inner git tags (if clone_dir given)
    so eliminated variants' numbers don't get reused.
    """
    nums: list[int] = []
    for v in population:
        if "-if-" in v["id"]:
            try:
                nums.append(int(v["id"].split("-if-")[1]))
            except ValueError:
                pass
    if clone_dir:
        inner = shadow_path(clone_dir)
        try:
            out, rc = git(["tag"], cwd=inner)
            if rc == 0:
                for t in out.strip().split("\n"):
                    m = re.search(r"/if-(\d+)$", t)
                    if m:
                        nums.append(int(m.group(1)))
        except Exception:
            pass
    return max(nums, default=0) + 1

agent-s-computer-use/agent_s4/harness/__init__.pycode · 1.7 KB

"""Harness registry. Lookup by string key; instantiate on demand."""

from __future__ import annotations

from agent_s4.harness.base import (
    BuilderHarness,
    EvaluatorHarness,
    BuilderResult,
    EvalResults,
)
from agent_s4.harness.opencode.builder import OpenCodeBuilder
from agent_s4.harness.prime.builder import PrimeAgentBuilder
from agent_s4.harness.ubuntu_vm import UbuntuVMEvaluator

BUILDER_HARNESSES: dict[str, type[BuilderHarness]] = {
    "prime_agent": PrimeAgentBuilder,
    "opencode": OpenCodeBuilder,
}

# The evaluator class is benchmark-bound (VM provisioning + scoring), not
# model-bound. OSWorld V2 and ScienceBoard both run through the shared worker
# (agent_s4.eval.osworld_v2 / agent_s4.eval.scienceboard_prime), which
# UbuntuVMEvaluator selects by benchmark.
EVALUATOR_HARNESSES: dict[str, type[EvaluatorHarness]] = {
    "prime_agent": UbuntuVMEvaluator,
    "opencode": UbuntuVMEvaluator,
}


def resolve_builder_harness(key: str) -> BuilderHarness:
    try:
        cls = BUILDER_HARNESSES[key]
    except KeyError:
        raise KeyError(
            f"unknown builder harness {key!r}; known: {sorted(BUILDER_HARNESSES)}"
        )
    return cls()


def resolve_evaluator_harness(key: str, benchmark: str = "osworld") -> EvaluatorHarness:
    try:
        cls = EVALUATOR_HARNESSES[key]
    except KeyError:
        raise KeyError(
            f"unknown evaluator harness {key!r}; known: {sorted(EVALUATOR_HARNESSES)}"
        )
    return cls()


__all__ = [
    "BUILDER_HARNESSES",
    "EVALUATOR_HARNESSES",
    "resolve_builder_harness",
    "resolve_evaluator_harness",
    "BuilderHarness",
    "EvaluatorHarness",
    "BuilderResult",
    "EvalResults",
]

agent-s-computer-use/agent_s4/harness/base.pycode · 1.7 KB

"""Harness interfaces + result dataclasses.

BuilderHarness and EvaluatorHarness are intentionally separate — they model
two different launch shapes (one subprocess per mutation vs. 25 parallel
VMs running an agent class) and have no shared method signatures worth
abstracting.
"""

from __future__ import annotations

from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from pathlib import Path


@dataclass
class BuilderResult:
    exit_code: int
    cost_usd: float
    log_path: Path
    crashed: bool  # auth / rate-limit / short-exit signal


@dataclass
class EvalResults:
    pass_rate: float
    total: int
    passed: int
    failed: int
    results: list[dict] = field(default_factory=list)
    timed_out: bool = False


class BuilderHarness(ABC):
    @abstractmethod
    def launch(
        self,
        *,
        clone_dir: Path,
        prompt: str,
        model: str,
        timeout: int | None,
        env_extra: dict[str, str] | None,
        variant_id: str,
        logs_dir: Path,
        log_name: str,  # e.g. "gui-if-007_mutation" or "gui-if-008_crossover"
        builder_bin: str | None = None,  # explicit claude binary path; None → PATH
    ) -> BuilderResult: ...


class EvaluatorHarness(ABC):
    @abstractmethod
    def run_tasks(
        self,
        *,
        clone_dir: Path,
        shadow_dir: Path,
        result_dir: Path,
        task_ids: list[str],
        model: str,
        region: str,
        timeout: int,
        max_steps: int | None = None,
        no_time_limit: bool = False,
        harness: str | None = None,
        benchmark: str = "osworld",  # selects the *_Base task tree
    ) -> EvalResults: ...

agent-s-computer-use/agent_s4/harness/builder_common.pycode · 6.9 KB

"""Shared subprocess + log-parsing helpers for builder harnesses.

The subprocess plumbing (env build, file-based stdout, timeout handling,
post-run log parse, crash detection, BuilderResult assembly) lives here so
the builder harness only owns its argv construction and auth env vars.
"""

from __future__ import annotations

import json
import logging
import os
import shutil
import subprocess
import time
from pathlib import Path

from agent_s4.harness.base import BuilderResult

logger = logging.getLogger("agent_s4.harness.builder_common")

_CRASH_ELAPSED_CEILING = 30.0
_EVALUATOR_ONLY_ENV_KEYS = {
    "GITLAB_URL",
    "GITLAB_PRIVATE_TOKEN",
    "WEBSITE_HOST_SUFFIX",
    "OSWORLD_FILE_BASE_URL",
    "PROXY_CONFIG_FILE",
    "PARASAIL_API_KEY",
}
_EVALUATOR_ONLY_ENV_PREFIXES = (
    "AWS_",
    "OSWORLD_EVAL_",
    "OSWORLD_USER_SIM_",
)


def build_subprocess_env(
    env_extra: dict | None,
    *,
    strip_keys: tuple[str, ...] = (),
) -> dict:
    """Copy os.environ minus strip_keys, merge env_extra on top."""

    def protected(key: str) -> bool:
        return key in _EVALUATOR_ONLY_ENV_KEYS or key.startswith(
            _EVALUATOR_ONLY_ENV_PREFIXES
        )

    env = {
        k: v for k, v in os.environ.items() if k not in strip_keys and not protected(k)
    }
    if env_extra:
        env.update({k: v for k, v in env_extra.items() if not protected(k)})
    return env


def parse_builder_log(log_path: Path) -> tuple[float, bool, bool]:
    """Read a stream-json log and return (cost, saw_result, saw_rate_limit_blocked)."""
    cost = 0.0
    saw_result = False
    saw_rate_limit_blocked = False
    with open(log_path) as f:
        for line in f:
            try:
                data = json.loads(line.strip())
            except (json.JSONDecodeError, TypeError):
                continue
            t = data.get("type", "")
            if t == "result":
                saw_result = True
                cost = float(data.get("total_cost_usd", 0) or 0)
            elif t == "rate_limit_event":
                info = data.get("rate_limit_info", {})
                if info.get("status") == "blocked":
                    saw_rate_limit_blocked = True
    return cost, saw_result, saw_rate_limit_blocked


def run_builder_subprocess(
    argv: list[str],
    *,
    clone_dir: Path,
    logs_dir: Path,
    log_name: str,
    timeout: int | None,
    env: dict,
    variant_id: str,
    binary_hint: str = "builder binary",
    sandbox_read_paths: list[Path] | None = None,
) -> BuilderResult:
    """Run a builder subprocess with file-based stdout, parse the log, and
    return a BuilderResult.

    ``sandbox_read_paths`` names extra read-only roots the Landlock ruleset
    must grant when the clone is an OSWorld V2 island. An agent whose runtime
    lives outside the clone cannot import itself without them.
    """
    log_path = Path(logs_dir) / f"{log_name}.jsonl"
    os.makedirs(logs_dir, exist_ok=True)

    timeout_label = "none" if timeout is None else f"{timeout}s"
    logger.info(
        "Builder for %s argv[0]=%s timeout=%s",
        variant_id,
        argv[0],
        timeout_label,
    )
    logger.info("  Log: %s", log_path)

    start_time = time.time()
    sandbox_tmp: Path | None = None
    proc: subprocess.Popen | None = None

    # The V2 Base checkout contains gated evaluator logic and sits beside the
    # writable island clone. Cwd alone is not an isolation boundary: enforce a
    # kernel filesystem allowlist before the agent gets autonomous tool use.
    # ScienceBoard_Base likewise holds every official evaluator and answer.
    if Path(clone_dir).name.startswith(("OSWorldV2_Evol_", "ScienceBoard_Evol_")):
        from agent_s4.builder_sandbox import (
            landlock_abi,
            sandboxed_builder_argv,
        )

        if landlock_abi() == 0:
            raise RuntimeError(
                "Landlock is unavailable; refusing an unsandboxed Builder "
                "beside gated benchmark tasks"
            )
        sandbox_tmp = Path(clone_dir) / ".builder_tmp"
        sandbox_tmp.mkdir(mode=0o700, exist_ok=True)
        argv = sandboxed_builder_argv(
            argv,
            clone_dir,
            read_paths=sandbox_read_paths,
        )

    try:
        with open(log_path, "w") as log_f:
            proc = subprocess.Popen(
                argv,
                cwd=str(clone_dir),
                stdout=log_f,
                stderr=subprocess.DEVNULL,
                text=True,
                env=env,
            )
            try:
                if timeout is None:
                    proc.wait()
                else:
                    proc.wait(timeout=timeout)
            except subprocess.TimeoutExpired:
                logger.warning("Builder timed out after %ds, terminating", timeout)
                proc.terminate()
                proc.wait(timeout=30)
    except FileNotFoundError:
        logger.error("%s not found on PATH (argv[0]=%r)", binary_hint, argv[0])
        return BuilderResult(
            exit_code=1,
            cost_usd=0.0,
            log_path=log_path,
            crashed=True,
        )
    finally:
        if sandbox_tmp is not None:
            try:
                if proc is not None and proc.returncode not in {None, 0}:
                    runtime_logs = sandbox_tmp / "prime-home" / "logs"
                    if runtime_logs.is_dir():
                        diagnostic_dir = Path(logs_dir) / f"{log_name}_runtime_logs"
                        if diagnostic_dir.exists():
                            shutil.rmtree(diagnostic_dir)
                        shutil.copytree(runtime_logs, diagnostic_dir)
                        logger.warning(
                            "Preserved failed Builder runtime logs at %s",
                            diagnostic_dir,
                        )
                if sandbox_tmp.is_symlink():
                    sandbox_tmp.unlink()
                elif sandbox_tmp.exists():
                    shutil.rmtree(sandbox_tmp)
            except OSError as exc:
                logger.warning("Could not remove Builder private temp dir: %s", exc)

    elapsed = time.time() - start_time
    cost, saw_result, saw_rate_limit_blocked = parse_builder_log(log_path)

    crashed = False
    if elapsed < _CRASH_ELAPSED_CEILING and not saw_result and cost == 0:
        logger.error(
            "Builder CRASHED for %s (elapsed=%.1fs, no result event, $0 cost)",
            variant_id,
            elapsed,
        )
        crashed = True
    if saw_rate_limit_blocked:
        logger.error("Builder hit BLOCKED rate limit for %s", variant_id)
        crashed = True

    exit_code = proc.returncode if proc.returncode is not None else 1
    logger.info(
        "Builder done for %s (exit=%d, cost=$%.2f, crashed=%s)",
        variant_id,
        exit_code,
        cost,
        crashed,
    )
    return BuilderResult(
        exit_code=exit_code,
        cost_usd=cost,
        log_path=log_path,
        crashed=crashed,
    )

agent-s-computer-use/agent_s4/harness/opencode/README.mdmarkdown · 2.3 KB

OpenCode harness

Builder (builder.py) and in-VM task agent (evaluator.py) built on the OpenCode CLI release binary.

Upstream version


Release	OpenCode `v1.18.34`, `opencode-linux-x64.tar.gz` from `https://github.com/anomalyco/opencode/releases`
Archive SHA-256	`0f22479647226d1d2dd99595d20082ee7bda3870b62dc6a90b41efc1a71d7e9a`
Local cache	`osworld_setup/s4/vm_assets/opencode/1.18.34/opencode`, built by `osworld_setup/s4/scripts/assets/prepare_opencode_runtime.py`

The pins live in source.py (OPENCODE_VERSION, RELEASE_ARCHIVE_SHA256, RELEASE_URL), which also checks the binary's reported version and the cache metadata before the Builder starts or any VM is launched.

What differs from upstream

The binary is used unmodified. It is configured, supervised and extended through OpenCode's own config file and plugin interface. Every setting is read from SHADOW_OPENCODE_* variables only, so a Prime run never picks one up by accident; see osworld_setup/s4/docs/OPENCODE_HARNESS.md for the full list.

config.py writes the opencode.json that the Builder and the task VM share: provider and model, context window and compaction, disabled tools (SHADOW_OPENCODE_DISABLED_TOOLS), and the plugins below.
runner.py supervises the CLI:
- takes file paths, never prompt text, in argv;
- enforces the idle timeout (SHADOW_OPENCODE_IDLE_TIMEOUT) and the step limit (SHADOW_OPENCODE_BUILDER_MAX_STEPS);
- continues a reply cut off at the output limit, up to SHADOW_OPENCODE_LENGTH_CONTINUATIONS times;
- retries a malformed tool call, up to SHADOW_OPENCODE_TOOLCALL_RECOVERIES times;
- cleans up only the CLI's own process group.
hermes_sanitizer.js is a plugin for prompt-emulated (Hermes <tool_call>) endpoints such as the LiteLLM proxy in osworld_setup/s4/scripts/proxy/. It strips malformed tool-call text from history, so the model does not copy it in later replies. Enabled with SHADOW_OPENCODE_HERMES_TOOL_FORMAT=1.
tool_limits.js is a plugin that caps the timeout the model can pass to the bash tool (SHADOW_OPENCODE_MAX_BASH_TIMEOUT_MS). Without the cap, a GUI app launched in the foreground keeps the call open until the supervisor's idle timeout ends the episode.

agent-s-computer-use/agent_s4/harness/opencode/__init__.pycode · 0.0 KB

agent-s-computer-use/agent_s4/harness/opencode/builder.pycode · 5.2 KB

"""OpenCode Builder. Uses the same configuration/supervisor as the VM path."""

from __future__ import annotations

import json
import os
from pathlib import Path
import shutil
import sys

from agent_s4.harness.base import BuilderHarness, BuilderResult
from agent_s4.harness.builder_common import (
    build_subprocess_env,
    run_builder_subprocess,
)
from agent_s4.harness.opencode.config import (
    HERMES_PLUGIN,
    TOOL_LIMITS_PLUGIN,
    build_config,
    load_context_config,
    runner_settings,
)
from agent_s4.harness.opencode.source import resolve_binary
from agent_s4.paths import PACKAGE_ROOT

RUNNER = PACKAGE_ROOT / "harness" / "opencode" / "runner.py"
BUILDER_SYSTEM_PROMPT = """You are the autonomous builder in a shadow-interface evolution run.
Inspect the permitted evidence, make the requested changes in the allowed
workspace, verify them, and finish without asking questions. Use your native
tools to run commands, edit files, and read screenshots. Keep working through
context compaction. Stop applications by their exact PID, not broad process
name patterns. Never terminate the agent, its supervisor, or unrelated runs.

The .builder_tmp directory contains your live runtime, sessions, and
configuration. Never delete, rename, or clean it or its contents. For
disposable validation files, create your own subdirectory under /tmp and clean
only that directory. Leave runtime cleanup to the orchestrator.
"""


class OpenCodeBuilder(BuilderHarness):
    def launch(
        self,
        *,
        clone_dir: Path,
        prompt: str,
        model: str,
        timeout: int | None,
        env_extra: dict[str, str] | None,
        variant_id: str,
        logs_dir: Path,
        log_name: str,
        builder_bin: str | None = None,
    ) -> BuilderResult:
        del builder_bin  # legacy Claude-only argument
        binary = resolve_binary()
        clone_dir = Path(clone_dir).resolve()
        temporary = clone_dir / ".builder_tmp"
        temporary.mkdir(mode=0o700, exist_ok=True)
        runtime = temporary / "opencode"
        # Each mutation is a new session. Only evaluator follow-ups resume.
        if runtime.exists():
            shutil.rmtree(runtime)
        runtime.mkdir(mode=0o700)
        runner = temporary / "opencode_runner.py"
        shutil.copy2(RUNNER, runner)
        source_env = {**os.environ, **(env_extra or {})}
        max_steps = int(source_env.get("SHADOW_OPENCODE_BUILDER_MAX_STEPS", "200"))
        plugin = runtime / HERMES_PLUGIN.name
        shutil.copy2(HERMES_PLUGIN, plugin)
        limits = runtime / TOOL_LIMITS_PLUGIN.name
        shutil.copy2(TOOL_LIMITS_PLUGIN, limits)
        config = build_config(
            model,
            BUILDER_SYSTEM_PROMPT,
            load_context_config(clone_dir / "shadow_interface"),
            max_steps=max_steps,
            env=source_env,
            hermes_plugin_path=plugin,
            tool_limits_plugin_path=limits,
        )
        settings = runner_settings(source_env)
        config_path, settings_path = runtime / "opencode.json", runtime / "runner.json"
        config_path.write_text(json.dumps(config, indent=2) + "\n")
        settings_path.write_text(json.dumps(settings) + "\n")
        task_file = runtime / "instruction.txt"
        task_file.write_text(prompt, encoding="utf-8")
        logs_dir = Path(logs_dir)
        logs_dir.mkdir(parents=True, exist_ok=True)
        # Preserve the effective config outside .builder_tmp's lifecycle.
        (logs_dir / f"{log_name}.opencode-config.json").write_text(
            config_path.read_text()
        )
        (logs_dir / f"{log_name}.opencode-runner.json").write_text(
            settings_path.read_text()
        )
        argv = [
            sys.executable,
            str(runner),
            "--binary",
            str(binary),
            "--config",
            str(config_path),
            "--settings",
            str(settings_path),
            "--task-file",
            str(task_file),
            "--state-dir",
            str(runtime / "state"),
            "--cwd",
            str(clone_dir),
            "--stderr-file",
            str(runtime / "stderr.log"),
            "--max-steps",
            str(max_steps),
        ]
        if timeout is not None:
            if timeout <= 0:
                raise ValueError("OpenCode builder timeout must be positive")
            argv.extend(["--timeout", str(timeout)])
        env = build_subprocess_env(env_extra, strip_keys=("CLAUDECODE",))
        # PARASAIL_API_KEY is evaluator-only in the generic helper; expose it
        # only when it is this Builder's explicitly selected model provider.
        if model.startswith("parasail/") and source_env.get("PARASAIL_API_KEY"):
            env["PARASAIL_API_KEY"] = source_env["PARASAIL_API_KEY"]
        home = runtime / "home"
        home.mkdir()
        env["HOME"] = str(home)
        result = run_builder_subprocess(
            argv,
            clone_dir=clone_dir,
            logs_dir=logs_dir,
            log_name=log_name,
            timeout=None if timeout is None else timeout + 15,
            env=env,
            variant_id=variant_id,
            binary_hint="pinned OpenCode",
            sandbox_read_paths=[binary.parent],
        )
        result.crashed = result.crashed or result.exit_code != 0
        return result

agent-s-computer-use/agent_s4/harness/opencode/config.pycode · 13.5 KB · preview truncated

"""Explicit, shared Builder/Evaluator configuration for pinned OpenCode.

Only SHADOW_OPENCODE_* variables are read: resuming a Prime experiment must
never pick up a different harness's configuration implicitly.
"""

from __future__ import annotations

import json
import math
import os
from copy import deepcopy
import warnings
from pathlib import Path
from urllib.parse import urlparse

DEFAULT_CONTEXT = 131072
DEFAULT_OUTPUT = 32000
COMPACTION_MARGIN = 4096
HERMES_PLUGIN = Path(__file__).resolve().parent / "hermes_sanitizer.js"
TOOL_LIMITS_PLUGIN = Path(__file__).resolve().parent / "tool_limits.js"
# Appended to every system prompt when the endpoint emulates tool calling in
# the prompt (Hermes <tool_call> text). Qwen otherwise drifts into its native
# `<function=NAME>` syntax when choosing among several tools: on the s_009
# opening this took unparseable first replies from 9/20 to 0/30. It states only
# the correct form: an earlier wording that named the forbidden syntax was read
# by the model as an instruction to use it (if-001 s_017).
HERMES_FORMAT_NOTE = (
    "\n\n## Tool-call format\n\nEvery tool call is one JSON object with exactly two keys, "
    '"name" and "arguments", inside <tool_call></tool_call> tags:\n<tool_call>\n'
    '{"name": "bash", "arguments": {"command": "ls /home/user"}}\n</tool_call>\n'
    "Write it as plain JSON: unescaped quotes around keys and string values, and the "
    "arguments object written directly (not as a string)."
)


def positive_int(name: str, value: object) -> int:
    if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
        raise ValueError(f"{name} must be a positive integer")
    return value


def load_context_config(shadow_dir: str | Path | None) -> dict:
    if shadow_dir is None:
        return {}
    path = Path(shadow_dir) / "context_compaction.json"
    if not path.exists():
        return {}
    value = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(value, dict):
        raise ValueError(f"{path} must contain an object")
    return value


def _integer(env: dict, name: str, fallback: int) -> int:
    raw = env.get("SHADOW_OPENCODE_" + name, "").strip()
    return positive_int(name, int(raw) if raw else fallback)


def _sampling(env: dict, name: str, upper: float) -> float | None:
    raw = env.get("SHADOW_OPENCODE_" + name, "").strip()
    if not raw:
        return None
    value = float(raw)
    if not math.isfinite(value) or not 0 <= value <= upper:
        raise ValueError(f"SHADOW_OPENCODE_{name} must be finite and in [0, {upper}]")
    return value


def hermes_tool_format(env: dict | None = None) -> bool:
    env = os.environ if env is None else env
    raw = env.get("SHADOW_OPENCODE_HERMES_TOOL_FORMAT", "").strip()
    if raw not in {"", "0", "1"}:
        raise ValueError("SHADOW_OPENCODE_HERMES_TOOL_FORMAT must be 0 or 1")
    return raw == "1"


# Native tools that may be withheld from the model. Under prompt-emulated tool
# calling, `write` carries whole files in one JSON string and broke its syntax
# on ~9% of calls (bash ~0.7%); without it the model writes files via bash.
DISABLEABLE_TOOLS = {"write", "edit", "todowrite", "webfetch", "skill", "glob", "grep"}


def disabled_tools(env: dict | None = None) -> list[str]:
    env = os.environ if env is None else env
    names = [
        n.strip()
        for n in env.get("SHADOW_OPENCODE_DISABLED_TOOLS", "").split(",")
        if n.strip()
    ]
    unknown = sorted(set(names) - DISABLEABLE_TOOLS)
    if unknown:
        raise ValueError(
            f"SHADOW_OPENCODE_DISABLED_TOOLS: cannot disable {unknown}; "
            f"allowed: {sorted(DISABLEABLE_TOOLS)}"
        )
    # v1.18.34 maps write/edit/patch onto one `edit` permission, last key
    # winning, so disabling either withholds both. Say so in the config.
    if {"write", "edit"} & set(names):
        names += ["write", "edit"]
    return sorted(set(names))


def max_bash_timeout_ms(env: dict | None = None) -> int | None:
    env = os.environ if env is None else env
    raw = env.get("SHADOW_OPENCODE_MAX_BASH_TIMEOUT_MS", "").strip()
    if not raw:
        return None
    value = positive_int("SHADOW_OPENCODE_MAX_BASH_TIMEOUT_MS", int(raw))
    if value > 1_800_000:
        raise ValueError(
            "SHADOW_OPENCODE_MAX_BASH_TIMEOUT_MS must stay below the 1800s idle budget"
        )
    return value


def build_config(
    model: str,
    system_prompt: str,
    context: dict | None = None,
    *,
    max_steps: int | None = None,
    env: dict | None = None,
    hermes_plugin_path: str | Path | None = None,
    tool_limits_plugin_path: str | Path | None = None,
) -> dict:
    """Produce OpenCode v1.18.34's native config, without embedding secrets.

    Unlike Prime's historical policy, enabled=false disables compaction but
    does not discard the declared model context limit.
    """
    env = os.environ if env is None else env
    context = {} if context is None else context
    if not isinstance(context, dict):
        raise ValueError("context compaction config must be an object")
    provider, sep, model_id = model.partition("/")
    if not sep or not model_id or provider not in {"openai", "openrouter", "parasail"}:
        raise ValueError(
            "OpenCode model must be openai/<id>, openrouter/<id>, or parasail/<id>"
        )
    enabled = context.get("enabled", True)
    if not isinstance(enabled, bool):
        raise ValueError("enabled must be a boolean")
    window = _integer(
        env, "CONTEXT_WINDOW", context.get("context_window_tokens", DEFAULT_CONTEXT)
    )
    output = _integer(env, "MAX_OUTPUT_TOKENS", DEFAULT_OUTPUT)
    # An explicit override wins over the interface, as for the other limits.
    # Prime never applied the interface field (its summary budget is
    # 0.8 * reserve); the override is how a run matches that.
    raw_summary = env.get("SHADOW_OPENCODE_SUMMARY_MAX_TOKENS", "").strip()
    summary_output = (
        int(raw_summary) if raw_summary else context.get("summary_max_tokens")
    )
    if summary_output is not None:
        summary_output = positive_int("summary_max_tokens", summary_output)
        if summary_output > output:
            raise ValueError("summary_max_tokens must not exceed MAX_OUTPUT_TOKENS")
    reserve = max(
        _integer(env, "RESERVE_TOKENS", context.get("reserve_tokens", 16384)),
        output + COMPACTION_MARGIN,
    )
    keep = _integer(env, "KEEP_RECENT_TOKENS", context.get("keep_recent_tokens", 20000))
    if output >= window or reserve >= window or (enabled and reserve + keep >= window):
        raise ValueError(
            "context window must exceed output/reserve and, with compaction, reserve + keep"
        )
    unknown = sorted(
        set(context)
        - {
            "enabled",
            "context_window_tokens",
            "reserve_tokens",
            "keep_recent_tokens",
            "summary_instructions",
            "summary_max_tokens",
        }
    )
    if unknown:
        warnings.warn(
            "OpenCode has no equivalent for context fields: " + ", ".join(unknown),
            UserWarning,
            stacklevel=2,
        )
    temperature = _sampling(env, "TEMPERATURE", 2)
    top_p = _sampling(env, "TOP_P", 1)
    sampling = {}
    if temperature is not None:
        sampling["temperature"] = temperature
    if top_p is not None:
        sampling["top_p"] = top_p
    request_timeout = _integer(env, "REQUEST_TIMEOUT", 1800)
    urls = {
        "openrouter": "https://openrouter.ai/api/v1",
        "parasail": "https://api.parasail.io/v1",
    }
    url = (
        env.get("SHADOW_OPENCODE_BASE_URL", "").strip()
        or urls.get(provider)
        or env.get("OPENAI_BASE_URL", "").strip()
    )
    if not url:
        raise ValueError(
            "openai/<id> requires OPENAI_BASE_URL or SHADOW_OPENCODE_BASE_URL"
        )
    parsed = urlparse(url)
    if (
        parsed.scheme not in {"http", "https"}
        or not parsed.netloc
        or parsed.username
        or parsed.password
        or parsed.query
        or parsed.fragment
    ):
        raise ValueError(
            "OpenCode base URL must be HTTP(S), without credentials, query, or fragment"
        )
    # OpenCode's built-in "openai" loader calls the Responses API even when
    # npm is overridden. A custom provider ID is required for Chat Completions.
    runtime_provider = "shadow-openai" if provider == "openai" else provider
    runtime_model = f"{runtime_provider}/{model_id}"
    hermes = hermes_tool_format(env)
    if hermes:
        system_prompt = system_prompt.rstrip() + HERMES_FORMAT_NOTE
    agent = {
        "mode": "primary",
        "model": runtime_model,
        "prompt": system_prompt,
        **sampling,
    }
    if max_steps is not None:
        agent["steps"] = positive_int("max_steps", max_steps)
    withheld = disabled_tools(env)
    if withheld:
        agent["tools"] = {name: False for name in withheld}
    summary = context.get("summary_instructions")
    if summary is not None and not isinstance(summary, str):
        raise ValueError("summary_instructions must be a string")
    compaction_agent = {"model": runtime_model, **sampling}
    if summary:
        compaction_agent["prompt"] = (
            "Summarize the conversation so another agent can continue the task. "
            "Do not execute the task or call tools.\n" + summary
        )
    key_name = {
        "openai": "OPENAI_API_KEY",
        "openrouter": "OPENROUTER_API_KEY",
        "parasail": "PARASAIL_API_KEY",
    }[provider]
    result = {
        "$schema": "https://opencode.ai/config.json",
        "model": runtime_model,
        "small_model": runtime_model,
        "default_agent": "shadow",
        "autoupdate": False,
        "share": "disabled",
        "snapshot": False,
        "lsp": False,
        "formatter": False,
        "enabled_providers": [runtime_provider],
        "provider": {
            runtime_provider: {
                "npm": "@ai-sdk/openai-compatible",
                "name": "Shadow " + provider,
                "options": {
                    "baseURL": url,
                    "apiKey": "{env:" + key_name + "}",
                    "timeout": request_timeout * 1000,
                    "headerTimeout": request_timeout * 1000,
                    "chunkTimeout": request_timeout * 1000,
                },
                "models": {
                    model_id: {
                        "id": model_id,
                        "name": model_id,
                        "temperature": True,
                        "tool_call": True,
                        "attachment": True,
                        "modalities": {"input": ["text", "image"], "output": ["text"]},
                        # v1.18.34 ignores compaction.reserved when limit.input is
                        # absent. Supplying input=context makes usable=context-reserve.
                        "limit": {"context": window, "input": window, "output": output},
                    }
                },
            }
        },
        "compaction": {
            "auto": enabled,
            "prune": False,
            "reserved": reserve,
            "preserve_recent_tokens": keep,
        },
        "agent": {
            "shadow": agent,
            "compaction": compaction_agent,
            "title": {"disable": True},
            "summary": {"disable": True},
        },
        # No interactive permission prompts or unbudgeted subagents in batch runs.
        "permission": {
            "*": "allow",
            "question": "deny",
            "task": "deny",
            "plan_enter": "deny",
            "plan_exit": "deny",
        },
    }
    plugins = []
    if hermes and hermes_plugin_path is not None:
        plugins.append("file://" + str(hermes_plugin_path))
    bash_cap = max_bash_timeout_ms(env)
    if bash_cap is not None and tool_limits_plugin_path is not None:
        plugins

agent-s-computer-use/agent_s4/harness/opencode/evaluator.pycode · 8.3 KB

"""Pinned OpenCode VM harness with shared configuration and supervised sessions."""

from __future__ import annotations

import base64
import json
import logging
import os
from pathlib import Path
import shlex

from agent_s4.harness.opencode.config import (
    HERMES_PLUGIN,
    TOOL_LIMITS_PLUGIN,
    build_config,
    runner_settings,
)
from agent_s4.harness.opencode.source import (
    OPENCODE_VERSION,
    VM_BINARY,
    assets_dir,
    validate_runtime_cache,
)
from agent_s4.paths import PACKAGE_ROOT
from agent_s4.vm.utils import (
    AgentCommand,
    _execute_shell_on_vm,
    _sudo_cmd,
    upload_file_to_vm,
)

logger = logging.getLogger(__name__)
TASK_ROOT = "/tmp/claude_task"
STATE_DIR = f"{TASK_ROOT}/opencode-state"
VM_RUNTIME_TARBALL = "/tmp/opencode_runtime.tar.gz"
RUNNER = PACKAGE_ROOT / "harness" / "opencode" / "runner.py"
VM_HERMES_PLUGIN = f"{TASK_ROOT}/{HERMES_PLUGIN.name}"
VM_TOOL_LIMITS_PLUGIN = f"{TASK_ROOT}/{TOOL_LIMITS_PLUGIN.name}"


def _resolve_runtime_cache() -> Path:
    return assets_dir() / "opencode_runtime.tar.gz"


def install_opencode_on_vm(controller, client_password="password"):
    """Never reuse a random system OpenCode or download at evaluation time."""
    cache = _resolve_runtime_cache()
    try:
        validate_runtime_cache(cache)
    except RuntimeError as exc:
        logger.error("%s", exc)
        return False
    result = _execute_shell_on_vm(controller, f"{VM_BINARY} --version", timeout=30)
    if (
        result
        and result.get("returncode") == 0
        and result.get("output", "").strip() == OPENCODE_VERSION
    ):
        return True
    if not upload_file_to_vm(controller, cache, VM_RUNTIME_TARBALL):
        logger.error("OpenCode runtime upload to %s failed", VM_RUNTIME_TARBALL)
        return False
    result = _execute_shell_on_vm(
        controller,
        _sudo_cmd(f"tar xzf {VM_RUNTIME_TARBALL} -C /", client_password)
        + f" && rm -f {VM_RUNTIME_TARBALL} && {VM_BINARY} --version",
        timeout=120,
    )
    if (
        result
        and result.get("returncode") == 0
        and result.get("output", "").strip() == OPENCODE_VERSION
    ):
        return True
    logger.error("OpenCode runtime extraction/verification failed: %s", result)
    return False


def _write(path: str, value: bytes) -> str:
    data = base64.b64encode(value).decode("ascii")
    return f"printf %s {shlex.quote(data)} | base64 -d > {shlex.quote(path)}"


def vm_config_env(model: str) -> dict[str, str]:
    """Select the VM-reachable deployment without changing the host Builder.

    SHADOW_EVOL_VM_OPENAI_BASE_URL is the existing shared deployment contract,
    not a Prime sampling/context setting. AWS VMs may need a VPC address while
    the orchestrator uses the same service's public address.
    """
    env = dict(os.environ)
    base = env.get("SHADOW_OPENCODE_VM_BASE_URL", "").strip()
    if (
        not base
        and model.startswith("openai/")
        and not env.get("SHADOW_OPENCODE_BASE_URL", "").strip()
    ):
        base = env.get("SHADOW_EVOL_VM_OPENAI_BASE_URL", "").strip()
    if base:
        env["SHADOW_OPENCODE_BASE_URL"] = base
    return env


def build_opencode_vm_command(
    instruction,
    system_prompt,
    model,
    timeout,
    max_turns=200,
    api_key=None,
    parasail_api_key=None,
    *,
    max_steps=None,
    no_time_limit=False,
    continue_session=False,
    context_compaction=None,
):
    config = build_config(
        model,
        system_prompt,
        context_compaction,
        max_steps=max_steps if max_steps is not None else max_turns,
        env=vm_config_env(model),
        hermes_plugin_path=VM_HERMES_PLUGIN,
        tool_limits_plugin_path=VM_TOOL_LIMITS_PLUGIN,
    )
    settings = runner_settings()
    secret_env = {}
    key = {
        "openai": "OPENAI_API_KEY",
        "openrouter": "OPENROUTER_API_KEY",
        "parasail": "PARASAIL_API_KEY",
    }[model.split("/", 1)[0]]
    value = {"OPENROUTER_API_KEY": api_key, "PARASAIL_API_KEY": parasail_api_key}.get(
        key
    ) or os.environ.get(key)
    if value:
        secret_env[key] = value
    steps = max_steps if max_steps is not None else max_turns
    argv = [
        "python3",
        f"{TASK_ROOT}/opencode_runner.py",
        "--binary",
        VM_BINARY,
        "--config",
        f"{TASK_ROOT}/opencode.json",
        "--settings",
        f"{TASK_ROOT}/runner.json",
        "--task-file",
        f"{TASK_ROOT}/instruction.txt",
        "--state-dir",
        STATE_DIR,
        "--cwd",
        "/home/user",
        "--stderr-file",
        f"{TASK_ROOT}/stderr.log",
        "--exit-file",
        f"{TASK_ROOT}/exit_code.txt",
    ]
    if steps is not None:
        argv.extend(["--max-steps", str(steps)])
    if not no_time_limit:
        if timeout is None or timeout <= 0:
            raise ValueError("timeout must be positive unless no_time_limit is set")
        argv.extend(["--timeout", str(timeout)])
    if continue_session:
        argv.append("--resume")
    setup = [
        "umask 077",
        "if [ -f /tmp/.shadow-agent-env ]; then . /tmp/.shadow-agent-env; rm -f /tmp/.shadow-agent-env; fi",
        "export DISPLAY=:0 HOME=/home/user",
        f"export SHADOW_GROUNDING_MODEL={shlex.quote(model)}",
        f"mkdir -p {TASK_ROOT}",
    ]
    if not continue_session:
        setup.append(f"rm -rf {STATE_DIR}")
    setup.extend(
        [
            f'trap \'status=$?; printf "%s\\n" "$status" > {TASK_ROOT}/exit_code.txt\' EXIT',
            _write(f"{TASK_ROOT}/instruction.txt", instruction.encode()),
            _write(f"{TASK_ROOT}/opencode.json", json.dumps(config).encode()),
            _write(f"{TASK_ROOT}/runner.json", json.dumps(settings).encode()),
            _write(f"{TASK_ROOT}/opencode_runner.py", RUNNER.read_bytes()),
            _write(VM_HERMES_PLUGIN, HERMES_PLUGIN.read_bytes()),
            _write(VM_TOOL_LIMITS_PLUGIN, TOOL_LIMITS_PLUGIN.read_bytes()),
            shlex.join(argv)
            + f" > {TASK_ROOT}/trajectory.jsonl 2> {TASK_ROOT}/stderr.log",
        ]
    )
    return AgentCommand(
        script=" && ".join(setup),
        secret_env=secret_env,
        accepted_exit_codes=(0, 124),
        activity_path=f"{TASK_ROOT}/trajectory.jsonl",
        idle_timeout=settings["idle_timeout"] + 60,
        process_pattern="[o]pencode_runner.py",
        runner_exit_grace=30,
    )


def validate_opencode_artifacts(artifacts: dict) -> str | None:
    raw = artifacts.get("claude_trajectory") or b""
    if isinstance(raw, bytes):
        raw = raw.decode("utf-8", errors="replace")
    records = []
    for line in raw.split("\n"):
        if not line.strip():
            continue
        try:
            event = json.loads(line)
        except ValueError:
            return "OpenCode emitted malformed JSONL"
        if not isinstance(event, dict):
            return "OpenCode emitted a non-object event"
        if event.get("type") == "error":
            return "OpenCode reported a provider/session error"
        records.append(event)
    if not records:
        return "OpenCode produced no trajectory"
    terminal = records[-1]
    if terminal.get("type") != "result" or terminal.get("is_error") is not False:
        return "OpenCode did not produce a successful terminal result"
    if terminal.get("subtype") not in {
        "completed",
        "step_limit",
        "wall_timeout",
        "length_limit",
    }:
        return "OpenCode ended without completion or an explicit task budget limit"
    if not terminal.get("session_id") or not terminal.get("steps", 0):
        return "OpenCode produced no completed model steps"
    exit_code = artifacts.get("exit_code", b"")
    if isinstance(exit_code, bytes):
        exit_code = exit_code.decode()
    expected = 124 if terminal["subtype"] == "wall_timeout" else 0
    if str(exit_code).strip() != str(expected) or terminal.get("exit_code") != expected:
        return "OpenCode exit marker disagrees with terminal result"
    return None


MODEL_CONFIG = {
    "opencode": {
        "binary": "opencode",
        "model_flag": None,
        "installer": install_opencode_on_vm,
        # Some V2 tasks (048) revoke sudo during setup; install from the clean
        # snapshot first, as Prime does. The post-setup call is then a no-op probe.
        "install_before_task_setup": True,
        "build_command": build_opencode_vm_command,
        "validate_artifacts": validate_opencode_artifacts,
        "accepted_exit_codes": (0, 124),
    }
}

agent-s-computer-use/agent_s4/harness/opencode/hermes_sanitizer.jscode · 1.4 KB

// OpenCode plugin for prompt-emulated (Hermes) tool calling.
//
// When a reply mixes a parseable tool call with a malformed one, the
// tool-calling proxy keeps the malformed block as visible assistant text.
// Replayed as history, that text is imitated: measured on the s_009 opening,
// a single leaked block raised the unparseable-reply rate from 0/30 to 28/30,
// which exhausts the proxy's retries and ends the episode with HTTP 422.
//
// This hook strips such blocks from the copy of the history sent to the model
// (normal turns and compaction summaries). Stored session parts, and so the
// recorded trajectory, keep the original text.
const BLOCK = /<tool_call>[\s\S]*?(?:<\/tool_call>|$)/g;
const XML = /<function=[\s\S]*?(?:<\/function>|$)/g;

export const ShadowHermesSanitizer = async () => ({
  "experimental.chat.messages.transform": async (_input, output) => {
    let cleaned = 0;
    for (const message of output.messages ?? []) {
      if (message?.info?.role !== "assistant") continue;
      for (const part of message.parts ?? []) {
        if (part?.type !== "text" || typeof part.text !== "string") continue;
        if (!part.text.includes("<tool_call") && !part.text.includes("<function=")) continue;
        part.text = part.text.replace(BLOCK, "").replace(XML, "").trim();
        cleaned += 1;
      }
    }
    if (cleaned) console.error(`[shadow-hermes-sanitizer] cleaned ${cleaned} text part(s)`);
  },
});

agent-s-computer-use/agent_s4/harness/opencode/runner.pycode · 15.9 KB · preview truncated

"""Standalone OpenCode supervisor, staged into Builder clones and task VMs.

The runner and CLI take file paths, never task/prompt text, in argv. Process
cleanup targets only the CLI's new process group, never pkill -f or killall.
This is accidental-termination protection, not a sandbox against malicious
tools running as the same Unix user.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import os
from pathlib import Path
import selectors
import signal
import subprocess
import sys
import time

EXPECTED_VERSION = "1.18.34"
TOOLCALL_PROMPT = (
    "Your previous reply could not be executed: its tool call was not valid JSON, "
    "so nothing ran. Continue the task in the same session and environment. Call "
    "one tool, written as plain JSON, for example\n<tool_call>\n"
    '{"name": "bash", "arguments": {"command": "ls /home/user"}}\n</tool_call>\n'
    "Write long file contents in several smaller steps instead of one large argument."
)
# The prompt-emulated tool-calling proxy's give-up message (HTTP 422).
TOOLCALL_ERROR = "No valid tool call"
LENGTH_PROMPT = (
    "Your last reply reached the output-token limit before finishing the task. "
    "Continue in the same session and environment. Keep reasoning short, act "
    "using a tool, and verify the result. Do not repeat your previous analysis."
)


def emit(kind: str, **fields) -> None:
    print(json.dumps({"type": kind, **fields}, ensure_ascii=False), flush=True)


def atomic_json(path: Path, value: dict) -> None:
    tmp = path.with_suffix(path.suffix + ".tmp")
    tmp.write_text(json.dumps(value) + "\n", encoding="utf-8")
    tmp.replace(path)


def isolated_env(config: dict, state_dir: Path, inherited: dict | None = None) -> dict:
    env = dict(os.environ if inherited is None else inherited)
    # Stale user flags must not disable compaction or replace the selected model.
    env = {k: v for k, v in env.items() if not k.startswith("OPENCODE_")}
    for name in ("config", "data", "state", "cache"):
        directory = state_dir / name
        directory.mkdir(parents=True, exist_ok=True)
        env[f"XDG_{name.upper()}_HOME"] = str(directory)
    provider = config["enabled_providers"][0]
    output = next(iter(config["provider"][provider]["models"].values()))["limit"][
        "output"
    ]
    env.update(
        {
            "OPENCODE_CONFIG_CONTENT": json.dumps(config),
            "OPENCODE_DISABLE_PROJECT_CONFIG": "true",
            "OPENCODE_DISABLE_CLAUDE_CODE": "true",
            "OPENCODE_DISABLE_EXTERNAL_SKILLS": "true",
            "OPENCODE_DISABLE_DEFAULT_PLUGINS": "true",
            "OPENCODE_DISABLE_MODELS_FETCH": "true",
            "OPENCODE_DISABLE_LSP_DOWNLOAD": "true",
            "OPENCODE_DISABLE_AUTOUPDATE": "true",
            "OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX": str(output),
        }
    )
    return env


def stop_group(proc: subprocess.Popen, grace: float = 2) -> None:
    """Reap the CLI and any surviving tools, including after a normal exit."""
    try:
        os.killpg(proc.pid, signal.SIGTERM)
    except ProcessLookupError:
        proc.wait()
        return
    deadline = time.monotonic() + grace
    while time.monotonic() < deadline:
        proc.poll()
        try:
            os.killpg(proc.pid, 0)
        except ProcessLookupError:
            proc.wait()
            return
        time.sleep(0.02)
    try:
        os.killpg(proc.pid, signal.SIGKILL)
    except ProcessLookupError:
        pass
    proc.wait()


def run(args: argparse.Namespace) -> int:
    state = args.state_dir.resolve()
    state.mkdir(mode=0o700, parents=True, exist_ok=True)
    config = json.loads(args.config.read_text(encoding="utf-8"))
    settings = json.loads(args.settings.read_text()) if args.settings else {}
    idle_timeout = settings.get("idle_timeout", 1920)
    max_continuations = settings.get("length_continuations", 0)
    max_recoveries = settings.get("toolcall_recoveries", 0)
    recoveries = 0
    env = isolated_env(config, state)
    # `opencode run` resolves its workspace from PWD before process.cwd().
    # Popen(cwd=...) alone leaves the parent's PWD in the environment.
    env["PWD"] = str(args.cwd.resolve())
    check = subprocess.run(
        [args.binary, "--version"], env=env, capture_output=True, text=True, timeout=30
    )
    if check.returncode or check.stdout.strip() != EXPECTED_VERSION:
        raise ValueError(f"OpenCode version must be {EXPECTED_VERSION}")
    session_path = state / "session.json"
    signature = hashlib.sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()
    session = None
    steps = 0
    if args.resume:
        record = json.loads(session_path.read_text())
        if record.get("config_sha256") != signature:
            raise ValueError(
                "refusing to resume an OpenCode session with changed configuration"
            )
        session = record["session_id"]
        steps = record["steps"]
    elif session_path.exists():
        raise ValueError(
            "OpenCode state already contains a session; use --resume or a fresh state directory"
        )
    total_cost = 0.0
    continuations = 0
    interrupted = []
    old_handlers = {}
    for signum in (signal.SIGTERM, signal.SIGINT):
        old_handlers[signum] = signal.signal(
            signum, lambda sig, frame: interrupted.append(sig)
        )
    started = time.monotonic()
    task = args.task_file.read_bytes()
    # A file rather than a pipe avoids deadlocks for large prompts and keeps
    # the task out of every parent/child argv, including the Python supervisor.
    input_path = state / "stdin.txt"
    reason = "incomplete"
    code = 1
    try:
        while True:
            if args.max_steps is not None and steps >= args.max_steps:
                reason, code = "step_limit", 0
                break
            input_path.write_bytes(task)
            command = [
                args.binary,
                "run",
                "--format",
                "json",
                "--model",
                config["model"],
                "--agent",
                "shadow",
                "--dir",
                str(args.cwd.resolve()),
            ]
            if session:
                command.extend(["--session", session])
            last_finish = None
            saw_error = False
            toolcall_error = False
            malformed = False
            seen_parts = set()
            last_activity = time.monotonic()
            buffer = b""
            stop_reason = None
            with input_path.open("rb") as stdin, args.stderr_file.open("ab") as stderr:
                proc = subprocess.Popen(
                    command,
                    cwd=args.cwd,
                    env=env,
                    stdin=stdin,
                    stdout=subprocess.PIPE,
                    stderr=stderr,
                    start_new_session=True,
                )
                selector = selectors.DefaultSelector()
                selector.register(proc.stdout, selectors.EVENT_READ)

                def consume(line: bytes) -> None:
                    nonlocal session, steps, total_cost, last_finish, saw_error, malformed, toolcall_error
                    if not line.strip():
                        return
                    try:
                        event = json.loads(line)
                        if not isinstance(event, dict):
                            raise ValueError("event must be an object")
                    except (ValueError, UnicodeDecodeError):
                        malformed = True
                        emit("opencode_protocol_error", message="malformed JSONL event")
                        return
                    if (
                        event.get("type") == "error"
                        and recoveries < max_recoveries
                        and TOOLCALL_ERROR in json.dumps(event.get("error"))
                    ):
                        # Recoverable: recorded under its own type so the
                        # trajectory validator does not treat it as fatal.
                        toolcall_error = True
                        emit(
                            "opencode_toolcall_error",
                            sessionID=event.get("sessionID"),
                            error=event.get("error"),
                        )
                        return
                    print(line.decode("utf-8"), flush=True)
                    event_session = event.get("sessionID")
                    if event_session:
                        if session and session != event_session:
                            malformed = True
                            return
                        session = event_session
                    if event.get("type") == "error":
                        saw_error = True
                    if event.get("type") == "step_finish":
                        part = event.get("part", {})
                        identifier = part.get("id")
                        if identifier is None or identifier not in seen_parts:
                            seen_parts.add(identifier)
                            steps += 1
                            total_cost += float(part.get("cost", 0) or 0)
                        last_finish = part.get("reason")
                    if session:
                        atomic_json(
                            session_path,
                            {
                                "session_id": session,
                                "steps": steps,
                                "config_sha256": signature,
                            },
                        )

                try:
                    while True:
                        now = time.monotonic()
                        if interrupted:
                            stop_reason = "signal"
                        elif args.timeout is not None and now - started >= args.timeout:
                            stop_reason = "wall_timeout"
                        elif now - last_activity >= idle_timeout:
                            stop_reason = "idle_timeout"
                        elif toolcall_error:
                            stop_reason = "toolcall_error"
                        elif malformed or saw_error:
                            stop_reason = (
                                "protocol_error" if malformed else "provider_error"
                            )
                        elif args.max_steps is not None and steps >= args.max_steps:
                            stop_reason = "step_limit"
                        if stop_reason:
                            break
                        ready = selector.select(timeout=0.1)
                        if ready:
                            chunk = os.read(proc.stdout.fileno(), 65536)
                            if not chunk:
                                break
                            last_activity = now
                            buffer += chunk
                            while b"\n" in buffer:
                                line, buffer = buffer.split(b"\n", 1)
                                consume(line)
                        elif proc.poll() is not None:
                            # Tools may inherit stdout after the CLI dies. Do
                            # not wait for those tools to close the pipe.
                            break
                    if buffer.strip() and stop_reason is None:
                        consume(buffer)
                    returncode = proc.poll()
                    if returncode is None and stop_reason is None:
                        try:
                            returncode = proc.wait(timeout=2)
                        except subprocess.TimeoutExpired:
                            stop_reason = "incomplete"
                finally:
                    selector.close()
                    stop_group(proc)
                    proc.stdout.close()

agent-s-computer-use/agent_s4/harness/opencode/source.pycode · 2.1 KB

"""Pin and verify OpenCode separately from the running Prime installation."""

from __future__ import annotations

import hashlib
import json
import os
from pathlib import Path
import shutil
import subprocess

from agent_s4.paths import SHARED_ROOT

OPENCODE_VERSION = "1.18.34"
RELEASE_ARCHIVE_SHA256 = (
    "0f22479647226d1d2dd99595d20082ee7bda3870b62dc6a90b41efc1a71d7e9a"
)
RELEASE_URL = f"https://github.com/anomalyco/opencode/releases/download/v{OPENCODE_VERSION}/opencode-linux-x64.tar.gz"
VM_BINARY = f"/opt/shadow-opencode/{OPENCODE_VERSION}/opencode"


def assets_dir() -> Path:
    return Path(os.environ.get("SHADOW_VM_ASSETS_DIR") or SHARED_ROOT / "vm_assets")


def resolve_binary() -> Path:
    explicit = os.environ.get("SHADOW_OPENCODE_BIN")
    local = assets_dir() / "opencode" / OPENCODE_VERSION / "opencode"
    name = explicit or (str(local) if local.is_file() else shutil.which("opencode"))
    if not name:
        raise RuntimeError(
            "Run scripts/assets/prepare_opencode_runtime.py or set SHADOW_OPENCODE_BIN"
        )
    binary = Path(name).expanduser().resolve()
    result = subprocess.run(
        [str(binary), "--version"], capture_output=True, text=True, timeout=30
    )
    if result.returncode or result.stdout.strip() != OPENCODE_VERSION:
        raise RuntimeError(
            f"OpenCode must be pinned to {OPENCODE_VERSION}; got {result.stdout.strip()!r}"
        )
    return binary


def validate_runtime_cache(path: Path) -> dict:
    try:
        metadata = json.loads(
            path.with_suffix(path.suffix + ".metadata.json").read_text()
        )
        with path.open("rb") as stream:
            digest = hashlib.file_digest(stream, "sha256").hexdigest()
    except (OSError, ValueError) as exc:
        raise RuntimeError(
            "OpenCode runtime cache missing or invalid; run scripts/assets/prepare_opencode_runtime.py"
        ) from exc
    if (
        metadata.get("opencode_version") != OPENCODE_VERSION
        or metadata.get("sha256") != digest
        or metadata.get("binary") != VM_BINARY
    ):
        raise RuntimeError("OpenCode cache version, layout or SHA256 mismatch")
    return metadata

agent-s-computer-use/agent_s4/harness/opencode/tool_limits.jscode · 1.0 KB

// OpenCode plugin: cap the bash tool's model-chosen `timeout`.
//
// The pinned CLI honours any timeout the model passes and emits nothing while
// a tool runs. A GUI launched with its stdout still attached then holds the
// call open for that long, and the supervisor's inactivity watchdog ends the
// whole episode (if-002 s_007 and 014: idle_timeout). With a cap, the CLI
// terminates the command at the cap and the agent continues.
export const ShadowToolLimits = async (_input, options = {}) => {
  const cap = Number(options.maxBashTimeoutMs);
  if (!Number.isFinite(cap) || cap <= 0) return {};
  return {
    "tool.execute.before": async (input, output) => {
      if (input?.tool !== "bash" || !output?.args) return;
      const requested = Number(output.args.timeout);
      if (Number.isFinite(requested) && requested > cap) {
        output.args.timeout = cap;
        console.error(`[shadow-tool-limits] bash timeout ${requested}ms capped to ${cap}ms`);
      }
    },
  };
};

agent-s-computer-use/agent_s4/harness/prime/README.mdmarkdown · 2.3 KB

Prime Agent harness

Builder (builder.py) and in-VM task agent (evaluator.py) built on the official Prime Agent CLI.

Upstream version


Project	PrimeIntellect-ai/prime-agent
Commit	`a9b5d88b50db50860088772a408e867f8071b507`
CLI package	`prime-agent-0.8.0-beta.546.1.a9b5d88.tgz` (SHA-256 `830d2b10656a7e04bece4b0d895b66fa0aa86e04bfc2088d03b00c27ef015831`)
Node.js	22.23.2 (official `linux-x64` build)
Python runtime	vendored from the same commit in `osworld_setup/s4/agent_harness/prime-agent-runtime/` (see its `PROVENANCE.md`)

The pins live in source.py (PINNED_PRIME_AGENT_COMMIT, PRIME_AGENT_VERSION, PRIME_AGENT_ARTIFACT_SHA256, NODE_VERSION), which also checks them before any VM is launched: the vendored runtime's commit marker and per-file digests, the CLI tarball hash, and the offline VM runtime built by osworld_setup/s4/scripts/assets/cache_prime_agent_assets.py.

What differs from upstream

The CLI and the vendored Python runtime are used unmodified on the host. The agent loop, sessions, tools and automatic context compaction stay Prime's own. Everything below is added around it, or applied to the VM copy only.

Added around the CLI:

runner.py relays prime-agent --mode json output unchanged and appends the terminal result record the shared Builder supervisor expects.
step_runner.py does the same in the VM and enforces a model-step ceiling.
compaction.py and context_window.py translate an interface's context_compaction.json into the settings Prime reads.
openrouter.py holds provider/model configuration.
temperature.py is an opt-in before_provider_request extension (SHADOW_PRIME_AGENT_TEMPERATURE). Prime itself sends no temperature.

Patched into the installed runtime on each task VM (evaluator.py applies both at install time):

overflow_patch.py makes Prime's overflow detector recognise the vLLM "decoder prompt ... is longer than the maximum model length" error, so that error triggers Prime's own compact-and-retry instead of a generic retry.
ipython_timeout_patch.py adds a retriable 60-second limit to each IPython cell. The overlong cell is interrupted and the model is told to retry with smaller work; the episode continues.

agent-s-computer-use/agent_s4/harness/prime/__init__.pycode · 0.0 KB

agent-s-computer-use/agent_s4/harness/prime/builder.pycode · 12.9 KB · preview truncated

"""Host-side builder harness backed by the official Prime Agent CLI."""

from __future__ import annotations

import json
import os
import shutil
import subprocess
import sys
from pathlib import Path

from agent_s4.harness.base import BuilderHarness, BuilderResult
from agent_s4.harness.builder_common import (
    build_subprocess_env,
    parse_builder_log,
    run_builder_subprocess,
)
from agent_s4.paths import PACKAGE_ROOT
from agent_s4.harness.prime.source import PRIME_AGENT_VERSION
from agent_s4.harness.prime.context_window import interface_window_is_honoured
from agent_s4.harness.prime.temperature import (
    EXTENSION_FILENAME,
    sampling_temperature,
    temperature_extension_source,
)
from agent_s4.harness.prime.compaction import (
    compaction_settings,
    context_window_from_config,
    load_compaction_config,
)
from agent_s4.harness.prime.openrouter import (
    DEFAULT_MAX_OUTPUT_TOKENS,
    OPENAI_COMPATIBLE_PROVIDER,
    openai_compatible_models_config,
    openrouter_routing,
)

RUNNER = PACKAGE_ROOT / "harness" / "prime" / "runner.py"

BUILDER_SYSTEM_PROMPT = """\
You are the autonomous builder inside a shadow-interface evolution run.
Follow the task exactly, inspect the available evaluation evidence, make the
requested changes directly in the allowed files, verify them, and finish
without asking the user questions.

Your only model-facing tool is a persistent IPython kernel. Run shell commands
from IPython subprocess APIs or a %%bash cell. To inspect the GUI visually,
first use a subprocess to run the shadow interface screenshot command, for
example:

    subprocess.run(["python3", "-m", "shadow_interface", "call",
                    "gui.take_screenshot", "/tmp/screenshot.png"], check=True)

Then, in a subsequent IPython cell, load the image into model context with:

    print(await attach_image("/tmp/screenshot.png"))

Do not Base64-encode screenshots or infer their visual contents from file
metadata. Prime Agent owns the agent loop, session history, and configured
context compaction; continue working through compaction when it occurs.

The .builder_tmp directory contains your live runtime, daemon registry,
sessions, and configuration. Never delete, rename, or clean it or its contents.
For disposable validation files, create your own subdirectory under /tmp and
clean only that directory. Leave runtime cleanup to the orchestrator.
"""


def resolve_prime_agent_binary() -> Path:
    """Resolve the explicitly pinned binary or ``prime-agent`` on PATH."""
    configured = os.environ.get("SHADOW_PRIME_AGENT_BIN")
    if configured:
        path = Path(configured).expanduser()
        if not path.is_file():
            raise RuntimeError(f"SHADOW_PRIME_AGENT_BIN does not name a file: {path}")
        resolved = path.resolve()
        validate_prime_agent_binary(resolved)
        return resolved
    resolved = shutil.which("prime-agent")
    if resolved is None:
        raise RuntimeError(
            "prime-agent is not installed or is not on PATH; install the "
            "official CLI or set SHADOW_PRIME_AGENT_BIN"
        )
    resolved_path = Path(resolved).resolve()
    validate_prime_agent_binary(resolved_path)
    return resolved_path


def validate_prime_agent_binary(binary: Path) -> None:
    """Require the host builder CLI to match the vendored/runtime pin."""
    try:
        version = subprocess.check_output(
            [str(binary), "--version"],
            text=True,
            stderr=subprocess.STDOUT,
            timeout=30,
        ).strip()
    except (OSError, subprocess.SubprocessError) as exc:
        raise RuntimeError(
            f"could not verify Prime Agent binary {binary}: {exc}"
        ) from exc
    if version != PRIME_AGENT_VERSION:
        raise RuntimeError(
            "Prime Agent host binary does not match the vendored integration: "
            f"expected {PRIME_AGENT_VERSION}, found {version or '<empty>'}"
        )


def prime_agent_runtime_root(binary: Path) -> Path:
    """Return the CLI package root that Landlock must make readable.

    Release/source JS launchers commonly resolve to
    ``<package>/dist/bundle/cli.js``. Granting only ``cli.js``'s directory
    hides sibling bundled skills and ``prime-agent-runtime``. A standalone
    native release has no ``dist`` ancestor and needs only its containing
    directory.
    """
    resolved = Path(binary).resolve()
    for parent in resolved.parents:
        if parent.name == "dist":
            return parent.parent
    # The pinned offline runtime uses <package>/bin/prime-agent as a shell
    # launcher, which execs sibling <package>/node/bin/node and
    # <package>/node_modules/.bin/prime-agent.  Granting only bin/ lets the
    # launcher start under Landlock but makes its exec fail with EACCES (126).
    if resolved.parent.name == "bin":
        package = resolved.parent.parent
        if (package / "node_modules").is_dir() and (package / "node").is_dir():
            return package
    return resolved.parent


def bundled_kernel_python(runtime_root: Path) -> Path | None:
    """Return Prime's pre-provisioned kernel Python when the runtime has one."""
    candidate = Path(runtime_root) / "python" / "bin" / "python3.11"
    return candidate.resolve() if candidate.is_file() else None


def context_window_tokens(clone_dir: Path) -> int | None:
    """Read the interface's explicit Prime context-window override.

    A missing file, a disabled policy, or an absent field all mean "no
    override", so Prime keeps the window its own model catalog advertises.
    Only a present-but-invalid value is an error, because that is a mutation
    that wrote something it should not have. A run pinned to Prime's catalog
    window ignores the field entirely; see
    agent_s4.harness.prime.context_window.
    """
    if not interface_window_is_honoured():
        return None
    try:
        return context_window_from_config(
            load_compaction_config(Path(clone_dir) / "shadow_interface")
        )
    except ValueError as exc:
        raise RuntimeError(str(exc)) from exc


class PrimeAgentBuilder(BuilderHarness):
    """Run Prime Agent headlessly while retaining its native session engine."""

    def launch(
        self,
        *,
        clone_dir: Path,
        prompt: str,
        model: str,
        timeout: int | None,
        env_extra: dict[str, str] | None,
        variant_id: str,
        logs_dir: Path,
        log_name: str,
        builder_bin: str | None = None,
    ) -> BuilderResult:
        # builder_bin remains the legacy Claude-only pin.
        del builder_bin
        if timeout is not None and timeout < 90:
            raise ValueError("prime-agent builder timeout must be at least 90 seconds")
        prime_binary = resolve_prime_agent_binary()
        runtime_root = prime_agent_runtime_root(prime_binary)
        kernel_python = bundled_kernel_python(runtime_root)
        clone_dir = Path(clone_dir)
        sandbox_tmp = clone_dir / ".builder_tmp"
        sandbox_tmp.mkdir(mode=0o700, exist_ok=True)
        runner = sandbox_tmp / "prime_agent_runner.py"
        shutil.copy2(RUNNER, runner)
        session_dir = sandbox_tmp / "prime-sessions"
        agent_dir = sandbox_tmp / "prime-home"
        agent_dir.mkdir(mode=0o700, parents=True, exist_ok=True)
        context_compaction = load_compaction_config(clone_dir / "shadow_interface")
        model_context_window = context_window_from_config(context_compaction)
        max_output_tokens = None
        # The Builder writes the interface that the evaluator will run under,
        # so both must reach the same deployment through the same declared
        # capabilities. Prime knows nothing about a self-hosted
        # OpenAI-compatible endpoint, so declare it here exactly as the
        # evaluator does.
        prime_model = model
        openai_base_url = os.environ.get("OPENAI_BASE_URL", "").strip()
        # Without a base URL this is a real OpenAI model and Prime's own
        # provider handles it; only a self-hosted deployment needs declaring.
        if model.startswith("openai/") and openai_base_url:
            model_id = model.removeprefix("openai/")
            prime_model = f"{OPENAI_COMPATIBLE_PROVIDER}/{model_id}"
            models_config = openai_compatible_models_config(
                model_id,
                openai_base_url,
                model_context_window,
            )
            declared_model = json.loads(models_config)["providers"][
                OPENAI_COMPATIBLE_PROVIDER
            ]["models"][0]
            model_context_window = declared_model["contextWindow"]
            max_output_tokens = min(
                declared_model["maxTokens"], DEFAULT_MAX_OUTPUT_TOKENS
            )
            (agent_dir / "models.json").write_text(
                models_config + "\n",
                encoding="utf-8",
            )
        elif model.startswith("openrouter/"):
            max_output_tokens = DEFAULT_MAX_OUTPUT_TOKENS
            provider_config = {
                "compat": {
                    "openRouterRouting": openrouter_routing(),
                },
            }
            # The Builder writes the interface that the evaluator will run
            # under, so it has to reason inside the same window.
            if model_context_window is not None:
                provider_config["modelOverrides"] = {
                    model.removeprefix("openrouter/"): {
                        "contextWindow": model_context_window,
                    },
                }
            (agent_dir / "models.json").write_text(
                json.dumps(
                    {
                        "providers": {
                            "openrouter": provider_config,
                        },
                    },
                    indent=2,
                )
                + "\n",
                encoding="utf-8",
            )
        (agent_dir / "settings.json").write_text(
            json.dumps(
                {
                    "compaction": compaction_settings(
                        context_compaction,
                        context_window=model_context_window,
                        max_output_tokens=max_output_tokens,
                    ),
                    "autoRefine": {"enabled": False},
                    "telemetry": {"enabled": False},
                },
                indent=2,
            )
            + "\n",
            encoding="utf-8",
        )
        argv = [
            sys.executable,
            str(runner),
            "--prime-bin",
            str(prime_binary),
            "--task",
            prompt,
            "--system-prompt",
            BUILDER_SYSTEM_PROMPT,
            "--model",
            prime_model,
            "--cwd",
            str(clone_dir),
            "--session-dir",
            str(session_dir),
        ]
        temperature = sampling_temperature()
        if temperature is not None:
            extension = agent_dir / EXTENSION_FILENAME
            extension.write_text(
                temperature_extension_source(temperature),
                encoding="utf-8",
            )
            argv.extend(("--extension", str(extension)))
        env = build_subprocess_env(env_extra, strip_keys=("CLAUDECODE",))
        # Isolate all Prime configuration and session writes inside the clone.
        # Set both current and legacy upstream names for version compatibility.
        env.update(
            {
                "HOME": str(agent_dir),
                "PRIME_AGENT_CODING_AGENT_DIR": str(agent_dir),
                "PI_CODING_AGENT_DIR": str(agent_dir),
                "PRIME_AGENT_SESSION_DIR": str(session_dir),
                "PI_SESSION_DIR": str(session_dir),
            }
        )
        if kernel_python is not None:
            # The pinned offline package already carries ipykernel, rlm
            # (prime-agent-runtime), and the default analysis packages. Using
            #

agent-s-computer-use/agent_s4/harness/prime/compaction.pycode · 4.6 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/context_window.pycode · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/evaluator.pycode · 21.4 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/ipython_timeout_patch.pycode · 11.5 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/openrouter.pycode · 4.6 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/overflow_patch.pycode · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/runner.pycode · 8.6 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/source.pycode · 13.2 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/step_runner.pycode · 8.9 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/prime/temperature.pycode · 1.5 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/harness/ubuntu_vm.pycode · 30.7 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/base.pycode · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/crossover.pycode · 17.3 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/elimination.pycode · 5.1 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/migration.pycode · 14.9 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/mutation.pycode · 16.8 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/operators/reevaluation.pycode · 10.1 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/orchestrator.pycode · 32.4 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/osworld_v2_preflight.pycode · 32.9 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/paths.pycode · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/pyproject.tomlcode · 1.6 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/scienceboard_synthetic_runtime.pycode · 11.1 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/scoring.pycode · 1.3 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/scratch_vm.pycode · 30.5 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/state.pycode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/synthetic_tasks.pycode · 10.4 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/tournament.pycode · 5.4 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/utils.pycode · 5.5 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/vm/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/vm/owner_tag.pycode · 3.1 KB
No inline preview for this entry.
agent-s-computer-use/agent_s4/vm/utils.pycode · 25.8 KB
No inline preview for this entry.
agent-s-computer-use/evaluation_sets/test_all.jsoncode · 16.1 KB
No inline preview for this entry.
agent-s-computer-use/evaluation_sets/test_small_new.jsoncode · 3.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/README.mdmarkdown · 9.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/WindowsAgentArena.mdmarkdown · 2.4 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/ACI.pycode · 0.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/LinuxOSACI.pycode · 31.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/MacOSACI.pycode · 20.6 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/WindowsOSACI.pycode · 18.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/aci/windowsagentarena/GroundingAgent.pycode · 24.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/cli_app.pycode · 8.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/AgentS.pycode · 14.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/BaseModule.pycode · 0.6 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/Knowledge.pycode · 9.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/Manager.pycode · 9.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/ProceduralMemory.pycode · 15.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/Worker.pycode · 9.7 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/core/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/mllm/MultimodalAgent.pycode · 9.7 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/mllm/MultimodalEngine.pycode · 8.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/mllm/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/utils/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/utils/common_utils.pycode · 28.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/utils/ocr_server.pycode · 1.4 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s1/utils/query_perplexica.pycode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/README.mdmarkdown · 13.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/WAA_setup.mdmarkdown · 10.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/agents/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/agents/agent_s.pycode · 16.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/agents/grounding.pycode · 23.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/agents/manager.pycode · 11.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/agents/worker.pycode · 10.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/cli_app.pycode · 11.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/core/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/core/engine.pycode · 17.6 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/core/knowledge.pycode · 15.7 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/core/mllm.pycode · 10.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/core/module.pycode · 0.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/memory/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/memory/procedural_memory.pycode · 17.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/utils/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/utils/common_utils.pycode · 6.3 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2/utils/query_perplexica.pycode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/README.mdmarkdown · 7.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/agents/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/agents/agent_s.pycode · 3.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/agents/grounding.pycode · 24.1 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/agents/worker.pycode · 7.8 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/cli_app.pycode · 11.1 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/core/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/core/engine.pycode · 16.1 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/core/mllm.pycode · 11.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/core/module.pycode · 0.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/memory/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/memory/procedural_memory.pycode · 6.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/utils/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s2_5/utils/common_utils.pycode · 3.6 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/README.mdmarkdown · 10.1 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/agents/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/agents/agent_s.pycode · 3.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/agents/code_agent.pycode · 12.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/agents/grounding.pycode · 27.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/agents/worker.pycode · 15.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/bbon/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/bbon/behavior_narrator.pycode · 11.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/bbon/comparative_judge.pycode · 5.1 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/cli_app.pycode · 11.9 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/core/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/core/engine.pycode · 16.4 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/core/mllm.pycode · 14.2 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/core/module.pycode · 0.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/memory/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/memory/procedural_memory.pycode · 25.4 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/utils/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/utils/common_utils.pycode · 6.5 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/utils/formatters.pycode · 1.9 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/s3/utils/local_env.pycode · 2.4 KB
No inline preview for this entry.
agent-s-computer-use/gui_agents/utils.pycode · 1.5 KB
No inline preview for this entry.
agent-s-computer-use/images/agent_s.pngbinary · 144.8 KB
Binary — download to inspect.
agent-s-computer-use/images/agent_s2_architecture.pngbinary · 106.2 KB
Binary — download to inspect.
agent-s-computer-use/images/agent_s2_osworld_result.pngbinary · 212.0 KB
Binary — download to inspect.
agent-s-computer-use/images/agent_s2_teaser.pngbinary · 420.0 KB
Binary — download to inspect.
agent-s-computer-use/images/agent_s_architecture.pdfbinary · 79.1 KB
Binary — download to inspect.
agent-s-computer-use/images/osworld_result.pngbinary · 226.0 KB
Binary — download to inspect.
agent-s-computer-use/images/results.pdfbinary · 14.9 KB
Binary — download to inspect.
agent-s-computer-use/images/results.pngbinary · 53.3 KB
Binary — download to inspect.
agent-s-computer-use/images/s3_results.pngbinary · 221.3 KB
Binary — download to inspect.
agent-s-computer-use/images/s3_results_new.pngbinary · 234.4 KB
Binary — download to inspect.
agent-s-computer-use/images/teaser.pngbinary · 915.1 KB
Binary — download to inspect.
agent-s-computer-use/images/waa_setup/fig1.pngbinary · 189.3 KB
Binary — download to inspect.
agent-s-computer-use/images/waa_setup/fig2.pngbinary · 409.8 KB
Binary — download to inspect.
agent-s-computer-use/images/windows_result.pngbinary · 89.5 KB
Binary — download to inspect.
agent-s-computer-use/integrations/openclaw/README.mdmarkdown · 7.5 KB
No inline preview for this entry.
agent-s-computer-use/integrations/openclaw/SKILL.mdmarkdown · 3.3 KB
No inline preview for this entry.
agent-s-computer-use/integrations/openclaw/agent_s_tasktext · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/integrations/openclaw/agent_s_wrapper.pycode · 5.5 KB
No inline preview for this entry.
agent-s-computer-use/models.mdmarkdown · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s1/OSWorld.mdmarkdown · 4.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s1/lib_run_single.pycode · 2.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s1/run.pycode · 11.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2/OSWorld.mdmarkdown · 4.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2/lib_run_single.pycode · 2.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2/run.pycode · 13.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2_5/OSWorld.mdmarkdown · 0.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2_5/lib_run_single.pycode · 2.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2_5/lib_run_single_local.pycode · 2.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2_5/run.pycode · 19.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s2_5/run_local.pycode · 13.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/OSWorld.mdmarkdown · 2.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/bbon/generate_facts.pycode · 7.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/bbon/run_judge.pycode · 8.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/bbon/utils.pycode · 10.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/lib_run_single.pycode · 3.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/run.pycode · 19.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/run.shcode · 1.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s3/run_local.pycode · 13.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/.env.exampletext · 4.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/.gitignoretext · 5.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/README.mdmarkdown · 10.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-kernel/pyproject.tomlcode · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-kernel/uv.locktext · 142.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/LICENSEtext · 1.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/PROVENANCE.mdmarkdown · 2.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/SOURCE_MANIFEST.jsoncode · 0.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/UPSTREAM_COMMITtext · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/pyproject.tomlcode · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/src/rlm/__init__.pycode · 12.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/src/rlm/harness.pycode · 31.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/src/rlm/mcp.pycode · 26.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/src/rlm/mcp_base.pycode · 13.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/src/rlm/skill.pycode · 1.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/agent_harness/prime-agent-runtime/uv.locktext · 186.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/config/osworld_v2_release_pin.jsoncode · 1.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/conftest.pycode · 1.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/data/README.mdmarkdown · 1.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/docs/OPENCODE_HARNESS.mdmarkdown · 11.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/docs/PRIME_AGENT_HARNESS.mdmarkdown · 6.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/docs/osworld_smoke_test_iam_policy.jsoncode · 0.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_full_minus_val_93.jsoncode · 0.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_heldout_68.jsoncode · 0.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_test.jsoncode · 0.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_train_25.jsoncode · 0.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_train_50.jsoncode · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_val_15.jsoncode · 0.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/osworld_v2_val_25.jsoncode · 0.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/eval_sets/synthetic_train_30.jsoncode · 0.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/prompts/combined.mdmarkdown · 2.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/prompts/crossover.mdmarkdown · 0.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/prompts/scratch_vm_addendum.mdmarkdown · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/prompts/synthetic_trajectories.mdmarkdown · 2.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/pytest.initext · 0.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/requirements.txtcode · 0.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/assets/cache_prime_agent_assets.pycode · 11.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/assets/prepare_opencode_runtime.pycode · 2.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/eval/create_osworld_v2_eval_sets.pycode · 8.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/eval/prepare_osworld_synthetic.pycode · 4.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/eval/run_osworld_v2_qwen_synthetic_30_15.shcode · 31.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/eval/run_osworld_v2_test_split_variant.shcode · 18.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/eval/stop_evolution.shcode · 2.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/ops/list_orphan_scratch_vms.pycode · 2.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/ops/rerun_infra_failures.pycode · 9.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/proxy/hermes_tools_handler.pycode · 30.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/proxy/litellm_config.yamlcode · 1.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/proxy/run_litellm.shcode · 1.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/vm/scratch_vm_deploy_seed.pycode · 3.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/vm/scratch_vm_down.pycode · 0.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/vm/scratch_vm_exec.pycode · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/scripts/vm/scratch_vm_up.pycode · 2.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/__main__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/cli.pycode · 4.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/context_compaction.jsoncode · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/helpers/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/helpers/atree.pycode · 2.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/helpers/browser.pycode · 1.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/helpers/cli_ops.pycode · 1.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/helpers/gui.pycode · 10.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/seeds/combined/setup.shcode · 0.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/shadow_evol_test_network_guard.pycode · 0.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/sitecustomize.pycode · 0.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/_fixtures/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/_fixtures/marker_operator.pycode · 1.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/_fixtures/test_marker_operator.pycode · 1.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/conftest.pycode · 0.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/e2e/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/fixtures/prime_compaction_runtime.mjscode · 5.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/integration/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/integration/test_scienceboard_vlog_result_handler.pycode · 3.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/integration/test_ubuntu_vm_evaluator.pycode · 29.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/integration/test_val_set_smoke.pycode · 2.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_builder_prompts.pycode · 1.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_cli_v2.pycode · 2.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_create_osworld_v2_eval_sets.pycode · 5.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_create_scienceboard_eval_sets.pycode · 1.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_crossover_v2.pycode · 14.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_db_schema_v2.pycode · 2.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_driver.pycode · 12.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_e2e_v2_smoke.pycode · 15.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_island_costs.pycode · 14.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_island_db.pycode · 12.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_island_scratch_vm.pycode · 21.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_launch_builder_env.pycode · 2.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_list_orphan_scratch_vms.pycode · 2.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_load_val_tasks.pycode · 0.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_migration_v2.pycode · 17.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_mutation_v2.pycode · 15.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_reevaluation.pycode · 14.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_reevaluation_driver.pycode · 4.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_resume.pycode · 1.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_scheduler.pycode · 2.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_scienceboard_scratch_vm_builder.pycode · 3.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_stage_results.pycode · 9.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_tournament_selection_signal.pycode · 2.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_tournament_v2.pycode · 1.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_val_config.pycode · 4.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_val_db.pycode · 2.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/test_val_set_resume_safety.pycode · 1.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/__init__.pycode · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_assemble_prompt.pycode · 6.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_benchmark_spec.pycode · 2.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_builder_landlock.pycode · 9.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_builder_scratch_vm.pycode · 3.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_collect_per_task.pycode · 5.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_elimination_backup.pycode · 2.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_experiment_config.pycode · 5.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_git_utils_variant_scope.pycode · 1.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_harness_base.pycode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_harness_registry.pycode · 1.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_hermes_tools_handler.pycode · 26.2 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_island_config.pycode · 0.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_loop_state.pycode · 2.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_network_guard.pycode · 0.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_opencode_builder.pycode · 1.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_opencode_dispatch.pycode · 4.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_opencode_harness.pycode · 11.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_opencode_live_runtime.pycode · 10.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_opencode_runtime.pycode · 10.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_orchestrator_seed_resume.pycode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_failure_manifest.pycode · 0.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v1_split.pycode · 6.0 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v2_adapter.pycode · 39.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v2_clone_provenance.pycode · 3.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v2_preflight.pycode · 15.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v2_run_splits.pycode · 5.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_osworld_v2_tasks.pycode · 4.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_owner_tag_hook.pycode · 2.4 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_paths.pycode · 3.1 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_agent_builder.pycode · 11.8 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_agent_evaluator.pycode · 35.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_agent_runner.pycode · 5.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_agent_source.pycode · 3.6 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_compaction.pycode · 4.3 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_prime_step_runner_compaction.pycode · 3.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_seed_grounding_provider.pycode · 0.7 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_synthetic_tasks.pycode · 9.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_v2_public_ip_addressing.pycode · 3.5 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/tests/unit/test_vm_agent_driver.pycode · 13.9 KB
No inline preview for this entry.
agent-s-computer-use/osworld_setup/s4/vm_assets/.gitignoretext · 0.0 KB
No inline preview for this entry.
agent-s-computer-use/requirements.txtcode · 0.3 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/.env.exampletext · 1.5 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/.gitignoretext · 0.4 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/README.mdmarkdown · 4.7 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/data/README.mdmarkdown · 1.3 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/eval_sets/scienceboard_full_169.jsoncode · 3.1 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/eval_sets/scienceboard_synthetic_train_48.jsoncode · 1.0 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/eval_sets/scienceboard_test.jsoncode · 1.7 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/eval_sets/scienceboard_val_15.jsoncode · 0.3 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/prompts/scienceboard_eval_format_addendum.mdmarkdown · 1.8 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/prompts/scienceboard_synthetic_trajectories.mdmarkdown · 3.7 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/prompts/scienceboard_synthetic_trajectories_60m.mdmarkdown · 3.7 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/prompts/scratch_vm_addendum_scienceboard.mdmarkdown · 2.0 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/scripts/eval/create_scienceboard_eval_sets.pycode · 7.2 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/scripts/eval/run_scienceboard_full_variant.shcode · 5.8 KB
No inline preview for this entry.
agent-s-computer-use/scienceboard_setup/s4/scripts/eval/run_scienceboard_qwen_synthetic_50_15.shcode · 19.2 KB
No inline preview for this entry.
agent-s-computer-use/setup.pycode · 2.1 KB
No inline preview for this entry.
agent-s-computer-use/tests/test_providers.pycode · 3.0 KB
No inline preview for this entry.

🥳 Updates

🚀 Quick Start

📚 Agent S Versions

💬 Citations

Star History

TalentDelta provenance record (added at packaging)

Packaging notes (TalentDelta)

Introduction

Initial WAA Setup

Fixing Setup Issues

Office Apps Aren’t Installed

Google Chrome Pop-ups

VSCode Pop-ups

Note: set_cell_values

Double Checking…

Set up Agent S2.5 with WAA Locally

Agent S2.5 with WAA on Azure

agent_s4

Results

Setup guides

Install

Entry points

How a run works

Module map

Where files go

Extending

OpenCode harness

Upstream version

What differs from upstream

Prime Agent harness

Upstream version

What differs from upstream

Note: `set_cell_values`