BENCHMARK-TARGETED TRAINING
Professional computing tasks in isolated terminal environments.
01 / DATASET OVERVIEW
SCOPE & PURPOSE
The training task.
- ISOLATED TERMINALS
- CROSS-DISCIPLINARY TASKS
- STATE VERIFIERS
Designed to evaluate agents' end-to-end ability to perform professional computing tasks in isolated terminal environments, the dataset provides challenging and continuously extensible evaluation tasks. It supports terminal agent training and evaluation, as well as research on tool use, environment interaction, and failure modes.
02 / DATASET PROFILE & TECHNICAL SIGNALS
- 01EXECUTION ENVIRONMENT
- Isolated terminals
- Containerized task environments.
- 02TASK COVERAGE
- Cross-disciplinary
- Professional computing scenarios.
- 03SOLVABILITY
- Oracle validation
- Reference solutions and solvability checks.
- 04VERIFICATION
- Terminal state
- Task-specific state verifiers.
03 / DATA PRODUCTION & CONSTRUCTION
Data production strengths
We have expert-level, cross-disciplinary task production capabilities for challenging terminal-based scenarios, with professional task designers across software engineering, scientific computing, finance, engineering, and system operations. Our workflow covers task design, containerized environment construction, reference solution development, terminal-state verifier development, instruction–verification alignment, Oracle solvability validation, and reward-hacking resistance review, supporting scalable production and quality assurance.
- 01DESIGN
Professional computing tasks
Design challenging tasks across software engineering, scientific computing, finance, engineering, and system operations.
- 02BUILD
Containerized environments
Construct task environments and develop reference solutions.
- 03VERIFY
Terminal-state verifiers
Develop verifiers and align task instructions with verification.
- 04VALIDATE
Solvability & resistance review
Validate Oracle solvability and review resistance to reward hacking.
04 / COVERAGE & CAPABILITY MATRIX
- 01Professional domains
- Software engineeringScientific computingFinanceEngineeringSystem operations
- 02Task environment
- Isolated terminalsContainerized environments
- 03Verification
- Terminal-state verifiersInstruction–verification alignmentOracle validation
- 04Research & training
- Terminal agent trainingTool useEnvironment interactionFailure modes