BENCHMARK-TARGETED TRAINING

Expert-reviewed evaluation for complex cross-file refactoring.

Overview Production
CROSS-FILE REFACTORINGMULTILINGUAL TASKSEXPERT REVIEW

01 / DATASET OVERVIEW

EXPERT-LEVEL TASK DATA

SCOPE & PURPOSE

The training task.

  • CROSS-FILE REFACTORING
  • MULTILINGUAL TASKS
  • EXPERT REVIEW

Designed to evaluate coding agents on large-scale, cross-file code refactoring, SWE-bench Pro Max provides expert-reviewed, multilingual software engineering evaluation data. It is used to assess the real-world upper bound of AI coding models on complex, long-horizon software engineering tasks.

02 / DATASET PROFILE & TECHNICAL SIGNALS

SCOPE · SCALE · CONSTRUCTION
01TASK SCOPE
Cross-file
Large-scale software refactoring.
02LANGUAGE SCOPE
Multilingual
Task design and review across programming languages.
03QUALITY REVIEW
Expert-reviewed
Professional task design and review resources.
04PRODUCTION
Automated pipeline
Scalable production and expert quality evaluation.

03 / DATA PRODUCTION & CONSTRUCTION

METHOD & WORKFLOW

Data production strengths

We have expert-level data production capabilities for large-scale, cross-file code refactoring scenarios. Through close collaboration with the SWE-bench Pro Max team, we have access to professional task design and review resources across multiple programming languages. Our workflow covers task design, repository-instance selection, issue-description rewriting, test-suite review, reference-patch alignment, and behavioral preservation verification, supported by an automated data-production pipeline for scalable production and expert-based quality evaluation.

  1. 01DESIGN

    Task & repository selection

    Design tasks and select repository instances for large-scale, cross-file refactoring.

  2. 02PREPARE

    Issue-description rewriting

    Rewrite issue descriptions and review test suites.

  3. 03ALIGN

    Reference-patch alignment

    Align reference patches and verify preservation of software behavior.

  4. 04REVIEW

    Expert quality evaluation

    Combine an automated data-production pipeline with expert-based quality evaluation.

04 / COVERAGE & CAPABILITY MATRIX

TASKS · METHODS · APPLICATIONS
DIMENSIONCOVERAGE
01Task complexity
Large-scale refactoringCross-file changesLong-horizon engineering
02Design resources
Multiple programming languagesProfessional task designersExpert reviewers
03Review workflow
Issue descriptionsTest suitesReference patchesBehavior preservation
04Model application
Coding-agent evaluationReal-world capability assessment
Explore the production method

NEXT STEP