Climb
Frontier-CS Open-Ended
Ten open-ended optimisation problems with no known best answer, from Frontier-CS (MIT licence) as chosen by the FrontierSmith authors. With thanks to both.
The first run, with the original starter Skill, scored 0.043 with it against 0.128 without it. With the rewritten starter Skill, a second run still scored lower than no Skill, though by less: a mean of 0.100 with it against 0.131 without it, better on two problems and worse on three. See that Result
- Question
- Does adding the Frontier-CS Skill raise the mean score across 10 open-ended optimisation problems?
- Changed
- One skill is added. Nothing else may differ.
- Tasks
- 10, fixed before either Run
- Input
- A problem statement. The agent writes a C++17 program, which is run on each of the problem's 10 test inputs.
- Expected output
- For each test input, an answer that keeps every rule of the problem.
- Scoring
- Each test input scores from 0 to 1 by the problem's own checker, and 0 when the program breaks a rule, fails to compile, crashes or runs over the problem's time or memory limit. A problem scores the mean of its 10 inputs; the Climb, the mean of its 10 problems.
- Held fixed
- Problems and their order, test inputs, checkers, compiler, each problem's time and memory limits, model, Hermes harness, runtime, tools, sampling, and budget.