Climb
HF Tasksmith
Six real changes from Hugging Face's trl, transformers, diffusers, peft and accelerate repositories, from the HF ML Tasksmith collection. With thanks to its authors. Not a measure of broad capability.
- Question
- Does adding the Tasksmith Skill solve more of 6 real changes to Hugging Face code?
- Changed
- One skill is added. Nothing else may differ.
- Tasks
- 6, fixed before either Run
- Held-out tasks
- 6 more, kept apart. They are run once, on the winning Skill, and never decide the winner.
- Input
- A repository with one real change taken back out, and a description of the change to make. Each task brings its own image.
- Expected output
- The repository's files with the change made again.
- Scoring
- Solved: the change's own tests run in a separate box on only the files the task names. Each task is 0 or 1, then the 6 task values are averaged.
- Held fixed
- Tasks and their order, each task's own images for working and for grading, model, Hermes harness, tools, sampling, and budget.