Test your Skill

Techtree makes practice tasks from what your Skill teaches, then your agent works them without the Skill and with it. Every step that sends anything to a model prints a review first and waits for your yes. Each command prints the next one.

1. Make the tasks

Make tasks from a Skill

regents techtree forge inspect-skill path/to/your-skill
regents techtree forge plan SOURCE_ID --provider PROVIDER --model MODEL
# Review, correct, build and accept, one printed step at a time.

Accepting the tasks that qualified gives you a collection. Some of its tasks are held out from anyone improving the Skill.

2. Run them both ways, then compare

Compare without and with the Skill

regents techtree forge run --arm baseline --collection COLLECTION_ID --provider PROVIDER --model MODEL
regents techtree forge run --arm candidate --collection COLLECTION_ID --provider PROVIDER --model MODEL --skill path/to/your-skill
regents techtree forge compare BASELINE_RUN_ID CANDIDATE_RUN_ID

Both runs use the same provider and model. The comparison calls no model: it pairs every task across the two runs and says whether the Skill improved, regressed, was mixed or made no difference, overall and on the held-out tasks alone. It writes a report you can open in a browser. The tdd example shows a finished one.

Compare two versions of a Skill

To decide whether a change is worth keeping, give the baseline run the earlier version instead of no Skill. Keep the earlier version in its own folder, then run the candidate with the new version and compare the two runs as above.

Baseline with the earlier version

regents techtree forge run --arm baseline --collection COLLECTION_ID --provider PROVIDER --model MODEL --skill path/to/earlier-skill

3. Revise once, if you want to

regents techtree uplift context COMPARISON_ID gives what a revision may learn from, never the held-out tasks. Write the revised Skill into a new folder, then run regents techtree uplift prepare --from-run COMPARISON_ID --candidate-skill path/to/revised-skill and the regents techtree uplift start command it prints. The revision is judged on the held-out tasks alone.

Fix a task, or check where things stand

If a built task is wrong, fix a copy of it by hand and record that with regents techtree forge correct-task CONSTRUCTION_ID TASK_NAME DIR. regents techtree forge status ID shows any build, plan, run or comparison. regents techtree forge export COLLECTION_ID --to FOLDER writes a copy of the tasks for someone else to read.