Demonstration · 14 Nov 2026
Dependabot
A guide for setting up and managing Dependabot, GitHub’s tool that keeps a repository’s dependencies up to date.
A demonstration Techtree ran itself on GitHub’s public Dependabot Skill, from its awesome-copilot collection (MIT license), used unchanged as the starting point. It is not customer work.
Task
Given a repository’s file list, reply with the Dependabot package ecosystems it needs, as an alphabetical, comma-separated list and nothing else; an answer counts only when it matches exactly.
Result
The final test: 38 cases nothing had seen, each asked 3 times of each version, 114 attempts per side.
- Verdict
- Improved
- Difference
- +52 attempts
- Cases
- 25 better, 12 the same, 1 worse
- Chance
- Under 1 in 1,000
Microsoft’s SkillOpt revision solved 52 more attempts than the original, more than chance explains. If the two versions were equally good, a difference this large would turn up by chance fewer than 1 time in 1,000. The line agreed before anything ran is 5 in 100, so this difference is beyond chance.
Method
-
Split
We wrote 84 cases for this test and split them once, before anything ran: 25 to work on the Skill, 21 to choose between versions and 38 kept apart for the final test. Every version was asked each case 3 times.
-
Improve
Two revisions were made from the 25 working cases alone: a single plain rewrite by Anthropic’s Claude Opus 5.5, and a revision by Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes. On those cases the agent solved 9% of its attempts with no Skill (7 of 75) and 28% with the original (21 of 75).
-
Choose
The original and both revisions were tried on the 21 choosing cases. Microsoft’s SkillOpt revision solved the most and became the improved Skill.
-
Final test
One run on the 38 cases nothing had seen, 114 attempts per version, with the same agent and settings on both sides.
Cases
One case did worse with the improved Skill: rust-service-container went from 3 of 3 to 2 of 3.
| Case | Original | Revised | Change |
|---|---|---|---|
| android-app-gradle | 2 of 3 | 3 of 3 | +1 |
| blog-theme-submodule | 0 of 3 | 3 of 3 | +3 |
| dart-package-ci | 0 of 3 | 1 of 3 | +1 |
| django-compose | 0 of 3 | 3 of 3 | +3 |
| django-pipenv-tests | 0 of 3 | 3 of 3 | +3 |
| dotfiles-submodules | 1 of 3 | 1 of 3 | 0 |
| elixir-library | 1 of 3 | 1 of 3 | 0 |
| fastapi-uv-container | 0 of 3 | 2 of 3 | +2 |
| flask-api-container | 2 of 3 | 2 of 3 | 0 |
| flutter-app | 1 of 3 | 1 of 3 | 0 |
| go-library-hooks | 0 of 3 | 3 of 3 | +3 |
| go-vendored-submodule | 0 of 3 | 1 of 3 | +1 |
| ios-app-fastlane | 0 of 3 | 3 of 3 | +3 |
| java-library-maven | 2 of 3 | 3 of 3 | +1 |
| java-react | 0 of 3 | 2 of 3 | +2 |
| jekyll-site | 3 of 3 | 3 of 3 | 0 |
| laravel-app | 1 of 3 | 2 of 3 | +1 |
| local-services-compose | 0 of 3 | 3 of 3 | +3 |
| maven-service-build | 0 of 3 | 3 of 3 | +3 |
| node-devcontainer | 0 of 3 | 2 of 3 | +2 |
| poetry-service-ci | 0 of 3 | 2 of 3 | +2 |
| rails-monolith | 3 of 3 | 3 of 3 | 0 |
| rails-react-ci | 0 of 3 | 2 of 3 | +2 |
| react-spa-container | 2 of 3 | 2 of 3 | 0 |
| ruby-gem-hooks | 1 of 3 | 3 of 3 | +2 |
| rust-cli-release | 0 of 3 | 2 of 3 | +2 |
| rust-devcontainer | 0 of 3 | 2 of 3 | +2 |
| rust-service-container | 3 of 3 | 2 of 3 | −1 |
| rust-wasm-site | 0 of 3 | 3 of 3 | +3 |
| scraper-pipenv | 3 of 3 | 3 of 3 | 0 |
| spring-maven-container | 3 of 3 | 3 of 3 | 0 |
| static-site-image | 1 of 3 | 3 of 3 | +2 |
| svelte-pnpm-app | 3 of 3 | 3 of 3 | 0 |
| swift-server-container | 1 of 3 | 2 of 3 | +1 |
| uv-web-app | 1 of 3 | 3 of 3 | +2 |
| uv-worker-container | 0 of 3 | 3 of 3 | +3 |
| vue-yarn-workspace | 3 of 3 | 3 of 3 | 0 |
| wordpress-plugin | 3 of 3 | 3 of 3 | 0 |
Tools
- Agent
- NVIDIA’s Nemotron 3 Nano 4B (nemotron-3-nano:4b), run on our own machine.
- Scoring
- Prime Intellect’s Verifiers 0.4.1, with the same settings on both sides.
- Check
- NVIDIA’s SkillEvaluator 0.9.0 reviewed the original Skill as written.
- Rewrite
- Anthropic’s Claude Opus 5.5, one plain rewrite from the working cases.
- Revision
- Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes, on the working cases only.
- Skill
- GitHub’s Dependabot Skill, the starting point, MIT license.
Report
The report lists every final-test case with the attempts each version solved, so the figures above can be counted again from it.