Demonstration · 14 Nov 2026

Dependabot

A guide for setting up and managing Dependabot, GitHub’s tool that keeps a repository’s dependencies up to date.

A demonstration Techtree ran itself on GitHub’s public Dependabot Skill, from its awesome-copilot collection (MIT license), used unchanged as the starting point. It is not customer work.

Task

Given a repository’s file list, reply with the Dependabot package ecosystems it needs, as an alphabetical, comma-separated list and nothing else; an answer counts only when it matches exactly.

Result

The final test: 38 cases nothing had seen, each asked 3 times of each version, 114 attempts per side.

Original 35% 40 of 114
Revised 81% 92 of 114
Verdict
Improved
Difference
+52 attempts
Cases
25 better, 12 the same, 1 worse
Chance
Under 1 in 1,000

Microsoft’s SkillOpt revision solved 52 more attempts than the original, more than chance explains. If the two versions were equally good, a difference this large would turn up by chance fewer than 1 time in 1,000. The line agreed before anything ran is 5 in 100, so this difference is beyond chance.

Method

  1. Split

    We wrote 84 cases for this test and split them once, before anything ran: 25 to work on the Skill, 21 to choose between versions and 38 kept apart for the final test. Every version was asked each case 3 times.

  2. Improve

    Two revisions were made from the 25 working cases alone: a single plain rewrite by Anthropic’s Claude Opus 5.5, and a revision by Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes. On those cases the agent solved 9% of its attempts with no Skill (7 of 75) and 28% with the original (21 of 75).

  3. Choose

    The original and both revisions were tried on the 21 choosing cases. Microsoft’s SkillOpt revision solved the most and became the improved Skill.

    Original 35% 22 of 63
    Rewrite 57% 36 of 63
    SkillOpt 73% 46 of 63
  4. Final test

    One run on the 38 cases nothing had seen, 114 attempts per version, with the same agent and settings on both sides.

Cases

One case did worse with the improved Skill: rust-service-container went from 3 of 3 to 2 of 3.

Case Original Revised Change
android-app-gradle 2 of 3 3 of 3 +1
blog-theme-submodule 0 of 3 3 of 3 +3
dart-package-ci 0 of 3 1 of 3 +1
django-compose 0 of 3 3 of 3 +3
django-pipenv-tests 0 of 3 3 of 3 +3
dotfiles-submodules 1 of 3 1 of 3 0
elixir-library 1 of 3 1 of 3 0
fastapi-uv-container 0 of 3 2 of 3 +2
flask-api-container 2 of 3 2 of 3 0
flutter-app 1 of 3 1 of 3 0
go-library-hooks 0 of 3 3 of 3 +3
go-vendored-submodule 0 of 3 1 of 3 +1
ios-app-fastlane 0 of 3 3 of 3 +3
java-library-maven 2 of 3 3 of 3 +1
java-react 0 of 3 2 of 3 +2
jekyll-site 3 of 3 3 of 3 0
laravel-app 1 of 3 2 of 3 +1
local-services-compose 0 of 3 3 of 3 +3
maven-service-build 0 of 3 3 of 3 +3
node-devcontainer 0 of 3 2 of 3 +2
poetry-service-ci 0 of 3 2 of 3 +2
rails-monolith 3 of 3 3 of 3 0
rails-react-ci 0 of 3 2 of 3 +2
react-spa-container 2 of 3 2 of 3 0
ruby-gem-hooks 1 of 3 3 of 3 +2
rust-cli-release 0 of 3 2 of 3 +2
rust-devcontainer 0 of 3 2 of 3 +2
rust-service-container 3 of 3 2 of 3 −1
rust-wasm-site 0 of 3 3 of 3 +3
scraper-pipenv 3 of 3 3 of 3 0
spring-maven-container 3 of 3 3 of 3 0
static-site-image 1 of 3 3 of 3 +2
svelte-pnpm-app 3 of 3 3 of 3 0
swift-server-container 1 of 3 2 of 3 +1
uv-web-app 1 of 3 3 of 3 +2
uv-worker-container 0 of 3 3 of 3 +3
vue-yarn-workspace 3 of 3 3 of 3 0
wordpress-plugin 3 of 3 3 of 3 0

Tools

Agent
NVIDIA’s Nemotron 3 Nano 4B (nemotron-3-nano:4b), run on our own machine.
Scoring
Prime Intellect’s Verifiers 0.4.1, with the same settings on both sides.
Check
NVIDIA’s SkillEvaluator 0.9.0 reviewed the original Skill as written.
Rewrite
Anthropic’s Claude Opus 5.5, one plain rewrite from the working cases.
Revision
Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes, on the working cases only.
Skill
GitHub’s Dependabot Skill, the starting point, MIT license.

Report

The report lists every final-test case with the attempts each version solved, so the figures above can be counted again from it.