Demonstration · 14 Nov 2026
Gitmoji
Writes commit messages in the gitmoji convention, choosing the emoji that matches the intent of each change.
A demonstration Techtree ran itself on GitHub’s public Gitmoji Skill, from its awesome-copilot collection (MIT license), used unchanged as the starting point. It is not customer work.
Task
Given a plain description of one change, reply with the gitmoji shortcode the commit should use, such as :bug:, and nothing else; an answer counts only when it matches exactly.
Result
The final test: 39 cases nothing had seen, each asked 3 times of each version, 117 attempts per side.
- Verdict
- Improved
- Difference
- +20 attempts
- Cases
- 17 better, 17 the same, 5 worse
- Chance
- About 0.1 in 100
The plain rewrite solved 20 more attempts than the original, more than chance explains. If the two versions were equally good, a difference this large would turn up by chance about 0.1 times in 100. The line agreed before anything ran is 5 in 100, so this difference is beyond chance.
Method
-
Split
We wrote 87 cases for this test and split them once, before anything ran: 26 to work on the Skill, 22 to choose between versions and 39 kept apart for the final test. Every version was asked each case 3 times.
-
Improve
Two revisions were made from the 26 working cases alone: a single plain rewrite by Anthropic’s Claude Opus 5.5, and a revision by Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes. On those cases the agent solved 0% of its attempts with no Skill (0 of 78) and 69% with the original (54 of 78).
-
Choose
The original and both revisions were tried on the 22 choosing cases. The plain rewrite solved the most and became the improved Skill.
-
Final test
One run on the 39 cases nothing had seen, 117 attempts per version, with the same agent and settings on both sides.
Cases
5 cases did worse with the improved Skill: require-node-20 went from 2 of 3 to 1 of 3; sdk-region-required went from 2 of 3 to 1 of 3; shared-price-formatter went from 3 of 3 to 1 of 3; split-report-service went from 3 of 3 to 1 of 3; webpack-5-94 went from 3 of 3 to 2 of 3.
| Case | Original | Revised | Change |
|---|---|---|---|
| backend-style-guide-format | 3 of 3 | 3 of 3 | 0 |
| batched-price-lookups | 3 of 3 | 3 of 3 | 0 |
| c-implicit-conversion-warnings | 0 of 3 | 3 of 3 | +3 |
| cart-count-stale | 3 of 3 | 3 of 3 | 0 |
| checkout-down-empty-url | 2 of 3 | 3 of 3 | +1 |
| cli-config-toml | 0 of 3 | 0 of 3 | 0 |
| config-parse-cached | 2 of 3 | 2 of 3 | 0 |
| dark-theme-tokens | 1 of 3 | 3 of 3 | +2 |
| dashboard-migration-guide | 2 of 3 | 2 of 3 | 0 |
| discount-calculator-tests | 3 of 3 | 3 of 3 | 0 |
| download-path-traversal | 3 of 3 | 3 of 3 | 0 |
| duplicate-project-action | 3 of 3 | 3 of 3 | 0 |
| editorconfig-markdown | 1 of 3 | 2 of 3 | +1 |
| eslint-no-console-rule | 1 of 3 | 3 of 3 | +2 |
| eu-region-timeouts | 2 of 3 | 3 of 3 | +1 |
| gallery-resize-on-upload | 3 of 3 | 3 of 3 | 0 |
| go-line-wrapping | 2 of 3 | 2 of 3 | 0 |
| heading-brand-font | 3 of 3 | 3 of 3 | 0 |
| import-command-reference | 3 of 3 | 3 of 3 | 0 |
| kotlin-coroutines-1-8 | 0 of 3 | 3 of 3 | +3 |
| last-active-sort-order | 0 of 3 | 1 of 3 | +1 |
| login-open-redirect | 3 of 3 | 3 of 3 | 0 |
| notification-retry-tests | 2 of 3 | 3 of 3 | +1 |
| order-export-deletes-file | 1 of 3 | 3 of 3 | +2 |
| prettier-frontend | 3 of 3 | 3 of 3 | 0 |
| pricing-table-restyle | 2 of 3 | 3 of 3 | +1 |
| pylint-billing-warnings | 0 of 3 | 3 of 3 | +3 |
| react-18-3-bump | 1 of 3 | 3 of 3 | +2 |
| remove-compact-view-setting | 2 of 3 | 3 of 3 | +1 |
| remove-pdf-preview | 2 of 3 | 3 of 3 | +1 |
| require-node-20 | 2 of 3 | 1 of 3 | −1 |
| sdk-region-required | 2 of 3 | 1 of 3 | −1 |
| search-pagination-offset | 2 of 3 | 3 of 3 | +1 |
| session-cookie-flags | 3 of 3 | 3 of 3 | 0 |
| shared-price-formatter | 3 of 3 | 1 of 3 | −2 |
| shared-retry-wrapper | 2 of 3 | 3 of 3 | +1 |
| split-report-service | 3 of 3 | 1 of 3 | −2 |
| streaming-log-parser | 3 of 3 | 3 of 3 | 0 |
| webpack-5-94 | 3 of 3 | 2 of 3 | −1 |
Tools
- Agent
- NVIDIA’s Nemotron 3 Nano 4B (nemotron-3-nano:4b), run on our own machine.
- Scoring
- Prime Intellect’s Verifiers 0.4.1, with the same settings on both sides.
- Check
- NVIDIA’s SkillEvaluator 0.9.0 reviewed the original Skill as written.
- Rewrite
- Anthropic’s Claude Opus 5.5, one plain rewrite from the working cases.
- Revision
- Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes, on the working cases only.
- Skill
- GitHub’s Gitmoji Skill, the starting point, MIT license.
Report
The report lists every final-test case with the attempts each version solved, so the figures above can be counted again from it.