Demonstration · 14 Nov 2026

Gitmoji

Writes commit messages in the gitmoji convention, choosing the emoji that matches the intent of each change.

A demonstration Techtree ran itself on GitHub’s public Gitmoji Skill, from its awesome-copilot collection (MIT license), used unchanged as the starting point. It is not customer work.

Task

Given a plain description of one change, reply with the gitmoji shortcode the commit should use, such as :bug:, and nothing else; an answer counts only when it matches exactly.

Result

The final test: 39 cases nothing had seen, each asked 3 times of each version, 117 attempts per side.

Original 68% 79 of 117
Revised 85% 99 of 117
Verdict
Improved
Difference
+20 attempts
Cases
17 better, 17 the same, 5 worse
Chance
About 0.1 in 100

The plain rewrite solved 20 more attempts than the original, more than chance explains. If the two versions were equally good, a difference this large would turn up by chance about 0.1 times in 100. The line agreed before anything ran is 5 in 100, so this difference is beyond chance.

Method

  1. Split

    We wrote 87 cases for this test and split them once, before anything ran: 26 to work on the Skill, 22 to choose between versions and 39 kept apart for the final test. Every version was asked each case 3 times.

  2. Improve

    Two revisions were made from the 26 working cases alone: a single plain rewrite by Anthropic’s Claude Opus 5.5, and a revision by Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes. On those cases the agent solved 0% of its attempts with no Skill (0 of 78) and 69% with the original (54 of 78).

  3. Choose

    The original and both revisions were tried on the 22 choosing cases. The plain rewrite solved the most and became the improved Skill.

    Original 64% 42 of 66
    Rewrite 86% 57 of 66
    SkillOpt 73% 48 of 66
  4. Final test

    One run on the 39 cases nothing had seen, 117 attempts per version, with the same agent and settings on both sides.

Cases

5 cases did worse with the improved Skill: require-node-20 went from 2 of 3 to 1 of 3; sdk-region-required went from 2 of 3 to 1 of 3; shared-price-formatter went from 3 of 3 to 1 of 3; split-report-service went from 3 of 3 to 1 of 3; webpack-5-94 went from 3 of 3 to 2 of 3.

Case Original Revised Change
backend-style-guide-format 3 of 3 3 of 3 0
batched-price-lookups 3 of 3 3 of 3 0
c-implicit-conversion-warnings 0 of 3 3 of 3 +3
cart-count-stale 3 of 3 3 of 3 0
checkout-down-empty-url 2 of 3 3 of 3 +1
cli-config-toml 0 of 3 0 of 3 0
config-parse-cached 2 of 3 2 of 3 0
dark-theme-tokens 1 of 3 3 of 3 +2
dashboard-migration-guide 2 of 3 2 of 3 0
discount-calculator-tests 3 of 3 3 of 3 0
download-path-traversal 3 of 3 3 of 3 0
duplicate-project-action 3 of 3 3 of 3 0
editorconfig-markdown 1 of 3 2 of 3 +1
eslint-no-console-rule 1 of 3 3 of 3 +2
eu-region-timeouts 2 of 3 3 of 3 +1
go-line-wrapping 2 of 3 2 of 3 0
heading-brand-font 3 of 3 3 of 3 0
import-command-reference 3 of 3 3 of 3 0
kotlin-coroutines-1-8 0 of 3 3 of 3 +3
last-active-sort-order 0 of 3 1 of 3 +1
login-open-redirect 3 of 3 3 of 3 0
notification-retry-tests 2 of 3 3 of 3 +1
order-export-deletes-file 1 of 3 3 of 3 +2
prettier-frontend 3 of 3 3 of 3 0
pricing-table-restyle 2 of 3 3 of 3 +1
pylint-billing-warnings 0 of 3 3 of 3 +3
react-18-3-bump 1 of 3 3 of 3 +2
remove-compact-view-setting 2 of 3 3 of 3 +1
remove-pdf-preview 2 of 3 3 of 3 +1
require-node-20 2 of 3 1 of 3 −1
sdk-region-required 2 of 3 1 of 3 −1
search-pagination-offset 2 of 3 3 of 3 +1
shared-price-formatter 3 of 3 1 of 3 −2
shared-retry-wrapper 2 of 3 3 of 3 +1
split-report-service 3 of 3 1 of 3 −2
streaming-log-parser 3 of 3 3 of 3 0
webpack-5-94 3 of 3 2 of 3 −1

Tools

Agent
NVIDIA’s Nemotron 3 Nano 4B (nemotron-3-nano:4b), run on our own machine.
Scoring
Prime Intellect’s Verifiers 0.4.1, with the same settings on both sides.
Check
NVIDIA’s SkillEvaluator 0.9.0 reviewed the original Skill as written.
Rewrite
Anthropic’s Claude Opus 5.5, one plain rewrite from the working cases.
Revision
Microsoft’s SkillOpt, with Anthropic’s Claude Opus 5.5 proposing its changes, on the working cases only.
Skill
GitHub’s Gitmoji Skill, the starting point, MIT license.

Report

The report lists every final-test case with the attempts each version solved, so the figures above can be counted again from it.