qa-toolkit: Packaging Our QA Team's Expertise into a Claude Plugin

Every team that adopts AI assistants hits the same wall sooner or later. The assistant is capable, but it has amnesia. It does not know your naming conventions, your CI templates, your defect report format, or the years of knowledge sitting in your Confluence spaces. So every engineer compensates by prompting from scratch, and every engineer gets slightly different results. On our team, we decided to fix this properly. We packaged our QA team's expertise into qa-toolkit — a Claude plugin with ten specialized skills and a shared, versioned knowledge base of roughly 2,350 notes migrated from twelve Confluence spaces. This is the story of what we built, why, and what changed.

The Problem: Ten Engineers, Ten Different Prompts

Our team had already crossed the adoption threshold. Everyone used Claude — in Claude Code from the terminal, or in the desktop app's Cowork mode. The productivity gains were real. So was a new kind of inconsistency. One engineer's AI-generated Playwright test used web-first assertions and page objects; another's used hard waits and inline locators. One Postman collection followed our platform conventions; the next invented its own structure. Every defect report looked different depending on who asked the AI to write it.

The root cause was obvious once we named it. Our conventions existed — in Confluence pages, in senior engineers' heads, in old merge request comments. But none of that context reached the AI. Each conversation started from zero. We were paying the context tax on every task, multiplied across the whole team, every day.

The Solution: A Plugin, Not a Prompt Library

Our first instinct was a shared prompt library — a wiki page of copy-paste snippets. We rejected it quickly. Prompt libraries rot. Nobody updates them, nobody versions them, and they still require every engineer to remember they exist. What we needed was infrastructure: something installable, versioned, and distributed like any other piece of software.

Claude plugins are that infrastructure. A plugin is a bundle of skills — structured instruction sets that activate automatically when relevant — plus any supporting assets: templates, scaffolds, scripts, and in our case an entire knowledge base. It installs once, updates centrally, and works identically in Claude Code and the Claude desktop app. One GitLab repository became the single source of truth for the plugin, its marketplace, and the team's knowledge.

Skills: Conventions as Code

Each skill is essentially our team's know-how written down in a form the AI can act on. The Postman skill carries the exact folder structure, naming rules, test script patterns and environment conventions we use across our API platform, plus ready-made scaffolds for our core services. The CI skill ships our canonical GitLab CI template with newman validation stages, error extraction and Slack notifications. The defect skill produces a Jira issue and a markdown report with environment details, request and response payloads, and actual-versus-expected results — in the same format, every time, regardless of who files it.

Writing the skills forced a healthy exercise: we had to agree on what our conventions actually were. Several turned out to exist only as tribal knowledge, with two or three competing variants in circulation. Codifying them into skills settled those debates permanently — and the skill files themselves became the documentation.

The toolkit also refused to stay inside QA. Once the pattern proved itself, neighboring workflows moved in: cross-notes turns status-meeting transcripts into structured Confluence pages, nps-report processes weekly NPS exports into analysis, Jira tickets and a Slack summary, onepager generates project briefs from the internal template, and po-summary compiles a weekly Product Owner digest from Jira and Slack. The plugin started as a QA tool and became the team's shared workbench.

The Memex: Giving the AI Our Confluence

Skills solve the conventions problem. They do not solve the knowledge problem. Years of team documentation lived across a dozen Confluence spaces — development, product management, QA, customer care, payments, the player. Valuable, but invisible to the AI — and increasingly hard for humans to find.

So we built a migration pipeline. A deterministic Python exporter pulls whole spaces through the Confluence REST API and converts every page to Obsidian markdown — HTML to markdown, macros to callouts, internal page links to wikilinks, page hierarchy to folders — for roughly zero LLM tokens: no per-page AI judgement is needed, so none is spent. The result is a vault: about 2,350 interlinked notes across twelve spaces, versioned directly in the plugin repository. A sync-state file records each page's exported version, so re-runs skip unchanged pages and update existing notes in place — preserving hand-added tags — and the exporter never deletes files; every run gets reviewed with git diff.

This dual-consumer design is the part I am most happy with. The vault is Obsidian-compatible, so engineers browse it as a linked knowledge graph. Claude reads the very same files through the memex skill — and can write back. When a session uncovers something worth keeping, it becomes a note in the vault, goes through review, and is available to everyone's next session. Knowledge finally compounds instead of evaporating in chat history.

Governance: Treating the Plugin Like a Product

A team tool that nobody maintains dies within months. From day one, we treated qa-toolkit as an internal product with real release engineering:

• All changes — skills and vault alike — go through merge requests. No direct commits to main.
• Semantic versioning: fixes bump the patch, new skills or capabilities bump the minor version.
• Pre-merge checks: valid plugin manifests, every skill fully described, no broken wikilinks in new vault notes.
• Two distribution channels: Claude Code users update with a single marketplace command; desktop app users receive a packaged plugin build after each release.

None of it is glamorous, but it is the reason the toolkit is still alive and trusted instead of joining the graveyard of internal tools that worked great in the demo.

What Changed

The most visible change is consistency. AI-generated Playwright tests, Postman collections, CI pipelines and defect reports now look like they came from one team, because in a real sense they did — the team's standards are embedded in every generation. Review time dropped accordingly: reviewers check logic and coverage instead of correcting structure and naming.

Onboarding changed too. A new engineer installs one plugin and immediately works the way the team works — the conventions, the templates and the accumulated knowledge come as a package. Questions that used to interrupt a senior colleague — Which environment? What's the defect format? Where are the notes on this service? — are now answered by the memex in seconds.

And there is a quieter, structural change: the team now has a habit of writing knowledge down, because written knowledge is no longer passive. A note in the vault is not documentation that might be read someday. It is context that actively shapes every AI-assisted task tomorrow.

Lessons Learned

• Start from conventions you already have. A skill that codifies existing practice gets adopted instantly; a skill that invents new practice gets debated.
• Version the knowledge with the tool. The vault living in the same repository as the skills means one MR can update a convention and the knowledge about it atomically.
• Distribution is half the battle. The marketplace update command and the packaged desktop build are what turned a nice repo into something the whole team actually runs.
• Keep deterministic work out of the LLM. Migrating ~2,350 pages is a job for a script, not an agent — the exporter does the whole conversion for roughly zero tokens, and AI judgement is saved for where it actually matters.
• Treat prompts as code. MRs, semver, and pre-merge validation feel heavy for "just instructions" — until the first bad change ships to ten engineers at once.

The broader point goes beyond QA. Any team using AI assistants seriously will eventually need what a plugin provides: a way to make its collective expertise portable, versioned and automatically present. We built ours for testing — and within months, product-owner workflows moved in beside the QA skills. The pattern is universal.