Bitwarden Testing Tools

Skill 71 of 83

Setting Up Regression Runs

Use when setting up the QA manual regression runs for a Bitwarden release in Testmo — e.g. “set up the 2026.8.0 regression runs”, “create the partial release regression suite”…

2 minutes · 402 words · 3 sections

Install

$ npx skills add bitwarden/ai-plugins --skill setting-up-regression-runs

$ npx skills add bitwarden/ai-plugins

» /plugin marketplace add bitwarden/ai-plugins

The first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.

Bitwarden’s manual regression suite for a release is a fixed set of Testmo runs, one per spec in specs/, all linked to the release’s milestone. This skill decides which specs a release needs; Skill(bitwarden-testmo-tools:creating-test-runs) creates them. If that skill is not available, stop and ask the user to install bitwarden-testmo-tools.

Release profiles

release-profiles.json (opens in a new tab) lists every spec in each profile:

partial — Web (Password Manager, Admin Console, Admin Portal / SM / Providers), Old Client / New Server, Mobile (iOS, Android), and Directory Connector (BWDC).
full — the same Web, Old Client / New Server, and Mobile runs, plus CLI, 3 Desktop and 4 Extension configuration variants. No BWDC.

full is not a superset of partial. Use exactly the list in the file.

Workflow

Collect the inputs from the user: the profile (partial or full), the milestone’s exact name (e.g. 2026.8.0 Manual Regression), and any spec to skip this release. Ask for whichever is missing.
Confirm the milestone exists. The user creates it in the Testmo UI — the API cannot. Releases use a parent YYYY.M.0 Release milestone with … Manual Regression and … Automated Regression children; manual regression runs link to the … Manual Regression child.
Build the spec list. Read the profile from release-profiles.json and drop any skipped specs. A name the user asks to skip that is not in the profile is a typo — stop and ask rather than continuing. Turn each remaining name into ${CLAUDE_PLUGIN_ROOT}/skills/setting-up-regression-runs/specs/<name>.json, written out as a literal absolute path. Never pass ${CLAUDE_PLUGIN_ROOT} itself to creating-test-runs — in that skill it means the bitwarden-testmo-tools root, not this plugin’s.
Hand off. Invoke Skill(bitwarden-testmo-tools:creating-test-runs) and follow it, passing every spec path as its own --spec and the milestone as --milestone-name. That skill owns the dry-run, the user’s review, and --create — never run the dry-run and --create in the same turn.
Finish the multi-configuration runs. Every Mobile, Desktop, and Extension run needs a manual Testmo UI pass after creation (full has all three; partial only Mobile). Tell the user which runs still need it, using references/regression-configurations.md (opens in a new tab) and each spec’s _comment.

To add one run that is missing from an already-populated milestone, pass only that spec — the script’s per-run duplicate check leaves the existing runs alone.

Changing the suite

The specs and profiles here are shipped in the plugin; edits in the installed copy are overwritten by /plugin update. If a spec is skipped every cycle, or a new run joins the suite, change specs/ and release-profiles.json in a PR to bitwarden/ai-plugins.

Case counts drift as the repository changes. Compare each dry-run against the previous cycle; a large swing usually means a renamed folder or tag.

About this skill

Trigger

Use when setting up the QA manual regression runs for a Bitwarden release in Testmo — e.g. "set up the 2026.8.0 regression runs", "create the partial release regression suite", "create the full regression runs for this milestone". Picks the release profile (partial or full), applies any per-release exclusions, and hands the profile's shipped specs to the bitwarden-testmo-tools creating-test-runs skill. Do NOT use it for a one-off Testmo run outside a release (use creating-test-runs directly) or to write test cases.

The verbatim description from this skill’s front matter — the string an agent matches on to decide whether to load it.

Allowed tools: ReadSkill(bitwarden-testmo-tools:creating-test-runs)
Source: plugins/bitwarden-testing-tools/skills/setting-up-regression-runs/SKILL.md

Colophon for this issue

Colophon

How this issue was made

Source

github.com/bitwarden/ai-plugins, branch main, last pushed 6 November 2026.

Licence

Other — the text of every skill is reproduced unmodified, frontmatter included, under the upstream licence.

Discovery

83 skills found by walking the repository tree for SKILL.md, not by matching a directory convention. 20 distinct layouts observed: plugins/bitwarden-abacus-tools/skills/*/SKILL.md, plugins/bitwarden-adr-tools/skills/*/SKILL.md, plugins/bitwarden-atlassian-tools/skills/*/SKILL.md, plugins/bitwarden-code-review/skills/*/SKILL.md, plugins/bitwarden-delivery-tools/skills/*/SKILL.md, plugins/bitwarden-design-tools/skills/*/SKILL.md, plugins/bitwarden-designer/skills/*/SKILL.md, plugins/bitwarden-devops-engineer/skills/*/SKILL.md, plugins/bitwarden-doc-parity/skills/*/SKILL.md, plugins/bitwarden-mailcatcher-tools/skills/*/SKILL.md, plugins/bitwarden-product-analyst/skills/*/SKILL.md, plugins/bitwarden-retrospective-tools/skills/*/SKILL.md, plugins/bitwarden-security-engineer/skills/*/SKILL.md, plugins/bitwarden-shepherd/skills/*/SKILL.md, plugins/bitwarden-stripe-tools/skills/*/SKILL.md, plugins/bitwarden-tech-lead/skills/*/SKILL.md, plugins/bitwarden-testing-tools/skills/*/SKILL.md, plugins/bitwarden-testmo-tools/skills/*/SKILL.md, plugins/claude-config-validator/skills/*/SKILL.md, plugins/claude-retrospective/skills/*/SKILL.md.

Issue colours

Resolved from a deterministic hash of the owner name. Two accent tones are generated per issue and each is proven against its own ground before it ships: a single accent that passes AA on both light and dark paper is arithmetically impossible.

Heading repairs

1 repair applied to this skill so the document has one h1 and no skipped levels:

Removed “Setting Up Regression Runs”, a leading h1 that duplicated the skill title.

Spec compliance

10 editorial notes across 6 of 83 skills. They are printed in the margin of each skill rather than as errors here.

Requirements Elicitation — 2 notes
Work Breakdown — 2 notes
Auditing Hackerone Vulns — 2 notes
Assessing Requirement Readiness — 1 note
Assessing Test Coverage — 2 notes
Designing Test Plans — 1 note

Images

Images inside a skill come from the upstream repository. Where the author gave no alternative text we mark the image decorative rather than inventing a description — a plausible caption we made up is worse than none for the reader who depends on it.

Marketplace

A plugin manifest is published at .claude-plugin/marketplace.json by Bitwarden, declaring 23 plugins. It is read for editorial metadata only — never as the skill index, which is always the repository tree.

Signal

Install counts come from skills.sh. They measure downloads, not quality, and an unranked repository is not an unread one.

Agent surfaces

The whole issue is available as one markdown document at /bitwarden/ai-plugins.md, and each skill at its own .md URL.

Publication

Set by Skills Docs from the source repository. Body text is Literata at the reader’s chosen size and measure; code is Geist Mono. Nothing on this page was written by us except this paragraph.

Release profiles

release-profiles.json (opens in a new tab) lists every spec in each profile:

partial — Web (Password Manager, Admin Console, Admin Portal / SM / Providers), Old Client / New Server, Mobile (iOS, Android), and Directory Connector (BWDC).
full — the same Web, Old Client / New Server, and Mobile runs, plus CLI, 3 Desktop and 4 Extension configuration variants. No BWDC.

full is not a superset of partial. Use exactly the list in the file.

Workflow

Collect the inputs from the user: the profile (partial or full), the milestone’s exact name (e.g. 2026.8.0 Manual Regression), and any spec to skip this release. Ask for whichever is missing.
Confirm the milestone exists. The user creates it in the Testmo UI — the API cannot. Releases use a parent YYYY.M.0 Release milestone with … Manual Regression and … Automated Regression children; manual regression runs link to the … Manual Regression child.
Build the spec list. Read the profile from release-profiles.json and drop any skipped specs. A name the user asks to skip that is not in the profile is a typo — stop and ask rather than continuing. Turn each remaining name into ${CLAUDE_PLUGIN_ROOT}/skills/setting-up-regression-runs/specs/<name>.json, written out as a literal absolute path. Never pass ${CLAUDE_PLUGIN_ROOT} itself to creating-test-runs — in that skill it means the bitwarden-testmo-tools root, not this plugin’s.
Hand off. Invoke Skill(bitwarden-testmo-tools:creating-test-runs) and follow it, passing every spec path as its own --spec and the milestone as --milestone-name. That skill owns the dry-run, the user’s review, and --create — never run the dry-run and --create in the same turn.
Finish the multi-configuration runs. Every Mobile, Desktop, and Extension run needs a manual Testmo UI pass after creation (full has all three; partial only Mobile). Tell the user which runs still need it, using references/regression-configurations.md (opens in a new tab) and each spec’s _comment.

To add one run that is missing from an already-populated milestone, pass only that spec — the script’s per-run duplicate check leaves the existing runs alone.

Changing the suite

Case counts drift as the repository changes. Compare each dry-run against the previous cycle; a large swing usually means a renamed folder or tag.

Setting Up Regression Runs

Release profiles

Workflow

Changing the suite

About this skill

Trigger

Colophon

Setting Up Regression Runs

Release profiles

Workflow

Changing the suite

Appendix 71.1–71.17

ReferencesMarkdown · 1 file

Also bundledJSON · 16 files

About this skill

Trigger

Colophon

Appendix 71.1–71.17

ReferencesMarkdown · 1 file

Also bundledJSON · 16 files