1{2 "skill_name": "architecting-solutions",3 "eval_type": "behavior",4 "purpose": "Prove the `architecting-solutions` skill changes the design output in the two Bitwarden-specific places the skill is load-bearing: (a) the ADR check — does the model consult `https://contributing.bitwarden.com/architecture/adr/` before recommending, and does it handle the ADR's three real branches (ADR applies, ADR conflicts with in-place code, no ADR governs) correctly; and (b) the six Bitwarden-Specific Principles — multi-client reality, dual data-access parity across Dapper and EF, open-source stewardship, self-hosted deployment constraint, V±2 client compatibility, and additive-only changes on existing endpoints. Each case is designed so a design that misses the constraint would look reasonable on its face; the impact-vs-baseline delta reflects whether the skill reliably pulls the Bitwarden constraint into the model's reasoning.",5 "evals": [6 {7 "id": 1,8
9 "prompt": "I'm adding a new background job in `src/Api` that runs a cleanup task on soft-deleted Ciphers. I want to log lifecycle events (`started`, `processed batch of N`, `finished`, `failed with error X`) so support can see what's happening in the pod logs. My first instinct is `Console.WriteLine(...)` — it goes straight to stdout, no plumbing. Reasonable, or is there a Bitwarden-specific pattern here?",
10 "expected_output": "WebFetches the ADR index at `https://contributing.bitwarden.com/architecture/adr/`, identifies the applicable logging ADR (likely `standard-output-logging`) and cites it by name, rejects `Console.WriteLine` in favor of the ADR-prescribed mechanism (structured `ILogger<T>` with correlation and level), and includes an explicit ADR reference section per the skill's artifact requirement.",
11 "expectations": [
12 "WebFetches the ADR index at `https://contributing.bitwarden.com/architecture/adr/` before recommending an approach",
13 "Identifies and cites a specific applicable ADR by name (e.g., `standard-output-logging`)",
14 "Recommends the ADR-prescribed mechanism (structured `ILogger<T>` or equivalent), not `Console.WriteLine`",
15 "Includes an ADR reference section in the response naming the ADR consulted and how it applies"
16 ]
17 },
18 {
19 "id": 2,
20 "name": "adr-conflicts-with-in-place-code",
21 "prompt": "I'm adding a small caching layer in front of a hot read path in `src/Core`. Looking at the surrounding services, they all use `MemoryCache` directly — same shape everywhere in this neighborhood. Do I just match that pattern for consistency, or is there something I need to check first?",
22 "expected_output": "WebFetches the ADR index and scans for caching / distributed-state ADRs. If an ADR governs and its guidance differs from the in-place `MemoryCache` pattern, surfaces the conflict to the human rather than either silently adopting the ADR (unscoped refactor) or blindly matching the code (ignoring the ADR). Explicitly names the tradeoff — new code should complement the existing pattern, and adopting the ADR wholesale is a separate refactor that needs human approval. Does not recommend a large refactor bundled with this change.",
23 "expectations": [
24 "WebFetches the ADR index and scans for caching / distributed-state ADRs",
25 "When an ADR conflicts with the in-place code pattern, surfaces the conflict to the human rather than deciding unilaterally in either direction",
26 "Does NOT recommend a large refactor to bring existing code into ADR alignment as part of this change",
27 "If no ADR governs, states so explicitly after actually scanning the index"
28 ]
29 },
30 {
31 "id": 3,
32 "name": "no-adr-governs",
33 "prompt": "I need to name a new custom HTTP response header we return from `POST /accounts/login`. It carries a UUID for session-scoped telemetry correlation. Naming options: `X-Bitwarden-Session-Id`, `X-Session-Id`, `Bitwarden-Session-Id`, `Session-Id`. Preference? Is there anything I need to check before picking?",
34 "expected_output": "WebFetches the ADR index and scans it. Concludes no ADR governs HTTP header naming and states that explicitly (not a silent skip). Offers a naming recommendation grounded in general practice (RFC 6648 / no `X-` prefix per modern convention) rather than fabricating a Bitwarden-specific rule. Includes an ADR reference section explicitly noting the absence of applicable ADRs.",
35 "expectations": [
36 "WebFetches the ADR index at `https://contributing.bitwarden.com/architecture/adr/` and scans it",
37 "States explicitly that no ADR governs this concern, rather than silently skipping the ADR check",
38 "Includes an ADR reference section that documents the absence of an applicable ADR",
39 "Gives a naming recommendation grounded in generic best practices (e.g., RFC 6648, avoiding `X-` prefix) rather than fabricating a Bitwarden-specific rule"
40 ]
41 },
42 {
43 "id": 4,
44 "name": "multi-client-shared-code-storage",
45 "prompt": "In `libs/common`, I'm adding a `RecentSearchesService` that remembers the last 20 vault searches per user. Plan is to persist them via `localStorage.setItem('recentSearches:{userId}', JSON.stringify(items))` on every search, and read them back on service init. That's simple and works — anything I'm missing?",
46 "expected_output": "Rejects direct `localStorage` in `libs/common`. Reasons: Chrome MV3 service workers (the browser extension's background context) have no `localStorage` — only `chrome.storage`. The CLI runs headless with no browser APIs at all. Desktop uses Electron's own storage layer. Shared code in `libs/common` must go through the platform-abstracted storage service so each client can provide its own backing. Recommends using the existing `AbstractStorageService` (or the storage abstraction already in place) rather than introducing a new one.",
47 "expectations": [
48 "Rejects direct `localStorage` (or any `window`/DOM API) in `libs/common`",
49 "Cites at least two of the four client runtimes (web / browser extension / desktop / CLI) as having incompatible storage constraints",
50 "Specifically calls out that browser extension MV3 service workers or the CLI's headless runtime lack `localStorage`",
51 "Recommends routing through the platform-abstracted storage service already in place, not inventing a new abstraction"
52 ]
53 },
54 {
55 "id": 5,
56 "name": "dual-data-access-parity",
57 "prompt": "I need to add a nullable `LastSyncedRevisionDate` datetime column to the `User` table. My plan: add the migration script, update the EF entity in `src/Infrastructure.EntityFramework/Auth/Models/User.cs`, update the fluent config, and add a mapper. Then bump the `User` model and the repository interface. That covers it, right?",
58 "expected_output": "Rejects the EF-only plan. Bitwarden ships both a Dapper stack (`src/Sql/dbo` — stored procedures like `User_ReadById`, `User_Update`, `User_Create`, plus `dbo.UserView` and the raw table SQL) AND an EF stack. Both must move together — the column has to be added to the table SQL, threaded through the affected stored procedures, added to `UserView`, and the EF entity + fluent config + migration + mapper all need to move in lockstep. Frames this as the dual data-access parity invariant, not just 'don't forget the SQL side.' Does not suggest deprecating Dapper or migrating off it as part of this change.",
59 "expectations": [
60 "Rejects the EF-only plan and calls out that Dapper stored procedures under `src/Sql/dbo` must also be updated",
61 "Names at least one specific SQL artifact that needs to change: stored procedure(s) touching `User`, `dbo.UserView`, or the raw table SQL script",
62 "Frames this as a Bitwarden-specific parity requirement (dual data-access architecture), not just generic ORM completeness",
63 "Does NOT recommend deprecating or migrating off Dapper as part of this change"
69 "prompt": "For an internal-only 'staff pilot' feature flag, I want to hardcode a list of Bitwarden staff account IDs directly in `src/Core/Services/Implementations/FeatureFlagService.cs` — something like `private static readonly HashSet<Guid> _staffPilotUserIds = { ... };` — so we don't need any config plumbing. Reasonable?",
70 "expected_output": "Rejects hardcoding the IDs in source. Cites the open-source nature of `bitwarden/server` — the moment the PR opens, the IDs are public, and git history preserves them even after a later commit removes them. Even non-secret identifiers reveal internal targeting mechanics to anyone reading the repo. Recommends an out-of-repo configuration mechanism: environment variable, config service, DB-driven allowlist, or a feature-flag service that already handles targeted roll outs.",
71 "expectations": [
72 "Rejects hardcoding the user IDs in source code",
73 "Cites the open-source / public-repo nature of `bitwarden/server` as the specific reason (not just 'hardcoding is bad')",
74 "Notes that git history preserves the values even if a later commit removes them",
75 "Recommends an out-of-repo configuration mechanism (env var, config service, DB-driven allowlist, or feature-flag service)"
76 ]
77 },
78 {
79 "id": 7,
80 "name": "self-hosted-degradation",
81 "prompt": "For the new 'Password Health Insights' feature, I'm planning to have the vault clients call a Bitwarden-hosted scoring service at `https://ml-scoring.bitwarden.com/api/score` — each password gets scored on breach probability. Clients call directly, cache locally. That's the cleanest architecture. Anything I'm missing?",
82 "expected_output": "Rejects the plan as-is. Self-hosted customers run `bitwarden/server` on their own infrastructure and cannot reach an internal `ml-scoring.bitwarden.com` endpoint. The design must include a graceful degradation path: (a) the feature is disabled on self-hosted with a clear UI signal, (b) scoring runs client-side with no server dependency, or (c) there's a documented opt-in cloud proxy or on-prem alternative. Bonus if it questions the client-to-internal-domain shape (usually server-mediated) and flags this as an architectural precedent worth surfacing to the Architecture group.",
83 "expectations": [
84 "Recognizes that self-hosted deployments cannot reach a Bitwarden-hosted-only endpoint",
85 "Requires the design to include a graceful degradation path for self-hosted (disabled with UI signal, client-side alternative, or opt-in cloud proxy)",
86 "Does NOT accept 'self-hosted just won't have this feature' without an explicit UX statement — the design has to acknowledge and handle the delta",
87 "Bonus: flags this as an architectural precedent (client-to-internal-domain shape) worth surfacing to the Architecture group"
88 ]
89 },
90 {
91 "id": 8,
92 "name": "v-plus-minus-2-response-shape",
93 "prompt": "The `GET /accounts/profile` response currently returns `email: string`. For the domain-scoped identity work I'm doing, we want it to return `email: { address: string, domain: string, verified: boolean }` instead — a structured object. I'll update the response DTO, update the client models, ship it. Ok?",
94 "expected_output": "Rejects the shape change. This is a breaking change on the response of an endpoint that older clients still call. Bitwarden's V±2 policy requires the server to support clients up to two major versions behind, and those clients expect `email` to be a string. Recommends the additive path: keep `email: string` on the response and add a new sibling field (e.g., `emailIdentity: { ... }`) that new clients read; deprecate `email` in server code but keep serving it. Cites V±2 as the invariant that governs the choice.",
95 "expectations": [
96 "Rejects changing the shape of the existing `email` field",
97 "Cites the V±2 policy (server supports clients up to two major versions behind) as the specific reason",
98 "Recommends an additive change: keep `email` as-is, add a new sibling field for the structured data",
99 "Does NOT recommend 'just bump the API version' or 'clients can update' — older clients still in the field must keep working"
105 "prompt": "I'm updating the existing `POST /organizations` endpoint. For a compliance requirement I need to add `dataResidencyRegion` to the request — every new org needs it to route data properly. Plan: add the field as required on `OrganizationCreateRequestModel`, update the clients to send it, done. Reasonable?",
106 "expected_output": "Rejects making the new field required on the existing endpoint. Because Bitwarden has no formal API versioning, existing endpoints must stay backwards-compatible on the request side — any older client that hits this endpoint without the new field would break. Recommends: make the field optional with a server-side default, gate the behavior on a header or client capability flag that only new clients set, or mint a new endpoint (e.g., `POST /organizations/v2`) and route new clients to it. Cites the additive-only invariant explicitly.",
107 "expectations": [
108 "Rejects making the new field required on the existing endpoint",
109 "Cites the 'no formal API versioning' / additive-only-changes principle explicitly (not just generic 'be backwards-compatible')",
110 "Proposes a concrete alternative that keeps older clients working: server-side default, optional field with fallback, new endpoint, or client-capability gating",
111 "Does NOT accept 'just require the newer client' — V±2 and no formal versioning together rule this out"