Rp Source CSV

48 skills · 466 min

This is not a stage in the migration flow — it is a reference consulted by two stages:

• rp-discovery consults the Capture section to sample the files and produce the canonical source-profile.md + source-schema.json.
• rp-import-codegen consults the Read contract section to generate a reader that extracts CSV data correctly (grouping, derived entities, empty-vs-absent) into durable project-local files for the later import step.

rp-execute-import never consults this skill — by the time execution runs, the CSV-specific knowledge is already baked into the generated reader.

Platform identity

• Source platform: csv — a file-provided flow, not a URL probe.
• Set "platform": "csv" in the emitted source-schema.json, and record the detected vendor per file in sourceMeta.vendor / sourceMeta.sourceFiles[].
• Record sourcePlatform=csv, sourceMode=files_only, and every input file in fileInputPaths in orchestration/decisions.json. Preflight requires no source env keys for csv; the files_only branch enforces a non-empty fileInputPaths instead.

Three layers

1. A generic core — lib/csv-parse.js (parse, dialect detection, type sniffing) and lib/csv-layout.js (layout inference, derived entities). Knows nothing about commerce and handles any CSV, including hand-rolled ones.
2. Per-vendor overlays — declarative JSON profiles under vendors/ that specialize the core with a vendor’s fingerprint anchors, known layout, column→Wix pre-map, and quirks. v1 ships shopify, woocommerce, magento, bigcommerce.
3. A fingerprint detector — lib/csv-fingerprint.js picks the overlay from the header set, or honors a user-stated vendor, and falls back to custom below the threshold.

The overlay is advisory, never authoritative. The authoritative record of which columns a file contains is always the header row read at discovery time. The overlay only (a) identifies the vendor and (b) pre-fills mappings the LLM would otherwise propose.

Capture (discovery-time)

Sampling the files to learn their shape — not a bulk load.

Before capture, verify migrations/<project>/config/source.csv.env. Create it if missing, using empty values for the user to fill:

Unlike source.wordpress.env, source.csv.env is not secret-bearing — it holds delimiter/encoding/vendor/rewrite hints only, no credentials. Regular file handling applies. There is no acquisition-mode fork and no credentials request for a CSV run.

1. Run the deterministic capture script from this skill’s directory (the folder containing this SKILL.md; see CONVENTIONS.md):

   Copy

   node scripts/csv-discovery.js --file <path> [--file <path> ...] \
     --out-dir <migrations-root>/<project>/data/csv-discovery

   --file is repeatable; pass the whole file set in one run so roles and split files are resolved together. For long runs, pass --progress-log <path> and poll it per CONVENTIONS.md#progress-log-polling.

   Useful options: --vendor (override detection), --delimiter / --encoding (override detection), --role <path>:<role> (name a file’s entity when the script cannot), --allow-header-superset (union split files whose headers differ), --scan-rows.

2. Per run the script: reads each header → detects the dialect → fingerprints the vendor per file → assigns a role per file and concatenates split files → resolves the layout (overlay or generic classifier) → derives column-values entities → diffs the real header against the matched profile → samples head and tail rows → infers per-column type/cardinality/requiredness/examples → writes the raw dump. Because headers are read fresh every run, discovered fields are never limited to what the profiles know.

3. It writes into --out-dir:

   • README.md — the index: files, vendor detection with evidence, layout, entities, profile drift, warnings.
   • fileset.json — machine-readable sourceFiles[], streams, vendor, dialect, layout, drift, mapping hints, entities and relations. This is what rp-discovery synthesizes source-schema.json from.
   • raw-capture.json — per-column detail plus head/tail samples.
   • <role>--<entity>.md — one per entity.

4. Honor the halt. When the capture reports halt: true (ambiguous layout, unknown file role, conflicting split-file headers, mixed vendors for one role), stop and ask the user the question named in the warning rather than synthesizing a schema from a guess.

5. Record counts are scan-based. The script streams each file in full for its row count and retains the first --scan-rows rows (default 5000) for inference. When raw-capture.json reports scan.truncated, say so in source-profile.md instead of presenting a sampled count as exact.

The raw capture is evidence, not a hand-off artifact. rp-discovery synthesizes it into the canonical artifacts and records traceability pointers (rawDiscovery, per-entity rawFile).

Layer 1 — the generic core

lib/csv-parse.js is the CSV analogue of rp-source-wordpress‘s wp-http.js: dependency-free and reused by both the capture script and the generated reader so the sampler and the importer parse identically. It handles what breaks a naive split(','):

• delimiter detection (, ; \t |), scored by field-count consistency and field count, since a semicolon file whose text fields each contain a comma splits perfectly consistently on , too. CSV_DELIMITER always wins.
• RFC-4180 quoting: embedded delimiters, embedded newlines, "" escapes. A quote opens a field only at field position 0; a"b is three literal characters.
• encoding / BOM: the mark is stripped before the header is read. A UTF-16 BOM fails fast naming CSV_ENCODING rather than producing mojibake — full transcoding is out of scope.
• line endings: \n, \r\n, and lone \r, including a \r\n split across a chunk boundary. A newline inside quotes is data and is preserved verbatim.
• empty vs absent (below).
• streaming: rows are yielded without loading the file.

Empty vs absent is detected, not assumed

Whether an empty cell means “absent” is a property of the writer’s quoting policy, not of the datum: exporters that quote every field (Excel, Magento) can only ever produce a quoted empty, and exporters that quote nothing unnecessary can only ever produce an unquoted one.

So the parser returns strings only, never null, with an optional parallel quoted[] array, and detectEmptyPolicy decides once per file:

• present-if-quoted — the file contains both forms, so it genuinely distinguishes them.
• always-empty — it contains only one form, so the distinction carries no information.

The policy is recorded in sourceMeta.dialect.emptyPolicy and applied through coerceEmpty(value, isQuoted, policy). Generated readers must use it, so a required Wix field is never fed an empty string the source did not actually have. This also keeps split files (shape C) consistent when two parts were written by different tools.

Layer 2 — vendor overlays

Each supported vendor is a declarative JSON profile under vendors/. Overlays are data, not code, because they are numerous, additive, and pure lookup: adding or updating a vendor is a data-file edit plus a header fixture, with no change to any skill logic.

Column matching is normalized + alias-based: lowercase, NFKC, strip punctuation and whitespace. Body (HTML) → bodyhtml, and Option1 Name ≡ Option 1 Name. This absorbs casing changes and renames without a code change.

columnGroups is the one thing the data cannot supply. Shopify’s Image Src and Variant SKU both vary within a product group, so only a declaration can say that one is a third collection and the other is the variant. Generic derivation still covers every column the overlay does not name — including new columns a vendor adds later.

An overlay feeds two stages:

• Discovery — layout + quirks resolve grouping deterministically, avoiding the generic ambiguity halt. Layout traps are stable and hard to infer, which is what makes them worth encoding.
• Mapping — columnMap pre-fills sourceMeta.mappingHints[] so rp-mapper reviews and corrects instead of authoring from scratch.

A custom file has no overlay: it runs core-only, and the mapper leans more on the LLM and the user at the same review checkpoint. A fully stale overlay degrades to exactly that experience — the floor, not a failure.

Layer 3 — fingerprint detection

lib/csv-fingerprint.js identifies the vendor from the header row:

• Anchors never alias-match. Aliases are permissive for mapping; permissiveness is poison for identification — Shopify’s variant.sku aliases include a bare SKU, which every WooCommerce and Magento export also has.
• Ratios, not counts. Anchor ratios are bounded 0..1, so a wide CSV cannot inflate a match. Do not reintroduce a header-count normalization: dividing by header count penalizes wide files and inflates narrow ones, which is what lets a 6-column near-miss flip to a false match.
• Ties, sub-threshold scores, and files with fewer than two strong anchors → custom.
• A near miss (a drifted vendor export that lost a required anchor) is reported as nearMiss so it becomes a user question, never silence.
• CSV_VENDOR / --vendor overrides detection, but detection still runs: a disagreement is recorded as conflict: { stated, detected } and surfaced.

Layout — how rows group

lib/csv-layout.js resolves the layout: flat, grouped-by-key (blank-key or repeat-key continuation), or sectioned. It runs for custom files and as a fallback when a matched overlay does not pin the layout — including when the overlay’s declared groupKey is missing from the header, which is recorded as overlay-groupkey-missing rather than crashing.

A pinned layout is verified, not believed

An overlay’s continuation is a claim about the rows, not a fact about the vendor, and it is checked against the sample before it is used. Shopify is the reason: most exports blank the Handle on continuation rows, but some repeat it on every row. Pinning blank-key against a repeating file fails silently — no key is ever blank, so no row ever continues a group, and a 50-product file is captured as 82 single-row products with 32 blank titles.

The check fires only when the two modes would group differently: with one row per group both readings produce the same boundaries, and reporting drift there would be noise.

On rejection the generic classifier is consulted about the disputed field only. Which column groups the rows is the part vendors do not change, and the overlay’s groupKey has just been confirmed present in the header — so a generic candidate naming that same column corroborates the observed mode even when the classifier as a whole halted, because its halt is about picking a key, not about this one. When corroborated, the continuation is corrected and the rest of the overlay (entity names, columnGroups) is kept, with source: overlay-continuation-corrected and the classifier’s confidence instead of the pin’s 1. When no candidate groups on the declared key either, nothing is trustworthy enough to pin and the whole inferred layout is used (inferred-after-overlay-mismatch).

Both outcomes append to layoutConflicts[] and raise a warning — never a silent correction. Do not “fix” this by re-pinning repeat-key in the overlay: exports vary, so that only moves the same silent failure to the other half of them.

Deterministic-first: when signals are clear it sets the pattern; when they are ambiguous it returns pattern: 'unknown' with halt: true and the ranked candidates, and discovery stops to ask the user.

Signals worth knowing:

• blank-key requires the key column’s non-blank values to be unique — a key never reappears after an intervening key. Two candidate columns that produce the same group boundaries (Shopify’s Handle and Title both go blank on continuation rows) are equivalent labels, not an ambiguity; different boundaries are a genuine ambiguity and halt.
• repeat-key additionally requires a supporting parent column — some other column constant inside each run and different between runs. Without it, a flat product list merely sorted by category is indistinguishable from a grouped one, and would be split into bogus parents and children.
• sectioned requires a column-population skew: partitioned by the discriminator, some other column’s blank rate must differ by ≥ 0.7 between partitions. Without that test every Published or Tax class column is a false discriminator.
• When a blank column’s blankness is a pure function of the discriminator value, the blank-key reading is already explained by the sectioning and does not compete with it — this is what keeps a WooCommerce export from looking ambiguous.

Parent vs child columns

A column belongs to the parent, to the child, or to a declared collection. The rule is within-group variance, not blankness:

• A column that varies inside a group in ≥80% of multi-row groups is a child column.
• A column blank on every continuation row and populated on the head row is a parent column.
• Anything else is reported as ambiguousColumns rather than guessed.

The naive rule — “child columns are the ones populated on continuation rows” — is wrong for the most important case: a Shopify product’s first variant lives on its group head row, and a single-variant product has no continuation rows at all.

Precedence: overlay columnGroups assigns the entity, generic derivation covers everything the overlay does not name, and a disagreement is recorded in layoutConflicts[] with the overlay kept. An overlay can never lose a column — that is what “advisory, never authoritative” means in practice.

layoutConflicts[] carries both kinds of disagreement, because a reviewer asks the same question of both — where did this capture stop believing the overlay? Column-level entries name a column; layout-level entries (overlay-continuation-mismatch, overlay-groupkey-missing, overlay-discriminator-missing) carry a kind and a resolution.

A child’s recordCount is the number of rows that populate at least one child column, not the group’s row count: a continuation row carrying only an extra image is an image, not a variant.

Entity origins — rows, row groups, and column values

An entity in a CSV migration comes from one of three origins, recorded per entity so the reader knows how to materialize it:

• file-rows — one row = one entity (a flat categories.csv; also a sectioned file’s parent or child rows, with filter: { column, includeValues | excludeValues }).
• row-group — a group of rows = one entity (a Shopify product, its variants, its images), with groupKey, continuation, and columnGroup where relevant.
• column-values — the distinct values of a column = an entity set. This is how categories and tags arrive in every named vendor’s export.

For column-values, capture records the source column, both separators, and the distinct-value count; the reader emits a synthetic entity file (one record per distinct path, ancestors included) plus the linking relation. Because the result is an ordinary entity from rp-mapper‘s point of view, the mapper maps it with no special handling and the existing dependency ordering creates categories before products.

Non-obvious rules the implementation depends on:

• Split the multi-value separator first, then the hierarchy separator. WooCommerce’s Clothing > Shirts, Sale is two categories, one nested. The other order produces a category literally named Shirts, Sale.
• A separator inside a value is escaped by quoting inside the cell ("Home, Garden > Tools", Sale), so the inner split is RFC-4180 aware too.
• Records are emitted depth-ascending, so a parent is always created before its child.
• Only the leaf is linked to the product (linkPolicy: "leaf"); whether ancestors are also attached is a mapping decision.
• Only group-head rows are read, so a Woo variation row does not double-count its product’s categories.
• Generic detection for custom files is deliberately narrow: > is auto-detected but / is not (it matches every URL column), markup-bearing columns are skipped, a multi-value split must collapse the distinct count, and a column is only auto-derived when its name says it is a taxonomy. Everything else is proposed to the user.

Set hierarchical: true on a nested derived entity. A hierarchical source taxonomy mapped to a flat Wix target triggers the mapper’s mandatory faithfulness-ledger entry; without the flag the flatten happens silently.

File sets — roles and concatenation

lib/csv-fileset.js resolves a list of input paths into logical entity streams. Four shapes occur in practice; A, B and C are v1, D is v1.1:

Role assignment walks a ladder, recording roleSource at each step: explicit --role → vendor fileRoles.filenameHints → generic filename tokens (whole tokens only, last token wins, so product-images.csv is media) → header anchors → unknown with a halt. A role is never invented from the filename stem; an entity map full of weird-export entities is worse than one honest question.

Same-entity concatenation (shape C) requires the same normalized column-name set:

• reordering, case, whitespace and BOM differences are tolerated — names are the identity, and each part keeps its own columnOrderMap;
• a subset/superset header does not silently concatenate. It is legitimate (a re-export after the vendor added a column) but it changes what a blank cell means for half the rows, so it halts with the exact diff; --allow-header-superset unions the parts and records driftAcrossParts;
• files sharing a role but only a small column overlap (Jaccard < 0.9) are a role-collision, not two parts of one export;
• part order is a numeric-suffix-aware natural sort, so part10 follows part2;
• partOf and rowOffset are recorded so any extract row traces back to its file.

Mixed vendors are allowed across roles (Shopify products plus a hand-rolled categories.csv is a real bundle) but two files of the same role from different vendors halt.

Read contract (codegen-time)

What a generated CSV reader must get right. Codegen selects it when source-schema.json.platform === "csv".

The reader is an extractor, not an in-memory loader: it parses with the shared parse core, applies the resolved layout to group rows into per-entity records, and writes durable per-entity NDJSON record streams (<entity>.ndjson, one record per line) plus a manifest.json into data/source-extract/ that rp-execute-import reads.

Emit records as they are grouped, streaming through ndjson.js‘s writer — do not accumulate an entity’s records in an array and serialise at the end. A 100k-row export must never require the whole entity in memory, and an interrupted extract should leave a valid readable prefix. See rp-import-codegen -> “Record streams are NDJSON”.

• Emit canonical records, or convert with lib/canonical-record.js — never hand-write the rename. rp-target-wix/lib/wix-build.js consumes the vendor-neutral vocabulary declared in wix-target-spec.js (description, brand, weight), while a reader naturally names fields the way its vendor thinks (bodyHtml, vendor, weightKg). lib/canonical-record.js holds that rename as per-vendor data (READER_FIELD_MAPS) and self-validates at require time, so a typo is a load error instead of a field that silently never reaches Wix. Vendor it alongside csv-parse.js and call it; do not re-derive the mapping in a project-local transform. It is regression-locked with the builder against 220 payloads from two live-verified imports (tests/mapping/wix-build-oracle-test.js).
• Reuse lib/csv-parse.js — do not regenerate it. Vendor a copy into the project (e.g. src/lib/) and import from it, exactly as the WordPress reader vendors wp-http.js. The reader then carries only per-project orchestration: which entities to emit, the grouping loop, type coercion, and transform glue. One tested parse core is what makes the sampler and the reader agree.
• Grouping is the reader’s core job. Replay the resolved layout: a blank key extends the current group; a discriminator column routes parent vs child rows, resolving parentRefColumn (Woo’s Parent carries id:123 or a SKU, and does not guarantee that a variation follows its parent).
• Iterate the file set by role, treating partOf entries as continuations of the same logical stream and honoring each part’s own column order.
• Materialize derived entities: collect the distinct values of a column-values column (and their ancestors when hierarchical) into their own entity file, depth-ascending, and emit the linking relation. Do not invent Wix ids at extract time — the import stage resolves them through the ImportCrosswalk.
• Coerce per inferred type, and apply coerceEmpty with the recorded emptyPolicy so required Wix fields are not fed empty strings.
• CSV values are plain text and are written as-is — no entity-decoding step unless a specific vendor is known to double-encode.
• No auth, no pagination, no rate limits: the entire WordPress transport section collapses to “open the file once”.

Media

Image columns hold either public URLs (Wix-reachable) or local paths / private URLs (not reachable — Wix Media import fetches from Wix servers, the same concern as a WordPress localhost source). Capture flags these as media-reachability warnings. Record the note in source-profile.md and offer the two choices:

• rewrite via CSV_MEDIA_URL_REWRITE_FROM / CSV_MEDIA_URL_REWRITE_TO, or expose the files at a public HTTPS URL, or
• skip/defer media import and continue with non-media entities.

Schema shape

source-schema.example.json (in this skill folder) is the template rp-discovery follows when emitting migrations/<project>/source-schema.json. It is a shape to follow, not a strict schema to validate against. Keep the platform-agnostic core stable; push CSV specifics (sourceFiles, vendor, dialect, drift, mappingHints, joins) into sourceMeta, and give every entity an origin.

Contract tests

Contract tests live in the repo’s tests/ tree, not in this bundle — a fixture shipped to a partner is dead weight. Run them from the repo root:

Or individually: node tests/source-csv/csv-parse-contract-test.js (also csv-fingerprint-, csv-layout-, csv-fileset-). Fixtures are in tests/fixtures/csv/.

They cover the dialect/quoting/BOM/line-ending/empty-policy matrix and chunk-boundary feeding; per-vendor header fixtures, the custom fallback, near misses and drifted headers; the layout patterns plus their false-positive guards, the ambiguity halt and the verification of a pinned continuation mode in both directions; derived-entity split order, ancestors and the URL guard; and role assignment, split-file concatenation, header-superset halts and part ordering.

When a vendor’s format drifts, fix vendors/<vendor>.json and add a header fixture to tests/fixtures/csv/headers/ — no skill logic changes.