Ingestion

Use upsert when:

• You’re writing a single document (e.g. a sentinel doc to verify end-to-end before a bulk load).
• You’re streaming writes from user actions and each request fits in a single batch.
• You want to fully replace a document by _id (upsert always replaces the whole document on conflict — for a true per-field patch, use documents.update below instead).

Each document is a dict keyed by field name (or a DocumentRecord/UpdateDocumentRecord instance — both accept a plain dict positionally or fields as keyword arguments). _id is required and must be a non-empty unique string within the namespace. Values must match the declared schema types for the few fields actually declared in the schema (FTS strings → str, dense vectors → list[float], sparse → {"indices": [...], "values": [...]}); every other field on the document — strings, numbers, booleans, string lists — is auto-indexed metadata with nothing declared for it. Field names that start with _ or $ are rejected; field names are limited to 64 bytes.

The endpoint returns 202 Accepted (async) and the body’s upserted_count is the number of items accepted, not the number that have finished indexing.

documents.batch_upsert — bulk loads

The SDK splits documents into batch_size-sized chunks and uploads them over max_concurrency parallel HTTP connections. show_progress=True prints a tqdm-style bar. max_concurrency must be a plain int (default 4) — it no longer accepts None.

Tuning batch_size and max_concurrency

• batch_size=50 (the SDK’s own default) is the sweet spot — comfortably below the per-request cap and small enough that transient failures cost less to redo.
• max_concurrency=4 (the SDK’s own default) is a safe default for large (thousands-of-docs) loads where you’re not simultaneously embedding. Ramp cautiously above 4 — you’ll hit Pinecone or upstream embedding-provider rate limits first.
• If you’re embedding on the fly (computing vectors inside the upsert loop), keep max_concurrency low so embedding latency dominates rather than index write latency.

Document and request size caps

Hard limits in 2026-07:

• Per document: max 2 MB (serialized JSON, all stored fields combined).
• Per full_text_search string field: max 100 KB AND max 10,000 tokens. Tokens longer than 256 bytes are silently truncated by the analyzer.
• Per upsert request: max 2 MB total AND max 1,000 documents.
• Per document filterable metadata (everything not in an FTS field): max 40 KB combined.
• Schema-level: up to 100 FTS string fields per index.

If any one of these is exceeded, the batch fails as a whole. The most common limit to hit on long-prose corpora is the per-FTS-field 100 KB / 10,000-token cap on a single body field — chunking is the standard fix (see below).

Dense-vector payload size

A high-dimensional dense field can silently turn a 50-doc batch into a 5–10 MB request, which the backend will reject wholesale. If every batch fails and the error message is opaque, the first thing to try is dropping the embedding dimension before debugging schema:

• Gemini: pass config=types.EmbedContentConfig(output_dimensionality=768). The model uses Matryoshka representations, so smaller dimensions are valid truncations of the native output. 768 is usually a 4× payload reduction vs. the native 3072 and costs very little quality.
• OpenAI text-embedding-3-*: pass dimensions=768 (or similar) to embeddings.create.
• Pinecone hosted / fixed-dim models: dimension is fixed; the only levers are batch_size (halve it to 25) and per-document body size.

The async-indexing footgun

After batch_upsert returns, your documents are written but not yet searchable. The server builds inverted indexes for FTS fields and ANN graphs for vector fields in the background. A search query issued immediately will return empty matches. Schemas with multiple indexed fields (e.g. text + dense + sparse) may take slightly longer.

This is distinct from — and in addition to — pc.indexes.create(...)‘s own polling for the index itself to become ready (which it does by default before returning). Even a fully-ready index needs this separate poll after each ingest.

Always poll with a deadline before trusting the index:

Pick a sentinel query likely to hit at least one document. For a typical corpus, a single common token works (e.g. "book" for a book-reviews corpus). For a small corpus, use a term you know appears in at least one document.

Chunking oversized text

Per the size caps above, the per-FTS-field hard limits are 100 KB and 10,000 tokens. In practice, plan for the token limit kicking in first on natural prose (~5,000 English words at ~2 tokens each is the rough ceiling). Probe before ingesting at scale — chunk anything that approaches either bound, with safety margin.

Strategy: probe first, then chunk if needed.

1. Find the longest document in your corpus: max(len(doc["body"]) for doc in docs).
2. Try upserting it as-is. If the upsert errors, chunk.

Chunking pattern:

Conventions:

• Shared key prefix. First chunk keeps the original _id; subsequent chunks append #p2, #p3. Easy to parse client-side.
• Duplicate identifying metadata. Fields like title, parent_doc_id, url, or whatever identifies the logical document should be present on every chunk so queries that filter or score against those fields work uniformly.
• Deduplicate at query time. After documents.search, group matches by parent_doc_id (or strip the #p* suffix from _id) and keep the highest-scoring chunk per parent. This preserves relevance ranking while collapsing duplicates in the UI.

Updating documents

2026-07 has real per-field updates via documents.update(...) — this replaces the fetch → modify → re-upsert workaround from the old preview API. Two shapes:

By _id, patching specific fields:

By filter, patching every matching document at once:

documents.update accepts documents= (a list of per-record patches) or filter= + set_fields=/remove_fields= (a bulk patch by metadata match) — not both. set_fields merges into the existing document; fields you don’t mention are left untouched. remove_fields deletes named fields outright. For a filtered update, matched_records on the response is a point-in-time count when the server accepted the request, the same caveat as documents.delete‘s matched_records — the update itself applies asynchronously.

If you need to fully replace a document (including its vector fields) rather than patch it, documents.upsert with the complete document under the same _id is still the right tool — upsert replaces, update patches.

Deletes

documents.delete accepts exactly one of ids: [...] (1–1000 items), filter: {...}, or delete_all: true:

delete_all=True wipes the entire namespace. Use carefully.

Namespace management

Confirmed working against document-schema indexes in 2026-07 — this was not supported in the old preview API, which could write to a namespace but not list, describe, create, or delete one via the API.

create_namespace‘s optional schema parameter ({"fields": {"<field>": {"filterable": True}}}) controls which metadata fields get indexed for filtering in that namespace specifically — omitting it means the namespace inherits the index’s own metadata-indexing configuration, which for the managed indexes this skill covers is “index everything” by default (see “Filterable metadata isn’t declared in the schema at all” in SKILL.md). There’s rarely a reason to pass it explicitly unless you’re deliberately restricting which fields are filterable in one namespace.

__default__ is reserved — it always exists and can’t be created or deleted; every namespace-taking call already defaults to it when namespace is omitted, but it’s worth knowing when you see it show up unbidden in a list_namespaces() result.

describe_index_stats — also now works

describe_index_stats(filter=...) is documented as rejected on every index type (there’s no operation that returns a filtered count) — call it with no arguments.

Integrating embedding providers

If your index has a dense or sparse vector field, you need embeddings. Three common paths:

Pinecone hosted inference

Cleanest integration — no extra API keys, same client as the index.

The distinction between input_type="passage" (stored content) and input_type="query" (runtime queries) matters for models that encode them asymmetrically (multilingual-e5-large is one). For sparse learned embeddings like pinecone-sparse-english-v0, the same convention applies, and each embedding has .sparse_indices / .sparse_values rather than .values.

Batch size: ~96 inputs per embed call is the typical server limit. Loop in chunks:

Generic pattern — any third-party provider

Wrap the provider-specific call in a thin adapter so ingestion logic doesn’t know which provider is in use:

This adapter also gives you a single chokepoint for retries, rate-limit backoff, and caching — add them once in embed() rather than at every call site.

Limits to be aware of

• Bulk import (from object storage) is out of scope for this skill, not unsupported by Pinecone. Pinecone supports bulk import for document-shaped indexes in 2026-07 (JSON Lines format — see Import data (opens in a new tab)); this skill just doesn’t implement it. Load through documents.upsert / documents.batch_upsert here, or use a dedicated import skill when one exists.
• No backup/restore. If you need recoverability, snapshot your source data, not the index.
• No CMEK projects alongside any full_text_search field — such indexes can’t be created in CMEK-enabled projects.
• Indexing latency: documents become searchable in ≲1 minute typically; multi-field schemas can take slightly longer.