Subchapter 10.51
helpers/file-upload-recovery.mdMarkdown8 KBView on GitHub
After partial File creation, retain the source. Never rerun creation, infer MI from redacted GET, or recover by recreation/deletion/reset/new roles. Generic File MI reuse/provenance guards remain unchanged.
python helpers/file_source.py --input <original-approved.json> --cleanup-receipt-dir <private-receipts> --upload-receipt-dir <separate-private-empty-directory>Use the private directory helper; no automatic directory/ ACL changes. The required journal retains original approval/context, actual conditional source PUT ACK/request ID/ETag, exclusive pre-POST attempts and separate upload responses. Never edit/remove/reconstruct entries. Integrity isn’t a service signature: retain exclusive control, no concurrent writers.
File CLI creation requires protected --cleanup-receipt-dir and
--upload-receipt-dir pre-write; separate directories, no guessed paths. Plan with
--execution-output <new-private-plan.json>; show reference/counts, not
hash-bearing envelopes. Plan JSON/library and non-File policies are unchanged.
Persistence failure stops requests. An attempt without result stays uncertain,
even if POST wasn’t sent. Missing ACK/ETag cannot come from GET. Journals neither
replace cleanup provenance nor authorize deletion.
Without originals: retain source, file-upload-provenance-missing, not
“recreate required”.
Closed request:
{"schema_version":"1.0","receipt_directory":"<absolute-original-private-directory>"}python helpers/file_upload.py --plan resume-request.json --execution-output <new-private-resume.json>
python helpers/file_upload.py --input <newly-approved-resume.json>Planning is read-only: an unapproved, closed version 1.0 resume-file-uploads
envelope binds original plan/creation ACK/journal digests, owner, exact
never-attempted ordinals and receipt directory;
cleanup_approved stays false. Review and newly approve these uploads. An empty
eligible list needs no mutation or execution; use the returned observation.
Never edit ordinals/hashes to authorize attempted files.
Execution validates the original unchanged approved corpus (paths, full hashes,
size, timestamps, mode, metadata), tenant/subscription/principal, source URL/API/
definition/ACK ETag, and applicable CU account/Search MI/role/network snapshots.
It neither acquires CU keys nor changes authentication. It invokes only existing
metadata GETs and direct File upload POSTs using the original Search identity;
no source PUT, source update, new role, indexer operation, deletion, KB or model call.
POST targets the original validated
/knowledgesources('<name>')/files?api-version=2026-08-01-preview&pageSize=200
multipart contract; approval/privacy gates apply.
Continuation excludes previously attempted files, including exhausted 429 retries, timeout/409/5xx and interrupted attempts. Only the original operation may retry its received File-upload 429 once, as below. An immediate or delayed empty inventory never proves in-flight termination for unknown outcomes. Positive exact file-ID/ marker/hash/size readback can prove completed ingestion, but never invents an upload ACK or source ownership. Drift blocks; never repair by write.
File upload is synchronous: extraction, chunking, embedding, indexing and metadata
persistence finish before success. No File indexer, schedule or asynchronous status
polling. Each sequential POST has a 180-second processing/response allowance,
separate from the 60-second read budget; stricter approved outer deadlines win.
Timeout is not remote cancellation. Retain 200/201 ACKs without relisting source/
full inventory after every healthy file. One final bounded inventory checks the
batch; follow validated @odata.nextLink exactly, without rebuilding parameters.
A per-file 415 permits independent continuation. Auth/access, deadlines,
persistence, drift and unknown outcomes stop new uploads. Repeated 429, or later
429 after the operation’s one retry opportunity, stops without more reads/sleeps.
Preserve prior confirmed files; never hammer the remaining corpus.
429 recovery supplies one server-respecting delay/read
opportunity, at most 30 seconds waiting/60 seconds per complete read sequence,
including all pages and response reads. Overlong delays stop, not shorten.
Journal-retained UTC backoff blocks resume planning/execution before any network;
fresh approval never waives it. file_batch.backoff reports waiting/unresolved/
elapsed. Legacy 429 without timing stays unresolved; no invented deadline.
--no-progress disables observations. Otherwise stderr emits counts, a waiting
event with the bounded HTTP-429 delay, and terminal state. During POST, an immediate
and then five-second content-free heartbeat reports ordinal/total/attempt, elapsed
time and already ingested count: “1/25 ingested; processing file 2/25”. This observer
never makes requests. No within-file percentage, ETA, filenames or private content.
Increment completion only after ACK/metadata proof and required receipt persistence.
file_batch.files records bounded relative names/full hashes, upload state,
verification state and sanitized request IDs. Counts distinguish:
accepted (original 200/201 ACK), confirmed (exact file-ID/markers without
reported error), failed, pending (ACK but not confirmed),
unverified (uncertain attempt), and not_attempted.
Accepted overlaps verification counts; do not sum it with them.
ingested counts synchronous completion proved by valid ACK metadata or exact
completed-file readback. A status-only ACK is insufficient. Original ACK file IDs
remain binding even if its remaining metadata is incomplete. Failed readback stays
partial while preserving known prior ingestion; retrieval stays unverified.
File completion does not prove OCR quality or KB retrieval.
Report “24 ingested, one failed/unverified” only with this proof; otherwise distinguish accepted/pending. Preserve outcomes, not a blanket blocker. A subset handoff must disclose pending files may appear and subset isolation is not proven; any KB use still needs its existing separate approval. No automatic KB writes.
The official File guide identifies upload HTTP 429 as a full processing queue and recommends bounded parallelism/exponential backoff; Retry-After is not guaranteed. Only this File upload route permits one same-operation retry across the batch, not per file. No private throttle marker/error text is required. Honor valid Retry-After; missing/invalid metadata uses the diagnosed first backoff step of one second. Shared 30-second wait/60-second complete preflight limits apply. After waiting, revalidate unchanged source/corpus and one fresh bounded inventory. Positive proof avoids another POST; absence is not the retry authority: the received queue-rejection 429 is. Persist exclusive original/retry attempt and result records when journaling. Retry the original URL, token, multipart bytes/ boundary and approved payload; source creation happens once.
The MI canary’s stricter one-upload approval forbids this retry. No PUT/DELETE/
update retry, new roles or auth substitution. Each upload creates a new file ID,
even for the same filename; no filename-idempotency key exists. Timeout/504/409/
5xx remain unknown, never replayable from empty inventory or elapsed time.
file-upload-retry-safety-unproven retains that blocker. Exhausted files stay
failed; continuation uploads only never-attempted files under fresh approval,
not another attempt at an exhausted file.
Authorities: failure/conflict/uncertainty only. File upload/troubleshooting (opens in a new tab).