Local Upload

1. Determine Intent

First, check the source path. If the user provides an S3 URI (e.g., s3://...) as the source, stop and use s3-files.md instead. This workflow is for local files only.

Parse the user’s request to route:

• Upload only? (“put this in S3”, “upload my file”, “move to AWS”) -> Path A
• Upload + make queryable? (“load this into a table”, “ingest this CSV”, “make it queryable”) -> Path B

If ambiguous and the file is structured (CSV, JSON, Parquet, TSV, Avro, ORC), ask: “Do you want this queryable as a table, or just stored in S3?”

2. Discover Local Data

1. Validate path: Confirm the file or directory exists and is readable
2. Detect format: Infer from extension (.csv, .json, .parquet, .tsv, .avro, .orc) or ask
3. Check size: ls -lh for files, du -sh for directories
4. For structured files, peek at content:
   • CSV/TSV: head -5 to check headers, delimiter, encoding
   • JSON: head -20 to check structure (records vs. arrays)
   • Parquet/Avro/ORC: note format, skip content peek

Encoding check (CSV/TSV/JSON only):

If not UTF-8 or ASCII, warn the user before upload. Non-UTF-8 files can cause downstream parsing failures.

3. Choose S3 Destination

1. Ask for target bucket or list available buckets:

   bashCopy

   aws s3 ls

2. Suggest prefix structure: s3://<bucket>/<domain>/<dataset>/<filename>

3. Confirm with user before uploading

Default: preserve original filename. Override: user specifies a different key.

4. Upload

Single file – check for existing objects before uploading (aws s3 cp silently overwrites):

If the object exists, warn the user and get explicit confirmation before proceeding.

Directory – check for existing objects before syncing. Use a bounded existence check to avoid enumerating every object under the prefix (which can be very slow on large prefixes):

If the result contains any Contents, objects exist and the user should be warned before proceeding. aws s3 sync skips unchanged files but overwrites modified ones without prompting.

Single file upload:

Directory upload:

For files over 8 MB, aws s3 cp uses multipart upload automatically. No special flags needed.

Verify upload:

5. Route Based on Intent

Path A: Upload Only

Report results and stop:

• S3 URI of uploaded file(s)
• File size and format
• Example command to download: aws s3 cp s3://... .

Path B: Upload + Table Ingestion

After upload completes, continue with the s3-files.md workflow using:

• S3 path where data was uploaded
• Detected file format
• Row/size estimate
• Encoding (if checked)

Do not reimplement schema inference or table creation – follow the S3 files workflow for those steps.

Gotchas

• aws s3 cp silently overwrites existing S3 objects. Always check first.
• aws s3 sync skips unchanged files but overwrites modified ones without prompting. Check destination before syncing directories.
• CSV files with mixed encodings (e.g., Latin-1 headers, UTF-8 body) upload fine but break downstream parsing. Always check encoding for text formats.
• Large uploads on slow connections can time out. For files over 5 GB, suggest running the upload in a screen or tmux session.
• Compressed files (.gz, .zip): upload as-is for Path A. For Path B, note the compression so the S3 files workflow can handle decompression.

Troubleshooting

References

• upload-options.md – Compression, multipart thresholds, sync vs cp tradeoffs