Publication Lifecycle

The two halves of this chapter are not equally hard. Publishing is a call three models get right unaided; retiring is the sequence none of them ever got right. The weight is on retirement deliberately.

Publishing: short, because the trap is the report, not the call

An asset in a project’s inventory is not in the catalog until it is published. A data source run adds it to the inventory; if the source has publishOnImport: false, consumers still cannot see it.

A data product publishes through the same call with --entity-type DATA_PRODUCT.

Do not report what the change set returned. Three things about a fresh listing are misleading, all measured:

So the publish is not finished until you have read it back:

Report the status and the id from GetListing. You MUST NOT tell a user an asset “is already published and discoverable” without having read a listing with status: ACTIVE in this session. That exact claim was made about an asset with zero listings in the domain, and a user who trusts it sends another team to subscribe to something that does not exist.

An unpublish leaves the listing readable at status: INACTIVE rather than removing it, so “is it still published” is a question about the status, not about whether the object resolves.

A failed publish is not retryable unchanged

If a publish fails, retrying the same call fails identically: the attempt is tied to the asset’s current revision and the service hands back the earlier failed attempt. Give the asset a new revision first, then publish again.

Retirement: the ordering, and both ways it goes wrong

This is the sequence. It is not optional and it is not reorderable:

Step 0 is the one confirmation in this sequence, and it is mandatory. Deleting an asset that has live subscribers removes access other teams are using, so the plan goes in front of the user before anything is revoked: every subscription id, the project that loses access through each, and the listing and asset that will be deleted. Then stop and wait for an explicit yes in a later message. A request that already says “delete it even though they are subscribed” does not replace this, because the user has not yet seen who is on the list. If the asset has no live subscriber, there is nothing to confirm: go straight to step 3.

An advance yes. A request that already names every subscription to revoke (by id), the project that loses access, and the asset is the yes, given with the list, so do not stop to ask it again. It does not skip the check: run step 0 anyway and compare. If the real subscribers are not exactly the ones named (one more, one fewer, or held by a different project), the user confirmed a different list: show the real plan and stop, as if nothing had been confirmed.

Two opposite failures are both common, and half of them need the procedure rather than a warning:

• Stopping in the wrong place. Asking a vague “are you sure?” before you have the plan, so the user answers without knowing who loses access; or asking again after the yes, at the settlement wait or before the unpublish. One stop, at step 0, with the plan. After the yes, run steps 1 to 5 through, then report what you revoked, whose access went, and what you deleted.
• Deleting first. DeleteAsset is permitted while the listing is published and a grant is live. Nothing protects the ordering. What survives is a listing at status: ACTIVE whose entity no longer resolves, visible to consumers and pointing at nothing.

Step 2 is where it actually breaks

RevokeSubscription returns REVOKED immediately while the grant is still being torn down in the background. In that window DeleteListing is refused, and its refusal names every blocker:

Three traps in the wait itself:

• Wait for the grant to disappear, not for a status. A grant that is still listed blocks DeleteListing whatever it reads, and straight after the revoke it still reads COMPLETED, the same status as a live grant. Poll for a status and you stop too early on COMPLETED, or never stop on GRANTED, which does not occur. (Checking that a consumer has lost access is a different question with a different answer; managing-data-asset-access covers it.)
• Poll by subscription id. The listing-level view goes stale.
• A grant being torn down reads PENDING then IN_PROGRESS, the same two statuses fulfilment passes through. The status alone does not tell you which direction it is moving; the subscription’s own status does.

Allow tens of seconds for a lake-backed grant and several minutes for a warehouse-backed one. Do the revokes for every subscription first, then poll for all of them together, never one wait per subscription.

Poll in SHORT calls, and repeat the call. Never sleep for minutes inside one. A single tool call that blocks for more than about a minute can exceed the agent runtime’s per-call read timeout: measured, a blocking wait killed the tool connection after ~107 seconds and the whole task was lost with no answer. So each invocation below polls for about forty seconds and then returns a verdict. If it says NOT SETTLED, run the same block again, as many times as it takes. Do not raise the sleep or the loop count to avoid re-running it; that is the thing that breaks.

Step 0, the plan. Read-only. Scope it to this asset’s listing: filtering by project instead (--approver-project-id) returns the subscriptions to every asset the project publishes, and revoking those cuts consumers off from data nobody asked you to retire.

If the plan is empty, go to step 3. If the request named the subscriptions (an advance yes), write the ids it named, one per line, and check each plan line carries the project it named:

Otherwise show the user every line of the plan, say that the listing and the asset will then be deleted, and stop. Proceed only on an explicit yes in their next message, and then record what they said yes to:

Step 1, once, after the yes. Revoke exactly what was confirmed. If the subscribers are not the confirmed list, whether they changed since the plan or the user named a different set, the yes was for a different list:

Capture the ids before revoking, as this does: once a subscription is revoked, the listing filter stops returning it, so a list taken afterwards is empty and the wait below would have nothing to wait on.

Step 2, repeated until it prints SETTLED. One collective poll over every subscription, bounded so the call always returns:

FAILED is terminal. A grant whose status (or an asset’s) reads GRANT_FAILED, REVOKE_FAILED, GRANT_AND_REVOKE_FAILED or INACCESSIBLE will not clear by waiting, and DeleteListing stays refused while it is there. Stop, name the subscription and the asset’s failureCause, and do not delete.

UNKNOWN is not SETTLED, and that is the point. A call that fails (denied, throttled, or made with credentials for another account) prints nothing on stdout, so folding its error into the count (2>/dev/null || echo 0) turns any failure into SETTLED: no grants remain. Measured: this block reported settlement while every call in it was being refused, because the refusal went to stderr and the substitution yielded zero. UNKNOWN is not the same answer as SETTLED and MUST NOT be reported as one. The whole reason to wait is that deleting into a live grant leaves a listing pointing at nothing. If you see UNKNOWN, read the error and fix the cause; references/troubleshooting.md covers the case where it is your tool surface rather than your permissions.

Steps 3 to 5, only once step 2 printed SETTLED. Do not ask again: the yes at step 0 covered the whole sequence.

If DeleteListing is refused, read the blockers it named and clear them: cancel a pending subscription, or delete a lingering grant, rather than retrying the same call.

Deleting an asset is destructive and the data is not the thing being deleted, so the user may not expect the blast radius: afterwards, say what was removed and name every consumer that lost access. Beyond step 0, ask only if you would go beyond the request: deleting a second asset, or removing a grant that belongs to something else.

Data products

A data product bundles listings under one name and publishes through the same change set with --entity-type DATA_PRODUCT. Create it with CreateDataProduct, revise with CreateDataProductRevision, then publish. The retirement sequence above applies to the product’s listing as well: revoke, wait, unpublish, delete.