Skill 73 · Managing Data Catalog
Subchapter 73.5
references/generating-metadata.mdMarkdown7 KBView on GitHub
SageMaker Catalog only. The Glue Data Catalog has no equivalent.
The gap here is pure discovery: the feature exists, and asked to “generate metadata suggestions” most agents describe the feature and then write the suggestions by hand. Hand-written suggestions are not the same thing: they live in a chat log, not in the catalog, so nothing durable is there for a human to review and accept later.
aws datazone start-metadata-generation-run --domain-identifier <domain> \
--owning-project-identifier <project> --type BUSINESS_DESCRIPTIONS \
--target '{"identifier":"<asset-id>","type":"ASSET","revision":"<revision>"}'
aws datazone get-metadata-generation-run --domain-identifier <domain> --identifier <run-id>
aws datazone list-metadata-generation-runs --domain-identifier <domain>
aws datazone cancel-metadata-generation-run --domain-identifier <domain> --identifier <run-id>
aws datazone accept-predictions --domain-identifier <domain> --identifier <asset-id> \
--revision <revision> --accept-rule '{"rule":"ALL"}' # see the warning below--type is one of BUSINESS_DESCRIPTIONS, BUSINESS_NAMES or
BUSINESS_GLOSSARY_ASSOCIATIONS. A run takes roughly thirty seconds per asset.
A prediction lands in SuggestionForm
(amazon.datazone.SuggestionFormType), reported under readOnlyFormsOutput
and not under formsOutput. Its content is a fieldSuggestions list keyed by
fieldPath: summary, plus columns[N].columnDescription per column.
Nothing reaches asset.description, AssetCommonDetailsForm.summary or
ColumnBusinessMetadataForm until a prediction is accepted. That is what makes
the review durable: the suggestion is a state in the catalog that a human can
come back to, rather than text in a conversation.
Starting a run bumps the asset’s revision, and the suggestion form cannot be removed from an asset by any API. So a run is not free of consequence even before anything is accepted, so do not run one on assets nobody asked about.
The second-commonest failure is finding the API and then running it over everything. Every asset costs a run and gains an irreversible form. The failure after that one is subtler and more common than it looks: the right mechanism, run over the wrong set, because the set was a judgement the agent held rather than a list it produced.
So the target set is a required first artefact, not a decision you carry. Resolve it with a query, print it, and only then start any run. An agent that starts the first run before printing the list is working from memory.
DOMAIN=<domain>; PROJECT=<project>
# One line per asset: its id, and which group it is in. Nothing is started yet.
aws datazone search --domain-identifier "$DOMAIN" --owning-project-identifier "$PROJECT" \
--search-scope ASSET --query 'items[].assetItem.{id:identifier,name:name}' --output text \
| while read -r id name; do
aws datazone get-asset --domain-identifier "$DOMAIN" --identifier "$id" > /tmp/a.json
jq -r --arg n "$name" --arg i "$id" '
( .description // "" ) as $d
| ( [.formsOutput[]|select(.formName=="AssetCommonDetailsForm")
|.content // "{}" |fromjson] | first // {} ) as $c
| ( [.formsOutput[]|select(.formName=="GlueTableForm")
|.content // "{}" |fromjson] | first // {} ) as $g
| [ $n, $i,
( if ($d|length)>0 or (($c.summary//"")|length)>0 or (($c.readMe//"")|length)>0
then "AUTHORED" elif (($g.tableDescription//"")|length)>0
then "INHERITED-only" else "NONE" end ) ] | @tsv' /tmp/a.json
done | tee /tmp/generation-targets.tsv
wc -l < /tmp/generation-targets.tsv # the number of runs you are about to startThen, before the first start-metadata-generation-run:
NONE group.
An asset carrying an inherited technical description is documented enough to
fall outside that phrasing, and including it is over-reach; say which group you
read the request as naming, and which field you read to decide.
references/measuring-coverage.md is the full treatment of that distinction.--accept-rule '{"rule":"ALL"}' unattended. It is one call
and it applies everything, including a generated summary of several thousand
characters, across every column.RejectPredictions is not a cleanup. It leaves the suggestion form in place, it
does not undo an accept that already happened, and it writes a stub
ColumnBusinessMetadataForm carrying a bare columnIdentifier per column. Do
not call it to tidy up; it adds debris. Naming a fieldPath as a
predictionTarget is refused outright.
AcceptPredictions performs the merge correctly. It wrote a generated
summary into AssetCommonDetailsForm.summary and a description onto every column
while preserving sourceIdentifier: the service does the read-then-write that
hand-authoring gets wrong.
So for bulk description work the generation path is safer than hand-authoring
through CreateAssetRevision, which is the reverse of what the merge rule in
SKILL.md would lead you to assume. This is a property of this one operation.
It is not evidence that the platform merges in general: every other write path
in this skill still replaces.