Setting the file. One moment.
Skill 73 · Managing Data Catalog
Subchapter 73.7
references/measuring-coverage.mdMarkdown5 KBView on GitHub
SageMaker Catalog. On the Glue Data Catalog the same question is easy.
Table.Description and StorageDescriptor.Columns[].Comment are the two fields,
glue get-tables returns both, and there is nothing to explain. This chapter is
about the SageMaker Catalog’s form model, where the same question is routinely
answered wrong in a specific way.
Scope note: reporting coverage inside a project so it can be fixed is this
skill. Counting gaps domain-wide by SQL over the exported metadata snapshot
belongs to querying-aws-sagemaker-catalog.
A crawled asset can carry a description that no steward ever wrote. The data source run inherits the source table’s description into the technical form, not into the asset’s business metadata:
| Kind | Where it lives | Written by |
|---|---|---|
| authored business metadata | asset.description, AssetCommonDetailsForm.summary / .readMe, ColumnBusinessMetadataForm, glossaryTerms | a steward, or an accepted AI prediction |
| inherited technical description | GlueTableForm.tableDescription, GlueTableForm.columns[].columnDescription | the crawl, from the source table |
The three groups below are about descriptions: an asset is AUTHORED when it has
asset.description, AssetCommonDetailsForm.summary or AssetCommonDetailsForm.readMe.
Glossary terms and column business metadata are authored too, but they don’t place an
asset in a group. The block prints the term count beside each asset (terms=), and
column business metadata is read separately.
An agent that reads only asset.description reports every asset as
undocumented. An agent that reads the forms and does not draw the distinction
reports the inherited ones as documented, which is the more damaging error:
the user closes the ticket and no steward ever writes the business meaning.
Both halves are the whole answer. You MUST report three groups, not two:
And the answer MUST also carry two things beyond the three counts, because a group total hides the work:
n of m columns.asset.description,
AssetCommonDetailsForm.summary and .readMe for authored metadata,
GlueTableForm.tableDescription and GlueTableForm.columns[].columnDescription
for the inherited kind. Naming the field is what lets the user check you, and
it is the distinction the whole answer turns on.DOMAIN=<domain>; PROJECT=<project>
aws datazone search --domain-identifier "$DOMAIN" --owning-project-identifier "$PROJECT" \
--search-scope ASSET --query 'items[].assetItem.{id:identifier,name:name}' --output text \
| while read -r id name; do
aws datazone get-asset --domain-identifier "$DOMAIN" --identifier "$id" > /tmp/a.json
jq -r --arg n "$name" '
( .description // "" ) as $d
| ( [.formsOutput[]|select(.formName=="AssetCommonDetailsForm")
|.content // "{}" |fromjson] | first // {} ) as $c
| ( [.formsOutput[]|select(.formName=="GlueTableForm")
|.content // "{}" |fromjson] | first // {} ) as $g
| [ $n,
( if ($d|length)>0 or (($c.summary//"")|length)>0 or (($c.readMe//"")|length)>0
then "AUTHORED" elif (($g.tableDescription//"")|length)>0
then "INHERITED-only" else "NONE" end ),
"cols_with_inherited=\([$g.columns//[]|.[]|select((.columnDescription//"")|length>0)]|length)/\(($g.columns//[])|length)",
"terms=\((.glossaryTerms//[])|length)" ] | @tsv' /tmp/a.json
doneFor a project of any size, generate this as one script and run it once rather than issuing a call per asset conversationally.
There may be no owner field to report on. Before saying an asset is “missing an
owner”, check whether any form in the project holds ownership at all. If none
does, the honest answer is that ownership is not recorded anywhere and capturing
it needs a custom metadata form or a glossary convention first; see
references/metadata-forms.md. Reporting fifteen assets as “missing an owner”
implies an empty field waiting to be filled, and there is none.
Rank the fix by cost, not by count:
references/generating-metadata.md, which is both cheaper
and safer than hand-authoring at volume.