Data Engineering

Skill 46 of 185

Data Quality Frameworks

Implement data quality validation with Great Expectations, dbt tests, and data contracts.

1 minute · 322 words · 10 sections

Install

$ npx skills add wshobson/agents --skill data-quality-frameworks

$ npx skills add wshobson/agents

» /plugin marketplace add wshobson/agents

The first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.

Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

When to Use This Skill

Implementing data quality checks in pipelines
Setting up Great Expectations validation
Building comprehensive dbt test suites
Establishing data contracts between teams
Monitoring data quality metrics
Automating data validation in CI/CD

Core Concepts

1. Data Quality Dimensions

Dimension	Description	Example Check
Completeness	No missing values	`expect_column_values_to_not_be_null`
Uniqueness	No duplicates	`expect_column_values_to_be_unique`
Validity	Values in expected range	`expect_column_values_to_be_in_set`
Accuracy	Data matches reality	Cross-reference validation
Consistency	No contradictions	`expect_column_pair_values_A_to_be_greater_than_B`
Timeliness	Data is recent	`expect_column_max_to_be_between`

2. Testing Pyramid for Data

          /\
         /  \     Integration Tests (cross-table)
        /────\
       /      \   Unit Tests (single column)
      /────────\
     /          \ Schema Tests (structure)
    /────────────\

Quick Start

Great Expectations Setup

# Install
pip install great_expectations

# Initialize project
great_expectations init

# Create datasource
great_expectations datasource new

# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.add_expectation_suite("orders_suite")

# Add expectations
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
)

# Validate
results = context.run_checkpoint(checkpoint_name="daily_orders")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Summary: {total_passed}/{total_tables} tables passed”)

    report.append("")

    for table, result in results.items():
        status = "✅" if result.passed else "❌"
        report.append(f"### {status} {table}")
        report.append(f"- Expectations: {result.total_expectations}")
        report.append(f"- Failed: {result.failed_expectations}")

        if not result.passed:
            report.append("- Failed checks:")
            for detail in result.details:
                if not detail["success"]:
                    report.append(f"  - {detail['expectation']}: {detail['observed_value']}")
        report.append("")

    return "\n".join(report)

Usage

context = gx.get_context() pipeline = DataQualityPipeline(context)

tables_to_validate = { “orders”: “orders_suite”, “customers”: “customers_suite”, “products”: “products_suite”, }

results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results)

Fail pipeline if any table failed

if not all(r.passed for r in results.values()): print(report) raise ValueError(“Data quality checks failed!”)


## Best Practices

### Do's

- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes

### Don'ts

- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too

About this skill

Trigger

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

The verbatim description from this skill’s front matter — the string an agent matches on to decide whether to load it.

Colophon for this issue

Colophon

How this issue was made

Source

github.com/wshobson/agents, branch main, last pushed 8 November 2026.

Licence

MIT — the text of every skill is reproduced unmodified, frontmatter included, under the upstream licence.

Discovery

185 skills found by walking the repository tree for SKILL.md, not by matching a directory convention. 52 distinct layouts observed: plugins/accessibility-compliance/skills/*/SKILL.md, plugins/agent-teams/skills/*/SKILL.md, plugins/api-scaffolding/skills/*/SKILL.md, plugins/avoid-ai-writing/skills/*/SKILL.md, plugins/backend-development/skills/*/SKILL.md, plugins/before-you-build/skills/*/SKILL.md, plugins/block-no-verify/skills/*/SKILL.md, plugins/blockchain-web3/skills/*/SKILL.md, plugins/brand-landingpage/skills/*/SKILL.md, plugins/business-analytics/skills/*/SKILL.md, plugins/cicd-automation/skills/*/SKILL.md, plugins/cloud-infrastructure/skills/*/SKILL.md, plugins/conductor/skills/*/SKILL.md, plugins/content-marketing/skills/*/SKILL.md, plugins/data-engineering/skills/*/SKILL.md, plugins/database-design/skills/*/SKILL.md, plugins/developer-essentials/skills/*/SKILL.md, plugins/dgx-spark-ops/skills/*/SKILL.md, plugins/documentation-generation/skills/*/SKILL.md, plugins/documentation-standards/skills/*/SKILL.md, plugins/dotnet-contribution/skills/*/SKILL.md, plugins/file-conversion/skills/*/SKILL.md, plugins/framework-migration/skills/*/SKILL.md, plugins/frontend-mobile-development/skills/*/SKILL.md, plugins/game-development/skills/*/SKILL.md, plugins/hermes-tweet/skills/*/SKILL.md, plugins/hr-legal-compliance/skills/*/SKILL.md, plugins/incident-response/skills/*/SKILL.md, plugins/javascript-typescript/skills/*/SKILL.md, plugins/kubernetes-operations/skills/*/SKILL.md, plugins/llm-application-dev/skills/*/SKILL.md, plugins/llm-finetuning/skills/*/SKILL.md, plugins/machine-learning-ops/skills/*/SKILL.md, plugins/observability-monitoring/skills/*/SKILL.md, plugins/payment-processing/skills/*/SKILL.md, plugins/plugin-eval/skills/*/SKILL.md, plugins/pptx-deck-creation/skills/*/SKILL.md, plugins/protect-mcp/skills/*/SKILL.md, plugins/python-development/skills/*/SKILL.md, plugins/quantitative-trading/skills/*/SKILL.md, plugins/reverse-engineering/skills/*/SKILL.md, plugins/review-agent-governance/skills/*/SKILL.md, plugins/security-scanning/skills/*/SKILL.md, plugins/shell-scripting/skills/*/SKILL.md, plugins/ship-mate/skills/*/SKILL.md, plugins/signed-audit-trails/skills/*/SKILL.md, plugins/skill-forge-essentials/skills/*/SKILL.md, plugins/social-publishing/skills/*/SKILL.md, plugins/startup-business-analyst/skills/*/SKILL.md, plugins/superself/skills/*/SKILL.md, plugins/systems-programming/skills/*/SKILL.md, plugins/ui-design/skills/*/SKILL.md.

Issue colours

Resolved from a deterministic hash of the owner name. Two accent tones are generated per issue and each is proven against its own ground before it ships: a single accent that passes AA on both light and dark paper is arithmetically impossible.

Heading repairs

3 repairs applied to this skill so the document has one h1 and no skipped levels:

Removed “Data Quality Frameworks”, a leading h1 that duplicated the skill title.
Shifted “10 headings” from h1 to h2 so the skill title is the only h1.
Repaired “When to Use This Skill” from h3 to h2, closing a skipped level.

Spec compliance

Every skill satisfies the Agent Skills specification’s frontmatter rules.

Images

Images inside a skill come from the upstream repository. Where the author gave no alternative text we mark the image decorative rather than inventing a description — a plausible caption we made up is worse than none for the reader who depends on it.

Marketplace

A plugin manifest is published at .claude-plugin/marketplace.json by Seth Hobson, declaring 94 plugins. It is read for editorial metadata only — never as the skill index, which is always the repository tree.

Signal

Install counts come from skills.sh. They measure downloads, not quality, and an unranked repository is not an unread one.

Agent surfaces

The whole issue is available as one markdown document at /wshobson/agents.md, and each skill at its own .md URL.

Publication

Set by Skills Docs from the source repository. Body text is Literata at the reader’s chosen size and measure; code is Geist Mono. Nothing on this page was written by us except this paragraph.

Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

When to Use This Skill

Implementing data quality checks in pipelines
Setting up Great Expectations validation
Building comprehensive dbt test suites
Establishing data contracts between teams
Monitoring data quality metrics
Automating data validation in CI/CD

Core Concepts

1. Data Quality Dimensions

Dimension	Description	Example Check
Completeness	No missing values	`expect_column_values_to_not_be_null`
Uniqueness	No duplicates	`expect_column_values_to_be_unique`
Validity	Values in expected range	`expect_column_values_to_be_in_set`
Accuracy	Data matches reality	Cross-reference validation
Consistency	No contradictions	`expect_column_pair_values_A_to_be_greater_than_B`
Timeliness	Data is recent	`expect_column_max_to_be_between`

2. Testing Pyramid for Data

          /\
         /  \     Integration Tests (cross-table)
        /────\
       /      \   Unit Tests (single column)
      /────────\
     /          \ Schema Tests (structure)
    /────────────\

Quick Start

Great Expectations Setup

# Install
pip install great_expectations

# Initialize project
great_expectations init

# Create datasource
great_expectations datasource new

# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.add_expectation_suite("orders_suite")

# Add expectations
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
)

# Validate
results = context.run_checkpoint(checkpoint_name="daily_orders")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Summary: {total_passed}/{total_tables} tables passed”)

    report.append("")

    for table, result in results.items():
        status = "✅" if result.passed else "❌"
        report.append(f"### {status} {table}")
        report.append(f"- Expectations: {result.total_expectations}")
        report.append(f"- Failed: {result.failed_expectations}")

        if not result.passed:
            report.append("- Failed checks:")
            for detail in result.details:
                if not detail["success"]:
                    report.append(f"  - {detail['expectation']}: {detail['observed_value']}")
        report.append("")

    return "\n".join(report)

Usage

context = gx.get_context() pipeline = DataQualityPipeline(context)

tables_to_validate = { “orders”: “orders_suite”, “customers”: “customers_suite”, “products”: “products_suite”, }

results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results)

Fail pipeline if any table failed

if not all(r.passed for r in results.values()): print(report) raise ValueError(“Data quality checks failed!”)


## Best Practices

### Do's

- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes

### Don'ts

- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too

Data Quality Frameworks

When to Use This Skill

Core Concepts

1. Data Quality Dimensions

2. Testing Pyramid for Data

Quick Start

Great Expectations Setup

Detailed patterns and worked examples

Summary: {total_passed}/{total_tables} tables passed”)

Usage

Fail pipeline if any table failed

About this skill

Trigger

Colophon

Data Quality Frameworks

When to Use This Skill

Core Concepts

1. Data Quality Dimensions

2. Testing Pyramid for Data

Quick Start

Great Expectations Setup

Detailed patterns and worked examples

Summary: {total_passed}/{total_tables} tables passed”)

Usage

Fail pipeline if any table failed

Appendix 46.1

ReferencesMarkdown · 1 file

About this skill

Trigger

Colophon

Appendix 46.1

ReferencesMarkdown · 1 file