Data Quality Frameworks

Implement data quality validation with Great Expectations, dbt tests, and data contracts.

1 minute · 322 words · 10 sections

Install

$ npx skills add wshobson/agents --skill data-quality-frameworks

$ npx skills add wshobson/agents

» /plugin marketplace add wshobson/agents

The first command installs just this skill, by the name in its SKILL.md; the second installs the whole repository.

Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

When to Use This Skill

Implementing data quality checks in pipelines
Setting up Great Expectations validation
Building comprehensive dbt test suites
Establishing data contracts between teams
Monitoring data quality metrics
Automating data validation in CI/CD

Core Concepts

1. Data Quality Dimensions

Dimension	Description	Example Check
Completeness	No missing values	`expect_column_values_to_not_be_null`
Uniqueness	No duplicates	`expect_column_values_to_be_unique`
Validity	Values in expected range	`expect_column_values_to_be_in_set`
Accuracy	Data matches reality	Cross-reference validation
Consistency	No contradictions	`expect_column_pair_values_A_to_be_greater_than_B`
Timeliness	Data is recent	`expect_column_max_to_be_between`

2. Testing Pyramid for Data

          /\
         /  \     Integration Tests (cross-table)
        /────\
       /      \   Unit Tests (single column)
      /────────\
     /          \ Schema Tests (structure)
    /────────────\

Quick Start

Great Expectations Setup

# Install
pip install great_expectations

# Initialize project
great_expectations init

# Create datasource
great_expectations datasource new

# great_expectations/checkpoints/daily_validation.yml
import great_expectations as gx

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.add_expectation_suite("orders_suite")

# Add expectations
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Summary: {total_passed}/{total_tables} tables passed”)

    report.append("")

    for table, result in results.items():
        status = "✅" if result.passed else "❌"
        report.append(f"### {status} {table}")
        report.append(f"- Expectations: {result.total_expectations}")
        report.append(f"- Failed: {result.failed_expectations}")

        if not result.passed:
            report.append("- Failed checks:")
            for detail in result.details:
                if not detail["success"]:
                    report.append(f"  - {detail['expectation']}: {detail['observed_value']}")
        report.append("")

    return "\n".join(report)

Usage

context = gx.get_context() pipeline = DataQualityPipeline(context)

tables_to_validate = { “orders”: “orders_suite”, “customers”: “customers_suite”, “products”: “products_suite”, }

results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results)

Fail pipeline if any table failed

if not all(r.passed for r in results.values()): print(report) raise ValueError(“Data quality checks failed!”)


## Best Practices

### Do's

- **Test early** - Validate source data before transformations
- **Test incrementally** - Add tests as you find issues
- **Document expectations** - Clear descriptions for each test
- **Alert on failures** - Integrate with monitoring
- **Version contracts** - Track schema changes

### Don'ts

- **Don't test everything** - Focus on critical columns
- **Don't ignore warnings** - They often precede failures
- **Don't skip freshness** - Stale data is bad data
- **Don't hardcode thresholds** - Use dynamic baselines
- **Don't test in isolation** - Test relationships too

Appendix 45.1

1 file · 11 KB

Everything this skill ships beside its prose. All of it is set here, as a subchapter of skill 45.

ReferencesMarkdown · 1 file

Documentation the agent loads on demand, rather than up front.

45.1Detailsreferences/details.md11 KB

About this skill

Trigger

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

The verbatim description from this skill’s front matter — the string an agent matches on to decide whether to load it.

Colophon for this issue

Colophon

How this issue was made

Source: github.com/wshobson/agents, branch main, last pushed 21 September 2026.
Licence: MIT — the text of every skill is reproduced unmodified, frontmatter included, under the upstream licence.
Discovery: 183 skills found by walking the repository tree for SKILL.md, not by matching a directory convention. 51 distinct layouts observed: plugins/accessibility-compliance/skills/*/SKILL.md, plugins/agent-teams/skills/*/SKILL.md, plugins/api-scaffolding/skills/*/SKILL.md, plugins/avoid-ai-writing/skills/*/SKILL.md, plugins/backend-development/skills/*/SKILL.md, plugins/before-you-build/skills/*/SKILL.md, plugins/block-no-verify/skills/*/SKILL.md, plugins/blockchain-web3/skills/*/SKILL.md, plugins/brand-landingpage/skills/*/SKILL.md, plugins/business-analytics/skills/*/SKILL.md, plugins/cicd-automation/skills/*/SKILL.md, plugins/cloud-infrastructure/skills/*/SKILL.md, plugins/conductor/skills/*/SKILL.md, plugins/data-engineering/skills/*/SKILL.md, plugins/database-design/skills/*/SKILL.md, plugins/developer-essentials/skills/*/SKILL.md, plugins/dgx-spark-ops/skills/*/SKILL.md.