Subchapter 2.100
references/components/state-backed/using.mdMarkdown11 KBView on GitHub
State-backed components produce Dagster definitions that depend on external systems (BI tools, ETL platforms, compiled artifacts). Instead of calling external APIs on every code server load, they split definition-building into two steps:
write_state_to_path — fetches external state and persists it locally (expensive, runs on refresh)build_defs_from_state — builds definitions from persisted state (cheap, runs on every load)The framework controls when refresh happens based on the configured management strategy.
| Strategy | Storage | Refresh Mechanism | Best For |
|---|---|---|---|
LOCAL_FILESYSTEM | .local_defs_state/ in project dir | dg utils refresh-defs-state in CI before building artifact | Most deployments |
VERSIONED_STATE_STORAGE | Cloud storage (S3 on Dagster+, or S3/GCS/Azure) | dg utils refresh-defs-state or independent of deploys | Dagster+ (automatic); decoupling state from builds |
LEGACY_CODE_SERVER_SNAPSHOTS | In-memory | Auto-refresh on every code server load | Not recommended; backwards-compat only |
State is written to .local_defs_state/ inside your project directory. This directory is auto-gitignored but must be included in your deployment artifact (Docker image, PEX).
my_project/
├── my_project/
│ ├── defs/
│ └── ...
├── .local_defs_state/
│ ├── DbtProjectComponent[my_dbt_project]/
│ └── FivetranAccountComponent/
├── pyproject.toml
└── .gitignore # includes .local_defs_state/State is stored in cloud storage with UUID-based versioning. Enables state updates without rebuilding deployment artifacts.
On Dagster+, VERSIONED_STATE_STORAGE works automatically with no user configuration. Dagster+ uploads and downloads state to an S3 bucket managed by Dagster Labs. Simply set management_type: VERSIONED_STATE_STORAGE on your component and Dagster+ handles the rest.
For OSS or self-hosted deployments, you must configure dagster.yaml with a storage backend:
# dagster.yaml (OSS / self-hosted only — not needed on Dagster+)
defs_state_storage:
module: dagster._core.storage.defs_state.blob_storage_state_storage
class: UPathDefsStateStorage
config:
base_path: s3://my-bucket/dagster/defs-state # or gs:// or az://LEGACY_CODE_SERVER_SNAPSHOTS is retained for backwards compatibility only. As of 1.13.0, bundled state-backed components default to LOCAL_FILESYSTEM. Use LOCAL_FILESYSTEM or VERSIONED_STATE_STORAGE in new components.
All code location loads and run processes require access to the configured DefsStateStorage, as state is downloaded in all of these cases. The configured storage must be high-availability.
Each state-backed component accepts a defs_state field in its YAML configuration:
type: dagster_fivetran.FivetranAccountComponent
attributes:
account: ...
defs_state:
management_type: LOCAL_FILESYSTEM
refresh_if_dev: false| Field | Type | Default | Description |
|---|---|---|---|
management_type | LOCAL_FILESYSTEM | VERSIONED_STATE_STORAGE | LEGACY_CODE_SERVER_SNAPSHOTS | Component-specific | Storage strategy for state |
key | string | Auto-generated from class name | Unique identifier for this component’s state |
refresh_if_dev | boolean | true | Auto-refresh state during dagster dev. Set false to reduce API calls during local dev |
When refresh_if_dev: true (the default), running dagster dev or using the dg CLI automatically refreshes state on startup. Set to false when:
Navigate to Deployment → [your code location] → Defs state to see registered state keys, last update timestamps, and version identifiers.
Keys uniquely identify a component’s state within a deployment. Default format: ClassName or ClassName[discriminator].
self.__class__.__name__ as the default keyDbtProjectComponent[analytics], DbtProjectComponent[marketing])key in the YAML defs_state block to use a custom keyBefore deploying, refresh state for all state-backed components:
uv run dg utils refresh-defs-stateThis fetches current state from all external systems and persists it according to each component’s configured strategy.
dg utils refresh-defs-state operates on one Dagster project at a time — the directory containing the project’s pyproject.toml (with [tool.dg.project]) or dg.toml. It does not read workspace.yaml and has no notion of a multi-location workspace; run it from inside the project directory.
To run successfully, the environment needs:
uv sync or pip install -e .). Installing only the integration libraries is insufficient — the full project must be importable so that components can be loaded and write_state_to_path() can execute.dg (from dagster-dg-cli) installed in that same environment.DAGSTER_HOME + cloud credentials — only required if any component uses VERSIONED_STATE_STORAGE. DAGSTER_HOME must point to a directory containing dagster.yaml with defs_state_storage configured.If a repo contains multiple code locations (each its own Dagster project, typically built into its own Docker image), refresh is per project. Run dg utils refresh-defs-state once per project, inside that project’s directory, as part of that project’s build. There is no top-level “workspace refresh”; the workspace.yaml that lives in the daemon/UI image is unrelated to state refresh.
Run dg utils refresh-defs-state before building your Docker image or PEX artifact. The .local_defs_state/ directory must be included when copying project files into the image. Each project owns its own .local_defs_state/.
Ensure the CI environment has:
DAGSTER_HOME pointing to a directory containing dagster.yaml with defs_state_storage configuredUse the Dagster+ variant of the refresh command:
uv run dg plus deploy refresh-defs-stateOr scaffold a complete CI/CD workflow with state refresh included:
uv run dg scaffold github-actionsA component can produce a job that calls self.refresh_state(project_root), decoupling state refresh from the deploy cycle entirely. This job can be triggered via manual runs, schedules, sensors, or the GraphQL API.
import asyncio
import dagster as dg
class MyApiComponent(dg.StateBackedComponent, dg.Model, dg.Resolvable):
api_url: str
# ... defs_state_config, write_state_to_path, build_defs_from_state ...
def build_defs_from_state(
self, context: dg.ComponentLoadContext, state_path: Path | None
) -> dg.Definitions:
assets = ... # build assets from state
op_name = f"refresh_state_{hash(self.defs_state_config.key) % 10**8}"
@dg.op(name=op_name)
def refresh_my_api_state():
asyncio.run(self.refresh_state(context.project_root))
@dg.job(name="refresh_my_api_component_state")
def refresh_job():
refresh_my_api_state()
return dg.Definitions(assets=assets, jobs=[refresh_job])After the job runs, the new state is persisted. On Dagster+, use “Refresh definitions state” in the UI or the refreshDefsState GraphQL mutation to pick up the new versions (see below). On OSS, the next code location reload automatically uses the latest state.
How state updates become visible depends on the deployment model:
refreshDefsState GraphQL mutation:mutation RefreshDefsStateMutation($locationName: String!) {
refreshDefsState(locationName: $locationName) {
... on WorkspaceEntry {
locationName
}
... on RefreshDefsStateError {
message
}
}
}This distinction matters when using programmatic refresh or CI/CD pipelines that update state independently of deploys — on Dagster+, you must explicitly request the new versions; on OSS, a reload is sufficient.
dagster_tableau.TableauComponentdagster_looker.LookerComponentdagster_sigma.SigmaComponentdagster_powerbi.PowerBIWorkspaceComponentdagster_fivetran.FivetranAccountComponentdagster_airbyte.AirbyteWorkspaceComponentdagster_dbt.DbtProjectComponentdagster_omni.OmniComponent