Using environment variables with components
With dg and components, you can easily configure components depending on the environment in which they are run. To demonstrate this, we'll walk through setting up an example ELT pipeline with a Sling component which reads Snowflake credentials from environment variables.
For more information on using environment variables with non-component Dagster code, see Using environment variables and secrets in Dagster code.
Step 1: Create a new Dagster project
First, we'll set up a basic ELT pipeline using Sling in an empty Dagster project:
uvx create-dagster@latest project ingestion
cd ingestion && source .venv/bin/activate
We'll install dagster-sling and scaffold an empty Sling connection component:
uv add dagster-sling
dg list components
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster.DefinitionsComponent │ An arbitrary set of Dagster definitions. │
├───────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ dagster.DefsFolderComponent │ A component that represents a directory containing multiple │
│ │ Dagster definition modules. │
├───────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ dagster.FunctionComponent │ Represents a Python function, alongside the set of assets or │
│ │ asset checks that it is responsible for executing. │
├───────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ dagster.PythonScriptComponent │ Represents a Python script, alongside the set of assets and │
│ │ asset checks that it is responsible for executing. │
├───────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ dagster.TemplatedSqlComponent │ A component which executes templated SQL from a string or file. │
├───────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────┤
│ dagster.UvRunComponent │ Represents a Python script, alongside the set of assets or asset │
│ │ checks that it is responsible for executing. │
├───────────────────────────────────────────────────┼───────────────────────────────────── ─────────────────────────────┤
│ dagster_sling.SlingReplicationCollectionComponent │ Expose one or more Sling replications to Dagster as assets. │
└───────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────┘
dg scaffold defs dagster_sling.SlingReplicationCollectionComponent ingest_to_snowflake
Creating defs at /.../ingestion/src/ingestion/defs/ingest_to_snowflake.
Step 2: Use environment variables in a component
Next, we will configure a Sling connection that will sync a local CSV file to a Snowflake database, with credentials provided with environment variables:
curl -O https://raw.githubusercontent.com/dbt-labs/jaffle-shop-classic/refs/heads/main/seeds/raw_customers.csv
source: LOCAL
target: SNOWFLAKE
defaults:
mode: full-refresh
object: "{stream_table}"
streams:
file://raw_customers.csv:
object: "sandbox.raw_customers"
We will use the env function to template credentials into Sling configuration in our defs.yaml file. Running dg check yaml will highlight that we
need to explicitly encode these environment dependencies at the bottom of the file:
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
connections:
SNOWFLAKE:
type: snowflake
account: "{{ env.SNOWFLAKE_ACCOUNT }}"
user: "{{ env.SNOWFLAKE_USER }}"
password: "{{ env.SNOWFLAKE_PASSWORD }}"
database: "{{ env.SNOWFLAKE_DATABASE }}"
replications:
- path: replication.yaml
dg check yaml --validate-requirements
/.../ingestion/src/ingestion/defs/ingest_files/defs.yaml:1 - requirements.env Component uses environment variables that are not specified in the component file: SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_PASSWORD, SNOWFLAKE_USER
|
1 | type: dagster_sling.SlingReplicationCollectionComponent
| ^ Component uses environment variables that are not specified in the component file: SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_PASSWORD, SNOWFLAKE_USER
2 |
3 | attributes:
4 | connections:
5 | SNOWFLAKE:
6 | type: snowflake
7 | account: "{{ env.SNOWFLAKE_ACCOUNT }}"
8 | user: "{{ env.SNOWFLAKE_USER }}"
9 | password: "{{ env.SNOWFLAKE_PASSWORD }}"
10 | database: "{{ env.SNOWFLAKE_DATABASE }}"
11 | replications:
12 | - path: replication.yaml
|
After adding the environment dependencies, running dg check yaml again will confirm that the file is valid:
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
connections:
SNOWFLAKE:
type: snowflake
account: "{{ env.SNOWFLAKE_ACCOUNT }}"
user: "{{ env.SNOWFLAKE_USER }}"
password: "{{ env.SNOWFLAKE_PASSWORD }}"
database: "{{ env.SNOWFLAKE_DATABASE }}"
replications:
- path: replication.yaml
requirements:
env:
- SNOWFLAKE_ACCOUNT
- SNOWFLAKE_USER
- SNOWFLAKE_PASSWORD
- SNOWFLAKE_DATABASE
dg check yaml
All component YAML validated successfully.
Next, you can invoke dg list env, which shows all environment variables configured or used by components in the project. Here we can see all of the Snowflake credentials we must configure in our shell or .env file in order to run our project:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ │ ingest_files │
│ SNOWFLAKE_DATABASE │ │ ingest_files │
│ SNOWFLAKE_PASSWORD │ │ ingest_files │
│ SNOWFLAKE_USER │ │ ingest_files │
└────────────────────┴───────┴──────────────┘
You can edit the .env file in your project root to specify environment variables for Dagster to use when running the project locally. You can run dg list env again to see that they are now set:
echo 'SNOWFLAKE_ACCOUNT=...' >> .env
echo 'SNOWFLAKE_USER=...' >> .env
echo 'SNOWFLAKE_PASSWORD=...' >> .env
echo "SNOWFLAKE_DATABASE=sandbox" >> .env
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │
└────────────────────┴───────┴──────────────┘
Step 3: Configure environment variables for components in Dagster+ (Optional)
If you are using Dagster+, you can also use the dg CLI to push environment variables to your deployment so that your code is ready to run in a production setting.
First, you will need to authenticate with Dagster+:
dg plus login
The dg list env command will now automatically show what scopes environment variables are configured for, in your configured deployment:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ │ │ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ │ │ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘
From here, you can use the dg plus create env command to push environment variables to your deployment. The --from-local-env flag will read the environment variables from your local shell or .env file.
Configure local environment variables
To enable the rest of your team to run your project locally, you can run the following command to push each Snowflake credential to the local scope:
dg plus create env SNOWFLAKE_ACCOUNT --from-local-env --scope local &&
dg plus create env SNOWFLAKE_USER --from-local-env --scope local &&
dg plus create env SNOWFLAKE_PASSWORD --from-local-env --scope local &&
dg plus create env SNOWFLAKE_DATABASE --from-local-env --scope local
Reading environment variable SNOWFLAKE_ACCOUNT from project .env file
Environment variable SNOWFLAKE_ACCOUNT set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_USER from project .env file
Environment variable SNOWFLAKE_USER set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_PASSWORD from project .env file
Environment variable SNOWFLAKE_PASSWORD set in local scope for location ingestion in deployment prod
Reading environment variable SNOWFLAKE_DATABASE from project .env file
Environment variable SNOWFLAKE_DATABASE set in local scope for location ingestion in deployment prod
Running dg list env again will confirm that the environment variables have been pushed to your deployment:
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ ✓ │ │ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ ✓ │ │ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘
Now that local scope environment variables have been pushed to your deployment, your teammates can pull them into a .env file for local use:
dg plus pull env
Environment variables saved to .env
Configure production environment variables
Finally, you can configure a different set of production environment variables for your Dagster+ deployment:
dg plus create env SNOWFLAKE_ACCOUNT ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_USER ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_PASSWORD ... --scope branch --scope full &&
dg plus create env SNOWFLAKE_DATABASE production --scope branch --scope full
Environment variable SNOWFLAKE_ACCOUNT set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_USER set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_PASSWORD set in branch, full scope for location ingestion in deployment prod
Environment variable SNOWFLAKE_DATABASE set in branch, full scope for location ingestion in deployment prod
dg list env
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━┓
┃ Env Var ┃ Value ┃ Components ┃ Dev ┃ Branch ┃ Full ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━┩
│ SNOWFLAKE_ACCOUNT │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_DATABASE │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_PASSWORD │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
│ SNOWFLAKE_USER │ ✓ │ ingest_files │ ✓ │ ✓ │ ✓ │
└────────────────────┴───────┴──────────────┴─────┴────────┴──────┘