| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: dbt-integration-tests | ||
|
|
||
| # Run dbt integration tests on PRs | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'sdk/python/feast/dbt/**' | ||
| - 'sdk/python/tests/integration/dbt/**' | ||
| - 'sdk/python/tests/unit/dbt/**' | ||
| - '.github/workflows/dbt-integration-tests.yml' | ||
|
|
||
| jobs: | ||
| dbt-integration-test: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ["3.11", "3.12"] | ||
| env: | ||
| PYTHON: ${{ matrix.python-version }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| architecture: x64 | ||
|
|
||
| - name: Install the latest version of uv | ||
| uses: astral-sh/setup-uv@v5 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Install dependencies | ||
| run: make install-python-dependencies-ci | ||
|
|
||
| - name: Install dbt and dbt-duckdb | ||
| run: | | ||
| uv pip install --system dbt-core dbt-duckdb | ||
|
|
||
| - name: Run dbt commands | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/test_dbt_project | ||
| dbt deps | ||
| dbt build | ||
| dbt test | ||
|
|
||
| - name: Setup Feast project for dbt import test | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt | ||
| mkdir -p feast_repo | ||
| cd feast_repo | ||
| cat > feature_store.yaml << EOF | ||
| project: feast_dbt_test | ||
| registry: data/registry.db | ||
| provider: local | ||
| online_store: | ||
| type: sqlite | ||
| path: data/online_store.db | ||
| EOF | ||
| mkdir -p data | ||
|
|
||
| - name: Test feast dbt import | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/feast_repo | ||
| feast -c feature_store.yaml dbt import \ | ||
| -m ../test_dbt_project/target/manifest.json \ | ||
| -e driver_id \ | ||
| -d file \ | ||
| --tag feast | ||
|
|
||
| - name: Verify feast objects were created | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/feast_repo | ||
| feast -c feature_store.yaml feature-views list | ||
| feast -c feature_store.yaml entities list | ||
|
|
||
| - name: Run dbt integration tests | ||
|
Comment thread
Copy link
Copy Markdown
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality@copillot we should also run the standard dbt commands; i.e,. dbt run
dbt build
dbt testafter that we should then test the new feast cli and test that feast materialization works from a dbt model into a sqlite online store.
Sorry, something went wrong.
All reactions
Copy link
Copy Markdown
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityAdded dbt commands (deps, build, test) and Feast CLI testing with SQLite online store in commit ba38da0. The workflow now:
Sorry, something went wrong.
All reactions
|
||
| run: | | ||
| cd sdk/python | ||
| python -m pytest tests/integration/dbt/test_dbt_integration.py -v --tb=short | ||
|
|
||
| - name: Minimize uv cache | ||
| run: uv cache prune --ci | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # dbt Integration Tests | ||
|
|
||
| Integration tests for Feast's dbt integration feature, which allows importing dbt models as Feast FeatureViews. | ||
|
|
||
| ## Overview | ||
|
|
||
| These tests verify the complete workflow of: | ||
| 1. Parsing dbt `manifest.json` files using dbt-artifacts-parser | ||
| 2. Extracting model metadata (columns, types, tags, descriptions) | ||
| 3. Creating Feast objects (Entity, DataSource, FeatureView) from dbt models | ||
| 4. Generating Python code with Feast definitions | ||
|
|
||
| ## Test Coverage | ||
|
|
||
| ### TestDbtManifestParsing | ||
| - Parse manifest metadata (dbt version, project name) | ||
| - Extract all models from manifest | ||
| - Filter models by dbt tags | ||
| - Filter models by name | ||
| - Parse model properties (database, schema, table, description, tags) | ||
| - Parse column metadata (name, type, description) | ||
|
|
||
| ### TestDbtToFeastMapping | ||
| - Create BigQuery data sources from dbt models | ||
| - Create Snowflake data sources from dbt models | ||
| - Create File data sources from dbt models | ||
| - Create Feast entities | ||
| - Create FeatureViews with proper schema | ||
| - Exclude entity and timestamp columns from features | ||
| - Handle custom excluded columns | ||
| - Create all objects together (Entity + DataSource + FeatureView) | ||
|
|
||
| ### TestDbtDataSourceTypes | ||
| - Test all supported data source types (bigquery, snowflake, file) | ||
| - Verify unsupported types raise errors | ||
|
|
||
| ### TestDbtCodeGeneration | ||
| - Generate Python code from dbt models | ||
| - Generate code for different data source types | ||
| - Verify generated code structure and imports | ||
|
|
||
| ### TestDbtTypeMapping | ||
| - Map dbt types to Feast types correctly: | ||
| - STRING → String | ||
| - INT32 → Int32 | ||
| - INT64 → Int64 | ||
| - FLOAT32 → Float32 | ||
| - FLOAT64 → Float64 | ||
| - TIMESTAMP → UnixTimestamp | ||
|
|
||
| ### TestDbtIntegrationWorkflow | ||
| - End-to-end workflow with multiple models | ||
| - Code generation workflow with file output | ||
|
|
||
| ## Test Data | ||
|
|
||
| Tests use a pre-generated dbt manifest from `test_dbt_project/`: | ||
| - 3 models: driver_features, customer_features, product_features | ||
| - Various column types and tags for comprehensive testing | ||
| - No external dependencies (database, dbt CLI) required | ||
|
|
||
| ## Running Tests | ||
|
|
||
| Run all dbt integration tests: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/ -v | ||
| ``` | ||
|
|
||
| Run specific test class: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/test_dbt_integration.py::TestDbtManifestParsing -v | ||
| ``` | ||
|
|
||
| Run specific test: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/test_dbt_integration.py::TestDbtManifestParsing::test_parse_manifest_metadata -v | ||
| ``` | ||
|
|
||
| ## Dependencies | ||
|
|
||
| Required: | ||
| - `dbt-artifacts-parser>=0.6.0` - For parsing dbt manifest files | ||
| - `feast` - Core Feast SDK with all data source types | ||
|
|
||
| ## CI/CD | ||
|
|
||
| Tests run in GitHub Actions: | ||
| - `.github/workflows/dbt-integration-tests.yml` - Dedicated dbt test workflow | ||
| - `.github/workflows/unit_tests.yml` - As part of general unit tests | ||
|
|
||
| ## Related Code | ||
|
|
||
| - `feast/dbt/parser.py` - dbt manifest parser | ||
| - `feast/dbt/mapper.py` - dbt to Feast object mapper | ||
| - `feast/dbt/codegen.py` - Python code generator | ||
| - `feast/cli/dbt_import.py` - CLI commands for dbt import |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # Integration tests for dbt import functionality |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| """ | ||
| Conftest for dbt integration tests. | ||
|
|
||
| This is a standalone conftest that doesn't depend on the main Feast test infrastructure. | ||
| """ | ||
|
|
||
| import pytest | ||
|
|
||
| # This conftest is minimal and doesn't import the main feast conftest | ||
| # to avoid complex dependency chains for dbt-specific tests |
| Back | FazBrowse Home | New Git URL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality@copilot we should use the Makefile to install dependencies here
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityUpdated to use make install-python-dependencies-ci in commit 621cefb.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.