FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Feat: Add model defaults support per gateway configuration by VaggelisD · Pull Request #3888 · SQLMesh/sqlmesh · GitHub

/ sqlmesh Public

Feat: Add model defaults support per gateway configuration - #3888

Merged
VaggelisD merged 4 commits into
mainfrom
vaggelisd/dialect_normalization
Feb 25, 2025
ConversationCommits (4)ChecksFiles changed
Merged

Feat: Add model defaults support per gateway configuration#3888
VaggelisD merged 4 commits into
mainfrom
vaggelisd/dialect_normalization

Conversation

VaggelisD commented Feb 24, 2025

Copy link
Copy Markdown
Collaborator

Users can employ different gateways on the same project:

gateways:
  redshift:
      connection:
        type: redshift
        ...

  snowflake:
    connection:
      type: snowflake
      ...

default_gateway: snowflake

model_defaults:
  dialect: snowflake
  start: 2025-02-05

However, the models will be normalized at the model_defaults.dialect even if the default_gateway is set to a different engine, which can lead to issues / inefficiencies e.g Redshift's INFORMATION_SCHEMA stores the tables as lower-case, so with Snowflake's upper-case normalization we'll fail to find the tables/views.

This PR adds support for model_defaults under gateways which override the global dialect, e.g:

gateways:
  redshift:
      connection:
        type: redshift
        ...

      model_defaults:
        dialect: snowflake, normalization_strategy=case_insensitive

This enables the following behavior:

# Snowflake dialect, Redshift gateway without model_defaults
❯ sqlmesh render vaggelis.test_model
[WARNING] The redshift engine is not recommended for storing SQLMesh state in production deployments. Please see 
https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#state-connection for a list of recommended engines and more information.
SELECT                                                                                                                                              
  "COL" AS "COL"                                                                                                                                    
FROM "TBL" AS "TBL"  

# Snowflake dialect, Redshift gateway with model_defaults.dialect: snowflake, normalization_strategy=case_insensitive
❯ sqlmesh render vaggelis.test_model
[WARNING] The redshift engine is not recommended for storing SQLMesh state in production deployments. Please see 
https://sqlmesh.readthedocs.io/en/stable/guides/configuration/#state-connection for a list of recommended engines and more information.
SELECT                                                                                                                                              
  "col" AS "col"                                                                                                                                    
FROM "tbl" AS "tbl"

georgesittas reviewed Feb 24, 2025
View reviewed changes
Comment thread sqlmesh/core/config/root.py Outdated
VaggelisD changed the title Feat: Set normalization strategy according to used gateway Feat: Add model defaults support per gateway configuration Feb 24, 2025
VaggelisD requested a review from a team February 24, 2025 20:33
georgesittas approved these changes Feb 24, 2025
View reviewed changes
Comment thread tests/core/test_context.py Outdated
VaggelisD force-pushed the vaggelisd/dialect_normalization branch from f54cafa to 5b9e246 Compare February 24, 2025 20:57
VaggelisD requested a review from a team February 25, 2025 15:29
VaggelisD merged commit 23697e1 into main Feb 25, 2025
VaggelisD deleted the vaggelisd/dialect_normalization branch February 25, 2025 19:16
This was referenced Aug 20, 2026
Merged
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

tobymao tobymao left review comments
georgesittas georgesittas approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

Back | FazBrowse Home | New Git URL