FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

microsoftexpert/terraform-databricks-metastore: Terraform module: terraform-databricks-metastore · GitHub

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

🧱 Databricks Metastore Terraform Module

Provisions a Unity Catalog metastore — the account-level root container for every catalog, schema, table, and grant in a Databricks Unity Catalog hierarchy — against the databricks/databricks provider ~> 1.117.0.

🧩 Overview

  • 🗄️ Creates one databricks_metastore — the account-level root of Unity Catalog.
  • 🔐 Defaults to no Delta Sharing, no external-client access, and no destroy-without-review.
  • 🌍 Carries the metastore's own regional and storage-root metadata (not provider authentication).
  • 🚫 Never accepts a credential, host, or account ID — the caller's account-level provider supplies those.

💡 Why it matters: every other Unity Catalog object in this library — catalogs, schemas, volumes, external locations, storage credentials, grants — exists inside exactly one metastore. Getting this module's secure defaults right (external access off, destroy protection on, bounded Delta Sharing token lifetimes) sets the security posture for the entire governance chain beneath it.


❤️ Support this project

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!


🗺️ Where this fits

flowchart LR
 THIS["terraform-databricks-metastore"]
 style THIS fill:#FF3621,color:#fff,stroke:#1B3139,stroke-width:1px

 ASSIGN["terraform-databricks-metastore-assignment"]
 style ASSIGN fill:#1B3139,color:#fff,stroke:#1B3139,stroke-width:1px

 CATALOG["terraform-databricks-catalog"]
 SCHEMA["terraform-databricks-schema"]
 VOLUME["terraform-databricks-volume"]
 style CATALOG fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px
 style SCHEMA fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px
 style VOLUME fill:#F2F2F2,color:#1B3139,stroke:#CCCCCC,stroke-width:1px

 THIS -->|"id becomes metastore_id"| ASSIGN
 ASSIGN -.->|"binds metastore to workspace (no direct reference)"| CATALOG
 CATALOG --> SCHEMA
 SCHEMA --> VOLUME
Loading

This module is the account-plane foundation of the Unity Catalog chain (first in this library's recommended authoring order). Its nearest sibling is terraform-databricks-metastore-assignment, which consumes this module's id output directly as metastore_id. Downstream modules (terraform-databricks-catalog, -schema, -volume) depend on a metastore existing and being assigned to their workspace, but do not reference this module's output directly — they operate against whichever metastore is assigned to the workspace their provider targets.

🧬 What this builds

flowchart TB
 subgraph INPUTS["var.*"]
 NAME["name"]
 STORAGE["storage_root / storage_root_credential_id|name"]
 REGION["region"]
 SHARING["delta_sharing_scope / recipient_token_lifetime / organization_name"]
 FLAGS["external_access_enabled / force_destroy / owner"]
 end

 KEYSTONE["databricks_metastore.this"]
 style KEYSTONE fill:#1B3139,color:#fff,stroke:#1B3139,stroke-width:1px

 subgraph OUTPUTS["outputs"]
 ID["id / metastore_id"]
 GLOBAL["global_metastore_id"]
 REST["cloud, storage_root_credential_*, default_data_access_config_id, privilege_model_version"]
 end

 NAME --> KEYSTONE
 STORAGE --> KEYSTONE
 REGION --> KEYSTONE
 SHARING --> KEYSTONE
 FLAGS --> KEYSTONE

 KEYSTONE --> ID
 KEYSTONE --> GLOBAL
 KEYSTONE --> REST
Loading

Resource inventory: one resource, databricks_metastore.this. No child collection — a metastore's storage-root and Delta Sharing settings are metastore-scoped fields, not independent objects.

✅ Provider / Versions

Requirement Value
Terraform >= 1.12.0
databricks/databricks ~> 1.117.0
Provider block None — the caller's root module configures provider "databricks" {}
tags / custom_tags Not supported by databricks_metastore — none added
timeouts Not supported by databricks_metastore — none added

Schema notes that bite:

  • storage_root forces replacement of the entire metastore if changed after creation. Replacing a metastore orphans every catalog/schema/table registered under the old one — this is the highest blast-radius immutable field in this module.
  • delta_sharing_organization_name, once set, cannot be removed — only changed to another valid value. Removing it requires tainting and recreating the metastore.
  • region is documented by the provider as "Mandatory for account-level" API use but optional (inferred) under a workspace-level provider. This module cannot detect which plane the caller's provider targets at plan time, so region stays nullable — set it explicitly under an account-level provider.
  • The live schema also exposes an api ("account" / "workspace") attribute and a provider_config { workspace_id } block for overriding which plane a single resource instance targets, independent of the provider block. This module deliberately does not expose either as a variable — plane selection is a provider-block concern per this library's authentication model, and a per-resource override here would let a module call silently contradict its own provider's configured plane. If a future use case genuinely needs this override, treat it as a new, explicit module variable added with its own design discussion, not a silent default.
  • The provider's own documentation lists storage_root_credential_name only under "Attribute Reference" (implying output-only), while the machine-readable schema (terraform validate's actual source of truth) marks it as a plain settable input. This module follows the machine-readable schema.

🔑 Required Databricks Permissions & Scopes

  • Account admin. Metastore creation is restricted to account admins even though the resource schema technically accepts a workspace-level provider.
  • If storage_root_credential_id / storage_root_credential_name is set: the referenced databricks_storage_credential (see terraform-databricks-storage-credential) must already exist and be usable by the applying identity.

Databricks Prerequisites

  • Account-level provider context — host = "https://accounts.azuredatabricks.net" (Azure) or "https://accounts.cloud.databricks.com" (AWS/GCP), plus account_id. This module accepts no auth-shaped variables.
  • Never call this module from a per-team workspace root module. This is rare, high-blast-radius, shared infrastructure — one metastore is typically assigned to many workspaces via terraform-databricks-metastore-assignment. Call it from a small number of platform-team root modules only.
  • No existing Unity Catalog metastore is required — this module creates the root of that hierarchy.

📁 Module Structure

terraform-databricks-metastore/
├── providers.tf # required_providers only — no provider {} block
├── variables.tf # name, storage_root(+credential), region, owner, Delta Sharing, flags
├── main.tf # databricks_metastore.this
├── outputs.tf # id first, then name and computed metastore metadata
├── SCOPE.md # cross-module contract
├── README.md # this file
└── examples/
 └── basic/
 └── main.tf # smallest real, runnable call

⚙️ Quick Start

module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-primary-metastore"
  region       = "eastus"
  storage_root = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner        = "uc-admins"
}

ℹ️ This is an account-plane module. The caller's root module must configure a provider with host = "https://accounts.azuredatabricks.net" (Azure) or "https://accounts.cloud.databricks.com" (AWS/GCP) and an account_id — this module accepts neither.

🔌 Cross-Module Contract

Consumes: none — this module is the root of the Unity Catalog chain and has no sibling dependencies.

Emits:

Output Description Consumed by
id System-generated metastore ID (identical to metastore_id) terraform-databricks-metastore-assignment (metastore_id input)
name Metastore name Operator tooling / auditing
metastore_id Unique identifier of the metastore Same as id
global_metastore_id Globally unique ID (cloud:region:metastore_id) Delta Sharing configuration, cross-account tooling
cloud Cloud vendor of the metastore's home shard Auditing / drift-detection tooling
storage_root_credential_id, storage_root_credential_name Echoed storage-credential reference, if set Auditing
default_data_access_config_id, privilege_model_version Metastore-managed computed attributes Auditing

📚 Example Library

1 · Minimal metastore (no storage root)
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name = "casey-minimal-metastore"
}

ℹ️ Every catalog created under this metastore must then define its own storage_root — this library's house recommendation regardless (see terraform-databricks-catalog).

2 · AWS metastore with S3 storage root
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-aws-metastore"
  region       = "us-east-1"
  storage_root = "s3://casey-unity-catalog-metastore/root"
  owner        = "uc-admins"
}
3 · Azure metastore with ADLS Gen2 storage root
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-azure-metastore"
  region       = "eastus"
  storage_root = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner        = "uc-admins"
}
4 · GCP metastore with GCS storage root
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-gcp-metastore"
  region       = "us-east1"
  storage_root = "gs://casey-unity-catalog-metastore"
  owner        = "uc-admins"
}
5 · Storage root with an explicit storage credential (by ID)
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                       = "casey-credentialed-metastore"
  region                     = "eastus"
  storage_root               = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  storage_root_credential_id = "11111111-2222-3333-4444-555555555555"
  owner                      = "uc-admins"
}

🔒 storage_root_credential_id and storage_root_credential_name are mutually exclusive — this module rejects a call that sets both at plan time.

6 · Delta Sharing enabled — internal only
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                = "casey-sharing-metastore"
  region              = "eastus"
  storage_root        = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner               = "uc-admins"
  delta_sharing_scope = "INTERNAL"
}

💡 delta_sharing_recipient_token_lifetime_in_seconds keeps its secure default of 604800 (7 days) even though it has no external-recipient effect under INTERNAL scope.

7 · Delta Sharing enabled — external, with a bounded token lifetime
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                                              = "casey-external-sharing-metastore"
  region                                            = "eastus"
  storage_root                                      = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner                                             = "uc-admins"
  delta_sharing_scope                               = "INTERNAL_AND_EXTERNAL"
  delta_sharing_recipient_token_lifetime_in_seconds = 259200 # 3 days
}

⚠️ INTERNAL_AND_EXTERNAL allows token-based sharing to recipients outside this Databricks account — reserve for a specific, reviewed integration.

8 · Databricks-to-Databricks sharing organization name
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                            = "casey-d2d-sharing-metastore"
  region                          = "eastus"
  storage_root                    = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner                           = "uc-admins"
  delta_sharing_scope             = "INTERNAL_AND_EXTERNAL"
  delta_sharing_organization_name = "casey-financial-partners"
}

⚠️ Once set, delta_sharing_organization_name cannot be removed — only changed to another valid value. Removing it requires tainting and recreating the metastore.

9 · External client access enabled (opt-in)
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                    = "casey-external-clients-metastore"
  region                  = "eastus"
  storage_root            = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner                   = "uc-admins"
  external_access_enabled = true
}

⚠️ Secure default is false. Only set true for a specific, reviewed non-Databricks-Runtime client integration.

10 · Destroy protection explicitly disabled (deliberate teardown)
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name          = "casey-decommission-metastore"
  region        = "eastus"
  force_destroy = true
}

⚠️ Secure default is false. Only set true immediately before a reviewed, deliberate teardown — never as a standing configuration.

11 · Full configuration surface exercised together
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name                                              = "casey-full-config-metastore"
  region                                            = "eastus"
  storage_root                                      = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  storage_root_credential_id                        = "11111111-2222-3333-4444-555555555555"
  owner                                             = "uc-admins"
  delta_sharing_scope                               = "INTERNAL_AND_EXTERNAL"
  delta_sharing_recipient_token_lifetime_in_seconds = 259200
  delta_sharing_organization_name                   = "casey-financial-partners"
  external_access_enabled                           = false
  force_destroy                                     = false
}
12 · Minimal least-privilege baseline (recommended starting point)
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-baseline-metastore"
  region       = "eastus"
  storage_root = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner        = "uc-admins"
  # delta_sharing_scope left null: sharing disabled
  # external_access_enabled left false (default): no non-DBR client access
  # force_destroy left false (default): no accidental teardown
}

💡 This is the recommended starting configuration for a new metastore in a regulated environment — every optional feature (Delta Sharing, external access) starts off, requiring a deliberate, separate change to enable.

13 · Multi-region reference (documentation only — one metastore per call)
module "metastore_us" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-metastore-us-east"
  region       = "us-east-1"
  storage_root = "s3://casey-unity-catalog-metastore-us-east/root"
  owner        = "uc-admins"
}

module "metastore_eu" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-metastore-eu-west"
  region       = "eu-west-1"
  storage_root = "s3://casey-unity-catalog-metastore-eu-west/root"
  owner        = "uc-admins"
}

ℹ️ Each metastore is a fully independent top-level container — there is no cross-metastore reference in this module. Multiple metastores per account are a real, supported topology (e.g. one per major region), each assigned to its own set of workspaces via terraform-databricks-metastore-assignment.

🏗️ 14 · End-to-end composition — metastore → assignment → catalog
module "metastore" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore.git?ref=v1.0.0"

  name         = "casey-primary-metastore"
  region       = "eastus"
  storage_root = "abfss://unity-catalog@caseyuc.dfs.core.windows.net/"
  owner        = "uc-admins"
}

module "metastore_assignment" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-metastore-assignment.git?ref=v1.0.0"

  metastore_id = module.metastore.id
  workspace_id = 123456789012345
}

module "analytics_catalog" {
  source = "git::https://github.com/microsoftexpert/terraform-databricks-catalog.git?ref=v1.0.0"

  name = "analytics"

  depends_on = [module.metastore_assignment]
}

ℹ️ terraform-databricks-metastore-assignment and terraform-databricks-catalog are seeded modules in this same catalog batch — this composition reflects their planned contract (per SCOPE.md), not yet a verified cross-module terraform plan. depends_on is required here because terraform-databricks-catalog has no direct Terraform reference to the metastore — Unity Catalog resolves the catalog against whichever metastore is assigned to the workspace at apply time, an implicit dependency Terraform's graph cannot see from resource references alone.

📥 Inputs

Variable Type Default Notes
name string — (required) Metastore name
storage_root string null Force-new if changed
storage_root_credential_id string null Mutually exclusive with storage_root_credential_name
storage_root_credential_name string null Mutually exclusive with storage_root_credential_id
region string null Mandatory under an account-level provider
owner string null
delta_sharing_scope string null INTERNAL | INTERNAL_AND_EXTERNAL
delta_sharing_recipient_token_lifetime_in_seconds number 604800 Secure default; API default is 31536000
delta_sharing_organization_name string null Cannot be unset once configured
external_access_enabled bool false Secure default
force_destroy bool false Secure default
Full variable declarations
variable "name" {
  type = string
}

variable "storage_root" {
  type    = string
  default = null
}

variable "storage_root_credential_id" {
  type    = string
  default = null
}

variable "storage_root_credential_name" {
  type    = string
  default = null
}

variable "region" {
  type    = string
  default = null
}

variable "owner" {
  type    = string
  default = null
}

variable "delta_sharing_scope" {
  type    = string
  default = null
  # validation: must be "INTERNAL" or "INTERNAL_AND_EXTERNAL" when non-null
}

variable "delta_sharing_recipient_token_lifetime_in_seconds" {
  type    = number
  default = 604800
  # validation: must be > 0
}

variable "delta_sharing_organization_name" {
  type    = string
  default = null
}

variable "external_access_enabled" {
  type    = bool
  default = false
}

variable "force_destroy" {
  type    = bool
  default = false
}

🧾 Outputs

Output Description Sensitive?
id System-generated metastore ID (same as metastore_id) No
name Metastore name No
metastore_id Unique identifier of the metastore No
global_metastore_id Globally unique ID (cloud:region:metastore_id) No
cloud Cloud vendor of the metastore's home shard No
storage_root_credential_id Echoed storage-credential ID, if set No
storage_root_credential_name Echoed storage-credential name, if set No
default_data_access_config_id Metastore's default data access configuration ID, if any No
privilege_model_version Privilege model version (major.minor) No

🧠 Architecture Notes

  • storage_root is force-new. Changing it after creation destroys and recreates the metastore, orphaning every catalog/schema/table registered underneath. Treat any storage_root change as a full-hierarchy migration event, not a routine update.
  • storage_root_credential_id / storage_root_credential_name are mutually exclusive by this module's own validation {} block — the provider schema does not enforce this itself. storage_root_credential_name's settable-input status is asserted from the machine-readable schema rather than the prose documentation, which lists it only as an attribute; re-verify at the next provider version bump.
  • region's requirement is plane-dependent (mandatory under an account-level provider, inferred under workspace-level) in a way this module cannot check at plan time — a caller misconfiguring this pairing will only find out at apply.
  • api / provider_config are intentionally absent. Plane selection is left entirely to the caller's provider configuration, per this library's authentication model — this module never lets a single resource call override the plane implied by its provider block.
  • No child resources, no for_each. A metastore has no meaningful child collection in this module's scope — Delta Sharing and storage-root settings are metastore-scoped fields, not separate objects.

🧱 Design Principles

Concern Secure default Opt-out (caller must set explicitly)
External client access external_access_enabled = false Explicit true, for a specific reviewed integration
Destroy protection force_destroy = false Explicit true, for a deliberate, reviewed teardown
Delta Sharing Disabled (delta_sharing_scope = null) Caller sets INTERNAL or INTERNAL_AND_EXTERNAL explicitly
Delta Sharing recipient token lifetime 604800 seconds (7 days), well under the API's 1-year default Caller sets a longer value explicitly

These defaults follow this library's Secure-by-default conventions — this module introduced all three (external_access_enabled, force_destroy at the metastore level, and the Delta Sharing token lifetime default) as new instances of that pattern.

🚀 Runbook

cd terraform-databricks-metastore
terraform init -backend=false
terraform validate
terraform fmt -check

Pin consumers to an immutable tag — ?ref=v1.0.0 — never a branch. This module is plan-only; a human applies from CI after review, against a sub-production environment first.

🧪 Testing

terraform validate / terraform fmt -check catch: missing required arguments (name), type mismatches, the storage_root_credential_id/_name mutual-exclusivity validation, the delta_sharing_scope enum validation, and malformed HCL. They do not catch: whether the applying identity actually holds account-admin rights, whether region is genuinely required for the caller's provider plane, whether a referenced storage_root_credential_id exists, or any real Unity Catalog API-side constraint (naming collision, region availability). Those require an actual plan/apply against a live account, which is out of scope for this authoring process.

💬 Example Output

$ terraform output
cloud = "azure"
default_data_access_config_id = null
global_metastore_id = "azure:eastus:12345678-90ab-cdef-1234-567890abcdef"
id = "12345678-90ab-cdef-1234-567890abcdef"
metastore_id = "12345678-90ab-cdef-1234-567890abcdef"
name = "casey-primary-metastore"
privilege_model_version = "1.0"
storage_root_credential_id = null
storage_root_credential_name = null

🔍 Troubleshooting

Symptom Cause Fix
terraform validate fails with "storage_root_credential_id or storage_root_credential_name, not both" Both mutually exclusive credential inputs were set Remove one — they identify the same storage credential two different ways
terraform validate fails on delta_sharing_scope Value other than INTERNAL / INTERNAL_AND_EXTERNAL Correct the value; re-verify against the live provider schema if the provider added a new scope value
Apply fails with a permissions error even though terraform validate passed Applying identity is not an account admin Confirm the identity/service principal running apply holds account-admin rights — validate cannot check live permissions
Apply attempts to replace the metastore unexpectedly storage_root was changed storage_root is force-new; treat any change as a deliberate, full-hierarchy migration, not a routine edit
terraform destroy fails with a "metastore not empty" style error force_destroy is false (the secure default) and the metastore still has registered catalogs/schemas Confirm the teardown is intentional, then set force_destroy = true explicitly for that run only

🔗 Related Docs

  • databricks_metastore provider resource
  • terraform-databricks-metastore-assignment (consumes this module's id)
  • terraform-databricks-catalog (downstream, same Unity Catalog chain)
  • This module's SCOPE.md

💙 "Infrastructure as Code should be standardized, consistent, and secure."

Releases

Packages

Contributors

Languages


Back | FazBrowse Home | New Git URL