Data Mutability Features

Data mutability applies INSERT, UPDATE, and DELETE operations from your warehouse to event data already in Amplitude, so corrections, late-arriving rows, and deletions in your source of truth propagate to your analytics. Mirror Sync delivers these operations from Snowflake, Databricks, Google BigQuery, and Amazon S3.

Turn on Mirror Sync when your warehouse is the source of truth and rows there change after first write, such as GDPR or CCPA deletions, backfilled corrections, or late-arriving updates. Stay on standard append-only ingestion when events never change after they land.

This functionality is especially valuable for organizations that need to:

• Correct historical data errors.
• Adhere to data privacy regulations (GDPR, CCPA).
• Maintain data consistency across systems.
• Handle late-arriving or corrected data.

Supported data sources

Data Mutability is available through the following warehouse integrations.

Snowflake

• Mirror Sync strategy with Change Data Capture (CDC).
• Supports INSERT, UPDATE, and DELETE operations.
• Requires Change Tracking enabled on source tables.
• Learn more about Snowflake integration .

Databricks

• Mirror Sync strategy with Change Data Feed (CDF).
• Supports INSERT, UPDATE, and DELETE operations.
• Requires Change Data Feed enabled on Delta tables.
• Learn more about Databricks integration .

Google BigQuery (Beta)

• Mirror Sync strategy with BigQuery's CHANGES() change history.
• Supports INSERT, UPDATE, and DELETE operations for the Event data type.
• Requires change history enabled on source tables (enable_change_history = TRUE).
• Learn more about BigQuery integration .

Amazon S3

• Mirror Sync strategy for file-based mutations.
• Supports INSERT, UPDATE, and DELETE operations.
• Requires structured mutation metadata in your data files.
• Learn more about Amazon S3 integration .

How Mirror Sync works

When you enable Mirror Sync with data mutability:

1. Change detection: The integration monitors your warehouse for data changes using native change tracking features (CDC for Snowflake, CDF for Databricks, CHANGES() change history for BigQuery, or file metadata for S3).

2. Operation processing: Amplitude processes three types of operations:

   • INSERT: Adds new events to Amplitude.
   • UPDATE: Modifies existing events in Amplitude.
   • DELETE: Removes events from Amplitude.

   Amplitude finds matching events based on the combination of user_id, insert_id, and event_time. All three fields must match before Amplitude can identify and modify the correct event.

3. Data synchronization: Changes apply to keep consistency between your warehouse and Amplitude.

Enrichment services

General requirements

• User ID required: All events must contain a user ID. Mirror Sync doesn't support anonymous events.
• Unique Insert ID: Each event should have a unique and immutable insert_id to prevent duplication.
• Chronological order: Process events in chronological order when possible.

Event volume considerations

Data retention

• Snowflake: DATA_RETENTION_TIME_IN_DAYS must be 1 (recommended: 7 days).
• Databricks: Change Data Feed retention must cover your sync frequency.
• BigQuery: The table's time-travel window must cover your sync frequency (default seven days, configurable from two to seven days).
• S3: Files must remain accessible throughout processing.

Best practices

Keep the following best practices in mind as you enable data mutability.

Plan your implementation

1. Start with a test project: Create a dedicated test environment to validate your mutation logic before implementing in production.

2. Design for idempotency: Ensure your mutation operations can be safely retried without causing data inconsistencies.

3. Monitor data quality: Implement validation checks to ensure mutations apply correctly.

Data privacy compliance

When using data mutability for privacy compliance:

1. Stop data flow first: Before you delete user data, ensure you send no new data about that user to Amplitude.

2. Use User Privacy API: For complete user deletion, use the User Privacy API with warehouse deletions.

3. Verify deletion: Confirm that deleted data no longer appears in your analytics.

Performance optimization

• Batch operations: Group related mutations together when possible.
• Optimize sync frequency: Balance data freshness needs with processing overhead.
• Monitor resource usage: Track warehouse compute costs associated with change tracking.

Migrate to data mutability

If you're migrating from a standard ingestion strategy to Mirror Sync, follow these steps.

Recommended migration steps

1. Create cutoff strategy:

   • Modify the existing connection with a time filter (for example, WHERE time < {cutOffDate}).
   • Set the cutoff date to tomorrow in milliseconds since epoch.

2. Wait for cutoff: Allow the cutoff date to pass and verify no new data flows through the old connection.

3. Create new Mirror Sync source:

   • Configure the new source with a complementary filter (for example, WHERE time >= {cutOffDate}).
   • Enable Mirror Sync with the mutation settings you want.

4. Clean up: Remove the old source connection after verifying the new one works correctly.

Common issues

Events don't update

• Verify that change tracking is enabled on source tables.
• Check that events contain required user IDs.
• Confirm sync frequency settings.

Missing deletions

• Ensure DELETE operations are properly configured in your source.
• Verify that deleted events had valid user IDs.
• Check that change retention periods haven't expired.

Data inconsistencies

• Review mutation operation ordering.
• Verify that Amplitude disabled enrichment services as expected.
• Check for timing issues between warehouse changes and sync execution.