🎨 UI for the Free Data Engineering Zoomcamp Course provided by DataTalksClub

Real-time data pipeline for collecting, processing, and visualizing telemetry data from Assetto Corsa Competizione using Mage, Kafka, PostgreSQL, and Metabase for interactive insights into vehicle performance and driver statistics.

Contains a Python script designed to validate data replication tasks in AWS Database Migration Service (DMS). The script checks for potential issues such as missing or duplicate data in the tables being replicated.

The Zoomcamp LLM Course focuses on tools for working with LLMs and RAG, including OpenAI API, HuggingFace, Elasticsearch, and Streamlit. It covers vector search, embedding creation, data ingestion with Mage, and monitoring using Grafana, emphasizing practical applications and best practices.

An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai

The AWS DMS Task Restart and Status Checker is a Python script designed to restart various AWS Database Migration Service (DMS) tasks and check their status. This script leverages AWS SDK (Boto3) and Mage.ai for DMS task integration and management, ensuring efficient and reliable task handling.

The Data Engineering Zoomcamp covers essential skills in containerization, workflow orchestration, data warehousing, analytics engineering, batch, and streaming processing. It includes tools like Docker, Terraform, BigQuery, dbt, Spark, Kafka, Kestra, Postgres, Google Data Studio, and Metabase.

This script automates cleaning up a specific folder in an S3 bucket, deleting all objects within it. It uses secure AWS credentials and is built on the Mage.ai platform. Additionally, the roadmap includes approach handling and logging for greater robustness and monitoring.

Automated script developed in Mage.ai to monitor and validate the status of AWS DMS CDC tasks, ensuring data integrity and synchronization. Sends notifications for any detected validation issues.

This project is a Data Engineering solution implemented to validate the data types of columns in PostgreSQL tables in a DataMart. It aims to validate whether the data stored in tables conforms to the expected data types, improving data integrity and reliability.

ETL pipeline on the New York Times data with apache superset for data visualization

pokéFastApi: A backend API with Docker Compose integration for retrieving and searching Pokémon data using FastAPI and DuckDB.

Using machine learning, feature engineering, and web scraping, I created an end-to-end laptop price prediction website by scraping data from a popular Iranian source. Empowering users with accurate pricing estimates and model comparisons

An ETL Pipeline built over GCP and orchestrated by Mage, which involves Extracting Data from GCS Bucket, building Dimensional Model, loading the Data into BigQuery and a Looker Dashboard for further analysis.

Code examples of Luigi, Prefect, Kedro, Dagster, and MageAI

Data Engineering Zoomcamp Final Project

Datatalks Club Free Data Engineering Zoomcamp Project

Modern data pipeline with Mage, dbt, PostgreSQL and Metabase.

A covid 19 Data engieneering pipeline

Swiss Air Quality Data Engineering Pipeline