This project automates an end-to-end ETL (Extract–Transform–Load) and Machine Learning (ML) pipeline for sentiment analysis on textual data such as product reviews.

It is designed to:

• Automatically extract and clean raw datasets
• Train a sentiment classification model (Logistic Regression + TF-IDF)
• Evaluate model performance
• Test predictions on unseen data
• Save results, models, and logs in an organized, production-ready format

The project is completely automated through a single entry point:

python run_pipeline.py

which runs the entire pipeline — from raw data to predictions.

ETL-Machine-Learning-Integration/
├── data/
│   ├── raw/
│   │   ├── train.csv
│   │   └── test.csv
│   └── processed/
│       ├── cleaned_with_predictions.csv
│       ├── test_cleaned.csv
│       └── test_predictions.csv
├── models/
│   ├── sentiment_model.pkl
│   └── vectorizer.pkl
├── sentiment_model.py
├── test_model_full.py
├── run_pipeline.py
├── etl_model.log
├── requirements.txt
└── README.md

• Reads train.csv from data/raw/
• Maps raw labels → {1: 0 (Negative), 2: 1 (Positive)}
• Removes unused columns
• Saves cleaned dataset as cleaned_with_predictions.csv

• TF-IDF Vectorization (max_features=5000, ngram_range=(1,2))
• Trains Logistic Regression classifier
• Evaluates on validation split (80/20)
• Prints accuracy and classification report
• Saves sentiment_model.pkl and vectorizer.pkl in /models/

• Cleans and prepares test.csv
• Loads saved model & vectorizer
• Combines title + review text
• Generates predictions and saves to test_predictions.csv
• Prints accuracy, precision, recall, and F1-score (if ground truth available)

• Controlled by run_pipeline.py
• Handles:
  • Directory creation
  • Data verification
  • Cleaning
  • Training (auto-skips if already exists)
  • Testing
  • Logging completion timestamps

pip install -r requirements.txt

Place your raw CSV files in:

data/raw/train.csv
data/raw/test.csv

Expected CSV Format:

"label","title","review_text"
2,"Great CD","My lovely Pat has one of the GREAT voices..."
1,"Batteries died within a year ...","I bought this charger in Jul 2003..."

python run_pipeline.py

This will:

1. Clean raw training data
2. Train model (if not already trained)
3. Prepare and test on new dataset
4. Save outputs and logs

Console Output:

🚀 Starting Full Sentiment Analysis Pipeline...

🧹 Preparing training data...
✅ Training data prepared -> data/processed/cleaned_with_predictions.csv

🤖 Training model (this may take a few minutes)...
✅ Model trained successfully (Accuracy: 0.869)
💾 Model and vectorizer saved to 'models/'

🧠 Testing on new dataset...
✅ Accuracy: 0.871
💾 Predictions saved to 'data/processed/test_predictions.csv'
🎯 Done! Check logs for details.

Sample of test_predictions.csv:

The model performs balanced on both classes and generalizes well for unseen reviews.

All runs are timestamped and saved to:

etl_model.log

Example log:

2025-10-18 18:32:44 [INFO] Model trained. Accuracy: 0.869
2025-10-18 18:35:07 [INFO] Model testing completed.
2025-10-18 18:35:07 [INFO] Pipeline run completed successfully.

• Automatically skips retraining if sentiment_model.pkl already exists.
• Automatically creates required directories (data/processed, models).
• Logs every event and prints progress in real-time via tqdm.
• Reproducible pipeline — run anytime to reprocess or test new data.

Vansh C. B.Tech (CSE), SRM University Kattankulathur (2023–2027) GitHub: VanshRajput-dev Email: vr7404560@gmail.com

This project represents a complete ML pipeline integrated with ETL automation — a professional-grade system for handling real-world sentiment analysis workflows.

• ✅ Fully modular
• ✅ Automatically executable
• ✅ Easily extensible
• ✅ Production-style logging & progress