Folders and files
-
Data ingestion into the project > Data Validation > Data Transformation
Data to Model - Model to Data (iteratively)
-
Model Training - Model Evaluation - Model Testing - Model Packaging (.pkl)
Supervised? Unsupervised? Classification? Regression?
- User App - Deployment
Virtual Machine Commands: sudo apt update sudo apt install python3-pip sudo apt install virtualenv mkdir mlflow cd mlflow pipenv install mlflow pipenv install awscli pipenv install boto3 pipenv shell aws configure
`python3 -m venv mlops`
git init
4. DVC (Data Version Control) Install DVC
DVC is very useful because it will run only the files in our pipeline that changed and not the entire pipeline. We will run our pipeline (all (.py) files+) through DVC. With DVC we avoid unnecessary computation.
-
pip install dvc
-
dvc init
-
git add .dvc .dvcignore
-
git commit -m "Initialize DVC"
-
create file in project's directory: dvc.yaml (configuration file) manually or by running e.g. dvc run -n preprocess -d src/preprocess.py -d data/raw -o data/processed
python src/preprocess.py data/raw data/processed -
dvc repro (this will execute dvc.yaml) , dvc dag , dvc metrics show
Split Jupyter Notebook into Python Scripts (Modular Approach):
- data_ingestion.py
- data_preprocessing.py
- feature_engineering.py
- model_building.py
- model_evaluation.py
- Output metrics.json
See here MLflow Repo
docker build -t username/app: latest docker run -p 8080:8080 username/app: latest docker run -d -p 8080:8080 username/app: latest
docker login docker push username/app: latest
Install Jenkins in AWS EC2 Ubuntu Instance sudo apt update sudo apt install openjdk-8-jdk -y visit: https://pkg.jenkins.io/debian-stable/ sudo systemctl start jenkins sudo systemctl enable jenkins sudo systemctl status jenkins add 8080 port to the security group sudo cat /var/lib/jenkins/secrets/initialAdminPassword
commands:
- git clone project in Google Cloud
- export PROJECT_ID=eng-hangar-466608-e9 (Google Cloud Project ID)
- docker build -t gcr.io/${PROJECT_ID}/{DOCKER-IMAGE} .
- docker images
- gcloud auth configure-docker gcr.io
- docker push gcr.io/${PROJECT_ID}/{DOCKER-IMAGE}
- gcloud config set compute/zone us-central1
- gcloud container clusters create app-cluster --num-nodes=1
- kubectl create deployment app --image=gcr.io/${PROJECT_ID}/{DOCKER-IMAGE}
- kubectl expose deployment app --type=LoadBalancer --port 80 --target-port 8080
- docker run -d -p 3000:3000 --name=grafana grafana/grafana-oss
- admin - admin
- docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus