| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
A production-ready AI-powered recommendation system built with microservices architecture. The system provides real-time model training, multiple data source integration, and comprehensive REST APIs for building intelligent recommendation features into any application.
VRecommendation is a complete recommendation engine that combines machine learning models with a modern web interface. It supports collaborative filtering, content-based filtering, and hybrid recommendation algorithms with real-time data processing from multiple sources including databases, CSV files, REST APIs, and Kafka message queues.
The system consists of three main services:
Supporting infrastructure:
All services run in Docker containers orchestrated with Docker Compose.
Optional for local development:
git clone https://github.com/yourusername/VRecommendation.git
cd VRecommendationCopy the example environment file and configure your settings:
cp example-env .envEdit the .env file with your configuration. Key variables:
# JWT Secret (REQUIRED - change this in production)
JWT_SECRET_KEY=your-secure-secret-key-here
# API Server Configuration
API_SERVER_HOST=0.0.0.0
API_SERVER_PORT=2030
# AI Server Configuration
AI_SERVER_HOST=0.0.0.0
AI_SERVER_PORT=9999
# Frontend Configuration
FRONTEND_PORT=5173
VITE_API_SERVER_URL=http://localhost:2030
VITE_AI_SERVER_URL=http://localhost:9999
# Redis Configuration
REDIS_HOST=redis
REDIS_PORT=6379
# Kafka Configuration
KAFKA_BOOTSTRAP_SERVERS=kafka:9093
KAFKA_PORT=9092To enable Google login, set up OAuth credentials:
GOOGLE_CLIENT_ID=your-client-id
GOOGLE_CLIENT_SECRET=your-client-secret
GOOGLE_CALLBACK_URL=http://localhost:2030/api/v1/auth/google/callback
# For LAN/ngrok access (optional)
GOOGLE_CALLBACK_URL_PUBLIC=https://your-ngrok-url.ngrok-free.app/api/v1/auth/google/callbackWindows:
start.cmd
# Or start everything including Admin Portal:
start.cmd allLinux/macOS:
docker-compose up -dThis will start all services in the background. First-time startup may take several minutes to download images and build containers.
Check that all services are running:
docker-compose psAll services should show status as "Up". Test the endpoints:
# Test API Server
curl http://localhost:2030/api/v1/ping
# Test AI Server
curl http://localhost:9999/api/v1/healthExpected responses indicate healthy services.
The Admin Portal allows you to manage which email addresses can register/login to the system.
Start Admin Portal:
start.cmd adminThen access: http://127.0.0.1:3456
Note: Admin Portal is only accessible from localhost for security reasons.
Or start everything together:
start.cmd allTo access the application from other devices on your local network:
start.cmd setupThis will:
HOST_IP=192.168.1.100
VITE_API_SERVER_URL=http://192.168.1.100:2030
VITE_AI_SERVER_URL=http://192.168.1.100:9999
FRONTEND_URL=http://192.168.1.100:5173docker-compose build frontend
docker-compose up -dnetsh advfirewall firewall add rule name="VRecom" dir=in action=allow protocol=tcp localport=5173,2030,9999Google OAuth does not allow private IP addresses as redirect URIs. Solutions:
ngrok http 2030See HUONG_DAN_OAUTH_LAN.md for detailed instructions.
VRecommendation/ ├── backend/ │ ├── ai_server/ # Python FastAPI ML service │ │ ├── src/ │ │ │ └── ai_server/ │ │ │ ├── handlers/ # Business logic handlers │ │ │ ├── models/ # ML model implementations │ │ │ ├── routers/ # API route definitions │ │ │ ├── services/ # Core services │ │ │ ├── tasks/ # Background tasks │ │ │ └── utils/ # Utility functions │ │ ├── config/ # Configuration files │ │ ├── models/ # Trained model files │ │ ├── tasks/ # Task definitions (JSON) │ │ ├── data/ # Data storage │ │ ├── Dockerfile │ │ └── pyproject.toml │ │ │ └── api_server/ # Go Fiber gateway service │ ├── app/ │ │ ├── controllers/ # HTTP handlers │ │ ├── middleware/ # Custom middleware │ │ └── models/ # Data models │ ├── internal/ │ │ ├── auth/ # Authentication logic │ │ ├── cache/ # Redis operations │ │ ├── initialize/ # App initialization │ │ └── proxy/ # AI server proxy │ ├── pkg/ # Reusable packages │ ├── config/ # Configuration files │ ├── Dockerfile │ ├── go.mod │ └── main.go │ ├── frontend/ │ └── project/ # React TypeScript application │ ├── src/ │ │ ├── components/ # React components │ │ ├── pages/ # Page components │ │ ├── services/ # API services │ │ ├── contexts/ # React contexts │ │ └── hooks/ # Custom hooks │ ├── public/ # Static assets │ ├── Dockerfile │ └── package.json │ ├── tests/ │ ├── demo-website/ # Demo e-commerce site │ ├── kafka-server/ # Standalone Kafka for testing │ └── test-data/ # Test datasets │ ├── scripts/ # Utility scripts ├── docs/ # Additional documentation ├── diagrams/ # Architecture diagrams ├── docker-compose.yml # Main orchestration file ├── Makefile # Build automation └── README.md # This file
The system uses a single .env file in the project root. Key configuration sections:
JWT_SECRET_KEY=your-secret-key # JWT signing secret
SESSION_SECRET=your-session-secret # Session encryption key
JWT_EXPIRE_MINUTES=1440 # Token expiration (24 hours)API_SERVER_HOST=0.0.0.0
API_SERVER_PORT=2030
AI_SERVER_HOST=0.0.0.0
AI_SERVER_PORT=9999
FRONTEND_PORT=5173# MySQL
MYSQL_HOST=your-mysql-host
MYSQL_PORT=3306
MYSQL_USER=your-username
MYSQL_PASSWORD=your-password
MYSQL_DATABASE=your-database
# MongoDB
MONGODB_HOST=your-mongodb-host
MONGODB_PORT=27017
MONGODB_USERNAME=your-username
MONGODB_PASSWORD=your-passwordKAFKA_BOOTSTRAP_SERVERS=kafka:9093 # Internal broker address
KAFKA_PORT=9092 # External broker port
KAFKA_GROUP_ID=vrecom_consumer_group # Consumer group IDEach service has additional configuration files:
AI Server: backend/ai_server/config/
API Server: backend/api_server/config/
Frontend: frontend/project/.env
Create a data chef to connect to your data source:
# Using CSV file
curl -X POST http://localhost:9999/api/v1/create_data_chef_from_csv \
-H "Content-Type: application/json" \
-d '{
"name": "my_interactions",
"path": "/app/data/interactions.csv",
"rename_columns": "userId:user_id,itemId:item_id,rating:rating"
}'
# Using SQL database
curl -X POST http://localhost:9999/api/v1/create_data_chef_from_sql \
-H "Content-Type: application/json" \
-d '{
"name": "my_interactions",
"query": "SELECT user_id, item_id, rating FROM interactions",
"rename_columns": ""
}'
# Using Kafka topic
# Configure in backend/ai_server/config/restaurant_data.yaml:
# my_kafka_data:
# type: messaging_queue
# brokers: kafka:9093
# topic: interactions
# group_id: my_consumer_groupCreate a JSON file in backend/ai_server/models/:
{
"model_name": "My Recommendation Model",
"model_id": "my_model",
"type": "svd",
"algorithm": "svd",
"hyperparameters": {
"n_components": 50,
"algorithm": "randomized",
"n_iter": 10,
"random_state": 42
},
"message": "Production recommendation model"
}Create a task file in backend/ai_server/tasks/:
{
"task_name": "my_model",
"model_id": "my_model",
"interactions_data_chef_id": "my_interactions",
"item_features_data_chef_id": null,
"user_features_data_chef_id": null,
"interval": 3600
}The interval is in seconds. The model will automatically retrain at this interval.
docker-compose restart ai_serverThe model will begin training according to the schedule.
curl "http://localhost:2030/api/v1/recommend?user_id=user123&model_id=my_model&n=10"Response format:
{
"user_id": "user123",
"model_id": "my_model",
"predictions": {
"user123": [
{"item_id": "item456", "score": 4.8},
{"item_id": "item789", "score": 4.6},
{"item_id": "item321", "score": 4.3}
]
},
"n_recommendations": 10,
"status": "completed",
"datetime": "2024-11-23T10:30:00Z"
}# Send interaction events
echo '{"user_id": "user1", "item_id": "item1", "rating": 5.0}
{"user_id": "user2", "item_id": "item2", "rating": 4.5}
{"user_id": "user3", "item_id": "item3", "rating": 4.0}' | \
docker exec -i vrecom_kafka kafka-console-producer \
--bootstrap-server localhost:9092 \
--topic interactionsdocker exec vrecom_kafka kafka-console-consumer \
--bootstrap-server localhost:9092 \
--topic interactions \
--from-beginning \
--max-messages 10docker exec vrecom_kafka kafka-consumer-groups \
--bootstrap-server localhost:9092 \
--group your_consumer_group \
--describeThis shows the current offset, lag, and consumption status.
# All services
docker-compose logs -f
# Specific service
docker-compose logs -f ai_server
docker-compose logs -f api_server
# Last N lines
docker-compose logs --tail=100 ai_server# Restart specific service
docker-compose restart ai_server
# Restart all services
docker-compose restart
# Stop and start (full restart)
docker-compose down
docker-compose up -d# Check service status
docker-compose ps
# Check resource usage
docker stats
# View metrics in Prometheus
# Open http://localhost:9090# Flush all Redis cache
docker exec vrecom_redis redis-cli FLUSHDB
# Clear specific keys
docker exec vrecom_redis redis-cli DEL "key_pattern"GET /api/v1/ping
Response: {"message": "pong", "timestamp": "2024-11-23T10:00:00Z"}
GET /api/v1/recommend?user_id={userId}&model_id={modelId}&n={count}
Parameters:
POST /api/v1/auth/login POST /api/v1/auth/logout GET /api/v1/auth/status
GET /api/v1/health
GET /api/v1/list_models
POST /api/v1/create_model
DELETE /api/v1/delete_model?model_id={modelId}
GET /api/v1/get_model?model_id={modelId}
GET /api/v1/list_data_chefs
POST /api/v1/create_data_chef_from_csv
POST /api/v1/create_data_chef_from_sql
POST /api/v1/create_data_chef_from_nosql
POST /api/v1/create_data_chef_from_api
DELETE /api/v1/delete_data_chef?name={chefName}
GET /api/v1/list_tasks
POST /api/v1/add_model_task
DELETE /api/v1/remove_model_task?task_name={taskName}
GET /api/v1/get_scheduler_status POST /api/v1/stop_scheduler POST /api/v1/restart_scheduler
POST /api/v1/recommend
{
"user_id": "user123",
"model_id": "my_model",
"n": 10
}
For AI Server (Python):
cd backend/ai_server
poetry install
poetry run serverFor API Server (Go):
cd backend/api_server
go mod download
go run main.gocd frontend/project
npm install
npm run devcd backend/ai_server
poetry run pytest tests/ -vcd backend/api_server
go test ./... -vcd frontend/project
npm test# Format code
poetry run black src/
# Lint code
poetry run flake8 src/
# Type checking
poetry run mypy src/# Format code
go fmt ./...
# Lint code
golangci-lint run
# Vet code
go vet ./...# Lint code
npm run lint
# Type checking
npm run type-check
# Format code
npm run formatfrom ai_server.models.base_model import BaseRecommendationModel
class MyNewModel(BaseRecommendationModel):
def __init__(self, **hyperparameters):
super().__init__(**hyperparameters)
def fit(self, interactions_df, **kwargs):
# Training implementation
pass
def predict(self, user_id, n=5):
# Prediction implementation
passRegister in backend/ai_server/src/ai_server/services/model_service.py
Add tests in backend/ai_server/tests/
Update documentation
def _cook_my_source(param1, param2):
# Data fetching logic
for record in data_source:
yield recordAdd to _cook_raw_data_source() switch statement
Create API endpoint in routers
Add configuration example
Before deploying to production:
Build optimized production images:
docker-compose -f docker-compose.prod.yml build
docker-compose -f docker-compose.prod.yml up -dRecommended minimum resources:
Adjust in docker-compose.yml:
services:
ai_server:
deploy:
resources:
limits:
cpus: '1.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 1GConfigure Prometheus targets in prometheus.yml:
scrape_configs:
- job_name: 'ai_server'
static_configs:
- targets: ['ai_server:9999']
- job_name: 'api_server'
static_configs:
- targets: ['api_server:2030']Set up Grafana dashboards for visualization (optional).
# Backup Redis data
docker exec vrecom_redis redis-cli SAVE
docker cp vrecom_redis:/data/dump.rdb ./backups/redis/
# Backup Kafka data
docker exec vrecom_kafka tar -czf /tmp/kafka-backup.tar.gz /var/lib/kafka/data
docker cp vrecom_kafka:/tmp/kafka-backup.tar.gz ./backups/kafka/# Backup trained models
docker cp vrecom_ai_server:/app/models ./backups/models/Automate with cron jobs or use volume backups.
Check Docker and Docker Compose versions:
docker --version # Should be 20.10+
docker-compose --version # Should be 2.0+Check for port conflicts:
# Linux/Mac
lsof -i :2030
lsof -i :9999
lsof -i :5173
# Windows
netstat -ano | findstr :2030View service logs for errors:
docker-compose logsVerify Kafka is running and accessible:
docker exec vrecom_kafka kafka-broker-api-versions \
--bootstrap-server localhost:9092Check consumer group status:
docker exec vrecom_kafka kafka-consumer-groups \
--bootstrap-server localhost:9092 \
--listReset consumer offset if needed:
docker exec vrecom_kafka kafka-consumer-groups \
--bootstrap-server localhost:9092 \
--group your_group_id \
--reset-offsets \
--to-earliest \
--topic your_topic \
--executeCheck if data chef is configured correctly:
curl http://localhost:9999/api/v1/list_data_chefsVerify data source connectivity from AI server container:
docker exec vrecom_ai_server python -c "
from ai_server.services.data_chef_service import DataChefService
service = DataChefService()
print(service.list_data_chefs())
"Check task scheduler status:
curl http://localhost:9999/api/v1/get_scheduler_statusView AI server logs for training errors:
docker-compose logs -f ai_server | grep -i "error\|exception"Verify the model exists and is trained:
curl http://localhost:9999/api/v1/list_modelsCheck if user exists in training data:
# View model metadata
docker exec vrecom_ai_server cat models/your_model_metadata.jsonClear recommendation cache:
docker exec vrecom_redis redis-cli KEYS "recommend:*"
docker exec vrecom_redis redis-cli DEL "recommend:user123:my_model:10"Monitor container resources:
docker statsReduce batch sizes in training configuration.
Clear unused data:
# Clear Redis cache
docker exec vrecom_redis redis-cli FLUSHDB
# Prune Docker system
docker system prune -aEnable debug logging by setting in .env:
DEBUG=true
LOG_LEVEL=debugRestart services:
docker-compose restartView detailed logs:
docker-compose logs -f --tail=100If issues persist:
The system uses Redis for multiple caching layers:
Configure cache TTL in backend/api_server/internal/cache/:
const (
RecommendationCacheTTL = 3600 // 1 hour
ModelMetadataCacheTTL = 7200 // 2 hours
)Configure optimal pool sizes in .env:
# API Server
REDIS_POOL_SIZE=20
REDIS_MAX_IDLE=10
# AI Server (adjust in code)
SQL_POOL_SIZE=10
SQL_MAX_OVERFLOW=20Adjust hyperparameters for faster training:
{
"hyperparameters": {
"n_components": 20,
"n_iter": 5,
"batch_size": 512
}
}Lower values train faster but may reduce accuracy.
Set appropriate limits in docker-compose.yml:
services:
ai_server:
deploy:
resources:
limits:
cpus: '2.0'
memory: 4GWe welcome contributions! Please follow these steps:
This project is licensed under the terms specified in the LICENSE.txt file.
For questions, issues, or feature requests:
Built with open-source technologies:
| Back | FazBrowse Home | New Git URL |