by 🅐🅛🅔🅧🅐🅝🅓🅔🅡 🅞🅝🅑🅨🅢🅗

This is a final project of Streaming programming with Scala at UCU. The aim of this project is to develop services for streaming data to Kafka topics then combine them using Kafka stream join.

The idea of the project is to get real stream of tweets and stock prices for given stock symbols, join them by symbol and forward to another Kafka topic for future processing.

Diagram shows rough architecture:

twitter-service - services gets real time stream of tweets with given keywords, all messages go to kafka tweets topic.
stocks-service - services gets real time stocks prices from IEX by given stock symbols, all prices go to kafka stocks topic.
combiner-service - service pulls messages from tweets and stocks, join them by stock symbol and sends to enriched topic.

This command will start zookeeper and kafka clusters. Default port is 9092.

make kafka-start

Shut down cluster:

make kafka-stop

1. Create stocks-credentials.env

TOKEN=sk_0000000000000000000000000

Token should be requested from IEX

2. Create twitter-credentials.env

ACCESS_TOKEN_SECRET=
ACCESS_TOKEN=
CONSUMER_SECRET=
CONSUMER_KEY=

Credentials should be requested from Twitter Developer

3. Create topics in Kafka

make kafka-create-topics

4. Run services

make services-start

# to stop
make services-stop

1. Install kaf tool github

make install-kaf

3. View topics

kaf topics

2. View topic's messages

kaf consume tweets
kaf consume stocks
kaf consume combined

All images are build automatically from master branch by DockerHub:

• alexanderonbysh/twitter-service:latest
• alexanderonbysh/stocks-service:latest
• alexanderonbysh/combiner-service:latest

If you want to update code just rebuild docker image:

cd twitter-service
docker build -t twitter-service .

• move data object like Tweet, Price, EnrichedTweet to common package 🙊
• handle rate limits for Tweeter and IEX API endpoints 🙉
• Tests 🙈