FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Farfetch fork from 0.7 to support databricks integration by rafalzydowicz · Pull Request #1076 · feast-dev/feast · GitHub

  1. feast-dev
  2. feast

Repository navigation

Farfetch fork from 0.7 to support databricks integration #1076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woop merged 11 commits into from
Feb 7, 2021
Conversation 8 Commits 11 Checks 0 Files changed
Merged

Farfetch fork from 0.7 to support databricks integration #1076

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
44a6b09
Farfetch databricks support
rafalzydowicz Oct 19, 2020
45e2ec3
Use classifier for uber jars to prevent conflicts during tests
rafalzydowicz Oct 19, 2020
b361efe
Fix python formatting
rafalzydowicz Oct 20, 2020
7060ade
Fix databricks emulator test
rafalzydowicz Oct 20, 2020
c1933ab
Add go package configuration to SparkRetrieval.proto
rafalzydowicz Oct 20, 2020
e89308a
Fix test precision.
rafalzydowicz Oct 20, 2020
fc301d1
Replace docker compose with testcontainers and launch containers dire…
rafalzydowicz Oct 20, 2020
a77221b
Increase sleep for slower hosts
rafalzydowicz Oct 20, 2020
9261470
Fix dummy statsd server
rafalzydowicz Oct 20, 2020
7eb25c9
Port Serving IT fix
rafalzydowicz Nov 2, 2020
87b4218
Revert notebook version
rafalzydowicz Nov 2, 2020
File filter

Filter by extension

Filter by extension .java  (1) .xml  (1) .yml  (1) All 3 file types selected
Only manifest files
Deleted files Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
Prev Previous commit
Next Next commit
Replace docker compose with testcontainers and launch containers dire… 
…ctly
  • Loading branch information
rafalzydowicz committed Oct 20, 2020
commit fc301d1ba323f2998c8e92802bf2366150533a95
6 changes: 6 additions & 0 deletions spark/spark-ingestion-job/pom.xml
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,12 @@
<version>1.14.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>kafka</artifactId>
<version>1.14.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
Expand Down
32 changes: 16 additions & 16 deletions spark/spark-ingestion-job/src/test/java/feast/spark/ingestion/SparkIngestionTest.java
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@
import org.junit.jupiter.api.io.TempDir;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.DockerComposeContainer;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import scala.collection.JavaConverters;
Expand All @@ -123,17 +124,11 @@ public class SparkIngestionTest {

private static final Logger LOGGER = LoggerFactory.getLogger(SparkIngestionTest.class.getName());

private static final String KAFKA_HOST = "localhost";
private static final int KAFKA_PORT = 9092;
private static final String KAFKA_BOOTSTRAP_SERVERS = KAFKA_HOST + ":" + KAFKA_PORT;
private static final String KAFKA_TOPIC = "topic_" + System.currentTimeMillis();
private static final String KAFKA_SPECS_TOPIC = "fstopic_" + System.currentTimeMillis();
private static final String KAFKA_ACK_TOPIC = "acktopic_" + System.currentTimeMillis();
private static final long KAFKA_PUBLISH_TIMEOUT_SEC = 10;

private static final String REDIS_HOST = "localhost";
private static final int REDIS_PORT = 6379;

// No of samples of feature row that will be generated and used for testing.
// Note that larger no of samples will increase completion time for ingestion.
private static final int IMPORT_JOB_SAMPLE_FEATURE_ROW_SIZE = 128;
Expand Down Expand Up @@ -183,7 +178,11 @@ public void setupStatsD() {

@BeforeEach
public void setupRedis() {
this.redisConfig = RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build();
this.redisConfig =
RedisConfig.newBuilder()
.setHost(redisContainer.getHost())
.setPort(redisContainer.getFirstMappedPort())
.build();

RedisURI redisuri =
new RedisURI(
Expand Down Expand Up @@ -216,16 +215,17 @@ public void teardownStatsD() {
}
}

@Container public KafkaContainer kafkaContainer = new KafkaContainer("5.2.1");

@Container
public static DockerComposeContainer environment =
new DockerComposeContainer(
new File("src/test/resources/docker-compose/docker-compose-it.yml"));
public GenericContainer<?> redisContainer =
new GenericContainer<>("redis:5-alpine").withExposedPorts(6379);

@Test
public void streamingQueryShouldWriteKafkaPayloadAsDeltaLakeAndRedis() throws Exception {
KafkaSourceConfig kafka =
KafkaSourceConfig.newBuilder()
.setBootstrapServers(KAFKA_HOST + ":" + KAFKA_PORT)
.setBootstrapServers(kafkaContainer.getBootstrapServers())
.setTopic(KAFKA_TOPIC)
.build();

Expand Down Expand Up @@ -270,12 +270,12 @@ public void streamingQueryShouldWriteKafkaPayloadAsDeltaLakeAndRedis() throws Ex
IngestionJobProto.SpecsStreamingUpdateConfig.newBuilder()
.setSource(
KafkaSourceConfig.newBuilder()
.setBootstrapServers(KAFKA_HOST + ":" + KAFKA_PORT)
.setBootstrapServers(kafkaContainer.getBootstrapServers())
.setTopic(KAFKA_SPECS_TOPIC)
.build())
.setAck(
KafkaSourceConfig.newBuilder()
.setBootstrapServers(KAFKA_HOST + ":" + KAFKA_PORT)
.setBootstrapServers(kafkaContainer.getBootstrapServers())
.setTopic(KAFKA_ACK_TOPIC)
.build())
.build();
Expand Down Expand Up @@ -329,7 +329,7 @@ public void streamingQueryShouldWriteKafkaPayloadAsDeltaLakeAndRedis() throws Ex

LOGGER.info("Publishing Feature Sets to Kafka");
TestUtil.publishToKafka(
KAFKA_BOOTSTRAP_SERVERS,
kafkaContainer.getBootstrapServers(),
KAFKA_SPECS_TOPIC,
featureSetSpecs.stream()
.map(f -> Pair.of(getFeatureSetStringRef(f), f))
Expand All @@ -339,7 +339,7 @@ public void streamingQueryShouldWriteKafkaPayloadAsDeltaLakeAndRedis() throws Ex

LOGGER.info("Publishing {} Feature Row messages to Kafka ...", allInputs.size());
TestUtil.publishToKafka(
KAFKA_BOOTSTRAP_SERVERS,
kafkaContainer.getBootstrapServers(),
KAFKA_TOPIC,
allInputs,
ByteArraySerializer.class,
Expand Down
29 changes: 0 additions & 29 deletions spark/spark-ingestion-job/src/test/resources/docker-compose/docker-compose-it.yml
Show comments
View file
Open in desktop

This file was deleted.

Back | FazBrowse Home | New Git URL