FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

BigDL/python/orca/example/learn/tf/basic_text_classification at main · intel/BigDL · GitHub

This repository was archived by the owner on Jul 6, 2026. It is now read-only.
/ BigDL Public archive

Breadcrumbs

  1. BigDL
  2. python
  3. orca
  4. example
  5. learn
  6. tf

basic_text_classification

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

Breadcrumbs

  1. BigDL
  2. python
  3. orca
  4. example
  5. learn
  6. tf

basic_text_classification

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
basic_text_classification.py
basic_text_classification.py
 
 

README.md

Orca Tensorflow Text Classification example with movie reviews

We demonstrate how to easily run synchronous distributed TensorFlow training using TensorFlow Estimator of Project Orca in BigDL. This example classifies movie reviews as positive or negative using the text of the review. See here for the original single-node version of this example provided by Tensorflow.

Prepare environments

We recommend you to use Anaconda to prepare the environments, especially if you want to run on a yarn cluster(yarn-client mode only).

conda create -n bigdl python=3.7 #bigdl is conda enviroment name, you can set another name you like.
conda activate bigdl
pip install tensorflow==1.15
pip install --pre --upgrade bigdl-orca

Prepare Dataset

The IMDB dataset comes packaged with TensorFlow. It has already been preprocessed such that the reviews (sequences of words) have been converted to sequences of integers, where each integer represents a specific word in a dictionary.

Run example

You can run this example on local mode and yarn client mode.

  • Run with Spark Local mode:
python basic_text_classification.py --cluster_mode local
  • Run with Yarn Client mode:
python basic_text_classification.py --cluster_mode yarn
  • Run with K8s Client mode:
python basic_text_classification.py --cluster_mode k8s --k8s_master "k8s://https://127.0.0.1:8443" --container_image "intelanalytics/bigdl-k8s:latest"

In above commands

  • --cluster_mode The mode of spark cluster, supporting local and yarn. Default is "local".

Results

You can find the logs for training:

 DistriOptimizer$:426 - [Epoch 10 5632/14848][Iteration 272][Wall Clock 16.805771461s] Trained 512 records in 0.034776978 seconds. Throughput is 14722.383 records/second. Loss is 0.38606125.

And after validation, test results will be seen like:

2020-12-25 15:52:43 INFO  DistriOptimizer$:111 - [Epoch 10 14848/14848][Iteration 290][Wall Clock 17.47923692s] Validate model...
2020-12-25 15:52:44 INFO  DistriOptimizer$:177 - [Epoch 10 14848/14848][Iteration 290][Wall Clock 17.47923692s] validate model throughput is 119.15809 records/second
2020-12-25 15:52:44 INFO  DistriOptimizer$:180 - [Epoch 10 14848/14848][Iteration 290][Wall Clock 17.47923692s] bigdl_metric_0 Loss is (Loss: 31.82238, count: 80, Average Loss: 0.39777976)
2020-12-25 15:52:44 INFO  DistriOptimizer$:180 - [Epoch 10 14848/14848][Iteration 290][Wall Clock 17.47923692s] bigdl_metric_1 Top1Accuracy is Accuracy(correct: 8548, count: 10000, accuracy: 0.8548)
Back | FazBrowse Home | New Git URL