FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [View Raw Code]   [Original HTTPS Page]

BigDL/python/orca/developer_guide.md at main · intel/BigDL · GitHub

This repository was archived by the owner on Jul 6, 2026. It is now read-only.
/ BigDL Public archive

Breadcrumbs

  1. BigDL
  2. python
  3. orca

developer_guide.md

Latest commit

 

History

History
58 lines (40 loc) · 1.55 KB

Breadcrumbs

  1. BigDL
  2. python
  3. orca

developer_guide.md

File metadata and controls

58 lines (40 loc) · 1.55 KB
Raw

BigDL Orca Developer Guide

This is a simple guide for developers to setup their development environment. You do not need to read this if you are just using Orca APIs.

Prerequisite

  • Linux OS
  • JDK (e.g. sudo apt-get install openjdk-8-jdk)
  • Maven (e.g. sudo apt install maven)
  • Conda (Optional)

Get BigDL source code

git clone https://github.com/intel-analytics/BigDL.git

Build BigDL scala code

cd BigDL/scala
./make-dist.sh

After running the above code there will be a BigDl/dist directory consisting of the built artifact.

Set up python environment

PyTorch Development (pytorch_ray_estimator, pytorch_pyspark_estimator, not including pytorch_spark_estimator)

  1. Install minimal dependencies
pip install pyspark==2.4.3 # ./make-dist.sh build against spark 2.4.3 by default
pip install ray==1.9.2
pip install aiohttp==3.9.0 # dependencies introduced by ray, latest version has api changes
pip install aioredis==1.3.1 # dependencies introduced by ray, latest version has api changes
pip install pytest pyarrow pandas
pip install bigdl-math bigdl-tf # dependencies introduced by bigdl-orca
pip3 install torch==1.10.1+cpu torchvision==0.11.2+cpu torchaudio==0.10.1+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html
  1. Set a few environment variables
cd BigDL/python/orca
source dev/prepare_env.sh
  1. Run tests to verify environment
cd BigDL/python/orca
pytest test/bigdl/orca/learn/spark/test_pytorch_estimator.py
  1. Please raise an issue if the above steps does not work
Back | FazBrowse Home | New Git URL