Scaleway submission job

 

We build 2 instances. 1 creates an instance from an image and the other once started runs the python scripts of the submission

Submission machine

- Installs miniconda

- Creates nmr environment

- clones code from gitlab

- Make sure you save an image of this machine that will be used in starting an instance from the starter machine

- Runs submission scripts on startup. Here is the crontab -e:

# Activate nmr environment

@reboot conda activate nmr&

@reboot date >> /root/test.txt&

# Account mabolfadl

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/sub_mabolfadl.py >> /root/logs_mabolfadl.txt&

# Account mabolfadl2, lgb_plain, lgb_rnd_drop_level_9

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/sub_lgbx.py >> /root/logs_sub_lgbx.txt&

# ACcounts mabolfadl3,5,6,7,8,9,10,11 lgb_rnd_drp_level_2,6,13,11,const

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/P4_live_submission.py >> /root/P4_live_submission_logs.txt&

@reboot date >> /root/test.txt&

Starter machine

The starter machine has a cronjob each Saturday night to start the instance. Make sure you copy the right image ID

0 10 * * SUN scw instance server create type=GP1-S zone=fr-par-1 image=5c38116d-aa00-441d-ad71-77355c0515ec root-volume=l:300G name=nmr-sub ip=new project-id=204d2eef-f4cc-4d7b-850d-acf2aa87eb2c

Then stop it

0 11 * * SUN cd /root && sh shutdown_nmr.sh>>logs_shutdown.log

Here are the contents of the shutdown_nmr.sh

#!/bin/bash

scw instance server list | grep nmr-sub | awk '{print $1}' > instance_id.txt

INSTANCE_ID=$(cat instance_id.txt)

#echo instance_id.txt | INSTANCE_ID

echo $INSTANCE_ID

scw instance server stop $INSTANCE_ID

sleep 300

scw instance server delete $INSTANCE_ID with-volumes=none zone=fr-par-1

Make sure the script is executable

chmod +777 shutdown_nmr.sh

Jupyter lab from cloud - NMR submission

 https://www.datacamp.com/community/tutorials/google-cloud-data-science

1. Create a VM instance on GCP

2. MAKE SURE THE OS IS UBUNTU AND NOT CONTAINER OPTIMIZED!!!!! For docker images use container optimized

2. Run the following in the ommand line (Make sure that the firewall settings below are configured)

sudo apt update
sudo apt-get install p7zip-full -y
cd ~

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt
wget https://nmr-dev.s3.fr-par.scw.cloud/nmr.zip
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

Replace pw
7z  x  -pXxXxXxXxX  nmr.zip -aoa

bash anaconda.sh -b -p ~/anaconda
~/anaconda/condabin/conda init bash
source ~/.bashrc
conda env create -f nmr_env.yml
conda activate nmr
cat requirements.txt | xargs -n 1 pip install
cd ~/nmr/
jupyter lab --allow-root --ip 0.0.0.0 --port 8888

===============================================

Replace XxXxXxXxXx with regular pw

sudo apt update
sudo apt-get install p7zip-full -y
cd ~

wget https://repo.anaconda.com/archive/Anaconda3-5.3.1-Linux-x86_64.sh -O ~/anaconda.sh
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt
wget https://nmr-dev.s3.fr-par.scw.cloud/nmr.zip
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

Replace pw
7z  x  -pXxXxXxXxX  nmr.zip -aoa

bash miniconda.sh -b -p ~/miniconda
~/miniconda/condabin/conda init bash
source ~/.bashrc
conda env create -f nmr_env.yml
conda activate nmr
cat requirements.txt | xargs -n 1 pip install
cd ~/nmr/
jupyter lab --allow-root --ip 0.0.0.0 --port 8888

Getting latest model code

git config --global user.email "mabolfadl@gmail.com"

git config --global user.name "Mohamed Abolfadl"

cd ~

git clone https://gitlab.com/mabolfadl/nmr_prd.git

Installing TAlib

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

conda install gcc_linux-64

apt install make

tar -xvf ta-lib-0.4.0-src.tar.gz

cd ~/ta-lib/

./configure --prefix=/usr

make

make install

apt upgrade

pip install TA-lib

Steps:

1. download from http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

2. untar tar -xvf ta-lib-0.4.0-src.tar.gz

3. cd /../ta-lib

4. ./configure --prefix=/usr

5. make

6. sudo make install

7. sudo apt upgrade

8. pip install ta-lib or pip install TA-Lib

9. Check import talib

3. Make Sure the firewall settings are applied

You first need to make the VM accessible from the web. To do that, you will create a firewall rule via the Google Cloud console. Go back to your Instances dashboard and in the top left menu, select "VPC Network > Firewall rules". Click on the "CREATE FIREWAL RULE" link and fill out the following values:

• Name: jupyter-rule (you can choose any name)
• Source IP ranges: 0.0.0.0/0
• Specified protocols and ports: tcp:8888
• and leave all the other variables to their default values.

The form should look like:

This firewall rule allows all incoming traffic (from all IPs) to hit the port 8888.

Using the "Equivalent command line link", you can see that firewall rule can also be created from the terminal with the following command line:

$ gcloud compute --project=datacamp-gcp firewall-rules create jupyter-rule --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:8888 --source-ranges=0.0.0.0/0

Now go back to the VM page (top left menu > Compute Engine > VM instances), click on your VM name.

Make a note of your VM IP address. This is the IP address that you will use in your browser to access your Jupyter environment. In my example, the IP address is: 35.196.81.49, yours will be different.

and make sure the Firewall rules are checked:

Docker approach

cd ~

mkdir work

docker run --rm -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -v /home/mabolfadl/work:/home/jovyan/work jupyter/tensorflow-notebook:latest

#-- Miniconda

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/home/root/work continuumio/miniconda /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet &&  mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

#-- ANaconda

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/opt/notebooks/work continuumio/anaconda /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet && pip install xgboost catboost lightgbm  && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/opt/notebooks/work continuumio/anaconda /bin/bash -c "pip install xgboost catboost lightgbm  && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

docker pull continuumio/anaconda

docker run --rm -p 8888:8888 -i -t -v /home/mabolfadl/work:/home/mabolfadl/work continuumio/anaconda /bin/bash

wget https://storage.googleapis.com/abolfadl-stk-dev/nmr_env_ws.yml -O ~/nmr_env_ws.yml

cd ~

conda env create -f nmr_env_ws.yml

conda activate nmr

jupyter notebook

#======================== jupyter data science with additional packages

FROM jupyter/datascience-notebook

MAINTAINER ole.bjorne@gmail.com

# -------Install XGBoost --------

USER root

# install gcc with openmp support in conda

RUN conda install -y gcc

# download and build xgboost

RUN cd /opt && \

git clone --recursive https://github.com/dmlc/xgboost && \

  cd xgboost && \

  make -j4

# set environment var to python package for both python2 and python3

ENV PYTHONPATH /opt/xgboost/python-package

# install R package - use pre-compiled CRAN version

RUN Rscript -e "install.packages('xgboost',repos='http://cran.rstudio.com/')"

# -------Install TensorFlow --------

# set up download links for both python2 and python3

ENV TF_BINARY_URL_PY2 https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl

ENV TF_BINARY_URL_PY3 https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.10.0rc0-cp35-cp35m-linux_x86_64.whl

# install python package for both environments

RUN pip2 install $TF_BINARY_URL_PY2 && \

pip3 install $TF_BINARY_URL_PY3

# -------Install OpenAI gym --------

RUN cd /opt && \

git clone https://github.com/openai/gym && \

cd gym && \

pip2 install -e . && \

  pip3 install -e .

USER $NB_USER

#======================================================

cd ~

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/nmr_env.yml

wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p ~/anaconda7

rm Anaconda3-4.2.0-Linux-x86_64.sh

echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc 

# Reload default profile

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

Trial 1

cd ~

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

curl -O https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh

bash Anaconda3-2019.03-Linux-x86_64.sh -b -p $HOME/anaconda3

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

eu.gcr.io/stock-288218/mabolfadl_stk

#===============================

$ sudo apt-get update

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash ~/miniconda.sh -b -p $HOME/miniconda

cd ~

conda env create -f pytorch_course_env.yml

############# MINICONDA

sudo apt update

sudo apt-get install p7zip-full -y

cd ~

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr.zip

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

7z  x  -phamadaguc2010  nmr.zip -aoa

bash miniconda.sh -b -p ~/miniconda

~/miniconda/condabin/conda init bash

source ~/.bashrc

conda env create -f nmr_env.yml

conda activate nmr

cat requirements.txt | xargs -n 1 pip install

cd ~/nmr/

jupyter lab --ip 0.0.0.0 --port 8888

Add to requirements.txt

scikit-learn==0.22.1

tensorflow==2.1.0

theano==1.0.4

keras==2.3.1

#############  ANACONDA

sudo apt update

sudo apt-get install p7zip-full -y

cd ~

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr.zip

curl -O https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh

7z  x  -phamadaguc2010  nmr.zip -aoa

bash Anaconda3-2019.03-Linux-x86_64.sh -b -p ~/anaconda

~/anaconda/condabin/conda init bash

source ~/.bashrc

conda env create -f nmr_env.yml

~/anaconda/condabin/conda activate nmr

cat requirements.txt | xargs -n 1 pip install

cd ~/nmr/

jupyter lab --ip 0.0.0.0 --port 8888

source ~/.bashrc

cd ~

# Go to home directory

cd ~

# You can change what anaconda version you want at 

# https://repo.continuum.io/archive/

wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p ~/anaconda

rm Anaconda3-4.2.0-Linux-x86_64.sh

echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc 

# Reload default profile

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Miniconda3-latest-Linux-x86_64.sh

rm Miniconda3-latest-Linux-x86_64.sh

source .bashrc

cd ~

conda env create -f pytorch_course_env.yml

wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Anaconda3-5.0.1-Linux-x86_64.sh

rm Anaconda3-5.0.1-Linux-x86_64.sh

source .bahsrc

cd ~

conda env create -f pytorch_course_env.yml