Jupyter lab from cloud - NMR submission

 https://www.datacamp.com/community/tutorials/google-cloud-data-science

1. Create a VM instance on GCP

2. MAKE SURE THE OS IS UBUNTU AND NOT CONTAINER OPTIMIZED!!!!! For docker images use container optimized

2. Run the following in the ommand line (Make sure that the firewall settings below are configured)

sudo apt update
sudo apt-get install p7zip-full -y
cd ~

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt
wget https://nmr-dev.s3.fr-par.scw.cloud/nmr.zip
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

Replace pw
7z  x  -pXxXxXxXxX  nmr.zip -aoa

bash anaconda.sh -b -p ~/anaconda
~/anaconda/condabin/conda init bash
source ~/.bashrc
conda env create -f nmr_env.yml
conda activate nmr
cat requirements.txt | xargs -n 1 pip install
cd ~/nmr/
jupyter lab --allow-root --ip 0.0.0.0 --port 8888

===============================================

Replace XxXxXxXxXx with regular pw

sudo apt update
sudo apt-get install p7zip-full -y
cd ~

wget https://repo.anaconda.com/archive/Anaconda3-5.3.1-Linux-x86_64.sh -O ~/anaconda.sh
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt
wget https://nmr-dev.s3.fr-par.scw.cloud/nmr.zip
wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

Replace pw
7z  x  -pXxXxXxXxX  nmr.zip -aoa

bash miniconda.sh -b -p ~/miniconda
~/miniconda/condabin/conda init bash
source ~/.bashrc
conda env create -f nmr_env.yml
conda activate nmr
cat requirements.txt | xargs -n 1 pip install
cd ~/nmr/
jupyter lab --allow-root --ip 0.0.0.0 --port 8888

Getting latest model code

git config --global user.email "mabolfadl@gmail.com"

git config --global user.name "Mohamed Abolfadl"

cd ~

git clone https://gitlab.com/mabolfadl/nmr_prd.git

Installing TAlib

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

conda install gcc_linux-64

apt install make

tar -xvf ta-lib-0.4.0-src.tar.gz

cd ~/ta-lib/

./configure --prefix=/usr

make

make install

apt upgrade

pip install TA-lib

Steps:

1. download from http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

2. untar tar -xvf ta-lib-0.4.0-src.tar.gz

3. cd /../ta-lib

4. ./configure --prefix=/usr

5. make

6. sudo make install

7. sudo apt upgrade

8. pip install ta-lib or pip install TA-Lib

9. Check import talib

3. Make Sure the firewall settings are applied

You first need to make the VM accessible from the web. To do that, you will create a firewall rule via the Google Cloud console. Go back to your Instances dashboard and in the top left menu, select "VPC Network > Firewall rules". Click on the "CREATE FIREWAL RULE" link and fill out the following values:

• Name: jupyter-rule (you can choose any name)
• Source IP ranges: 0.0.0.0/0
• Specified protocols and ports: tcp:8888
• and leave all the other variables to their default values.

The form should look like:

This firewall rule allows all incoming traffic (from all IPs) to hit the port 8888.

Using the "Equivalent command line link", you can see that firewall rule can also be created from the terminal with the following command line:

$ gcloud compute --project=datacamp-gcp firewall-rules create jupyter-rule --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp:8888 --source-ranges=0.0.0.0/0

Now go back to the VM page (top left menu > Compute Engine > VM instances), click on your VM name.

Make a note of your VM IP address. This is the IP address that you will use in your browser to access your Jupyter environment. In my example, the IP address is: 35.196.81.49, yours will be different.

and make sure the Firewall rules are checked:

Docker approach

cd ~

mkdir work

docker run --rm -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -v /home/mabolfadl/work:/home/jovyan/work jupyter/tensorflow-notebook:latest

#-- Miniconda

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/home/root/work continuumio/miniconda /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet &&  mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

#-- ANaconda

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/opt/notebooks/work continuumio/anaconda /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet && pip install xgboost catboost lightgbm  && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

docker run -i -t -p 8888:8888 -v /home/mabolfadl/work:/opt/notebooks/work continuumio/anaconda /bin/bash -c "pip install xgboost catboost lightgbm  && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

docker pull continuumio/anaconda

docker run --rm -p 8888:8888 -i -t -v /home/mabolfadl/work:/home/mabolfadl/work continuumio/anaconda /bin/bash

wget https://storage.googleapis.com/abolfadl-stk-dev/nmr_env_ws.yml -O ~/nmr_env_ws.yml

cd ~

conda env create -f nmr_env_ws.yml

conda activate nmr

jupyter notebook

#======================== jupyter data science with additional packages

FROM jupyter/datascience-notebook

MAINTAINER ole.bjorne@gmail.com

# -------Install XGBoost --------

USER root

# install gcc with openmp support in conda

RUN conda install -y gcc

# download and build xgboost

RUN cd /opt && \

git clone --recursive https://github.com/dmlc/xgboost && \

  cd xgboost && \

  make -j4

# set environment var to python package for both python2 and python3

ENV PYTHONPATH /opt/xgboost/python-package

# install R package - use pre-compiled CRAN version

RUN Rscript -e "install.packages('xgboost',repos='http://cran.rstudio.com/')"

# -------Install TensorFlow --------

# set up download links for both python2 and python3

ENV TF_BINARY_URL_PY2 https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl

ENV TF_BINARY_URL_PY3 https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.10.0rc0-cp35-cp35m-linux_x86_64.whl

# install python package for both environments

RUN pip2 install $TF_BINARY_URL_PY2 && \

pip3 install $TF_BINARY_URL_PY3

# -------Install OpenAI gym --------

RUN cd /opt && \

git clone https://github.com/openai/gym && \

cd gym && \

pip2 install -e . && \

  pip3 install -e .

USER $NB_USER

#======================================================

cd ~

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/nmr_env.yml

wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p ~/anaconda7

rm Anaconda3-4.2.0-Linux-x86_64.sh

echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc 

# Reload default profile

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

Trial 1

cd ~

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

curl -O https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh

bash Anaconda3-2019.03-Linux-x86_64.sh -b -p $HOME/anaconda3

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

eu.gcr.io/stock-288218/mabolfadl_stk

#===============================

$ sudo apt-get update

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash ~/miniconda.sh -b -p $HOME/miniconda

cd ~

conda env create -f pytorch_course_env.yml

############# MINICONDA

sudo apt update

sudo apt-get install p7zip-full -y

cd ~

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr.zip

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

7z  x  -phamadaguc2010  nmr.zip -aoa

bash miniconda.sh -b -p ~/miniconda

~/miniconda/condabin/conda init bash

source ~/.bashrc

conda env create -f nmr_env.yml

conda activate nmr

cat requirements.txt | xargs -n 1 pip install

cd ~/nmr/

jupyter lab --ip 0.0.0.0 --port 8888

Add to requirements.txt

scikit-learn==0.22.1

tensorflow==2.1.0

theano==1.0.4

keras==2.3.1

#############  ANACONDA

sudo apt update

sudo apt-get install p7zip-full -y

cd ~

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr_env.yml -O ~/nmr_env.yml

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/requirements.txt

wget https://storage.googleapis.com/abolfadl-nmr-dev/vm_nmr_project/nmr.zip

curl -O https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh

7z  x  -phamadaguc2010  nmr.zip -aoa

bash Anaconda3-2019.03-Linux-x86_64.sh -b -p ~/anaconda

~/anaconda/condabin/conda init bash

source ~/.bashrc

conda env create -f nmr_env.yml

~/anaconda/condabin/conda activate nmr

cat requirements.txt | xargs -n 1 pip install

cd ~/nmr/

jupyter lab --ip 0.0.0.0 --port 8888

source ~/.bashrc

cd ~

# Go to home directory

cd ~

# You can change what anaconda version you want at 

# https://repo.continuum.io/archive/

wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p ~/anaconda

rm Anaconda3-4.2.0-Linux-x86_64.sh

echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc 

# Reload default profile

source ~/.bashrc

cd ~

conda env create -f pytorch_course_env.yml

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Miniconda3-latest-Linux-x86_64.sh

rm Miniconda3-latest-Linux-x86_64.sh

source .bashrc

cd ~

conda env create -f pytorch_course_env.yml

wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh

wget https://storage.googleapis.com/abolfadl-stk-dev/pytorch_course_env.yml -O ~/pytorch_course_env.yml

bash Anaconda3-5.0.1-Linux-x86_64.sh

rm Anaconda3-5.0.1-Linux-x86_64.sh

source .bahsrc

cd ~

conda env create -f pytorch_course_env.yml

Connect to Ubuntu from WIndows using X2go

 https://www.howtoforge.com/tutorial/x2go-server-ubuntu-14-04/

Install X2Go Server

To install the server side application just type one of the following commands to install the prerequisites, depending on your Ubuntu version.
For an older version of Ubuntu, use the following command:

sudo apt-get install python-software-properties

For the recent version you may use:

sudo apt-get install software-properties-common

Then add the X2Go PPA software repository, update the packagelist and install the server:

sudo add-apt-repository ppa:x2go/stable

 sudo apt-get update

 sudo apt-get install x2goserver x2goserver-xsession

Install X2Go Client

Now, after installing the application on the server you will have to install the client side application on the system of your choice by typing:

sudo apt-get install x2goclient

This will install the client side application on your system. The X2Go client will get added automatically to your application menu.

How it works?

Every time a new session is started the script x2gostartagent is executed on your server. The x2gostartagent file creates a new session by finding the unique session ID and adds it to its own database.

Desktop Bindings

To improve the usability, X2Go provides Desktop Binding tools for various desktops. Here are the commands to install them:

Gnome Bindings

sudo apt-get install x2gognomebindings

(Not available for GNOMEv3 environment yet)

MATE Bindings

sudo apt-get install x2gomatebindings

KDE/Plasma Bindings

sudo apt-get install plasma-widget-x2go

LXDE Bindings

sudo apt-get all x2golxdebindings

XFCE4 Bindings

Bindings for XFCE4 have not been created yet, but you can install GNOME bindings for that purpose.

Using the client side application

The next steps describe the setup of the first connection to your X2Go server:



Session name: Fill out your session name.

Host: This can be your host name or IP address.

Login: This will be your user name with which you will be logging in on to the remote server.

SSH port: This will be your port number that you will be using to connect to your remote server, the default used is 22, you may change it as per your settings on SSH config file (/etc/ssh/sshd_config).

RSA/DSA key: You may specify the key, if created. The key decryption will prompt password for decryption.

Session type: Select your desktop manager that you are using currently on the remote server.

After the successful configuration of the session you are ready to go. You will have something like this on the top right corner of the window:



Click on that window you will get a login screen like:



Make sure you type in your own user name and password for the system and click on login. It should take you to the desktop of your server:



Here it is you have successfully logged in on to your system. You may also want to check Pyhoca-GUI as a client application.

Links.

• X2Go.
• PyHoca-GUI.

Insall graphics card driver

 

 

 

https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-20-04-focal-fossa-linux

 

ubuntu-drivers devices

sudo ubuntu-drivers autoinstall