Friday, June 28, 2019

Docker building Dockerfile to install R libraries

Reference

https://hub.docker.com/r/rocker/tidyverse/dockerfile

https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image


1. Create empty folder on host machine

2. Create Dockerfile with no extension.

3. Copy paste the following commands

FROM rocker/rstudio

COPY main/ /main/

RUN apt-get update -qq && apt-get -y --no-install-recommends install \
  libxml2-dev \
  libcairo2-dev \
  libsqlite3-dev \
  libmariadbd-dev \
  libmariadb-client-lgpl-dev \
  libpq-dev \
  libssh2-1-dev \
  unixodbc-dev \
  libsasl2-dev

4. Make sure the folder main is in the empty folder. This should container the folder structure which you want

4. In the terminal run [Dont forget the "." at the end]

docker build -t image_name .

5. After image is set up, run the image. The ROOT=TRUE argument is optional

docker run -e PASSWORD=hamada -e ROOT=TRUE -p 8787:8787 image_name



6. open in browser 127.0.0.1:8787

7. log in to rstudio and install all the needed libraries from rstudio

install.packages("data.table")




[DOCKER] Sharing data between host and container

Reference

https://hackernoon.com/docker-data-containers-cb250048d162


Trial 1


  • First approach was to use the "-v" when running a container to mount a folder on the container.
  • This did not work due to "Firewall" issues



Trial 2

The idea is to create a container which stores the data that needs to be shared with the container


  • Create a data container

docker create -v /shared_data --name dataContainer busybox


  • Copy data to the container
docker cp test.csv dataContainer:/shared_data/



  • Run the intended container with the argument --volumes-from pointing at the name of the data container


docker run --rm --volumes-from dataContainer -p 8787:8787 mabolfadl/rstudio_libs:v1



Saturday, June 22, 2019

Install Docker on offline machines

References
https://docs.docker.com/install/linux/docker-ce/centos/




OPTION A


  1. Download the static binary archive. Go tohttps://download.docker.com/linux/static/stable/ (or change stable to nightlyor test), choose your hardware platform, and download the .tgz file relating to the version of Docker CE you want to install.
  2. Extract the archive using the tar utility. The dockerd and docker binaries are extracted.
    $ tar xzvf /path/to/<FILE>.tar.gz
    
  3. Optional: Move the binaries to a directory on your executable path, such as /usr/bin/. If you skip this step, you must provide the path to the executable when you invoke docker or dockerd commands.
    $ sudo cp docker/* /usr/bin/
    
  4. Start the Docker daemon:
    $ sudo dockerd &
    
    If you need to start the daemon with additional options, modify the above command accordingly or create and edit the file /etc/docker/daemon.json to add the custom configuration options.
  5. Verify that Docker is installed correctly by running the hello-world image.
    $ sudo docker run hello-world
    
    This command downloads a test image and runs it in a container. When the container runs, it prints an informational message and exits.


OPTION B

1. Make sure no docker is installed

$ sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

2. Go to https://download.docker.com/linux/centos/7/x86_64/stable/Packages/ and download the .rpm file for the Docker version you want to install.


3. Install Docker CE, changing the path below to the path where you downloaded the Docker package.
  1. $ sudo yum install /path/to/package.rpm
    
    Docker is installed but not started. The docker group is created, but no users are added to the group.
  2. Start Docker.
  3. $ sudo systemctl start docker
    
  4. Verify that Docker CE is installed correctly by running the hello-world image.
    $ sudo docker run hello-world


To make non sudo users run docker


To create the docker group and add your user:
  1. Create the docker group.
    $ sudo groupadd docker
    
  2. Add your user to the docker group.
    $ sudo usermod -aG docker $USER
    
  3. Log out and log back in so that your group membership is re-evaluated.
    If testing on a virtual machine, it may be necessary to restart the virtual machine for changes to take effect.
    On a desktop Linux environment such as X Windows, log out of your session completely and then log back in.
  4. Verify that you can run docker commands without sudo.
    $ docker run hello-world
    
    This command downloads a test image and runs it in a container. When the container runs, it prints an informational message and exits.
    If you initially ran Docker CLI commands using sudo before adding your user to the docker group, you may see the following error, which indicates that your ~/.docker/ directory was created with incorrect permissions due to the sudocommands.
    WARNING: Error loading config file: /home/user/.docker/config.json -
    stat /home/user/.docker/config.json: permission denied
    
    To fix this problem, either remove the ~/.docker/ directory (it is recreated automatically, but any custom settings are lost), or change its ownership and permissions using the following commands:
    $ sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
    $ sudo chmod g+rwx "$HOME/.docker" -R
    

Configure Docker to start on boot

Most current Linux distributions (RHEL, CentOS, Fedora, Ubuntu 16.04 and higher) use systemd to manage which services start when the system boots. Ubuntu 14.10 and below use upstart.

systemd

$ sudo systemctl enable docker
To disable this behavior, use disable instead.
$ sudo systemctl disable docker



Transfer images to the offline machine



The workaround of this problem is to download Docker images on an Online system and then transfer and load these images to our Offline CentOS 7 server.
Note: You must have Docker-CE installed on the online system first. Because we will use the docker command to download and save images from Docker Hub.
[root@docker-online ~]# docker pull jenkins/jenkins
Using default tag: latest
latest: Pulling from jenkins/jenkins
741437d97401: Pull complete
34d8874714d7: Pull complete
0a108aa26679: Pull complete
7f0334c36886: Pull complete
aa29d9cbdbf5: Pull complete
e54d29f74413: Pull complete
eb5b24cf4e1f: Pull complete
5edfd6c9b475: Pull complete
b00dabba5e89: Pull complete
9f51dff87c48: Pull complete
0544e8830903: Pull complete
dd2464419c60: Pull complete
78125f701da6: Pull complete
5e3b2221f1a0: Pull complete
8700b2d54fbc: Pull complete
4613d2e35dec: Pull complete
08320da45709: Pull complete
8f947c5bbe77: Pull complete
51cf55002ec2: Pull complete
9537066ae19a: Pull complete
e156275467ac: Pull complete
Digest: sha256:20981c20164347728fca4774b3c45f5d24a73d857e8b9b8e6faf4100cfc0812d
Status: Downloaded newer image for jenkins/jenkins:latest
Similarly, pull more images according to your requirement.
Save Jenkins image in a tar file.
[root@docker-online ~]# docker save jenkins/jenkins > ~/jenkins.tar
[root@docker-online ~]# ls -lh
total 690M
-rw-------. 1 root root 1.5K Dec 22 11:29 anaconda-ks.cfg
-rw-r--r--. 1 root root 690M Feb 12 22:07 jenkins.tar
Transfer jenkins.tar to docker-offline.example.com.
Load jenkins.tar image into docker.
[root@docker-offline ~]# docker load < jenkins.tar
13d5529fd232: Loading layer  105.6MB/105.6MB
abc3250a6c7f: Loading layer  24.07MB/24.07MB
578414b395b9: Loading layer  8.005MB/8.005MB
6257fa9f9597: Loading layer  146.4MB/146.4MB
364be905de1c: Loading layer  2.332MB/2.332MB
57eab9d93a79: Loading layer  3.584kB/3.584kB
ad6eaafe7ab3: Loading layer  1.536kB/1.536kB
b98fdbf8cf7f: Loading layer  356.3MB/356.3MB
596ecd570594: Loading layer  1.698MB/1.698MB
f5ee7c2ae54f: Loading layer  338.9kB/338.9kB
349fe2545d85: Loading layer  3.584kB/3.584kB
b2b9702adfd1: Loading layer  9.728kB/9.728kB
ad16984b47fb: Loading layer  868.9kB/868.9kB
c469c008fbc0: Loading layer  4.608kB/4.608kB
5a8ce619bb31: Loading layer  77.33MB/77.33MB
8eae0810e454: Loading layer  4.608kB/4.608kB
5924ca705d38: Loading layer  9.216kB/9.216kB
852edd42bb1e: Loading layer  4.608kB/4.608kB
03b3a4ed2e5a: Loading layer  3.072kB/3.072kB
37dfb8384dfe: Loading layer  7.168kB/7.168kB
8f65ce1dc902: Loading layer  12.29kB/12.29kB
Loaded image: jenkins/jenkins:latest
jenkins.tar image has been loaded into Docker. Use following command to verify this.

[root@docker-offline ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jenkins/jenkins     latest              9b74eda1c268        32 hours ago        704MB
Now, we can create and run Docker containers from the Jenkins/Jenkins image.
Docker-CE has been installed on our Offline CentOS 7 server.




Setting up Docker for data science: rstudio and jupyter

References
https://towardsdatascience.com/a-short-guide-to-using-docker-for-your-data-science-environment-912617b3603e

https://jupyter-docker-stacks.readthedocs.io/en/latest/

https://hub.docker.com/r/rocker/rstudio/


Installation:

Tricky installation on Windows


  • download the desktop version
  • click nexts
  • run run docker hello-world  -> throws an error
  • Fixed by IT, issue with Hyper V
  • On boot took a long time to start -> Docker right click and reset


RStudio

docker run --rm -p 8787:8787 -e PASSWORD=yourpasswordhere rocker/rstudio

Use any password except rstudio

open the URL 127.0.0.1:8787

Enter rstudio and password


Jupyter

docker run -p 8888:8888 jupyter/scipy-notebook:17aba6048f44

Then open link which is printed out


Mounting a local folder
docker run -P -v "path_to_shared_local_directory":/home/jovyan/work/ -t cvbi/datascience-environment


Change "path_to_shared_local_directory" to a folder on the laptop

Got an error

docker: Error response from daemon: Drive sharing seems blocked by a firewall.
See 'docker run --help'.


Stop and remove a container

docker stop image_name
docker rm image_name



Save changes to container

After doing the changes

docker commit CONTAINER_ID new_name
docker stop CONTAINER_ID

docker rm CONTAINER_ID


Script to automate saving a container


https://github.com/zoobab/docker-save-running-containers

Restart

docker run -p xxxx:xxxx new_name





Loud fan of desktop

 Upon restart the fan of the desktop got loud again. I cleaned the desktop from the dust but it was still loud (Lower than the first sound) ...