Scaleway submission job

 

We build 2 instances. 1 creates an instance from an image and the other once started runs the python scripts of the submission

Submission machine

- Installs miniconda

- Creates nmr environment

- clones code from gitlab

- Make sure you save an image of this machine that will be used in starting an instance from the starter machine

- Runs submission scripts on startup. Here is the crontab -e:

# Activate nmr environment

@reboot conda activate nmr&

@reboot date >> /root/test.txt&

# Account mabolfadl

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/sub_mabolfadl.py >> /root/logs_mabolfadl.txt&

# Account mabolfadl2, lgb_plain, lgb_rnd_drop_level_9

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/sub_lgbx.py >> /root/logs_sub_lgbx.txt&

# ACcounts mabolfadl3,5,6,7,8,9,10,11 lgb_rnd_drp_level_2,6,13,11,const

@reboot /root/miniconda/envs/nmr/bin/python /root/nmr_prd/01_code/P4_live_submission.py >> /root/P4_live_submission_logs.txt&

@reboot date >> /root/test.txt&

Starter machine

The starter machine has a cronjob each Saturday night to start the instance. Make sure you copy the right image ID

0 10 * * SUN scw instance server create type=GP1-S zone=fr-par-1 image=5c38116d-aa00-441d-ad71-77355c0515ec root-volume=l:300G name=nmr-sub ip=new project-id=204d2eef-f4cc-4d7b-850d-acf2aa87eb2c

Then stop it

0 11 * * SUN cd /root && sh shutdown_nmr.sh>>logs_shutdown.log

Here are the contents of the shutdown_nmr.sh

#!/bin/bash

scw instance server list | grep nmr-sub | awk '{print $1}' > instance_id.txt

INSTANCE_ID=$(cat instance_id.txt)

#echo instance_id.txt | INSTANCE_ID

echo $INSTANCE_ID

scw instance server stop $INSTANCE_ID

sleep 300

scw instance server delete $INSTANCE_ID with-volumes=none zone=fr-par-1

Make sure the script is executable

chmod +777 shutdown_nmr.sh