devops,

Deep Learning Computing CI/CD Framework

Wen-Chieh-Lee Wen-Chieh-Lee Follow Aug 22, 2020 · 5 mins read
Deep Learning Computing CI/CD Framework
Share this

POCs on Small changes from Open Source

  1. Non-GPL license Open source projects are good basements to add value as POC espcially on Deep Learning areas.

POCs on Short but Full Cycle Deployments

  1. Projects use git/yaml script to setup CI/CD pipeline and deployment flows.

  2. Easy to trasnfer from localhost to server via gitlab-runner

Deployments based on Docker/K8s for Scalablity, Portablity

  1. Docker/K8s based deployment for scaiblity, portablity

All POCs setups as a Ecosystem

Live sites

Gitlab server

Git Repo Status Progress Comments
gitlab status progress User=root
gitlab grafana
 

Current runners

Servers Runner OS tag Monitoring
dlc.dlc.com dlc Ubuntu18.04 dlc, ubuntu, GPU Node GPU
dlc1.dlc.com dlc1 Ubuntu18.04 dlc1, ubuntu, GPU Node GPU
dlc2.dlc.com dlc2 Ubuntu18.04 dlc2, ubuntu Node

How to setup dlc/dlc1 to run a TensorFlow GPU project

with Gitlab runner

Step 1: Add project to Gitlab https://tailab.dlc.com:9443/deeplearningcomputing
Step 1.5: If your project is in https://git.dlc.com/

You will need to import your project for gitlab CI/CD only by add your project into https://tailab.dlc.com:9443/root/git-sync-mirror. After that, bitbucket code will be automatically syced to gitlab server.

Step 2: Enable dlc gitlab runner and setup CI/CD
Step 3: See the CI/CD results

with ssh or RDP + admin account

Step 1: Check with wj.lee@dlc.com and ask for admin account of dlc

Step 2: With ssh or RPD to login to dlc

How to setup your runner- How to install gitlab-runner in your ubuntu

Step 1:

sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

Step 2:

sudo chmod +x /usr/local/bin/gitlab-runner

Step 3:

sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

Step 4:

sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Step 5.0: Check https://docs.gitlab.com/runner/register/index.html

sudo gitlab-runner register

and register interactively.

or

Step 5.1:

First, by command line for docker gitlab-runner

sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor docker --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, docker, ${HOSTNAME}" --run-untagged="true" --docker-image "docker:stable" --docker-privileged --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr 

PS: YOUR-TOKEN can be obtained from Gitlab Server, in top menu, Admin Area->Runners to get the registration token. If you have no idea how to get ‘/etc/gitlab-runner/certs/ssl.csr’, please check step XX.

Second, by command line for shell gitlab-runner

sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor shell --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, shell, ${HOSTNAME}" --run-untagged="true" --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr

If you have no idea how to get ‘/etc/gitlab-runner/certs/ssl.csr’, please check step XX.

Step 6: Allow passwordless sudo

execute

sudo joe /etc/sudoers

, then check and edit/add one line as

gitlab-runner  ALL=(ALL) NOPASSWD: ALL

if no joe command, please install

sudo apt-get install joe

Step 6.1: Modify /etc/gitlab-runner/config.toml

sudo joe /etc/gitlab-runner/config.toml

and change concurrent from 1 to 40 or more. Also, for shell runner, please also add

environment = ["GIT_SSL_NO_VERIFY=true"]

Step 7: Install git-lsf

sudo apt-get -y install git-lfs

Step 8: Verify gitlab-runner

sudo gitlab-runner verify

Step 9:

For ubuntu 20.04, please do this to prevent Gitlab runner shell executor doesn’t work on Ubuntu focal

sudo rm /home/gitlab-runner/.bash_logout

Step X: If you want to upgrade gitlab-runner

This is optional step. If you want to upgrae gitlab-runner to newest one. Please do the following commands

sudo systemctl stop gitlab-runner.service
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo systemctl start gitlab-runner.service
sudo systemctl status gitlab-runner.service

Step XX: If you met the problem like below or you have no idea how to get ‘/etc/gitlab-runner/certs/ssl.csr’

ERROR: Registering runner... failed
runner=CtzAuyzs status=couldn't execute POST against https://gitlab.test.com.tw/api/v4/runners: 
Post https://gitlab.test.com.tw/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems 

then follow the steps below to get self-certification and note the filename is ‘ssl.crt’

SERVER=SERVER=tailab.dlc.com
PORT=9443

CERTIFICATE=/etc/gitlab-runner/certs/ssl.crt

sudo mkdir -p $(dirname "$CERTIFICATE")

openssl s_client -connect ${SERVER}:${PORT} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | sudo tee "$CERTIFICATE" >/dev/null

then you get ‘/etc/gitlab-runner/certs/ssl.crt’ you need for gitlab-runner register.

References

Wen-Chieh-Lee
Written by Wen-Chieh-Lee
Senior Software Architect