Deep Learning Computing CI/CD Framework¶

Overview¶

POCs on Small changes from Open Source¶

Non-GPL license Open source projects are good basements to add value as POC espcially on Deep Learning areas.

POCs on Short but Full Cycle Deployments¶

Projects use git/yaml script to setup CI/CD pipeline and deployment flows.
Easy to trasnfer from localhost to server via gitlab-runner

Deployments based on Docker/K8s for Scalablity, Portablity¶

Docker/K8s based deployment for scaiblity, portablity

All POCs setups as a Ecosystem¶

Live sites¶

Gitlab server¶

Git Repo	Status	Progress	Comments
		User=root

Current runners ¶

	OS
dlc	Ubuntu18.04	dlc, ubuntu, GPU
dlc1	Ubuntu18.04	dlc1, ubuntu, GPU
dlc2	Ubuntu18.04	dlc2, ubuntu

How to setup dlc/dlc1 to run a TensorFlow GPU project¶

with Gitlab runner¶

Step 1: Add project to Gitlab https://tailab.dlc.com:9443/deeplearningcomputing ¶

Step 1.5: If your project is in https://git.dlc.com/¶

You will need to import your project for gitlab CI/CD only by add your project into https://tailab.dlc.com:9443/root/git-sync-mirror. After that, bitbucket code will be automatically syced to gitlab server.

Step 2: Enable dlc gitlab runner and setup CI/CD¶

Step 3: See the CI/CD results¶

with ssh or RDP + admin account¶

Step 1: Check with wj.lee@dlc.com and ask for admin account of dlc

Step 2: With ssh or RPD to login to dlc

How to setup your runner- How to install gitlab-runner in your ubuntu¶

Step 1:

sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

Step 2:

sudo chmod +x /usr/local/bin/gitlab-runner

Step 3:

sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

Step 4:

sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Step 5.0: Check https://docs.gitlab.com/runner/register/index.html

1	`sudo gitlab-runner register`

and register interactively.

or

Step 5.1:

First, by command line for docker gitlab-runner

sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor docker --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, docker, ${HOSTNAME}" --run-untagged="true" --docker-image "docker:stable" --docker-privileged --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr 

PS: YOUR-TOKEN can be obtained from Gitlab Server, in top menu, Admin Area->Runners to get the registration token. If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Second, by command line for shell gitlab-runner

sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor shell --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, shell, ${HOSTNAME}" --run-untagged="true" --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr

If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Step 6: Allow passwordless sudo

execute

1	`sudo joe /etc/sudoers`

, then check and edit/add one line as

gitlab-runner  ALL=(ALL) NOPASSWD: ALL

if no joe command, please install

sudo apt-get install joe

Step 6.1: Modify /etc/gitlab-runner/config.toml

1	`sudo joe /etc/gitlab-runner/config.toml`

and change concurrent from 1 to 40 or more. Also, for shell runner, please also add

environment = ["GIT_SSL_NO_VERIFY=true"]

Step 7: Install git-lsf

sudo apt-get -y install git-lfs

Step 8: Verify gitlab-runner

1	`sudo gitlab-runner verify`

Step 9:

For ubuntu 20.04, please do this to prevent Gitlab runner shell executor doesn't work on Ubuntu focal

1	`sudo rm /home/gitlab-runner/.bash_logout`

Step X: If you want to upgrade gitlab-runner

This is optional step. If you want to upgrae gitlab-runner to newest one. Please do the following commands

sudo systemctl stop gitlab-runner.service
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo systemctl start gitlab-runner.service
sudo systemctl status gitlab-runner.service

Step XX: If you met the problem like below or you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr'

ERROR: Registering runner... failed
runner=CtzAuyzs status=couldn't execute POST against https://gitlab.test.com.tw/api/v4/runners: 
Post https://gitlab.test.com.tw/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems 

then follow the steps below to get self-certification and note the filename is 'ssl.crt'

SERVER=SERVER=tailab.dlc.com
PORT=9443

CERTIFICATE=/etc/gitlab-runner/certs/ssl.crt

sudo mkdir -p $(dirname "$CERTIFICATE")

openssl s_client -connect ${SERVER}:${PORT} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | sudo tee "$CERTIFICATE" >/dev/null

then you get '/etc/gitlab-runner/certs/ssl.crt' you need for gitlab-runner register.