Skip to content

DevOps

  • in DevOps
  • 1 min read
Draft

Gitlab-runner in Synology NAS

Overview

Figure 1 DevOps with AI
1
2
3
4
5
6
7
8
# For files with "RSA" in the name
openssl rsa -in RSA-privkey.pem -out rsa.key

# For files with "ECC" in the name
openssl ec -in ECC-privkey.pem -out ecc.key

# For certificate files (regardless of type)
openssl x509 -in cert.pem -out cert.crt

Refrences

  • in DevOps
  • 1 min read
Draft

Biz-Needs-Tech-Alignment

Technology must align with the goals of User needs and Businees Goals or it is just a works make things more complicated.

with this idea, you can work smarter, not harder.

Technology mainly focus on system integration and its related components and output as Biz,Process (flow), Services, Products and etc.

The component can be Algorithm, Application, Software, Hardware, System.

References

  • in DevOps
  • 6 min read

Deep Learning Computing CI/CD Framework

Overview

POCs on Small changes from Open Source

  1. Non-GPL license Open source projects are good basements to add value as POC espcially on Deep Learning areas.

POCs on Short but Full Cycle Deployments

  1. Projects use git/yaml script to setup CI/CD pipeline and deployment flows.

  2. Easy to trasnfer from localhost to server via gitlab-runner

Deployments based on Docker/K8s for Scalablity, Portablity

  1. Docker/K8s based deployment for scaiblity, portablity

All POCs setups as a Ecosystem

Live sites

Gitlab server

Git Repo Status Progress Comments
gitlab status progress User=root
gitlab grafana

Current runners

Servers Runner OS tag Monitoring
dlc.dlc.com dlc Ubuntu18.04 dlc, ubuntu, GPU Node GPU
dlc1.dlc.com dlc1 Ubuntu18.04 dlc1, ubuntu, GPU Node GPU
dlc2.dlc.com dlc2 Ubuntu18.04 dlc2, ubuntu Node

How to setup dlc/dlc1 to run a TensorFlow GPU project

with Gitlab runner

Step 1: Add project to Gitlab https://tailab.dlc.com:9443/deeplearningcomputing
Step 1.5: If your project is in https://git.dlc.com/

You will need to import your project for gitlab CI/CD only by add your project into https://tailab.dlc.com:9443/root/git-sync-mirror. After that, bitbucket code will be automatically syced to gitlab server.

Step 2: Enable dlc gitlab runner and setup CI/CD
Step 3: See the CI/CD results

with ssh or RDP + admin account

Step 1: Check with wj.lee@dlc.com and ask for admin account of dlc

Step 2: With ssh or RPD to login to dlc

How to setup your runner- How to install gitlab-runner in your ubuntu

Step 1: 

1
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

Step 2: 

1
sudo chmod +x /usr/local/bin/gitlab-runner
Step 3: 

1
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

Step 4: 

1
2
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start
Step 5.0: Check https://docs.gitlab.com/runner/register/index.html

1
sudo gitlab-runner register
and register interactively.

or

Step 5.1:

First, by command line for docker gitlab-runner

1
sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor docker --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, docker, ${HOSTNAME}" --run-untagged="true" --docker-image "docker:stable" --docker-privileged --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr

PS: YOUR-TOKEN can be obtained from Gitlab Server, in top menu, Admin Area->Runners to get the registration token. If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Second, by command line for shell gitlab-runner

1
sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor shell --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, shell, ${HOSTNAME}" --run-untagged="true" --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr
If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Step 6: Allow passwordless sudo

execute 

1
sudo joe /etc/sudoers
, then check and edit/add one line as

1
gitlab-runner  ALL=(ALL) NOPASSWD: ALL

if no joe command, please install

1
sudo apt-get install joe

Step 6.1: Modify /etc/gitlab-runner/config.toml

1
sudo joe /etc/gitlab-runner/config.toml

and change concurrent from 1 to 40 or more. Also, for shell runner, please also add 

1
environment = ["GIT_SSL_NO_VERIFY=true"]

Step 7: Install git-lsf

1
sudo apt-get -y install git-lfs

Step 8: Verify gitlab-runner 

1
sudo gitlab-runner verify

Step 9:

For ubuntu 20.04, please do this to prevent Gitlab runner shell executor doesn't work on Ubuntu focal

1
sudo rm /home/gitlab-runner/.bash_logout

Step X: If you want to upgrade gitlab-runner

This is optional step. If you want to upgrae gitlab-runner to newest one. Please do the following commands

1
2
3
4
sudo systemctl stop gitlab-runner.service
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo systemctl start gitlab-runner.service
sudo systemctl status gitlab-runner.service

Step XX: If you met the problem like below or you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr' 

1
2
3
4
ERROR: Registering runner... failed
runner=CtzAuyzs status=couldn't execute POST against https://gitlab.test.com.tw/api/v4/runners: 
Post https://gitlab.test.com.tw/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
then follow the steps below to get self-certification and note the filename is 'ssl.crt'

1
2
3
4
5
6
7
8
SERVER=SERVER=tailab.dlc.com
PORT=9443

CERTIFICATE=/etc/gitlab-runner/certs/ssl.crt

sudo mkdir -p $(dirname "$CERTIFICATE")

openssl s_client -connect ${SERVER}:${PORT} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | sudo tee "$CERTIFICATE" >/dev/null
then you get '/etc/gitlab-runner/certs/ssl.crt' you need for gitlab-runner register.

References

  • in DevOps
  • 1 min read

Deep Learning Computing wiki

Google Slides presentation in a Jekyll post

1
2
3
4
5
6
7
8
<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by Google -->
  <iframe src="https://docs.google.com/presentation/d/1F0DQTNPg3YG_By6LMGcgwT3icJ3eMhCiupAZm76CIfE/embed?start=false&loop=false&delayms=3000" frameborder="0" width="1024px" height="768px" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>
<!-- Google embed ends -->
</div>

Onedrive Powerpoint presentation in a Jekyll post

1
2
3
4
5
6
7
8
<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by MS -->
<iframe src="https://barcozone-my.sharepoint.com/personal/wj_lee_barco_com/_layouts/15/Doc.aspx?sourcedoc={565a0ed9-b548-42ad-9bcf-0c86c621c369}&amp;action=embedview&amp;wdAr=1.7777777777777777" width="1024px" height="768px" frameborder="0">This is an embedded <a target="_blank" href="https://office.com">Microsoft Office</a> presentation, powered by <a target="_blank" href="https://office.com/webapps">Office</a>.</iframe>
<!-- MS embed ends -->
</div>

References

  • in DevOps
  • 1 min read
Draft

Deep Learning Computing CI/CD Framework (II)

Delete all non-running PODs

1
2
3
kubectl get pod --all-namespaces | \
awk '{if ($4 != "Running") \
system ("kubectl -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

References