Skip to content

Blog

U2Net

SOD

SOD (Salient Object Detection) is a topics in deep learning that by given a image, SOD can automatically segmentize the most interested objects of the image without any hints. SOD learns how human see the interested objects by detecting the denisity of feature points and segmentize the most dense parts. So far, U2Net provide a state of art performance.

First results of U2Net

These are the first results of the U2Net on target benchmark images. For the full results can be checked in Chimay-SOD1 and asubset Chimay-SOD2 can be found.

{% include ideal-image-slider/slider.html selector="slider1" %}

Image sliders

In this page, image slider for jekyll and its js code is used for image slider. Also a Jekyll Ideal Image Slider Include Demo shows the possiblity of Ideal Image Slider.

References

  • in DevOps
  • 6 min read

Deep Learning Computing CI/CD Framework

Overview

POCs on Small changes from Open Source

  1. Non-GPL license Open source projects are good basements to add value as POC espcially on Deep Learning areas.

POCs on Short but Full Cycle Deployments

  1. Projects use git/yaml script to setup CI/CD pipeline and deployment flows.

  2. Easy to trasnfer from localhost to server via gitlab-runner

Deployments based on Docker/K8s for Scalablity, Portablity

  1. Docker/K8s based deployment for scaiblity, portablity

All POCs setups as a Ecosystem

Live sites

Gitlab server

Git Repo Status Progress Comments
gitlab status progress User=root
gitlab grafana

Current runners

Servers Runner OS tag Monitoring
dlc.dlc.com dlc Ubuntu18.04 dlc, ubuntu, GPU Node GPU
dlc1.dlc.com dlc1 Ubuntu18.04 dlc1, ubuntu, GPU Node GPU
dlc2.dlc.com dlc2 Ubuntu18.04 dlc2, ubuntu Node

How to setup dlc/dlc1 to run a TensorFlow GPU project

with Gitlab runner

Step 1: Add project to Gitlab https://tailab.dlc.com:9443/deeplearningcomputing
Step 1.5: If your project is in https://git.dlc.com/

You will need to import your project for gitlab CI/CD only by add your project into https://tailab.dlc.com:9443/root/git-sync-mirror. After that, bitbucket code will be automatically syced to gitlab server.

Step 2: Enable dlc gitlab runner and setup CI/CD
Step 3: See the CI/CD results

with ssh or RDP + admin account

Step 1: Check with wj.lee@dlc.com and ask for admin account of dlc

Step 2: With ssh or RPD to login to dlc

How to setup your runner- How to install gitlab-runner in your ubuntu

Step 1: 

1
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

Step 2: 

1
sudo chmod +x /usr/local/bin/gitlab-runner
Step 3: 

1
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

Step 4: 

1
2
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start
Step 5.0: Check https://docs.gitlab.com/runner/register/index.html

1
sudo gitlab-runner register
and register interactively.

or

Step 5.1:

First, by command line for docker gitlab-runner

1
sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor docker --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, docker, ${HOSTNAME}" --run-untagged="true" --docker-image "docker:stable" --docker-privileged --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr

PS: YOUR-TOKEN can be obtained from Gitlab Server, in top menu, Admin Area->Runners to get the registration token. If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Second, by command line for shell gitlab-runner

1
sudo gitlab-runner register -n --url https://tailab.dlc.com:9443/ --registration-token YOUR-TOKEN --executor shell --description ${HOSTNAME}.dlc.com --tag-list "ubuntu, shell, ${HOSTNAME}" --run-untagged="true" --tls-ca-file=/etc/gitlab-runner/certs/ssl.csr
If you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr', please check step XX.

Step 6: Allow passwordless sudo

execute 

1
sudo joe /etc/sudoers
, then check and edit/add one line as

1
gitlab-runner  ALL=(ALL) NOPASSWD: ALL

if no joe command, please install

1
sudo apt-get install joe

Step 6.1: Modify /etc/gitlab-runner/config.toml

1
sudo joe /etc/gitlab-runner/config.toml

and change concurrent from 1 to 40 or more. Also, for shell runner, please also add 

1
environment = ["GIT_SSL_NO_VERIFY=true"]

Step 7: Install git-lsf

1
sudo apt-get -y install git-lfs

Step 8: Verify gitlab-runner 

1
sudo gitlab-runner verify

Step 9:

For ubuntu 20.04, please do this to prevent Gitlab runner shell executor doesn't work on Ubuntu focal

1
sudo rm /home/gitlab-runner/.bash_logout

Step X: If you want to upgrade gitlab-runner

This is optional step. If you want to upgrae gitlab-runner to newest one. Please do the following commands

1
2
3
4
sudo systemctl stop gitlab-runner.service
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo systemctl start gitlab-runner.service
sudo systemctl status gitlab-runner.service

Step XX: If you met the problem like below or you have no idea how to get '/etc/gitlab-runner/certs/ssl.csr' 

1
2
3
4
ERROR: Registering runner... failed
runner=CtzAuyzs status=couldn't execute POST against https://gitlab.test.com.tw/api/v4/runners: 
Post https://gitlab.test.com.tw/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
then follow the steps below to get self-certification and note the filename is 'ssl.crt'

1
2
3
4
5
6
7
8
SERVER=SERVER=tailab.dlc.com
PORT=9443

CERTIFICATE=/etc/gitlab-runner/certs/ssl.crt

sudo mkdir -p $(dirname "$CERTIFICATE")

openssl s_client -connect ${SERVER}:${PORT} -showcerts </dev/null 2>/dev/null | sed -e '/-----BEGIN/,/-----END/!d' | sudo tee "$CERTIFICATE" >/dev/null
then you get '/etc/gitlab-runner/certs/ssl.crt' you need for gitlab-runner register.

References

  • in DevOps
  • 1 min read

Deep Learning Computing wiki

Google Slides presentation in a Jekyll post

1
2
3
4
5
6
7
8
<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by Google -->
  <iframe src="https://docs.google.com/presentation/d/1F0DQTNPg3YG_By6LMGcgwT3icJ3eMhCiupAZm76CIfE/embed?start=false&loop=false&delayms=3000" frameborder="0" width="1024px" height="768px" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>
<!-- Google embed ends -->
</div>

Onedrive Powerpoint presentation in a Jekyll post

1
2
3
4
5
6
7
8
<style>
.responsive-wrap iframe{ max-width: 100%;}
</style>
<div class="responsive-wrap">
<!-- this is the embed code provided by MS -->
<iframe src="https://barcozone-my.sharepoint.com/personal/wj_lee_barco_com/_layouts/15/Doc.aspx?sourcedoc={565a0ed9-b548-42ad-9bcf-0c86c621c369}&amp;action=embedview&amp;wdAr=1.7777777777777777" width="1024px" height="768px" frameborder="0">This is an embedded <a target="_blank" href="https://office.com">Microsoft Office</a> presentation, powered by <a target="_blank" href="https://office.com/webapps">Office</a>.</iframe>
<!-- MS embed ends -->
</div>

References

  • in DevOps
  • 1 min read
Draft

Deep Learning Computing CI/CD Framework (II)

Delete all non-running PODs

1
2
3
kubectl get pod --all-namespaces | \
awk '{if ($4 != "Running") \
system ("kubectl -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

References

Recommendation

Overview

Figure 1 DataForecast in AI

pinreset and its pin alogirthm

Amplitude Based Recommendation

amplitude user cohort lists

Here gives a demo for amplitude cohort download and query JSON-Server for Amplitude User Cohorts

References

  • in AI
  • 3 min read

Pytorch POC 2: OpenTTS

Git Repo Status Progress Comments
OpenTTS status progress Pytorch POC #2
mozillatts status progress Pytorch POC #3
MaryTTS status progress Pytorch POC #4

Based on last time keyword spotting topics on Chimay, I even mention items about TTS (text-to-speech) and showed POCs. Here I adopt Opentts to create a API server for speech and later ultrasound generation from Web.

Opentts

In live opentts demo site, you can check the conventional (non-deep learning) speech synthesis (marytts, nanotts) and deep-learning ones (Mozillatts with Tacotron and Tacotron2). Deep-learing ones provide a beeter speech quality. A public MOS test results as below also show similar conclusions.

MOS

Demo wave file as

Demo wave

Swagger API also includes the following:

Opentts swagger

The following diagram is from mozzila project. It shows the whole picture of nature lanaugege iteration with end users. But, of course, it will be a long way to go.

References

  • in AI
  • 2 min read

Dataset

Datasets

Datasets for Faces

Datasets for Images

Datasets for Images on Salient Object Detection
  • ECSSD
  • CSSD
  • DUTS-TE
  • DUTS-TR
  • salObj
  • DUT-OMRON

  • chimay-SOD1- chimay SOD testset #1

    • Download from wget 
    1
    wget -r -c -nH -np --cut-dirs=1 --content-disposition --no-check-certificate -U "Mozilla/5.0 (Android) Nextcloud-android/3.8.0" -O output.zip  http://dlc.barco.com:9980/s/m6dwLSan8M97YgW/download

Datasets for Audio

Datasets for WIFI locations

Datasets for movie rating

Datasets for animations

Datasets for NLP

References