Skip to content

AI

Tensorflow.js POC 8: Super Resolution with waifu2x

Overview

Figure 1 Computer Vision in AI
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<body>
    <script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>

    <script src="../../assets/plugins/jquery.event.move.js"></script>
    <script src="../../assets/plugins/jquery.twentytwenty.js"></script>
    <script>
      $(function()
      {

        $(".twentytwenty-container").eq(0).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'waifu2x',click_to_move: true});
        $(".twentytwenty-container").eq(1).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x',click_to_move: true});
        $(".twentytwenty-container").eq(2).twentytwenty({default_offset_pct: 0.5,before_label: 'SRGAN',after_label: 'waifu2x-NCNN',click_to_move: true});
        $(".twentytwenty-container").eq(3).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x',after_label: 'waifu2x-NCNN',click_to_move: true});
        $(".twentytwenty-container").eq(4).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x',after_label: 'SRMD-NCNN',click_to_move: true});
        $(".twentytwenty-container").eq(5).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x',after_label: 'Anime4K',click_to_move: true});
        $(".twentytwenty-container").eq(6).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'Anime4K',click_to_move: true});

        $(".twentytwenty-container").eq(7).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'waifu2x_artwork',click_to_move: true});
        $(".twentytwenty-container").eq(8).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x_artwork',click_to_move: true});
        $(".twentytwenty-container").eq(9).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_artwork_y',click_to_move: true});
        $(".twentytwenty-container").eq(10).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_photo',click_to_move: true});

        $(".twentytwenty-container").eq(11).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'lanczos3',click_to_move: true});
        $(".twentytwenty-container").eq(12).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'lanczos3',click_to_move: true});
        $(".twentytwenty-container").eq(13).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_photo',click_to_move: true});
        $(".twentytwenty-container").eq(14).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_artwork_y',click_to_move: true});
        $(".twentytwenty-container").eq(15).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x_photo',click_to_move: true});
      });
    </script>
</body>
Git Repo Status Progress Comments
waifu2x status progress tensorflow.js POC #8

Overview

Super Resolution of images are important for video quality. Common SR like bicubic or lanczos3, now embedded in GPU as default SR. But, in 4K display or larger display, common SR is not enough, SR with Deep learning provide significant improvements on differnt target images.

Super resolution of images highly depends the properties of the image, like animation, line draw, or a full color complex nature images will have different results on different algorithms. Here we benchmark several traditional SR algorithm of bicubic, lanczos3, (Bell, and ....) from imageenlarger and do a comparison with a DL SR on line draw or animation style image.

Now we provide A/B Test on the different target images to verify the results of deep learning and normal images.

A/B test on waifu2x-NCNN vs. waifu2x vs. lanczos3 vs. bicubic vs. SRGAN

1096x632_PPT_image

1096x632_PPT_image is a 1096x632 size PNG with words and graph PPT. We use it to identify the clearance of the SR.

SRMD-NCNN = waifu2x-NCNN = waifu2x > lanczos3 > bicubic > Anime4K > SRGAN (bicubic is most GPU interpolation algorithm. So lanczos3 and waifu2x show better results to normal GPU). SRGAN, Anime4K: SR result is GG..

Quality comparison: bicubic < waifu2x
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{% slider2020 %}
  ![bicubic](../../assets/images/1096x632_PPT_image-bicubic.png)
  ![waifu2x](../../assets/images/1096x632_PPT_image_waifu2x_art_noise1_scale_tta_1.png)
{% endslider2020 %}

#### Quality comparison: lanczos3 < waifu2x
{% slider2020 %}
  ![lanczos3](../../assets/images/1096x632_PPT_image-lanczos3.png)
  ![waifu2x](../../assets/images/1096x632_PPT_image_waifu2x_art_noise1_scale_tta_1.png)
{% endslider2020 %}

#### Quality comparison: SRGAN < waifu2x. 
SRGAN is GG

{% slider2020 %}
  ![SRGAN](../../assets/images/1096x632_PPT_image-SRGAN.png)
  ![waifu2x-NCNN](../../assets/images/1096x632_PPT_image_Waifu2x-NCNN-Vulkan.png)
{% endslider2020 %}

#### Quality comparison: waifu2x = waifu2x-NCNN

{% slider2020 %}
  ![waifu2x](../../assets/images/1096x632_PPT_image_waifu2x_art_noise1_scale_tta_1.png)
  ![waifu2x-NCNN](../../assets/images/1096x632_PPT_image_Waifu2x-NCNN-Vulkan.png)
{% endslider2020 %}

#### Quality comparison: waifu2x = SRMD-NCNN

{% slider2020 %}
  ![waifu2x](../../assets/images/1096x632_PPT_image_waifu2x_art_noise1_scale_tta_1.png)
  ![SRMD-NCNN](../../assets/images/1096x632_PPT_image_SRMD-NCNN-Vulkan.png)
{% endslider2020 %}


#### Quality comparison: waifu2x > Anime4K

{% slider2020 %}
  ![waifu2x](../../assets/images/1096x632_PPT_image_waifu2x_art_noise1_scale_tta_1.png)
  ![Anime4K](../../assets/images/1096x632_PPT_image_Anime4K.png)
{% endslider2020 %}

#### Quality comparison: bicubic > Anime4K

{% slider2020 %}
  ![bicubic](../../assets/images/1096x632_PPT_image-bicubic.png)
  ![Anime4K](../../assets/images/1096x632_PPT_image_Anime4K.png)
{% endslider2020 %}

Part II

Check Part II

Next step

  • WenGL version of waifu2x.js
  • Speedup of waifu2x

References

Jekyll image comparison slider

In this page, twentytwenty image compare slider is used to enable image comparison slider.

To enable it in your post, please add the following code into your markdown documents

1
2
3
4
{% slider2020 %}
  ![lanczos3](../../assets/images/1096x632_PPT_image-lanczos3.png)
  ![Waifu2x](../../assets/images/1096x632_PPT_image_Waifu2x-NCNN-Vulkan.png)
{% endslider2020 %}

After above, you can have first version of image compariosn slider. But, if you want to change the label of the images. you will need to find default_with_image_slider.html and modify the code as below in the beginning of the .md file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
<body>
    <script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>

    <script src="../../assets/plugins/jquery.event.move.js"></script>
    <script src="../../assets/plugins/jquery.twentytwenty.js"></script>
    <script>
      $(function()
      {

        $(".twentytwenty-container").eq(0).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x-NCNN',click_to_move: true});
      });
    </script>
</body>

You also can check the code of this site in gitbucket and search for all 'twentytwenty'

Tensorflow.js POC 8: Super Resolution with waifu2x (II)

Overview

Figure 1 Computer Vision in AI
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<body>
    <script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>

    <script src="../../assets/plugins/jquery.event.move.js"></script>
    <script src="../../assets/plugins/jquery.twentytwenty.js"></script>
    <script>
      $(function()
      {

        $(".twentytwenty-container").eq(0).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'waifu2x_artwork',click_to_move: true});
        $(".twentytwenty-container").eq(1).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x_artwork',click_to_move: true});
        $(".twentytwenty-container").eq(2).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_artwork_y',click_to_move: true});
        $(".twentytwenty-container").eq(3).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_photo',click_to_move: true});

        $(".twentytwenty-container").eq(4).twentytwenty({default_offset_pct: 0.5,before_label: 'bicubic',after_label: 'lanczos3',click_to_move: true});
        $(".twentytwenty-container").eq(5).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'lanczos3',click_to_move: true});
        $(".twentytwenty-container").eq(6).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_photo',click_to_move: true});
        $(".twentytwenty-container").eq(7).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_artwork_y',click_to_move: true});
        $(".twentytwenty-container").eq(8).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x_artwork',click_to_move: true});
        $(".twentytwenty-container").eq(9).twentytwenty({default_offset_pct: 0.5,before_label: 'waifu2x_artwork',after_label: 'waifu2x_artwork_y',click_to_move: true});
        $(".twentytwenty-container").eq(10).twentytwenty({default_offset_pct: 0.5,before_label: 'lanczos3',after_label: 'waifu2x_photo',click_to_move: true});

      });
    </script>
</body>
Git Repo Status Progress Comments
waifu2x status progress tensorflow.js POC #8

avatar255x287

avatar255x287 is a 255x287 size PNG animation. We use it to identify the aliasing reduction of the SR.

waifu2x-photo = waifu2x-artwork_y = waifu2x-artwork > lanczos3 > bicubic

Quality comparison: bicubic < waifu2x_artwork
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
{% slider2020 %}
  ![bicubic](../../assets/images/avatar255x287-bicubic.png)
  ![waifu2x_artwork](../../assets/images/avatar255x287-waifu2x_artwork.png)
{% endslider2020 %}

#### Quality comparison: lanczos3 < waifu2x_artwork
{% slider2020 %}
  ![lanczos3](../../assets/images/avatar255x287-lanczos3.png)
  ![waifu2x_artwork](../../assets/images/avatar255x287-waifu2x_artwork.png)
{% endslider2020 %}

#### Quality comparison: waifu2x_artwork = waifu2x_artwork_y

{% slider2020 %}
  ![waifu2x_artwork](../../assets/images/avatar255x287-waifu2x_artwork.png)
  ![waifu2x_artwork_y](../../assets/images/avatar255x287-waifu2x_artwork_y.png)
{% endslider2020 %}

#### Quality comparison: waifu2x_artwork = waifu2x_photo

{% slider2020 %}
  ![waifu2x_artwork](../../assets/images/avatar255x287-waifu2x_artwork.png)
  ![waifu2x_photo](../../assets/images/avatar255x287-waifu2x_photo.png)
{% endslider2020 %}

### lena400x400

lena400x400 is a 400x400 size JPG with natural human. We use it to identify the color richness and fine struture of the SR.

 waifu2x-artwork_y = waifu2x-artwork > waifu2x-photo > lanczos3 = bicubic

#### Quality comparison: bicubic = lanczos3

{% slider2020 %}
  ![bicubic](../../assets/images/lena400x400-bicubic.png)
  ![lanczos3](../../assets/images/lena400x400-lanczos3.png)
{% endslider2020 %}

#### Quality comparison: waifu2x_artwork > lanczos3

{% slider2020 %}
  ![waifu2x_artwork](../../assets/images/lena400x400-waifu2x_artwork.png)
  ![lanczos3](../../assets/images/lena400x400-lanczos3.png)
{% endslider2020 %}

#### Quality comparison: waifu2x_artwork > waifu2x_photo

{% slider2020 %}
  ![waifu2x_artwork](../../assets/images/lena400x400-waifu2x_artwork.png)
  ![lanczos3](../../assets/images/lena400x400-waifu2x_photo.png)
{% endslider2020 %}

#### Quality comparison: waifu2x_artwork = waifu2x_artwork_y

{% slider2020 %}
  ![waifu2x_artwork](../../assets/images/lena400x400-waifu2x_artwork.png)
  ![waifu2x_artwork_y](../../assets/images/lena400x400-waifu2x_artwork_y.png)
{% endslider2020 %}

#### Quality comparison: lanczos3 < waifu2x_photo

{% slider2020 %}
  ![lanczos3](../../assets/images/lena400x400-lanczos3.png)
  ![waifu2x_photo](../../assets/images/lena400x400-waifu2x_photo.png)
{% endslider2020 %}

Next step

  • WenGL version of waifu2x.js
Draft

Tensorflow.js POC 1: Pose-animator

Figure 1 Computer Vision in AI
Git Repo Status Progress Comments
pose-animator status progress tensorflow.js POC #1- and WhoIsTalking for PIC #9.

Demo

A demo for two people for pose-animators. Since pose-estmiation and facemesh are not correctly aligned so sometimes, face and body will be mis-aligned. So next step for pose-animator is to improve the pose-estmiation and facemesh alignment for multiple users.

Next Milestone

  1. To support pose-estmiation and facemesh alignment for multiple users