Select non-NA values from the Pandas dataframe.

Used to be like this.

iselect=np.nonzero(LMs.left_eye_center_x.notna() & LMs.right_eye_center_x.notna() &

         LMs.nose_tip_x.notna() & LMs.mouth_center_bottom_lip_x.notna())[0]

instead we can use dropna function.

iselect = LMs.dropna(subset=['left_eye_center_x', 'right_eye_center_x', 'nose_tip_x', 'mouth_center_bottom_lip_x'])

much simpler right?

Meaning of CNN filters

I thought CNN's filters are determined. but it is not!

When I construct CNN layers with the filter size(for instance 64, 32 and so on), I thought filter's weights were determined by CNN author and just use it but it wasn't!

Basically Conv Filter's weights are also trainable. To verify this is really easy.

1. Construct CNN layers without train.
2. Visualize all the images with filter applied.

and then compare it with

1. Construct CNN layers with train.
2. Visualize all the images with filter applied.

Show line number in the jupyter notebook

If you press a key 'L' in the jupyter notebook then you can see the line number in the editor.

python binary write/read example

import numpy as np
import json

def read32(bs):
    data = bs.read(4)
    return int.from_bytes(data, byteorder='big', signed=False)

def write32(bs, int_data):
    bs.write(int_data.to_bytes(4, byteorder='big', signed=False))

def writeTest():
    trainImageFile = open('eyedata_set/train-images.ubyte', 'wb')
    count = 4
    width = 1024
    height = 768

    write32(trainImageFile, count)
    write32(trainImageFile, width)
    write32(trainImageFile, height)
   
    trainImageFile.close()
   
    print('[DONE] write test')
   
def readTest():
    trainImageFile = open('eyedata_set/train-images.ubyte', 'rb')
    count = read32(trainImageFile)
    width  = read32(trainImageFile)
    height = read32(trainImageFile)
   
    trainImageFile.close()
       
    print('[DONE] read test')
    print('{}, {}, {}'.format(count, width, height))

MNIST 데이터 읽어서 이미지로 저장하기

train-images.idx3-ubyte 파일은 http://yann.lecun.com/exdb/mnist/에서 받으면 되고
보통은 만들어진 파서 사용하면 되지만 파이썬 공부도 할겸. 직접 만들어서 처리.

# train-images.idx3-ubyte
from PIL import Image

f = open('train-images.idx3-ubyte', 'rb')

#[offset] [type]          [value]          [description]
#0000     32 bit integer  0x00000803(2051) magic number
#0004     32 bit integer  60000            number of images
#0008     32 bit integer  28               number of rows
#0012     32 bit integer  28               number of columns
#0016     unsigned byte   ??               pixel
#0017     unsigned byte   ??               pixel
#........
#xxxx     unsigned byte   ??               pixel
#Pixels are organized row-wise. Pixel values are 0

def read32(bs):
    data =  bs.read(4)
    return int.from_bytes(data, byteorder='big', signed=False)

magic = read32(f)
imageCount = read32(f)
imageRow = read32(f)
imageCol = read32(f)

for i in range(0,imageCount):
    # 루프를 돌면서 28x28개수만큼 픽셀을 읽는다.
    imageBuffer = f.read(28*28)
    image = Image.frombytes('L', (28, 28), imageBuffer, 'raw')   
    image.save('extracted/' + str(i) + '.jpg', 'JPEG')
   
print('[DONE] Extracted all the images!')
f.close()