How to Make Hand Detector on Native Activity with OpenCV

2,976 views

Published on

Preparing
Benchmark
How to Load Files on NativeActivity

How to Make Hand Detector
Calculate Histgram of Skin Color
Detect Skin Area from CapImage
Calculate the Largest Skin Area
Matching Histgrams

Published in: Technology, Art & Photos
  • Be the first to comment

How to Make Hand Detector on Native Activity with OpenCV

  1. 1. How to Make Hand Detector on Native Activity with OpenCV Noritsuna Imamura noritsuna@siprop.org ©SIProp Project, 2006-2008 1
  2. 2. Agenda Preparing Benchmark How to Load Files on NativeActivity How to Make Hand Detector Calculate Histgram of Skin Color Detect Skin Area from CapImage Calculate the Largest Skin Area Matching Histgrams ©SIProp Project, 2006-2008 2
  3. 3. Hand Detector ©SIProp Project, 2006-2008 3
  4. 4. Chart of Hand Detector Calc Histgram of Skin Color Detect Skin Area from CapImage Calc the Largest Skin Area Matching Histgrams ©SIProp Project, 2006-2008 4
  5. 5. Chart of Hand Detector Calc Histgram of Skin Color Histgram Detect Skin Area from CapImage Convex Hull Calc the Largest Skin Area Labeling Matching Histgrams Feature Point Distance ©SIProp Project, 2006-2008 5
  6. 6. Mat vs IplImage Benchmark ©SIProp Project, 2006-2008 6
  7. 7. About Mat & IplImage cv::Mat IplImage Version 2.x and Upper Written by C++ Version 1.x and Upper Written by C Advantage Advantage Easy to Use Faster 1. 2. 3. 4. 5. 6. 7. 8. 9. cv::Mat_<cv::Vec3b> img; for (int r = 0; r < img.rows; r++ ) { for(int c = 0; c < img.cols; c++ ) { cv::Vec3b &v = img.at<cv::Vec3b>(r,c); v[0] = 0;//B v[1] = 0;//G v[2] = 0;//R } } Many Documents 1. 2. 3. 4. 5. 6. 7. 8. IplImage* img; for(int h = 0; h < img->height; h++) { for(int w = 0; w < img->width; w++){ img->imageData[img>widthStep * h + w * 3 + 0]=0;//B img->imageData[img>widthStep * h + w * 3 + 1]=0;//G img->imageData[img>widthStep * h + w * 3 + 2]=0;//R } 7 } ©SIProp Project, 2006-2008
  8. 8. Benchmark on Android Gray Scale Download Here: http://github.com/noritsuna/MatIplBenchmark ©SIProp Project, 2006-2008 8
  9. 9. How to Load File on NativeActivity ©SIProp Project, 2006-2008 9
  10. 10. AssetManager “assets” dir is your resource file dir on Android “res”(resource) dir is also same. But the file that is there is made “Resource ID” by R file. Ex. I18n How to Use NDK with Java AAssetManager Class (C++) NativeActivity AssetManager4NativeActivity by me http://github.com/noritsuna/AssetManager4NativeActivity ©SIProp Project, 2006-2008 10
  11. 11. AssetManager4NativeActivity Android.mk 1. 2. 3. 4. include $(CLEAR_VARS) LOCAL_MODULE:=assetmanager LOCAL_SRC_FILES:=../AssetManager4NativeActivity/li bassetmanager.a include $(PREBUILT_STATIC_LIBRARY) 5. LOCAL_STATIC_LIBRARIES += assetmanager Copy header File & static lib assetmanager.h libassetmanager.a ©SIProp Project, 2006-2008 11
  12. 12. AssetManager4NativeActivity assetmanager.h int setupAsset(const char *package_name); Copy "assets" directory from APK file to under "/data/data/[Package Name]" directory. int loadAseetFile(const char *package_name, const char *load_file_name); Copy File of "load_file_name" from APK file to under "/data/data/[Package Name]/assets" directory. 1. 2. 3. createAssetFile("assets/images/skincolorsample.jpg"); sprintf(file_path, "%s/%s/%s", “/data/data”, PACKAGE_NAME, "assets/images/skincolorsample.jpg"); skin_color_sample = cvLoadImage(file_path); ©SIProp Project, 2006-2008 12
  13. 13. How to Make Hand Detector ©SIProp Project, 2006-2008 13
  14. 14. Hand Detector Sample Source Code: http://github.com/noritsuna/HandDetector ©SIProp Project, 2006-2008 14
  15. 15. Chart of Hand Detector Calc Histgram of Skin Color Detect Skin Area from CapImage Calc the Largest Skin Area Matching Histgrams ©SIProp Project, 2006-2008 15
  16. 16. Chart of Hand Detector Calc Histgram of Skin Color Histgram Detect Skin Area from CapImage Convex Hull Calc the Largest Skin Area Labeling Matching Histgrams Feature Point Distance ©SIProp Project, 2006-2008 16
  17. 17. Calculate Histgram of Skin Color ©SIProp Project, 2006-2008 17
  18. 18. What’s Histgram? Frequency Distribution Chart. Why Use it? For Checking Skin Color. Each people’s Skin Color is NOT same. One of Leveling algorithm. ©SIProp Project, 2006-2008 18
  19. 19. Step 1/2 Convert RGB to HSV RGB color is changed by Light Color. Hue Saturation/Chroma Value/Lightness/Brightness 1. cvCvtColor( src, hsv, CV_BGR2HSV ); 2. IplImage* h_plane IPL_DEPTH_8U, 1 IplImage* s_plane IPL_DEPTH_8U, 1 IplImage* v_plane IPL_DEPTH_8U, 1 3. 4. = cvCreateImage( size, ); = cvCreateImage( size, ); = cvCreateImage( size, ); ©SIProp Project, 2006-2008 19
  20. 20. Step 2/2 cvCreateHist(); Prameter Dimension of Histgram Size Type Range of limit Over limit Use or Not 1. 2. 3. 4. 5. 6. 7. 8. IplImage* planes[] = { h_plane, s_plane }; *hist = cvCreateHist(2, hist_size, CV_HIST_ARRAY, ranges, 1); cvCalcHist( planes, *hist, 0, 0 ); cvMinMaxLoc(v_plane, vmin, vmax); ©SIProp Project, 2006-2008 20
  21. 21. Detect Skin Area from CapImage ©SIProp Project, 2006-2008 21
  22. 22. How to Get Skin Area? Use “Convex Hull” algorithm 1. 2. 3. 4. 5. Check Image From Left-Top. Found Black Color Pixel is Start Point. Search Black Pixel by Right Image. Go to Black Pixel that First Found, this is next point. Do 2-4 again, if back to Start Point, get Convex Hull. ※Convert to Black-White Image Image Source: http://homepage2.nifty.com/tsugu/sotuken/ronbun/sec3-2.html#0013 ©SIProp Project, 2006-2008 22
  23. 23. Step 1/3 Delete V(Lightness/Brightness) Color 1. Calculate Back Project Image by Skin Color Histgram. 2. Threshold by V(Lightness/Brightness) Color. 3. And Operation between Mask and Back Project. 4. Threshold to Back Project. (Adjustment) 1. cvCalcBackProject(planes, backProjectImage, hist); 2. cvThreshold(v_plane, maskImage, *v_min, *v_max, CV_THRESH_BINARY); cvAnd(backProjectImage, maskImage, backProjectImage); 3. 4. cvThreshold(backProjectImage, dstImage, 10, 255, CV_THRESH_BINARY); ©SIProp Project, 2006-2008 23
  24. 24. Step 2/3 Noise Reduction 1. Erode (scale-down) 2. Dilate (scale-up) 1/4 1. 2. cvErode(dstImage, dstImage, NULL, 1); cvDilate(dstImage, dstImage, NULL, 1); ©SIProp Project, 2006-2008 24
  25. 25. Step 3/3 Convex Hull cvFindContours(); Source Image Convex that is detected First Convex Pointer that detected 1. cvFindContours(dstImage, storage, &contours); ©SIProp Project, 2006-2008 25
  26. 26. Calculate the Largest Skin Area ©SIProp Project, 2006-2008 26
  27. 27. What’s Labeling? Labeling Area Marking Algorithm. 4-Connection 8-Connection Image Source: http://imagingsolution.blog107.fc2.com/blog-entry-193.html ©SIProp Project, 2006-2008 27
  28. 28. Labeling Algorithm 1/4 1, Scan Image by Raster 2, If you got a White Pixel, 1, Check Right Image Pixels 2, All “0”, Put the Latest Number + 1 in Pixel ©SIProp Project, 2006-2008 28
  29. 29. Labeling Algorithm 2/4 1, If you got a White Pixel, 1, Check Right Image Orange Pixels 2, Not “0”, The Lowest Orange Pixels Number in Pixel ©SIProp Project, 2006-2008 29
  30. 30. Labeling Algorithm 3/4 1, If got 2 more Number in Orange Pixeles, 1, Put The Lowest Number in Pixel, Change Other Numbers’ “Look up table” to The Lowest Number. ©SIProp Project, 2006-2008 30
  31. 31. Labeling Algorithm 4/4 1, After finish, Check “Look up Table”. 1, If Dst is NOT Serial Number, Change to Serial Number 2, Src is changed Dst Number. ©SIProp Project, 2006-2008 31
  32. 32. Get Area Size cvContourArea(); 1. 2. 3. 4. 5. 6. 7. for (CvSeq* c= contours; c != NULL; c = c->h_next){ double area = abs(cvContourArea(c, CV_WHOLE_SEQ)); if (maxArea < area) { maxArea = area; hand_ptr = c; } } ©SIProp Project, 2006-2008 32
  33. 33. Matching Histgrams ©SIProp Project, 2006-2008 33
  34. 34. Matching Histgrams Histgram of Oriented Gradients (HoG) Split Some Area, And Calc Histgram of each Area. ©SIProp Project, 2006-2008 34
  35. 35. Why Use HoG? Matching Hand Shape. Use Feature Point Distance with Each HoG. ©SIProp Project, 2006-2008 35
  36. 36. Step 1/3 Calculate each Cell (Block(3x3) with Edge Pixel(5x5)) luminance gradient moment luminance gradient degree=deg 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. for(int y=0; y<height; y++){ for(int x=0; x<width; x++){ if(x==0 || y==0 || x==width-1 || y==height-1){ continue; } double dx = img->imageData[y*img>widthStep+(x+1)] - img->imageData[y*img->widthStep+(x-1)]; double dy = img->imageData[(y+1)*img>widthStep+x] - img->imageData[(y-1)*img->widthStep+x]; double m = sqrt(dx*dx+dy*dy); double deg = (atan2(dy, dx)+CV_PI) * 180.0 / CV_PI; int bin = CELL_BIN * deg/360.0; if(bin < 0) bin=0; if(bin >= CELL_BIN) bin = CELL_BIN-1; hist[(int)(x/CELL_X)][(int)(y/CELL_Y)][bin] += m; } } ©SIProp Project, 2006-2008 36
  37. 37. Step 2/3 Calculate Feature Vector of Each Block (Go to Next Page) 1. 2. for(int y=0; y<BLOCK_HEIGHT; y++){ for(int x=0; x<BLOCK_WIDTH; x++){ 3. 4. 5. 6. 7. 8. 9. //Calculate Feature Vector in Block double vec[BLOCK_DIM]; memset(vec, 0, BLOCK_DIM*sizeof(double)); for(int j=0; j<BLOCK_Y; j++){ for(int i=0; i<BLOCK_X; i++){ for(int d=0; d<CELL_BIN; d++){ int index = j*(BLOCK_X*CELL_BIN) + i*CELL_BIN + d; vec[index] = hist[x+i][y+j][d]; } } } 10. 11. 12. 13. ©SIProp Project, 2006-2008 37
  38. 38. Step 3/3 (Continued) Normalize Vector Set Feature Vector 1. 2. 3. 4. 5. 6. 7. 8. //Normalization of Vector double norm = 0.0; for(int i=0; i<BLOCK_DIM; i++){ norm += vec[i]*vec[i]; } for(int i=0; i<BLOCK_DIM; i++){ vec[i] /= sqrt(norm + 1.0); } 9. 10. 11. //Put feat for(int i=0; i<BLOCK_DIM; i++){ int index = y*BLOCK_WIDTH*BLOCK_DIM + x*BLOCK_DIM + i; 12. 13. 14. 15. feat[index] = vec[i]; } } } ©SIProp Project, 2006-2008 38
  39. 39. How to Calc Approximation Calc HoG Distance of each block Get Average. ©SIProp Project, 2006-2008 39
  40. 40. Step 1/1 Calulate Feature Point Distance 1. 2. 3. 4. 5. double dist = 0.0; for(int i = 0; i < TOTAL_DIM; i++){ dist += fabs(feat1[i] - feat2[i])*fabs(feat1[i] - feat2[i]); } return sqrt(dist); ©SIProp Project, 2006-2008 40

×