Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DIY: Computer Vision with GWT.

800 views

Published on

DIY: Computer Vision with GWT.

#GWTcreate 2015 - Munich

Francesca Tosi & Alberto Mancini

Published in: Software
  • Be the first to comment

DIY: Computer Vision with GWT.

  1. 1. DIY: computer vision with GWT Francesca Tosi Alberto Mancini francesca@jooink.com alberto@jooink.com
  2. 2. Alberto Software Developer & Linux Sysadmin C/C++ ( since 80’s :/ ) GWT ( since 1.6 ) Javascript & Java alberto@jooink.com +AlbertoMancini
  3. 3. Francesca Web and mobile developer, software engineer and architect C/C++, CFD, HPC GWT since 2009 “with a passion for clean code and fine tuned details” francesca@jooink.com @francescatosi +FrancescaTosi
  4. 4. Devfest’s CodeProject A couple of years(**) ago we started working on a pet project trying to explore how mixing modern WebAPIs and an effective programming approach(*) can enable us to develop complex client-side applications running in a browser. (*) read: GWT, i.e. a typed language compiled into js (**) for the 2012’s GDG DevFest season
  5. 5. Anybody knows Browsers are a good “execution environnement” for evoluted web-pages ….
  6. 6. Browsers are enough ?
  7. 7. I Mean ... C’mon we are in 2015: browsers are certainly more than enough for any kind of application ! GoogleDocs, Inbox, Plus, Scala vm in js, JsDOSBox…
  8. 8. BETting The right tool(*) is enough to let us to develop - complex - compute-intensive browser-based applications ? (*) GWT of course :)
  9. 9. The RIGHT TOOL Should (at least) let us: ● master the (quite tricky) web platform ● code safely … in a type-safe language (but provide native-like flexibility and performance) ● reuse existing code … and of course produce a fast and reliable application
  10. 10. NO,no way Our forecast was
  11. 11. Target We were working on Augmented Reality and we had an expertize on it so we started trying to realize an AR webApp in order to show that this kind of applications is way too-complex for the performance of a browser.
  12. 12. Augmented Reality (browser based) Marker Based Augmented Reality
  13. 13. Marker Based AR Essentially composed of 3 steps ● Marker detection and identification (finding the marker in a picture) ● Computation of the marker pose (det. six degrees of freedom w.r.t camera, f.i.) ● Syntetic objects collocation (object space to camera space transf)
  14. 14. Marker Based AR
  15. 15. NyARToolkit, do not reinvent the wheel NyARToolKit is an ARToolKit built with 100% pure Java. [great performance, featurefull, GPL] http://nyatla.jp/nyartoolkit/wp/
  16. 16. GWT-NyARToolkit … given an image we get a 3x4 (3x3 + a translation vector actually) matrix that identifies the marker in the camera space nyar.update(i_sensor); ← ‘push’ an image found = nyar.isExistMarker(marker_id); ← query if a marker is detected mm = nyar.getMarkerMatrix(marker_id); ← compute the collocation matrix https://code.google.com/p/gwt-nyartoolkit/ http://jooink.blogspot.com
  17. 17. Browser Based AR <video/> <canvas/> Ny HTTP ? ?
  18. 18. GWT … glue
  19. 19. Acquiring Video (and eventually Audio)
  20. 20. WebRTC
  21. 21. Acquiring Video (and eventually Audio) Acquisition of Audio/Video stream via simple js api.
  22. 22. Why WebRTC ?
  23. 23. Why WebRTC ? Plugins for IE e.g. Temasys provides ad ad hoc crafted replacement of adapter.js
  24. 24. Audio/Video Acqusition Audio/Video stream via simple js apis. navigator.getUserMedia(constraints,successCallBack,errorCallBack)
  25. 25. WebRTC with GWT GWT (Elemental at the beginning) At the beginning elemental, but right now (using jnsi and now jsinterop) we are able to do all the operation without dependencies GWT. https://code.google.com/p/elemental-getusermedia-demo https://github.com/jooink/gwt-getusermedia
  26. 26. Browser Based AR <video/> the video stream can (has to be) be played in a <video/> tag “native” (jsni) video.src = URL.createObjectURL(stream);
  27. 27. Browser Based AR <canvas/> <video/> canvasCtx2D.drawImage(video, ...);
  28. 28. Browser Based AR <canvas/> <video/> canvasCtx2D.drawImage(video, ...); canvasCtx2D.getImageData(sx, sy, sw, sh)
  29. 29. Browser Based AR <video/> Ny <canvas/>
  30. 30. Browser Based AR <video/> <canvas/> Ny 3D models to be shown on the marker after the recognition
  31. 31. Browser Based AR <video/> <canvas/> Ny HTTP
  32. 32. WebGL We had to reinvent the wheel :( three.js, BabylonJS parallax (GWT !) awesome shading rendering and lighting tend to mix webgl-interaction with 3D graphics
  33. 33. WebGL, gwt-webgl <video/> Ny <canvas/> https://code.google.com/p/gwt-webgl based on Elemental with some modifications damn thin wrapper around webgl standard
  34. 34. Tiny3D ● 3D geometry (4x4 & affine matrices) ● Lambertian and Phong lighting ● Polygons, Polyhedra, normals etc.. Wavefront OBJ import
  35. 35. Browser Based AR <video/> <canvas/> Ny HTTP
  36. 36. picshare We lost the BET http://picshare.jooink.com
  37. 37. Browsers are enough!
  38. 38. GWT is more than enough!
  39. 39. mGWT GWT Widgets & more for mobile
  40. 40. FirefoxOS Web developers preferred mobile OS? B2G
  41. 41. Browser Based AR <video/> <canvas/> Ny HTTP
  42. 42. Browser Based AR <video/> <canvas/> Ny HTTP
  43. 43. Computer Vision Computer Vision As told by Alberto, after a while we needed to have some more complex features on this tool for image recognition, in order to have more information on the image we are working with.
  44. 44. Computer Vision Computer Vision There exists a lot of CV libs (complete, tested and fine tuned) for example: - OpenCV - VLFeat - VLX - … C/C++
  45. 45. Computer Vision Computer Vision There exists a lot of CV libs (complete, tested and fine tuned) for example: - OpenCV - VLFeat - VLX - … - BoofCV C/C++ Java
  46. 46. Computer Vision Computer Vision by BoofCV There is a great library for Computer Vision interely written in Pure Java: BoofCV → we decided to adapt BoofCV for GWT.
  47. 47. BoofCV BoofCV is a complete & complex project BoofCV is an opensource Java library for computer vision & robotics applications. BoofCV is organized into several packages: image processing, features, geometric vision, calibration, recognition, visualize and IO.
  48. 48. BoofCV Written by Peter Abeles www.boofcv.org Link to the source code: https://github.com/lessthanoptimal/BoofCV
  49. 49. BoofCV Adapting an entire big & complex library is not is a funny game. BoofCV depends on other Java libs, so we must adapt all the dependencies
  50. 50. BoofCV - DDogLeg: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more; - GeoRegression: open source java geometry lib for scientific computing; - EJML: dense matrix linear algebra library written in Java
  51. 51. BoofCV - GWT-DDogLeg: https://github.com/jooink/gwt-ddogleg http://gwtddogleg.appspot.com/ - GWT-GeoRegression: https://github.com/jooink/gwt-georegression http://gwtgeoregression.appspot.com/ - GWT-EJML: https://github.com/jooink/gwt-ejml http://gwtejml.appspot.com/
  52. 52. EJML - EJML: dense matrix linear algebra library written in Java https://github.com/lessthanoptimal/ejml
  53. 53. GWT-EJML GWT-EJML is an adapter-project around EJML that let us to use the original library: - provides .gwt.xml files - supersources many classes - contains running examples
  54. 54. GWT-EJML GWT-EJML published on github https://github.com/jooink/gwt-ejml
  55. 55. GWT-EJML org.EJML.gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.6.1. rc1//EN" "http://google-web-toolkit.googlecode.com/svn/tags/2.6.1. rc1/distro-source/core/src/gwt-module.dtd"> <module> <source path="ejml"/> </module>
  56. 56. GWT-EJML .gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0 //EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd"> <module rename-to='gwtejml'> <inherits name='org.EJML'/> ... <super-source path="jre" /> ... </module>
  57. 57. GWT-EJML The Strategy we decided to use the SuperSourcing in order to change/adapt the class of the basic lib and leave the ejml project unchanged → the maintainability is easier
  58. 58. GWT-EJML SuperSourcing We added the super source path in the gwt.xml config file <super-source path="jre" /> and we created the file to modify in that directory (using the right path of the file in the main project).
  59. 59. GWT-EJML .gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0 //EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd"> <module rename-to='gwtejml'> <inherits name='org.EJML'/> ... <super-source path="jre" /> ... </module>
  60. 60. GWT-EJML Supersourced parts of JRE java.io.ByteArrayOutputStream public class ByteArrayOutputStream extends OutputStream { public ByteArrayOutputStream() { … } public void write(int arg0) { … } }
  61. 61. GWT-EJML Supersourced parts of EJML org.ejml.ops.MatrixIO; public class MatrixIO { … } org.ejml.alg.dense.decomposition .eig.symm.SymmetricQREigenHelper org.ejml.alg.dense.decomposition .svd.implicitqr.SvdImplicitQrAlgorithm java.io. PrintStream printf()
  62. 62. GWT-EJML Supersourced parts of EJML java.io.PrintStream does not have printf() in the gwt emulated jre We could not supersource the PrintStream because already exist on emulated jre and we cannot super- sourcing just a method.
  63. 63. GWT-EJML Link https://github.com/jooink/gwt-ejml http://gwtejml.appspot.com/
  64. 64. DDogLeg - DDOGLEG: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more; https://github.com/lessthanoptimal/ddogleg
  65. 65. GWT-DDogLeg .gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0 //EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd"> <module rename-to='gwtddogleg'> <inherits name='org.DDOGLEG'/> <inherits name='com.jooink.gwtejml.gwtEJML_lib'/> ... <super-source path="jre" /> ... </module>
  66. 66. GWT-DDogLeg Supersourced parts of DDOGLEG: java.lang.OutOfMemoryError Unchecked Exception …
  67. 67. GWT-DDogLeg Supersourced parts of DDOGLEG: org.ddogleg.struct.FastQueue org.ddogleg.struct.FastQueueList → Reflection (not supported by gwt-jre)
  68. 68. GWT-DDogLeg Link https://github.com/jooink/gwt-ddogleg http://gwtddogleg.appspot.com/
  69. 69. GEOREGRESSION - GEOREGRESSION: open source java geometry lib for scientific computing; https://github.com/lessthanoptimal/georegression
  70. 70. GWT-GEOREGRESSION - GWT-GEOREGRESSION: minor modifications
  71. 71. GWT-GEOREGRESSION .gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0 //EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd"> <module rename-to='gwtgeoregression'> <inherits name='georegression.GEOREGRESSION'/> <inherits name='com.jooink.gwtejml.gwtEJML_lib'/> <inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/> ... </module>
  72. 72. GWT-GEOREGRESSION Link https://github.com/jooink/gwt-georegression http://gwtgeoregression.appspot.com/
  73. 73. BoofCV - BOOFCV: fast computer vision library written entirely in Java; https://github.com/lessthanoptimal/boofcv
  74. 74. .gwt.xml <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0 //EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd"> <module rename-to='gwtboofcv'> <inherits name='boofcv.BOOFCV'/> <inherits name='com.jooink.gwtejml.gwtEJML_lib'/> <inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/> <inherits name='com.jooink.gwtGeoRegression.gwtGeoRegression_lib'/> ... <super-source path="jre" /> ... </module>
  75. 75. BoofCV GWT-BOOFCV: ● Reflection ● Image managment
  76. 76. BoofCV GWT-BOOFCV: ● Reflection: ○ used mostly to handle Arrays of primitive types ○ probably unneeded upstream working with Peter to understand if can be avoided through, for instance, a few of factories. ● Image managment
  77. 77. BoofCV Reflection: java.lang.reflect.Array Array.newInstance( … ) Array.getLength( … ) ...
  78. 78. BoofCV GWT-BOOFCV: ● Reflection ● Image managment
  79. 79. BoofCV Image managment: of course we had to work-out the different point of view about Images: Browser → (<img/> <video/>) BoofCV → array of values → we used <canvas/> (also in this case); canvas.getContext2d().getImageData( … );
  80. 80. BoofCV EXAMPLES: - Interest Point detection - Interest point association
  81. 81. BoofCV EXAMPLE 1 Interest Point detection: ImageFloat32 DetectDescribePoint<ImageFloat32, SurfFeature>detDesc.detect(boofImage);
  82. 82. BoofCV - Interest point detection http://static.jooink.com/experiments/boofcv-gwt/interest-points/
  83. 83. BoofCV - Interest point association
  84. 84. Questions? Contact us : Francesca Tosi R&D at Jooink Team francesca@jooink.com Alberto Mancini Dev at Jooink Team alberto@jooink.com Thanks ! Please rate this presentation http://gwtcreate.com/agenda

×