Downloaded 11 times
![Copyright © 2016 Accenture All rights reserved. 3
Related Sessions in Java One 2016
Monday, Sep 19 Tuesday, Sep 20 Wednesday, Sep 21
Introduction to Java ME 8
[CON3189]
Java ME and Single-Board
Computers for Creating Industrial
Middleware [CON3187]
All Your Faces Belong to Us:
Building an Open Face
Recognition Platform [CON6217]
12:30 p.m. - 1:30 p.m 2:30 p.m. - 3:30 p.m 3:00 p.m. - 4:00 p.m
Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8
Kevin Hooke
Julio Palma
Jorge Hidalgo
Julio Palma
Jorge Hidalgo
Mariano Rodriguez](https://image.slidesharecdn.com/javaone-facescounter-160929125106/75/JavaOne-2016-Faces-Counter-3-2048.jpg)
JavaOne 2016 - Faces Counter
- 1. All Your Faces BelongTo Us Building an Open Face Recognition Platform
- 2. Copyright © 2016Accenture All rights reserved. 2 About Us Senior Technology Architect Accenture Spain Global Java Community Lead Twitter: @_deors GitHub: deors Senior Technology Analyst Accenture Spain Global Java SME Twitter: @locoporf1 GitHub: locoporf1
- 3. Copyright © 2016Accenture All rights reserved. 3 Related Sessions in Java One 2016 Monday, Sep 19 Tuesday, Sep 20 Wednesday, Sep 21 Introduction to Java ME 8 [CON3189] Java ME and Single-Board Computers for Creating Industrial Middleware [CON3187] All Your Faces Belong to Us: Building an Open Face Recognition Platform [CON6217] 12:30 p.m. - 1:30 p.m 2:30 p.m. - 3:30 p.m 3:00 p.m. - 4:00 p.m Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8 Hilton - Golden Gate 6/7/8 Kevin Hooke Julio Palma Jorge Hidalgo Julio Palma Jorge Hidalgo Mariano Rodriguez
- 4. Copyright © 2016Accenture All rights reserved. 4 • Internet of Things • Applications • OpenCV library • Cascade Classifier for Object Detection • Lbp vs Haar • Java API • Object Recognition Sequence Diagram • MQTT protocol • Quality of Service • Faces counter • Solution schema • Development environment • Demo live! Building an Open Face Recognition Platform
- 5. Copyright © 2016Accenture All rights reserved. 5 Internet Of Things • Objects: Pressure sensor in tires Monitor implant in hearts GPS in cabs Thermometers in offices … • With unique identifier • Connected to internet • Transfer data without requiring human interaction
- 6. Copyright © 2016Accenture All rights reserved. 6 IoT Applications • Smart Cities: smart street lights, traffic control, surveilance cameras, … • Healthcare: remote monitoring, ambulance telemetry, drug tracking, … • Smart Manufacturing: flow optimization, real time inventory, employee safety, … • Wearable: entertaiment, fitness, smart watch, location & tracking, … • Automotive: wire replacement, telemetry, predictive maintenance, … • Building & Home automation: light & temperature control, access control, energy optimization, … Smart homes Appliances Wearables Industrial internet Healthcare Smart places Service Industries Smart cities P
- 7. Copyright © 2016Accenture All rights reserved. 7 OpenCV Library • Coded in C/C++ • Free for academic or commercial use • Supports Windows, Linux, Mac OS, iOS and Android • Focused on real-time applications: optimized using hardware accelaration and multi-core processing • C++, Phyton and Java interfaces • Current version: 2.4.13 (May, 2016) • Repository: https://github.com/opencv/opencv
- 8. Copyright © 2016Accenture All rights reserved. 8 Cascade Classifier 1. A XML file applied to every frame of the video stream to detect a particular object 2. OpenCV distribution contains some examples to detect eyes, cats, human faces, … 3. Can be generated to detect any object by training
- 9. Copyright © 2016Accenture All rights reserved. 9 Cascade Classifier Types LBP Faster Calculations in integers Best option for embedded/mobile HAAR More accurate Calculations in floats Falses Positives Lost Frames / sec LBP 0 2 5.72 HAAR 2 0 4.24
- 10. Copyright © 2016Accenture All rights reserved. 10 • org.opencv.core • Mat • org.opencv.videoio • VideoCapture • org.opencv.objdetect • CascadeClassifier • org.opencv.imgproc • Imgproc • org.opencv.imgcodecs • Imgcodecs OpenCV Java API
- 11. Copyright © 2016Accenture All rights reserved. 11 Object Detection Sequence Diagram read VideoCapture Mat Imgproc cvtColor equalizeHist CascadeClassfier detectMultiScale Imgcodecs imwrite new new rectangle
- 12. Copyright © 2016Accenture All rights reserved. 12 Low network bandwidth Low power usage Perfect for IoT Publish / Subscribe mechanism Can use WebSockets Downloads: Client: http://www.eclipse.org/paho/ Broker: http://projects.eclipse.org/projects/tec hnology.mosquitto MQTT Protocol MQTT Broker Topic Device 1 publish Device 2 Device N subscribe subscribe …
- 13. Copyright © 2016Accenture All rights reserved. 13 MQTT: Quality of Service Publisher Broker Subscriber publish 0: At most once Guarantees a best effort delivery It won’t be acknowledged by the receiver or stored and redelivered by the sender Fire and forget publish
- 14. Copyright © 2016Accenture All rights reserved. 14 MQTT: Quality of Service Publisher Broker Subscriber publish 1: At least once Guarantees that a message will be delivered at least once to the receiver But the message can also be delivered more than once Application must be tolerating duplicates and process them accordingly publish puback store message delete message
- 15. Copyright © 2016Accenture All rights reserved. 15 MQTT: Quality of Service Publisher Broker Subscriber publish 1: Exactly once Guarantees that each message is received only once by the counterpart Safest and also the slowest quality of service level publish pubrec pubrel pubcomp store message store message
- 16. Copyright © 2016Accenture All rights reserved. 16 Face Detector Device • Raspberry PI (model 3) • Raspicam (V2) ~ 60 $}
- 17. Copyright © 2016Accenture All rights reserved. 17 Faces Dectector Solution Schema (I) … Facedetectordevices publish MQTT Broker Web Server subscribe MQTT MQTT WS
- 18. Copyright © 2016Accenture All rights reserved. 18 Face Counter Solution Schema (II) publish subscribe MQTT broker
- 19. Copyright © 2016Accenture All rights reserved. 19 Setting up a Raspberry PI: Install dependecies sudo apt-get update sudo apt-get install cmake ant build-essential pkg-config libpng12- 0 libpng12-dev libpng++-dev libpng3 libpnglite-dev zlib1g-dbg zlib1g zlib1g-dev pngtools libtiff4 libtiffxx0c2 libtiff-tools libjpeg8 libjpeg8- dev libjpeg8-dbg libjpeg-progs libavcodec-dev libavformat-dev libgstreamer0.10-0-dbg libgstreamer0.10-0libgstreamer0.10-dev libunicap2 libunicap2-dev libdc1394-22-dev libdc1394-22libdc1394-utils swig libv4l- 0 libv4l-dev
- 20. Copyright © 2016Accenture All rights reserved. 20 Setting up a Raspberry PI: Enviroment variables Edit ~/.bashrc file to add the following lines: export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/ export ANT_HOME=/usr/share/ant/
- 21. Copyright © 2016Accenture All rights reserved. 21 Setting up a Raspberry PI: Download OpenCV source code wget https://codeload.github.com/Itseez/opencv/zip/3.1.0 mv 3.1.0 opencv-3.1.0.zip unzip opencv-3.1.0.zip
- 22. Copyright © 2016Accenture All rights reserved. 22 Setting up a Raspberry PI: Create Makefile cd opencv-3.1.0 mkdir build cd build cmake -DCMAKE_BUILD_TYPE=RELEASE -DWITH_OPENCL=OFF -D BUILD_PERF_TESTS=OFF -DBUILD_SHARED_LIBS=OFF - DJAVA_INCLUDE_PATH=$JAVA_HOME/include - DJAVA_AWT_LIBRARY=$JAVA_HOME/jre/lib/amd64/libawt.so - DJAVA_JVM_LIBRARY=$JAVA_HOME/jre/lib/arm/server/libjvm.so - DCMAKE_INSTALL_PREFIX=/usr/local ..
- 23. Copyright © 2016Accenture All rights reserved. 23 Setting up a Raspberry PI: Compile, install and check • make install • ATTENTION!!! This process can take several hours! • Check if exist: build/libs/libopencv_java310.so and build/bin/opencv- 310.jar have been created!!!
- 24. Copyright © 2016Accenture All rights reserved. 24 Setting up a Raspberry PI: Install raspicam drivers • curl http://www.linux-projects.org/listing/uv4l_repo/lrkey.asc | sudo agt-key add – • Edit file /etc/apt/source.list to add the following line: deb http://www.linux-projects.org/listing/uv4l_repo/raspbian/ jessie main • sudo apt-get update • sudo apt-get install uv4l uv4l-raspicam • sudo apt-get install uv4l-raspicam-extras • sudo raspi-config (Enable raspicam interface) • sudo rpi-update (Update firmware)
- 25. Copyright © 2016Accenture All rights reserved. 25 Setting up a Raspberry PI: Transfer files & run • Using ftp client tool copy: • Faces detector properties and JAR files • MQTT Paho JAR file • Edit properties file to customize • Run java -Djava.library.path=build/libs/libopencv_java310.so -jar faces-counter.jar faces-counter.properties
- 26. Copyright © 2016Accenture All rights reserved. 26 Development Enviroment Remote Mode (option I): • Oracle JDK • NetBeans • Putty • FileZilla (client) Remote Mode (option II): • Oracle Java ME SDK • NetBeans “Embedded” Mode • BlueJ or Geany • Oracle JDK
- 27. Copyright © 2016Accenture All rights reserved. 27 • Faces Detector: https://github.com/locoporf1/faces-detector • WebApp Monitor: https://github.com/locoporf1/faces-detector-server Face Detector App Source Code
- 28.
- 29.
Editor's Notes
- #6 The problem is, people have limited time, attention and accuracy, I mean they are not very good at capturing data about things in the real world. If we had computers that knew everything there was to know about things – gathering data without any help from us -- we would be able to track and count everything and greatly reduce waste, loss and cost. We would know when things needed replacing, repairing or whatever. The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, … that are provided with unique identifiers and the ability to transfer data over a network without requiring human interaction.
- #7 Broadband Internet is become more widely available, the cost of connecting is decreasing, more devices are being created with Wi-Fi capabilities and sensors built into them, technology costs are going down, and smartphone penetration is sky-rocketing. All of these things are creating a “perfect storm” for the IoT. Practical applications of IoT technology can be found in many industries today, including precision agriculture, building management, healthcare, energy and transportation.
- #8 Open computer vision is an C library under BSD (Berkeley Software Distribution) license. It can be used in windows, linux or Mac or even in mobile such as Android or iOS. The library is optimized to use hardware accelerator (GPU) and multi-core processing; Recently, even it is optimized for ARM platforms. Java interfaces are provided within the current distribution. A jar file that have to be included in the classpath application and a DLL (dynamic link library) (or SO, shared object in linux) whose path has to be set to “java.library.path” system property. OpenCV has two versions: version 2 whose last update was in May and the version 3 that was updated in December of the last year. Benefits of version 3 on 2 are: - Improved and extended Java, Phyton and Matlab bindings - More clean up APIs - Improved Android support
- #9 It’s the XML file that defines the object or form to detect in a frame. OpenCV distribution and opencv_contrib (a module outside the official distribution) contain some cascade classifiers to detect human and cat faces, eyes, nose, ears and more It is possible to generate cascade classifiers by training. OpenCV provides tools for it and to generate the set of samples or dataset that are used as input for the training.
- #10 There two main types: Local binary patterns and haar. The main differences between both are the calculations time and accuracy. LBP is better intend for embedded or mobile devices. In this table shows a simple test result: 25 seconds working with both cascade classifiers in raspberry PI. The false positives column shows the number of wrong detections. The Lost column shows how many objects escaped from the detection. And the Frames column shows the total number of frames captured from the video stream during the test. As you can see, in this particular test, LBP was faster and as accurate as haar.
- #11 In this slide you can see the main packages and classes of the java api. - Core: has the core functionality, data structure and operations. Mat is the core data structure. It’s a matrix containing a frame or picture. - Videoio: video input / output. The main class here is VideoCapture captures video from files, image sequences, or cameras. - Objdetect. As the name says it is for object detection. The main class is cascade classifier - Imgproc: Operations with images such as dilate an image, convert from one color space to another or calculates the integral of an image. - Imgcodecs: has just one class with the same name. It’s used for read or write images from/to file.
- #12 This diagram shows how to use openCV Java API for detecting an object from a camera installed in the device. - First of all, we have to create a video capture object and an empty matrix which is the data structure to store each frame read from video stream. - The second step is: in an infinite loop, read from the video stream and store the frame in the matrix - The third step is: convert to gray color scale and equalize histograms that is, normalize frames. - The four step is the main: we have to apply the cascade classifier to the matrix in order to detect the desired object in the frame. The result is an array of rectangles demarcating the object in the frame. - Finally, this step is optional, for example for debugging proposes, you can use Imgproc class to draw these rectangles in the frame and Imgcodecs to save the frame to a file.
- #13 It is a lightweight messaging protocol for small sensors and mobile devices. Similar to XMPP used for Whatsapp this protocol is used (for example) for Facebook chat. Built as a low-overhead protocol with strong considerations towards bandwidth and CPU limitations - Consume little bandwidth - Low power or energy usage - For the previous two points, MQTT is perfect for IoT - Works using publish and subscribe mechanism; listeners subscribe to a topic and senders publish to this topic - Can goes over WebSockets, by default goes over TCP - You can find MQTT client libraries under Paho project and the broker server under Mosquitto project. Both under Eclipse for IoT project. A very useful development tool when you use MQTT is a chrome plugin called MQTTLens.
- #14 It is an input parameter that publishers have to set when they want to send a message and defines how the broker will work with the message. Its value is between zero and two. Value 0 is also called “fire and forget” because the broker will send the message to receivers once and won’t notify to sender if this message was really delivered.
- #15 For the value 1, the broker guarantees that the message was delivered at least once (may be more than one). So, your application has to support duplicated messages.
- #16 When this is not possible, you have the quality value two. In this case, the broker guarantees that the message was delivered exactly once
- #17 This is our face dectector device. One raspberry PI model 3 or model B plus. And a video camera for the raspberry, model V1 or v2, both are ok. All of this for sixty dolars more or less
- #18 This slide shows the face detector whole solution that we are going to show you in a live demo in a couple of minutes. First of all, the raspberries with camera sending to broker the number of detected faces for each frame using MQTT protocol and quality of service value 2. The broker server is iot.eclipse.org, ready to use on internet for testing porpoises. Installed in this laptop, a web application to monitor the face detectors activity and connected to the same broker server And finally we will connect to this web application using a web browser with web sockets support (or any other push technology such as server send events) and we will tell to web application to subscribe to same topic where raspberries are publishing.
- #19 The previous one with more detail. At the left side the raspberries containing in the lower layer the raspbian operative system. Above this, the raspicam driver, uv4l. The Open computer vision libraries (in this case version 3) The Java runtime version 8. And running on it, the face detector java application using as dependencies OpenCV and mqtt paho libraries. In the other hand, the monitor web application is a spring root application using vaadin and vaading charts and running over java runtime 8
- #20 Preparing a Raspberry for object detection is a little bit complex and can take a lot of time. The starting point is a raspbian operative system already installed. So the first thing to do is to install all required packages.
- #21 Next is set the JAVA_HOME and ANT_HOME environment variable because opencv compilation process will look them
- #22 Download the opencv source code from github and unzip it.
- #23 Next step is to create a makefile using the CMAKE command line tool with java flags
- #24 Once we have created makefile, we can run “MAKE” command to start the opencv compilation and then install it.
- #25 The next, install raspicam drivers, enable raspicam interfaces via raspi-config menú, update firmware and reboot the raspberry pi
- #26 Finally, we have to transfer faces-detector and MQTT client paho libraries jar files using a FTP client tool like filezilla. And customize faces-detector properties. And that is it!!!, We are ready to run. In this point is very important don’t forget set java.library.path system property to the right value of opencv dynamic link library
- #27 When you are going to develop a program for raspberry PIs, you have three options. The first of them is to develop over the raspberry PI. The current models have enough power to run a lightweight integrated development environment such as BlueJ or Granny. Also, the current raspbian distribution already includes oracle jdk 8. But the more comfortable options are any of these. In both cases you use a personal computer, where you have your preferred IDE and usually these are the most efficient way to develop. The only difference between both are if you use Java Micro Edition or the standard one. To use java micro edition has some adventages like: + remote application management + debugging + profiling + or even emulation
- #28 And to finish, you can download all source code from git hub: + The application to run into detector devices + And the web application to monitor activity detectors.