Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi

261 views

Published on

Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi.

Ingesting JSON Crime Feeds, XML Feeds, Twitter feeds, Traffic Camera Images.

Published in: Data & Analytics
  • Be the first to comment

Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi

  1. 1. Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi HENRY SOWELL Technical Director Cloudera Government Solutions TIMOTHY SPANN Field Engineer, Data in Motion Cloudera
  2. 2. © 2019 Cloudera, Inc. All rights reserved. 2 Introduction - Henry Sowell Henry Sowell Technical Director, Cloudera Government Solutions Inc hsowell@cloudera.com
  3. 3. © 2019 Cloudera, Inc. All rights reserved. 3 Introduction - Tim Spann • Tim has been running meetups in Princeton on Big Data technologies since 2015. • Tim has spoken at many international conferences on Apache NiFi, Deep Learning and Streaming. • Tim is the Field Engineer for the Data in Motion team for the East Coast. https://www.meetup.com/futureofdata-princeton/ https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://dzone.com/articles/integrating-keras-tensorflow-yolov3-into-apache-ni
  4. 4. © 2019 Cloudera, Inc. All rights reserved. 4 Introduction • Using Apache NiFi we can ingest various sources of crime related data in real-time as activities happen. Simultaneously, we can monitor live traffic cameras for infractions and correlation to other crime related activities (Source: TrafficLand). • We can do a lot of alerting, routing and reacting to crime data as it arrives, but that’s only the beginning. • Apache HBase and Apache Phoenix are great tools to tackle this problem. • We’ll populate tables with ease and speed! https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html https://github.com/tspannhw/nifi-cdsw
  5. 5. © 2019 Cloudera, Inc. All rights reserved. 5 DATAFLOW
  6. 6. © 2019 Cloudera, Inc. All rights reserved. 6
  7. 7. © 2019 Cloudera, Inc. All rights reserved. 7 CONTROL DATA-IN-MOTION FROM EDGE-TO-ENTERPRISE Cloudera DataFlow - Collect, Curate and Analyze Data-in-Motion ● Edge-to-enterprise streaming data platform for management, security and governance of real-time streaming data ● Edge data collection, processing and content routing of sensor data from edge devices ● Continuous data ingestion from any streaming source or IoT device ● Ease-of-use in building sophisticated data flows with drag-and-drop user interface ● Real-time stream processing and content syndication at the scale of millions of messages per second ● Predictive and prescriptive analytics from streaming analytics engines to gain actionable intelligence DataFlow & Steaming
  8. 8. © 2019 Cloudera, Inc. All rights reserved. 8 CLOUDERA FLOW MANAGEMENT ● Web-based user interface ● Highly configurable ● Out-of-the-box data provenance ● Designed for extensibility ● Secure ● NiFi Registry ○ DevOps support ○ FDLC ○ Versioning ○ Deployment
  9. 9. © 2019 Cloudera, Inc. All rights reserved. 9 300+ PROCESSORS FOR DEEPER ECOSYSTEM INTEGRATION Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP AMQP WebSocket
  10. 10. © 2019 Cloudera, Inc. All rights reserved. 10 ARCHITECTURE
  11. 11. © 2019 Cloudera, Inc. All rights reserved. 11
  12. 12. © 2019 Cloudera, Inc. All rights reserved. 12 Crime JSON { "cartodb_id" : "2513323", "the_geom" : "0101000020E61000009B2FFECF04CA52C0D3EE8080EFF94340", "the_geom_webmercator" : "0101000020110F0000672C610B4DEA5FC1556E4BF02C895241", "objectid" : "33869482", "dc_dist" : "06", "psa" : "2", "dispatch_date_time" : "2019-05-19T00:00:00Z", "dispatch_date" : "2019-05-19", "dispatch_time" : "00:16:00", "hour_" : "0", "dc_key" : "201906023125", "location_block" : "1000 BLOCK FILBERT ST", "ucr_general" : "600", "text_general_code" : "Thefts", "point_x" : "-75.15654373", "point_y" : "39.95262152", "shape" : { "type" : "Point", "coordinates" : [ -75.15654373, 39.95262152 ]}}
  13. 13. © 2019 Cloudera, Inc. All rights reserved. 13 DEMO NiFi
  14. 14. © 2019 Cloudera, Inc. All rights reserved. 14
  15. 15. © 2019 Cloudera, Inc. All rights reserved. 15
  16. 16. © 2019 Cloudera, Inc. All rights reserved. 16
  17. 17. © 2019 Cloudera, Inc. All rights reserved. 17 HBase
  18. 18. © 2019 Cloudera, Inc. All rights reserved. 18 SPRING BOOT APPLICATION TO PHOENIX TABLE CREATE TABLE PHILLYCRIME (dc_dist varchar, dc_key varchar not null primary key,dispatch_date varchar,dispatch_date_time varchar,dispatch_time varchar,hour varchar,location_block varchar,psa varchar, text_general_code varchar,ucr_general varchar); java -Xms512m -Xmx2048m -Dhdp.version=3.1 -Djava.net.preferIPv4Stack=true -jar target/phoenix-0.0.1-SNAPSHOT.jar @RequestMapping("/query/{query}")
  19. 19. © 2019 Cloudera, Inc. All rights reserved. 19 SPRING BOOT APPLICATION TO PHOENIX https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://github.com/tspannhw/phillycrime-springboot-phoenix
  20. 20. © 2019 Cloudera, Inc. All rights reserved. 21 DEMO HBase
  21. 21. © 2019 Cloudera, Inc. All rights reserved. 22 Beyond Collection: Gaining Actionable Intelligence ● Visualization and Reporting ● Optimizing the Allocation of Police Resources ○ Analyze trends and activity to make timely resource assignments ● Data Driven Policing ○ Holistic allsource analysis: Criminals are not defined by a single data source (e.g., social media account, etc.) ● Develop machine learning approaches to enable law enforcement missions ● Add additional datasets
  22. 22. © 2019 Cloudera, Inc. All rights reserved. 23 THANK YOU!

×