Utilizing Groovy based closures for flexible real-time analytics

  • 220 views
Uploaded on

Speaker: Mark Johnson …

Speaker: Mark Johnson
Real-time analytics is the process of monitoring and often reacting to web based events or machine-to-machine communication and then quickly deliver the appropriate response to an end-user or machine process. The challenge though is how can we modify the analytics algorithms and reference data without shutting down the data feeds.
In this session we will demonstrate architectural and implementation patterns utilizing Spring Integration Framework, HBase, and of course Groovy to illustrate how it is possible to dynamically change your enterprise’s real-time algorithms without a system restart.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
220
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
9
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Utilizing Groovy based closures for flexible real-time analytics Mark Johnson markfjohnson@gmail.com © Hortonworks Inc. 2013 Page 1
  • 2. Who Am I? •  Director Consulting Hortonworks •  Distributed Compute fan •  Lazy programmer who prefers to reuse rather than re-implement code •  President New England Java Users Group Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 2
  • 3. Real – Time Analytics Discovery Analytics Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 3
  • 4. Real-Time Analytics Real-time analytics is the use of, or the capacity to use, all available enterprise data and resources when they are needed. It consists of dynamic analysis and reporting, based on data entered into a system less than one minute before the actual time of use. Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 4
  • 5. Changing Analytic models Lost Bits too much time to execute Real-Time Analytic Challenges Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 5
  • 6. Architectural Tradeoffs Performance Requirements: •  Required reaction time •  Resource consumption Architecting the Future of Big Data © Hortonworks Inc. 2013 Changeability Requirements: •  24 x 7 processing •  Models change •  JVM class reload issues Page 6
  • 7. Groovy Closures A Groovy Closure is a code block which is usable via a variable instance for later execution. Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 7
  • 8. Groovy Closure: Example Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 8
  • 9. GroovyShell Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 9
  • 10. Groovy closures and Hotspot 140 120 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 •  Groovy script improves with hotspot execution Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 10
  • 11. Architecture Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 11
  • 12. Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 12
  • 13. Solution: Dynamic Code deployment •  Embed GroovyShell() into service activator •  Reference an externally defined Groovy script from a service activator base method implementation. •  Groovy script may be located most anywhere: – Property file – Database – External text file Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 13
  • 14. Service Activator Example Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 14
  • 15. Solution: Load Management Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 15
  • 16. Solution: Fast / High Capacity Archival Storage Real – Time Analytics Discovery Analytics •  Archival data necessary to back test real-time analytics •  Sensor Data samples can lead to erroneous conclusions Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 16
  • 17. Demo Architecting the Future of Big Data © Hortonworks Inc. 2013 Page 17