Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A global consulting and IT solutions leader
Understanding
the State of your Web Application
using Apache Kafka, Spark
What if webapps ….
… have feelings?
java.lang.NullPointerException
at com.insideout.riley.anger.doAngry(Riley.java:123)
at com.insideout.riley.joy.doHappy(Ril...
Exception in thread "main" java.lang.
OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at...
Log Data Tell You Everything/Anything/Nothing
● Who tried to login to your app?
● Was the login attempt successful?
● Whic...
More Users = More Logs = TMI !!!
How to make sense of it all???
Data Analyst!
All Your Data!
NOW!
Real-Time Insights Are More:
● Relevant
● Actionable
LOG
DATA
REAL-TIME
INSIGHTS
???
Logback Appender
● Natural integration to your webapp
● Take advantage of logback framework (i.e. specific appenders per m...
Apache Kafka
● Publish-Subscribe Messaging
● Fast, Scalable, Durable
● Distributed by Design
● Well Integrated to
Hadoop E...
Analytics Platform
● Apache Spark Streaming (micro-batch)
● Apache Storm (real-time stream processing)
Summary
● Application logging is important! Tells the story of your webapp.
● Not all useful data are stored in your datab...
Questions?
Understanding the state of your web application using Apache Kafka, Spark
Upcoming SlideShare
Loading in …5
×

Understanding the state of your web application using Apache Kafka, Spark

276 views

Published on

A presentation on understanding the state of your web application using Apache Kafka and Spark by Adrian Co, one of Exist's software architects during the DevCon Summit 2015.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Understanding the state of your web application using Apache Kafka, Spark

  1. 1. A global consulting and IT solutions leader
  2. 2. Understanding the State of your Web Application using Apache Kafka, Spark
  3. 3. What if webapps …. … have feelings?
  4. 4. java.lang.NullPointerException at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.doPuberty(Riley.java:58) 15/11/09 20:57:42 DEBUG Invalid character in customer name: ‘s@dness’ 15/11/09 20:54:56 INFO Successfully saved customer record: customer.name=’Joy’
  5. 5. Exception in thread "main" java.lang. OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Unknown Source) at java.lang.String.<init>(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley .main(ScaryClown.java:1)
  6. 6. Log Data Tell You Everything/Anything/Nothing ● Who tried to login to your app? ● Was the login attempt successful? ● Which pages the user visited? ● What did the user do? ● Did ze make any purchases? ● Did the customer encountered any problems while using your app? ● What were they doing at your website when ‘Tamang Panahon’ was ongoing at Philippine Arena?
  7. 7. More Users = More Logs = TMI !!! How to make sense of it all???
  8. 8. Data Analyst! All Your Data! NOW!
  9. 9. Real-Time Insights Are More: ● Relevant ● Actionable
  10. 10. LOG DATA REAL-TIME INSIGHTS ???
  11. 11. Logback Appender ● Natural integration to your webapp ● Take advantage of logback framework (i.e. specific appenders per module, log level, etc.) ● Easier if you use structured data for your kafka messages (i.e. JSON, XML, etc.) ● Couple of OSS appenders available, but doesn’t seem to have an “official logback version”
  12. 12. Apache Kafka ● Publish-Subscribe Messaging ● Fast, Scalable, Durable ● Distributed by Design ● Well Integrated to Hadoop Ecosystem ● … makes you yell ‘Hooray!’
  13. 13. Analytics Platform ● Apache Spark Streaming (micro-batch) ● Apache Storm (real-time stream processing)
  14. 14. Summary ● Application logging is important! Tells the story of your webapp. ● Not all useful data are stored in your database. ● Listening to your webapp will give you better insights to your business. ● Data is becoming the hottest commodity right now. The faster you are able to utilize it, the more edge you will have.
  15. 15. Questions?

×