SlideShare a Scribd company logo
1 of 20
Download to read offline
Transaction Mining for Deeper Machine Data Intelligence 
Ariel Smoliar
Analyzing Related Sequences of Logs - Use Cases 
Phone registrations failures over specific period 
Tracking transactions in payment processing platform 
Tracking a renewal or new signup transaction 
E-commerce: typical user session, anomalous checkout 
transactions, catching drop off in checkout 
Tracking users on-boarding process 
Attribution modeling - Determining the origin of a user action 
How Sumo Logic handles a search query and on-boarding of 
new users 
2
Transaction (operator) Capability 
The new capability provides tools to analyze related 
sequences of logs 
Two main modes of operation: unordered and 
ordered transaction analysis 
Several result type view: 
– Unordered analysis by transaction, states (and filtering) 
– Ordered analysis by flow (and drill-down from the graph) 
3
Transaction Operator - Required Components 
The operator requires the following components: 
– Transaction IDs (Session ID, IP, user name, email, etc.) to 
group related messages together 
– States mapping from the logs 
4
Transaction Operator - Transaction IDs (examples) 
transaction on ip 
transaction on userid, usersessionid 
transaction on sessionid 
transaction on location, part 
5
Transaction Operator - Mapping States (examples) 
| transaction on sessionid 
with "Starting session *" as init, 
with "Initiating countdown *" as countdown_start, 
with "Countdown reached *" as countdown_done, 
with "Launch *” as launch 
_sourceCategory=ecom "/login" OR "/checkout” 
| parse regex "(?<ip>d{1,3}.d{1,3}.d{1,3}.d{1,3})" 
| parse regex "GET (?<url>[^" ]+)" 
| where url matches "/login" or url matches "/checkout*" 
| parse regex field=url "^(?:/checkout)?/(?<step>[A-Za-z0-9_]+)" 
| transaction on ip 
with states login, cart, checkout, shipping_method, billing, review, progress, confirmation in step 
6
Transaction Operator - fringe cut-off 
Queries are constrained by a time window 
Some transactions may be cut off if they occur near 
the edges of the window 
Filter the transactions by using the fringe argument 
7
Unordered Analysis 
Not taking into account the ordering of the messages 
within a transaction 
Covering many of the use cases 
8
Results for Unordered Analysis (1/3) 
9 
by transactions - counts the number of times a transaction hits a state 
Transactions can be filtered by using where states="___110” 
Threshold (on count) for a state can be added, with the thresh argument with "…" thresh=2 as 
Aggregates other than count can be specified using the showing clause, the first aggregate definition applies globally, 
additional aggregates may relate to a specific state. To count, use the function sum(“1”)
Results for Unordered Analysis (2/3) 
10 
by states - number transactions with specific states combination
Results for Unordered Analysis (3/3) 
11 
by logs - shows the actual logs for the transactions that satisfy the 
filter, where statues=“101_1110”
Ordered Analysis 
Monitoring transition between (two distinct) states 
Which transitions does a transaction go through 
Number of transactions between transitions 
Latency between transitions 
Supports the Sankey diagram (new chart type) 
12
Results for Ordered Analysis 
13 
by flow - The default aggregate between states is count, but users can add other aggregates 
(max(latency) or avg(latency))
Sankey Diagram - A New Chart Type 
Sankey diagram is used to visualize the magnitude of 
flow between states in ordered analysis 
New chart icon in the Search page, enabled only for 
the relevant syntax (otherwise grayed out) 
14
Sankey Diagram - Sumo’s Site 
15
Sankey Diagram - UI Features (1/3) 
Hovering over the state box exposes inbound and outbound flow 
16
Sankey Diagram - UI Features (2/3) 
17 
Hovering over the link exposes the count and flow direction
Sankey Diagram - UI Features (3/3) 
Try to drag the state boxes vertically 
18
Sankey Diagram - Drilldown from the graph! 
Clicking on a link/edge between two states will launch a new 
search showing only the relevant result for the transition 
19
Sankey Diagram - Specified Topology 
20 
E-commerce website

More Related Content

Similar to Transaction Analytics

Lead Time System (LTS) Detail Presentation
Lead Time System (LTS) Detail PresentationLead Time System (LTS) Detail Presentation
Lead Time System (LTS) Detail Presentationdynamicp
 
Step by step lsmw tutorial
Step by step lsmw tutorialStep by step lsmw tutorial
Step by step lsmw tutorialraonivaz
 
The art of the event streaming application: streams, stream processors and sc...
The art of the event streaming application: streams, stream processors and sc...The art of the event streaming application: streams, stream processors and sc...
The art of the event streaming application: streams, stream processors and sc...confluent
 
Kafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming appKafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming appNeil Avery
 
Enterprise applications in the cloud: a roadmap to workload characterization ...
Enterprise applications in the cloud: a roadmap to workload characterization ...Enterprise applications in the cloud: a roadmap to workload characterization ...
Enterprise applications in the cloud: a roadmap to workload characterization ...Leonid Grinshpan, Ph.D.
 
Click, View & Do! - English
Click, View & Do! - EnglishClick, View & Do! - English
Click, View & Do! - EnglishDafna Levy
 
0103 navigation
0103 navigation0103 navigation
0103 navigationvkyecc1
 
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...confluent
 
Kakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming appKakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming appNeil Avery
 
unified modeling language diagrams
unified modeling language diagramsunified modeling language diagrams
unified modeling language diagramsmuhammadali0014
 
Online shopping cart system file
Online shopping cart system fileOnline shopping cart system file
Online shopping cart system fileSunil Jaiswal
 
How the monitors work
How the monitors workHow the monitors work
How the monitors workraghutrv
 
Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0Anyscale
 
The State of Stream Processing
The State of Stream ProcessingThe State of Stream Processing
The State of Stream Processingconfluent
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12Ala Qunaibi
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12Ala Qunaibi
 
Serverless London 2019 FaaS composition using Kafka and CloudEvents
Serverless London 2019   FaaS composition using Kafka and CloudEventsServerless London 2019   FaaS composition using Kafka and CloudEvents
Serverless London 2019 FaaS composition using Kafka and CloudEventsNeil Avery
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemAccumulo Summit
 
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew SkeltonWhy and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew SkeltonSkelton Thatcher Consulting Ltd
 

Similar to Transaction Analytics (20)

Lead Time System (LTS) Detail Presentation
Lead Time System (LTS) Detail PresentationLead Time System (LTS) Detail Presentation
Lead Time System (LTS) Detail Presentation
 
Step by step lsmw tutorial
Step by step lsmw tutorialStep by step lsmw tutorial
Step by step lsmw tutorial
 
The art of the event streaming application: streams, stream processors and sc...
The art of the event streaming application: streams, stream processors and sc...The art of the event streaming application: streams, stream processors and sc...
The art of the event streaming application: streams, stream processors and sc...
 
Kafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming appKafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming app
 
Enterprise applications in the cloud: a roadmap to workload characterization ...
Enterprise applications in the cloud: a roadmap to workload characterization ...Enterprise applications in the cloud: a roadmap to workload characterization ...
Enterprise applications in the cloud: a roadmap to workload characterization ...
 
Click, View & Do! - English
Click, View & Do! - EnglishClick, View & Do! - English
Click, View & Do! - English
 
0103 navigation
0103 navigation0103 navigation
0103 navigation
 
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
The Art of The Event Streaming Application: Streams, Stream Processors and Sc...
 
Kakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming appKakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming app
 
Autonomous transaction
Autonomous transactionAutonomous transaction
Autonomous transaction
 
unified modeling language diagrams
unified modeling language diagramsunified modeling language diagrams
unified modeling language diagrams
 
Online shopping cart system file
Online shopping cart system fileOnline shopping cart system file
Online shopping cart system file
 
How the monitors work
How the monitors workHow the monitors work
How the monitors work
 
Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0
 
The State of Stream Processing
The State of Stream ProcessingThe State of Stream Processing
The State of Stream Processing
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12
 
Serverless London 2019 FaaS composition using Kafka and CloudEvents
Serverless London 2019   FaaS composition using Kafka and CloudEventsServerless London 2019   FaaS composition using Kafka and CloudEvents
Serverless London 2019 FaaS composition using Kafka and CloudEvents
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
 
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew SkeltonWhy and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
 

Recently uploaded

Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxFinatron037
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxdhiyaneswaranv1
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsThinkInnovation
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 

Recently uploaded (16)

Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptx
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in Logistics
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 

Transaction Analytics

  • 1. Transaction Mining for Deeper Machine Data Intelligence Ariel Smoliar
  • 2. Analyzing Related Sequences of Logs - Use Cases Phone registrations failures over specific period Tracking transactions in payment processing platform Tracking a renewal or new signup transaction E-commerce: typical user session, anomalous checkout transactions, catching drop off in checkout Tracking users on-boarding process Attribution modeling - Determining the origin of a user action How Sumo Logic handles a search query and on-boarding of new users 2
  • 3. Transaction (operator) Capability The new capability provides tools to analyze related sequences of logs Two main modes of operation: unordered and ordered transaction analysis Several result type view: – Unordered analysis by transaction, states (and filtering) – Ordered analysis by flow (and drill-down from the graph) 3
  • 4. Transaction Operator - Required Components The operator requires the following components: – Transaction IDs (Session ID, IP, user name, email, etc.) to group related messages together – States mapping from the logs 4
  • 5. Transaction Operator - Transaction IDs (examples) transaction on ip transaction on userid, usersessionid transaction on sessionid transaction on location, part 5
  • 6. Transaction Operator - Mapping States (examples) | transaction on sessionid with "Starting session *" as init, with "Initiating countdown *" as countdown_start, with "Countdown reached *" as countdown_done, with "Launch *” as launch _sourceCategory=ecom "/login" OR "/checkout” | parse regex "(?<ip>d{1,3}.d{1,3}.d{1,3}.d{1,3})" | parse regex "GET (?<url>[^" ]+)" | where url matches "/login" or url matches "/checkout*" | parse regex field=url "^(?:/checkout)?/(?<step>[A-Za-z0-9_]+)" | transaction on ip with states login, cart, checkout, shipping_method, billing, review, progress, confirmation in step 6
  • 7. Transaction Operator - fringe cut-off Queries are constrained by a time window Some transactions may be cut off if they occur near the edges of the window Filter the transactions by using the fringe argument 7
  • 8. Unordered Analysis Not taking into account the ordering of the messages within a transaction Covering many of the use cases 8
  • 9. Results for Unordered Analysis (1/3) 9 by transactions - counts the number of times a transaction hits a state Transactions can be filtered by using where states="___110” Threshold (on count) for a state can be added, with the thresh argument with "…" thresh=2 as Aggregates other than count can be specified using the showing clause, the first aggregate definition applies globally, additional aggregates may relate to a specific state. To count, use the function sum(“1”)
  • 10. Results for Unordered Analysis (2/3) 10 by states - number transactions with specific states combination
  • 11. Results for Unordered Analysis (3/3) 11 by logs - shows the actual logs for the transactions that satisfy the filter, where statues=“101_1110”
  • 12. Ordered Analysis Monitoring transition between (two distinct) states Which transitions does a transaction go through Number of transactions between transitions Latency between transitions Supports the Sankey diagram (new chart type) 12
  • 13. Results for Ordered Analysis 13 by flow - The default aggregate between states is count, but users can add other aggregates (max(latency) or avg(latency))
  • 14. Sankey Diagram - A New Chart Type Sankey diagram is used to visualize the magnitude of flow between states in ordered analysis New chart icon in the Search page, enabled only for the relevant syntax (otherwise grayed out) 14
  • 15. Sankey Diagram - Sumo’s Site 15
  • 16. Sankey Diagram - UI Features (1/3) Hovering over the state box exposes inbound and outbound flow 16
  • 17. Sankey Diagram - UI Features (2/3) 17 Hovering over the link exposes the count and flow direction
  • 18. Sankey Diagram - UI Features (3/3) Try to drag the state boxes vertically 18
  • 19. Sankey Diagram - Drilldown from the graph! Clicking on a link/edge between two states will launch a new search showing only the relevant result for the transition 19
  • 20. Sankey Diagram - Specified Topology 20 E-commerce website