Presentation from "121 test automation day Berlin". This talk covers the interesting questions we started having with increasing traction on the Prioridata platform and how we were able to answer quite a few of them by capturing structured log data for our platform usage by setting up an ELK stack with kibana dashboards for visualization.

1. Uncovering hidden
stories in logs !
121 Test Automation Day, Berlin 11-04-2018

2. About me
● VP Engineering @ Priori Data
● Hands-on Techie
● Broad expertise in software consulting
and delivery
● Recent “Dad” :)
@jog_chandan

3. Let’s set some context!

4. ● We build tools that provide “competitive” and “market” intelligence into
mobile apps in Apple and Google play stores.
● Broad client categories
○ App Publishers
○ Consultancies & Agencies
○ Investors
○ SDK Vendors
○ “Bots” :)
- What do we do?

5. ● Web Platform & API offering
● Stats we provide
○ App Rank, Downloads and Store Revenue - By Country/Category
○ Usage & Retention (DAU, MAU, Day 1-7-30 day Retention)
○ App Rank by Keyword & Keyword Popularity
○ Market Intelligence on Apps & Publishers
○ Audience Intelligence
- What do we do?

6. Product Screenshot

7. High level Tech Stack

8. High level Data Flow
Data Collectors:
Public + Partner
data, ETL
Data Modelling Platform

9. And we lived happily ever after!

10. Questions that needed Answering...
● What’s the reason for increased slowness and performance degradation?
● Capacity Planning - Should we spend more on infrastructure?
● Who’s accessing our website and from where?
● Are we getting scraped?
○ Are bots scraping us and should we thottle them?
○ Are free trial users abusing our service?
○ Which bots are scraping us?
○ Are paid users misusing our service?
● Who are our most active users in an organization?
○ What’s there usage pattern?
○ Can we drive product decisions based on their product usage?
○ Can we upsell them?
● Who are our most active API users?

11. ELK stack to the rescue
● Multiple Log Files
● Pushed via Rsyslog
Parse, Transform, Enhance
logs
Data Storage ● Real Time Analytics
● Visualisations
● Dashboards

12. Application Log line
Apr 10 20:40:52 dolphin website_production I, [2018-04-10T20:40:51.489590
#7536] INFO -- : [130.211.1.47] method=GET
path=/apps/nhat-ky-cua-toi-1366079020/GTM-TSTP28 format=html
controller=AppsPrototypeController action=show status=404 duration=16.61
view=1.20 db=1.15 params={"name"=>"nhat-ky-cua-toi", "id"=>"1366079020",
"tab"=>"GTM-TSTP28"} ip=66.249.79.30 ua=Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html) user_id= org_id= org_name=
plan_id= old_plan_id= user_name=

13. Grok Pattern

14. Transform

15. Demo Time
http://35.192.32.176/app/kibana

16. Our “Automation” & “Deployment”
toolbox

17. Setup / Deploy using Ansible
$ansible-playbook -i hosts elk.yml ---key-file ~/.ssh/google_compute_engine

18. Ansible Concepts
● Infrastructure as code, no more handcrafting servers.
● SSH based deploys. Agentless. Applies only changes.
● Inventory: Static vs Dynamic
● Ad-hoc commands
● Playbooks
● Modules
● Enhance using Plugins

19. Talk is Easy. Show me the code!
https://github.com/chandanjog/ops-spikes/tree/master/ansible

20. Thank You!
Questions?