The presentation belonging to the ALTEN Playground of november 2, 2017. More information on this playground can be found here: https://www.gitbook.com/book/matthijsmali/user-metrics/

1. Playground:
User Monitoring
Matthijs Mali
UX Designer
Joel Witteveen
Software Engineer

2. Actually, let’s call it
Data visualization
We’re going to be doing more than user monitoring

3. Welcome
• Slides available here
• [slide link]
• Preparation docs
• https://www.gitbook.com/book/matthijsmali/user-metrics/

4. Contents
• Why User Monitoring
• Tools
• Docker
• ELK-Stack | X-Pack
• Log Files
• Use cases
• Playground goal
• Hands-on
• Share results
• Wrap-up

5. Who did not
go through prep?

6. Why Data Visualization?

7. Yes, why?
• Visualize user behavior
• What is the status of our usage Hypothesis?
• Is the application working as expected?
• Easy insight in technical logging
• Visually explore large datasets

8. TOOLS

9. DOCKER

10. Docker basics

12. ELK + X

16. LOGS

17. Logs
• System logs
• System did what and when?
• Request logs
• User requested what and when?
• Event logs
• User did what without requesting something when?
• …

18. Use cases

19. Check potential security breach

20. Check potential security breach
Has unsafe API endpoint been used from an external
IP address?

21. Check usage hypothesis

22. Check usage hypothesis
50% of all sessions should contain usage of function
X, is that the case?

23. Usage monitoring

24. Usage monitoring
How often does the user search?
At what time interval is the application used?

25. Exploring data

26. Exploring data
We need insight in a large dataset and
do not have a BI tool.

27. What had you crawling through
log files recently?

28. Playground goal

29. RandomLogger
Logfiles
ELKX Container
Application Container
Own
Logs
Real-life logging situation

30. RandomLogger
Logfiles
ELKX Container
Application Container
Own
Logs
Dataset.csv
Real-life logging situation
Data exploration situation

31. Hands-on
Form pairs if you don’t have a suitable laptop

32. Hands-on
• Laptops ready & powered?
• Pair up if you feel that is necessary
• Working individually is also fine
• The ‘random’-logger we use is not similar to a real-life
situation. But gives a good idea.

33. Rough steps
1. Gathering information
2. Configure Filebeat (altenlogger)
3. Configure Logstash (elkx)
4. Restarting Logstash
5. Run Filebeat
6. Start logging
7. Create visualizations
8. Create a dashboard

34. Gathering information
Terminal 1
- docker run –d –name altenlogger:altenlogger tail –f
/dev/null
- Docker exec –it altenlogger /bin/bash -l
- Apt-get install net-tools
- Apt-get install vim
- Ifconfig (write down IP)
Terminal 2
- docker run –m4G -p 5601:5601 -p 9200:9200 -p 5044:5044 -p
9300:9300 -it --name elkx sebp/elkx
- Docker exec –it elkx /bin/bash –l
- Ifconfig (write down IP)
For linux users

35. Configure Filebeat (Altenlogger)
- vim /etc/filebeat/filebeat.yml
- Loglocatie: /tmp/testlogfiles/error/*.log
- Comment elasticsearch lines (92 – 100)
- Uncomment logstash and write down correct host (“x.x.x.x:5044”)
- Consider spacing and indenting while making edits.

36. Configure Logstash (elkx)
- Look at /etc/logstash/conf.d/ directory
- Pipeline, all files are executed
-vim /etc/logstash/conf.d/02-beats-input.conf
- Set SSL to false
- Save the document :wq
-cp 10-syslog.conf 12-altenlog.conf
-vim 12-altenlog.conf

37. Gitbook > Cheatsheet > Log Grok pattern
filter {
if ([fields][log_type] == "error") {
grok {
match => { “message” => “%{IP:client} [%{TIMESTAMP_ISO8601:timestamp}] - %{WORD:username} - %{URIPATHPARAM:request}
%{WORD:method} %{NUMBER:response}“ }
}
}
else if ([fields][log_type] == "event") {
grok {
match => { "message" => "[%{TIMESTAMP_ISO8601:timestamp}] - %{WORD:username} - %{URIPATHPARAM:path} %{DATA:category}
%{DATA:action} %{DATA:label}" }
}
}
else if ([fields][log_type] == "request") {
grok {
match => { “message” => “%{IP:client} [%{TIMESTAMP_ISO8601:timestamp}] - %{WORD:username} - %{URIPATHPARAM:request}
%{WORD:method} %{NUMBER:response}“ }
}
}
}
https://logz.io/blog/logstash-grok/

38. Restart Logstash
• service --status-all
• service logstash restart
• ps aux | grep logstash
• tail –f /var/log/logstash/logstash-plain.log

39. Run Filebeat from altenlogger
-filebeat.sh –e
- (-e shows debug information)
- If it doesn’t show connection issues, all seems okay!

40. Start logging from altenlogger
-docker exec –it altenlogger /bin/bash –l
-java –jar com.altenlogger-etc…

41. Open Kibana
• Navigate to localhost:5601
• Login with elastic/changeme
• Create index pattern: “filebeat-*”
• Choose “timestamp” for Time Filter field name
• Click create
• See no data? Adjust time filter on the top right and set to
month to date

42. Discovering data
• Try some queries
• Username:Sydney
• Username:”Sydney” method:post
• Click Add Filter
• Choose response is 200
• Drag across time to zoom in

43. Visualize
• Demo
• Most Active Users
• Total amount of requests within selection
• Requests per day

44. Steps to create a dashboard
• Create new Visualization
• Choose type
• Choose index (filebeat-*)
• Choose aggregation
• Choose fields
• Press “Play button” and finetune
• Save Visualization
• Choose a good name, ie. “Altenlogger-ActiveUsers”
• Create new dashboard / Open dashboard
• Click edit > Add
• Choose visualizations to add
• Save dashboard

45. Share results

46. What about other data?
Open data sets ☺

47. Demo Fifa dataset
• Demonstrate ‘finalized’ dashboard

48. Configuring an open data set
• See Gitbook > Playing with other data
• Stop logstash service
• Import the CSV
• Logstash configuration
• Mutate data
• CSV imported as strings by default, but we need numbers
• Start Logstash service
• Check logging elastic / logstash for errors

49. Discussion

50. Questionnaire
http://bit.ly/2A7a2Xj

51. Thanks!

52. Thanks!
• Slides available here
• [slide link]
• Preparation docs
• https://tiny.cc/altenplayground
• Questionnaire
• http://bit.ly/2A7a2Xj
{