随着应用场景的越来越复杂和微服务的兴起，各种软件系统、模块和服务产生的数据越来越多，应用性能正经历着大数据时代的来临，传统应用性能管理技术已经逐渐不能适应数据的增长以及业务需求。应用性能低下和应用出错已经成为困扰软件开发和运维人员最大的挑战之一。排查这类问题往往需要几个小时甚至几天的时间，严重影响了效率和商业业务。在本演讲中，我们将对应用性能中常见的半结构化数据进行详细的分析。针对这些数据类型，我们将阐述和展示智能根源分析(RCA)的方法，从而大大降低了应用性能排查的时间至几分钟甚至几秒钟，极大提高了软件开发和运维的效率和成本。 演讲提纲： 1. 应用性能（APM）中的大数据； 1.1 大数据场景； 1.2 应用性能排查的痛点； 2. 对半结构化数据的智能根源分析； 2.1 智能根源分析（RCA）案例； 2.2 系统设计； 2.3 在日志（Log）上的应用； 3. 总结和未来方向； 听众受益： 1.理解APM场景中大数据已经到来； 2.在大数据场景中，对应用性能中不同的数据类型如何处理； 3.在应用性能降低或出错时，如何迅速的智能化的找到真正原因。

1. Principal Data Scientist, AppDynamics

2. :
+
• Principal Data Scientist AppDynamics
• APM
• 2016 Gartner Forrester
• /
• KDD,ICDM,IJCAI
• APM Log
• Java/Python/Scala production code
• Sumo Logic, Linkedin, eBay, IBM Watson

3. APM
■
■
■ RCA
■
■ Log

4. APM
■
■
■ RCA
■
■ Log

5. Offline Online

6. App

7. How?
APM

8. APM
■
■
■ RCA
■
■ Log

9. 1
2

10. app

11. ...

12. ...

13. Monitor Everything

14. Monitor Everything:
Application Performance
Management
Database
Monitoring
Synthetic
Monitoring
Browser Real-User
Monitoring
Mobile Real-User
Monitoring
Server
Monitoring
Database
API
Server
ApplicationWeb
Server
Business
Transaction
Browser
Mobile
Unified Monitoring

15. • Metrics, KPI
• CPU Java GC
•
• IP
•
• cluster —> machine —> process —> thread
•
•
• stacktraces,

16. • Metrics, KPI
•
•
•
•
PB

17. 1
Big Data

18. 2
Big Data

19. =
+

20. APM
■
■
■ RCA
■
■ Log

21. RCA
Login
Flight Status
Search Flight
Purchase
Mobile
SOA
NOSQL
Cloud
Agile
Web
IOT

22. Login
Flight Status
Search Flight
Purchase
Mobile
SOA
Cloud
Agile
Web

23. • •
• •

24. • •
• •

25. APM
■
■
■ RCA
■
■ Log

28. (Filtering)

30. “airline:AA” transactions :
• 2%
• 82% “AA”.
• 41

32. take actions!

33. APM
■
■
■ RCA
■
■ Log

36. Instrumentation
clustering, classification

37. return value
method parameters

39. transaction
online classifier
• Normal
• Slow
• Stall
• Error

41. Now we know when it gets wrong,
but can we know why?
online classifier
•Normal
•Slow
•Stall
•Error

43. all transactions
(baseline)
/ /
transactions
(query)
baseline v.s. query
transactions
(baseline)

44. all transactions
(baseline)
/ /
transactions
(100 )
baseline v.s. query
transactions
(1000 )
from = ‘Beijing’:
Baseline Query
32 3
3.2% 3%
attribute

45. all transactions
(baseline)
/ /
transactions
(100 )
baseline v.s. query
transactions
(1000 )
from = ‘Beijing’:
Baseline Query
32 3
3.2% 3%
attribute

46. all transactions
(baseline)
/ /
transactions
(100 )
baseline v.s. query
transactions
(1000 )
airline = ‘the Bus’:
Baseline Query
150 80
15% 80%
attribute

47. all transactions
(baseline)
/ /
transactions
(100 )
baseline v.s. query
transactions
(1000 )
airline = ‘the Bus’:
Baseline Query
150 80
15% 80%
attribute

48. all transactions
(baseline)
/ /
transactions
(100 )
measurements
transactions
(1000 )

49. / /
transactions
entity baseline
transactions
of entity 1
transactions
of entity 2
transactions
of entity n
entity 1 entity 2 entity n
….

50. baseline:
• cluster tier node
• app
•

51. normalization
sigmoid :

52. normalization

53. APM
■
■
■ RCA
■
■ Log

55. Error, Exception
Lots of errors

56. Error, Exception
Lots of errors
sources

57. root cause
log source

58. errors/exceptions
≠
root cause

59. errors/exceptions ≠ root cause
significant errors/exceptions
root cause

63. APM
■
■
■ RCA
■
■ Log

64. •
• production
• index
• ElasticSearch Nested Documents
• ES count overcount
undercount

65. •
• +
• RCA
•
•
•
• transaction
• log

66. • Metrics, KPI
•
•
•
•
unified RCA

67. Next Time: RCA Framework
> KPI
production