Benjamin McAlary (Atlassian) discusses how to manage and monitor the performance of critical applications hosted in your IaaS provider.

2. BENJAMIN MCALARY | PRINCIPLE NETWORK ENGINEER
BMCALARY@ATLASSIAN.COM
How ThousandEyes saved the sanity of the
Atlassian Network Team

3. Agenda
Atlassian’s Network
How we deploy and use Thousandeyes
War Stories

4. Agenda
Atlassian’s Network
How we deploy and use Thousandeyes
War Stories

5. Agenda
Atlassian’s Network
How we deploy and use Thousandeyes
War Stories

6. We believe behind every great human
achievement, there is a team.
Our mission is to unleash the potential of
every team.
OUR MISSION

7. Atlassian Today
Customers
120,000 business
Employees
3000
Locations
Sydney
San Francisco
Mountain View
Austin
Yokohama
Amsterdam
Manila
Gdansk
New York

9. Atlassian Values
They guide what we do, why we create, and who we hire.
Open company,
no bullshit
Build with heart
& balance
Be the change
you seek
Play,
as a team
Don’t #@!%
the customer

10. Atlassian’s Network

11. Infinite
bandwidth
No latencyConnectivity to
everything
Free
(or cheap)
Features of a ‘good network’

12. 100+
AWS accounts

13. 200+
connected VPCs

14. 5
regions + Points-of-Presence

15. ~10
offices

16. 2
datacenter cages

17. 10-40+
Gbps

19. ASH1 POP
ASH2 POP
SC1 POP
SEA1 POP
SYD POP
DUB1 POP

20. WHAT IS IN A NETWORK POP?
Internet
Networking
•Multiple Redundant 10GB+ AWS Direct Connect
•10GB+ Public Connectivity (Tier 1) + CloudFront
•Akamai Prolexic or AWS DDOS protection
•Links to BU or Components
Global Edge
•Load Balancers

21. Network POPs
Internet
ASH2
SC1
SEA1
ASH1
Bitbucket
SYD1
DUB1
AWS EU-WE-1
AWS US-WE-1
AWS US-WE-2
AWS US-EA-1
AWS AP-SE-2
AWS EU-WE-1
MPLS
MPLS
MPLS
AWS
Softlink

22. AWS Softlink - low cost backhaul via AWS
ask me later :)

23. peerd
Patent pending… 15/788,229 ;)
Will be open sourced on Bitbucket

24. Deploying Thousandeyes

25. Before
Gigs of logs going into Splunk
Nagios/SNMP triggering PagerDuty
Loads of Custom Checks
Auto-discovery of links and devices
sFlow and NetFlow
MTR and hping
Grafana graphs (so many!)
Pingdom
.. but missing actual end-to-end distributed
visibility

26. • Offices (double as cloud agents)
• On x86 in our cages / PoPs
• VPCs in AWS (including sometimes one per route table!)
• Using puppet, ansible, as containers - depending on the
environment
Deployed Agents everywhere!

27. • Each other (agent-to-agent)
• Public and Private loopbacks
• Key Servers and Systems
• Anycast IPs (DNS)
• Lots of AWS endpoints
Monitoring all-the-things!
• IPsec tunnels
• MPLS Backbone
• Devices
• Offices
• Internet Links + Load Balancers
• VPCs and AWS indirectly

28. Wide-shallow, Tall-Thin alerting
• Alert to PagerDuty, HipChat, Stride, email
• Alert when loss is high for a few agents
• Alert when loss is low for many agents

29. Scrape API with 1
minute Lambda
(will open source on Bitbucket…)
Correlating with Datadog
Correlate and Ingest
with Datadog
Monitor with
Thousandeyes

35. Problem: Target behind arbitrary proxy

36. root:/etc# vi te-agent.cfg
log-path=/var/log
log-file-size=
log-level=
num-log-files=
account-token=
bind-addr=
crash-reports=1
proxy-type=PAC
proxy-location=https://my.internal.server/te-agent-proxy.pac
Problem: Target behind arbitrary proxy

37. function FindProxyForURL(url, host) {
if (url.match(".*via-proxy.*")) {
var proxy_url = url.match(".*via-proxy=(.*)");
return "PROXY " + proxy_url[1];
}
return "DIRECT";
}
Problem: Target behind arbitrary proxy

38. Problem: Target behind arbitrary proxy

39. War Stories

40. The setup

41. The expected behaviour

43. What was actually happening

44. AWS trying out new things in eu-central-1

45. AWS trying out new things in eu-central-1
3
2
1

46. External Vendor on AWS - Who’s at fault?

47. External Vendor on AWS - Who’s at fault?

48. After
40+ Agents (FP&A wtf?)
Thousandeyes triggers a HipChat/Stride
alert + PagerDuty
We open TE Dashboard + Datadog
Correlate and (most cases) identify
immediate problem
Fix the issue or notify the vendor
Vendors and Providers still an issue..
Thousandeyes saved our sanity + time
Before
Gigs of logs going into Splunk
Nagios/SNMP triggering PagerDuty
Custom Checks
Auto-discovery of links and devices
sFlow and NetFlow
MTR and hping
Grafana graphs (so many!)
Pingdom
.. but missing actual end-to-end distributed
visibility

49. BENJAMIN MCALARY | PRINCIPLE NETWORK ENGINEER
BMCALARY@ATLASSIAN.COM
Thank you!