Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lXj8Ub. Armon Dadgar talks about HashiCorp Research, its long tradition of basing their tools and products on academic research, how they incorporate research, and what has been particularly useful for them. Filmed at qconnewyork.com. Armon Dadgar is currently the CTO of HashiCorp, where he brings distributed systems into the world of DevOps tooling. He has worked on Terraform, Consul, and Serf at HashiCorp, and maintains the Statsite and Bloomd OSS projects as well.

1. ⁄
Leaving the Ivory Tower:
Research in the Real World

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
hashicorp-research/

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Armon Dadgar
Co-Founder and CTO at HashiCorp

5. Copyright © 2018 HashiCorp ⁄ !3
HashiCorp Suite
C++
Provision
Operations
Secure
Security
Deploy
Development
Connect
Networking
Private Cloud AWS Azure GCP
Common Cloud Operating Model

6. Research
Origins
Mitchell Hashimoto Armon Dadgar

7. Contributing Back

8. Standing on the Shoulder of Giants
Or The Value of Research
▪ Discover the “State of the Art”
▪ Relevant works to challenge thinking
▪ Understand fundamental tradeoffs (e.g. FLP Theorem)
▪ Metrics for evaluation

9. ⁄
Building Consul:
A Story of (Service)
Discovery

10. Immutable + Micro-services
Front End
API Layer
Data Layer
Immutable Artifact

11. Common Solutions
Circa 2012
▪ Hard Coded IP of Host / Virtual IP / Load Balancer
▪ Config Management “Convergence Runs”
▪ Custom Zookeeper based systems

12. Imagining Solutions
API Layer
Data Layer
Database:3306
10.0.1.25:3306
API Layer
Data Layer
10.0.1.25:3306

13. Entirely Peer to Peer
B
C
A
D

14. Exploring the Literature
Centralized Decentralized
Central Servers “Super Peers” Peer To Peer

15. Exploring the Literature
Structured Unstructured
Rings
Spanning Trees
Binary Trees
Adaptive Structure
Hybrid Structures
Epidemic Broadcast
Mesh Network
Randomized

16. Exploring the Literature
Limited
Visibility
Full
Visibility
Few Members Known “Neighbors” Known All Members Known

17. Imposing Constraints
Cloud Datacenter Environment
Low Latency and
High Bandwidth
We are operating within a
cloud datacenter, where we
expect low latencies and high
bandwidth, relative to IoT or
Internet-wide applications.
Few Nodes (< 5K)
The operating environment
was not large scale peer-to-
peer public networks for file
sharing, but private
infrastructure. The scale is
much smaller than some other
target environments.
Simple To Implement
Keep It Simple Stupid (KISS)
was a goal. We wanted the
simplest possible
implementation, and no
simpler. Complex protocols
are more difficult to implement
correctly.

18. The SWIM Approach

19. SWIM Properties
▪ Completely Decentralized
▪ Unstructured, with Epidemic Dissemination
▪ Full Visibility, All Members Known
▪ Trades more bandwidth use for simplicity and fault tolerance

20. Closely Considered
▪ Plumtree. Hybrid tree and epidemic style.
▪ T-Man. Adaptive, can change internal style.
▪ HyParView. Limited view of membership.
▪ Complexity of implementation deemed not worthy
▪ Size of clusters not a concern for full view
▪ Expected traffic minimal

21. Adaptations Used
▪ Bi-Modal Multicast. Active Push/Pull Synchronization.
▪ Steady State vs Recovery Messages. Optimize for efficient distribution
in steady state.
▪ Lamport Clocks. Provide a causal relationship between messages.
▪ Vivaldi. Network Coordinates to determine “distance” of peers.

22. Serf Product (serf.io)

23. Gossip For Service Discovery
B
C
A
D
“Web” at IP1
“DB” at IP2
“Cache” at IP3
“LB” at IP4

24. Serf in Practice
▪ (+) Immutable Simplified
▪ (+) Fault Tolerant, Easy to Operate
▪ (-) Eventual Consistency
▪ (-) No Key/Value Configuration
▪ (-) No “Central” API or UI

25. Rethinking Architecture
B CA D
“Web” at IP1
“DB” at IP2 “Cache” at IP3“LB” at IP4
Server

26. Central Servers Challenges
▪ High Availability
▪ Durability of State
▪ Strong Consistency

27. Paxos or How Hard is it to Agree?

28. Paxos Made Simple (?)

29. Exploring The Literature
▪ Multi Paxos
▪ Egalitarian Paxos
▪ Fast Paxos
▪ Cheap Paxos
▪ Generalized Paxos

31. Raft or Paxos Made Simple

32. Consul Product (consul.io)
Hybrid CP / AP Design
- Strongly consistent servers (Raft)
- Weekly consistent membership (SWIM)
- Centralized API and State
- Decentralized Operation

33. Work Embedded in Consul (and Serf)
▪ Consensus
▪ Gossip Protocols
▪ Network Tomography
▪ Capabilities Based Security
▪ Concurrency Control (MVCC)
▪ Lamport / Vector Clocks

34. Research across Products
- Security Systems (Kerberos)
- Security Protocols
- Access Control Systems
- Cryptography
- Graph Theory
- Type Theory
- Automata Theory
- Scheduler Design (Mesos,
Borg, Omega)
- Bin Packing
- Pre-emption
- Consensus
- Gossip

35. ⁄
Forming HashiCorp
Research

36. Industrial Research Group
Jon Currey joins as Director of Research

37. Focus on industrial research,
working 18 to 24 months ahead of
engineering, on novel work.
HashiCorp Research Charter

38. Research Goals
Problem Novel Solution
Existing
Solution
Publish
Integrate
Product

39. Customer Problem
Frontend BackendInternet

40. Customer Problem
Frontend BackendInternet

41. Research Process
Collect Data
Make
Hypothesis
Design
Solution
Design
Experiment
Validate
Hypothesis
Validate
Solution

42. Gossip FSM
SuspectHealthy
Dead
Ping Timeout
Suspect Timeout
Refute Dead
Refute Suspect

43. Untimely Processing
SuspectHealthy
Dead
Ping Timeout
Suspect Timeout
Refute Dead
Refute Suspect

44. Reducing Sensitivity
Exponential
Convergence
- Replace Fixed Timers
- Use Redundant
Confirmations
- Insight from Bloom Filters, K
independent hashes
Local Health
Awareness
- Measure Local Health
- Tune sensitivity as health
changes
Early Notification
- Send Suspicion Early
- Send Suspicion Redundant
- Enable faster refute

45. Evaluation of Solution

46. Publishing Lifeguard

47. Integration with Product

48. ⁄
Picking the Problem

49. Vault Audit Logs
User Action Audit Log

50. Vault Anomaly Detector
Anomaly Detection
Audit LogUser Action

51. Anomaly Detector
Unexpected
ExpectedEvent Detector Model

52. Exploring the Literature
Few False
Negatives
Few False
Positives
Lots of false positives Lots of false negatives

53. Applications to Vault
Screen Millions
of Events
Security Issues
Missed

54. Defining a Model
Unexpected
ExpectedEvent Detector Model

55. Refining Configuration
Vault Advisor
Audit LogUser Action
Configuration

56. Vault Advisor in Depth

57. Research Status
Problem Novel Solution
Existing
Solution
Publish
Integrate
Product

58. Lifeguard Integration
Pull Request
Upstream
Research Team Project Fork Eng Team

59. Product-ization
Research Team
| Advisor
Prototype
Eng Team Train Develop
Publish
Research
Embedded

60. What’s Coming
Problem Novel Solution
Existing
Solution
Publish
Integrate
Product

61. ⁄
Research Culture

62. Fostering Research Culture
▪ Product / Engineering is 100x bigger than Research
▪ Cultural approach needed
▪ Consuming research

63. Publishing PRD / RFCs

64. Slack #talk-research

65. Brown bags and Conferences

66. Sponsorships & Memberships

67. Cultural Goals
▪ Build awareness of research
▪ Give access to published academic work
▪ Create channels to engage internally
▪ Promote involvement in external community
▪ Involve Research in Engineering, and visa versa

68. ⁄
Conclusion

69. Real world value
▪ Leverage the “State of the Art”, instead of naive design
▪ Apply domain constraints against fundamental tradeoffs
▪ Improve product performance, security, and usability

70. Research used from Day 1
▪ Academic research fundamental to HashiCorp Products
▪ Day 1 core designs based on the literature
▪ Day 2+ improvements from literature

71. HashiCorp Research
▪ Focused on Industrial Research
▪ Publishing work, not just consuming
▪ Advocate for research culture internally
▪ Features like Lifeguard
▪ New products like Vault Advisor

72. Promoting Research
▪ Build a culture around research
▪ Enable access, encourage consumption
▪ Create bridges between Research and Engineering
▪ Vocalize the benefits

73. Thank You
www.hashicorp.com

74. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
hashicorp-research/