Downloaded 44 times
Uploaded byPivotalOpenSourceHub
#GeodeSummit - Design Tradeoffs in Distributed Systems
The document discusses how Southwest Airlines utilizes Apache Geode to optimize operational decisions across various data centers, handling extensive real-time data for thousands of users and critical systems. It highlights trade-offs in software and organizational domains while emphasizing the importance of scalability, recovery optimization, and effective structured communications within distributed teams. Additionally, it outlines best practices, safety tips, and important configurations for managing memory and performance in distributed systems.
More Related Content
Similar to #GeodeSummit - Design Tradeoffs in Distributed Systems
#GeodeSummit - Design Tradeoffs in Distributed Systems
- 1. Design Tradeoffs In DistributedSystems How Southwest Uses Apache Geode
- 2. Brian Dunlap @brianwdunlap Technical Lead- Aircraft Systems March 8th, 2016
- 3. APACHE GEODE AT SOUTHWEST OPSSUITE CARGO CREW SOUTHWEST.COM
- 4. Optimize decisions with integratedschedule information.
- 5. Grow over the next10 years.
- 6. Show a real-timeview of our operational day… to 1,000s of web users.
- 7. Scale across datacenters and ensure consistency.
- 8. Support our mostcritical operational systems.
- 10. Southwest’s Network Operations Control integratesdecision makers. BOISE ALBANY OKLAHOMA CITY AUSTIN PANAMA CITY BEACH CHARLESTON GREENVILLE-SPARTANBURG TUCSON LUBBOCK AMARILLO MIDLAND/ODESSA EL PASO LITTLE ROCK NASHVILLE DALLAS (LOVE FIELD) SACRAMENTO OAKLAND SAN JOSE BURBANK LOS ANGELES (LAX) ORANGE COUNTY ONTARIO SAN DIEGO SAN FRANCISCO (SFO) BIRMINGHAM LOUISVILLE CLEVELAND OMAHA TULSA RENO/TAHOE HARLINGEN/SOUTH PADRE ISLAND PUERTO VALLARTA CORPUS CHRISTI ALBUQUERQUE DES MOINES MEMPHIS CABO SAN LUCAS/LOS CABOS ROCHESTER AKRON/ CANTON WICHITA PENSACOLA MEXICO CITY NASSAU PUNTA CANA SAN JUAN MONTEGO BAY ARUBA CANCÚN FLINT GRAND RAPIDS CHARLOTTE DAYTON MINNEAPOLIS/ ST. PAUL PHOENIX DENVER INDIANAPOLIS COLUMBUS RALEIGH/DURHAM CHICAGO (MIDWAY) FT. LAUDERDALE (MIAMI AREA) DETROIT HOUSTON (HOBBY) SEATTLE/TACOMA LAS VEGAS NEW ORLEANS ST. LOUIS MILWAUKEE BUFFALO/ NIAGARA FALLS ATLANTA ORLANDO FT. MYERS/NAPLES JACKSONVILLE TAMPA WEST PALM BEACH SAN ANTONIO KANSAS CITY BELIZE CITY SAN JOSÉ LIBERIA PORTLAND WASHINGTON, D.C. (REAGAN NATIONAL) RICHMOND MANCHESTER PROVIDENCE HARTFORD/SPRINGFIELD NORFOLK/VIRGINIA BEACH BOSTON LOGAN PHILADELPHIA BALTIMORE/WASHINGTON (BWI) WASHINGTON, D.C. (DULLES) PITTSBURGH NEW YORK (LAGUARDIA) LONG ISLAND/ISLIP NEW YORK (NEWARK) SALT LAKE CITY SPOKANE PORTLAND
- 11.
- 12.
- 13.
- 14. a response timemeasured in seconds RECOVERY OPTIMIZATION DRIVES
- 15.
- 16. Thinking about boundaries TEAMS SOFTWARE FOCUS ORG FOCUS DOMAINS CORE DOMAIN SUPPORTING DOMAIN GEODE NODES ACROSSAZs GC, WAN PATTERNS PARALLEL PROCESSING ASYNC BEHAVIOR
- 17.
- 18. What do youown? What do you need? How long can you keep it? Domain tradeoffs
- 19. Books by: @ericevans0 @VaughnVernon Get Organized! DomainDriven Design (DDD) CORE DOMAIN SUPPORTING DOMAIN UBIQUITOUS LANGUAGE AGGREGATES DOMAIN EVENTS
- 20. What do youown? (core) <invest> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
- 21. Crew Maint PaxCargo Flight Gate Existing domain silos… OVER 15 YEARS OF COMPLEXITY
- 22. Crew Maint PaxCargo Flight Gate 100% 100% CORE DOMAINSSUPPORTING DOMAINS INCREMENTAL STEPS SELECTED INTEGRATION
- 23. What do youown? (core) <focus> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
- 24. Adding is veryeasy. Watch out for data that’s around for too long. Do all of these data need to be in-memory? Data at rest for a long time? (>365 days) GEODE REGION SIZES
- 25. Determine if eachsubdomain should use Geode. Don’t make an automatic decision. Domain tradeoffs
- 26. Maybe it needsan entirely different home? Domain tradeoffs
- 27.
- 28. How far? How fast? Patterntrade-offs
- 29.
- 30. OLD NEW NORMALIZED JOINS REGIONSFOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN ServiceManagerHandlerImpl
- 31.
- 32. OLD NEW NORMALIZED JOINS REGIONSFOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN
- 33. We write immutabledomain events into event regions. Clients receive events using Geode CQs. Clients checkpoint their position into separate regions. Event regions expire messages. checkpointing
- 34. Akka Cluster managesActor Singletons which coordinate parallel processing based on a logical groupId. Backpressure is implemented through a competing consumer pattern. Take a look at Akka Streams! All Geode replicate regions use distributed ack. We don’t want to converge. (some write wins) coordination (*important concept)
- 35. JMS adapter Command adapters Commandhandlers - to CQ clients View model builders - to CQ clients JMS publishers data flow
- 36. PUSH or PULL Howdo we scale expensive read I/O? Contain expensive reads With CQRS view model builders, perform heavy state enriching “select *” once. Push read updates vs. polling (Geode CQs) Conflate triggering view model rebuild events
- 37. Be careful withtimeouts! Be careful with alerts! Be careful with joins! Be careful with large values! Be careful with old habits! safety tips
- 38.
- 39. Distributed systems are createdby distributed teams. Communication coordination is a thing.
- 40. Integrate Geode securitywith a directory Tune JVM size and GC Deploy and upgrade environments Size and configure VMs Support production events Enable WAN Gateway Sender / Receivers Load snapshots between environments Automate starting and stopping clusters Teaching distributed concepts - like CAP How do we share new distributed system responsibilities? DBAs UNIX DEVs Middleware Release Management Offshore Support New Geode Team DevOps EARLIER IS BETTER
- 41. Learn to luvconversation tension. When there’s tension, you’re on the right track!
- 42. opssuite-all schedule-core.jar Use separate reposto help with boundaries. Align Teams with repo ownership. Minimize jar dependencies across teams.
- 43. EMBRACE a 100X MENTALITY! Whatdoes 100x mean? (msg/sec) Normal rate: 50 Busy rate: 500 Recover rate: 5,000 HOW FAST CAN WE RECOVER?
- 44. Create great learning resources Watchout for old habits!
- 45.
- 46. Prefer less-shared diskI/O. (local to a VM rack, or dedicated) Prefer larger + fewer Geode nodes. (4 larger nodes vs. 8 smaller ones) Take advantage of availability zones (AZs). CONVERSATION LEADERSHIP ACROSS TEAMS SHARED or SHARED LESS What infrastructure supports Geode?
- 47. Know your memory(and GC) limits. Watch out for slow heap growth that triggers continuous GC. -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -Xloggc:/your/path/node-name.GC.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=5M Check out GCViewer for GC log analysis.
- 48. Essential tool forreal-time decision optimization testing! Helpful for QA performance and functional testing. Wonderful Geode feature! WAN Gateway
- 49. Optimization binary consumesPDX via C++ Native Client Moving > 200 MB per optimization request Be careful with refactoring PDX data types! C++ Native Client
- 50.
- 51. 51 Join the ApacheGeode Community! • Check out http://geode.incubator.apache.org • Subscribe: user-subscribe@geode.incubator.apache.org • Download: http://geode.incubator.apache.org/releases/ `