Your SlideShare is downloading. ×
0
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
AppJam 2012: When SOA Meets Big Data at FamilySearch
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

AppJam 2012: When SOA Meets Big Data at FamilySearch

618

Published on

Bob Hartley, Principle Engineer from Family Search …

Bob Hartley, Principle Engineer from Family Search
Family Search’s distributed platform generates over 1.5 terabytes of data every day, servicing a user base of 3 million (and growing). Furthermore, Family Search’s application environment is highly dynamic, with code release cycles as short as 40 minutes. In this session Bob will discuss common challenges associated with managing big data in a dynamic and distributed environment, as well as his own technologies, methodologies and processes for maintaining uptime and availability in Family Search’s applications.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
618
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • First generationFully relational databaseJava/EJB application stackSingle data center – 300+ servers, 1 PB dataNever became publicly available – never able to achieve desired performance or scaleSecond generationCreated a cluster of search nodes based upon Lucene and a proprietary search technology which closely resembles map-reduce.Relational database used only for detail and long-term persistent storageJava/Struts application stackMultiple geographically distributed data centers – 1800 servers, 5 PB dataLimited public availability – achieved acceptable performance, never able to achieve desired scaleThird generationIntroduced SOA with multiple clusters of specialized services.Moved to SOLR search technologiesDatabase uses a mixture of relational and object storageJava/Spring application stackMultiple geographically distributed data centers, cloud providers – 3000 servers, 10 PB dataFull public availability of most services – finally achieved acceptable scale and performanceFourth generation (current/coming)Greater use of SOA with multiple clusters of specialized servicesMultiple specialized databases for detail and long-term persistent storageJava/Spring on backend with Node.js frontendMultiple geographically distributed data centers, multiple cloud providers – 5000 servers, 20 PB dataFull public availability of all services – improved scale and performance, more rich user experience
  • Visibility Provides insight into the details of deploying and operating computer based services Ensure that existing services are functioning within their anticipated operational envelope and react properly when they are not Identify inefficiencies in processes and services Analyze the effects of change Manage resource utilization Plan for the futureResponsiveness Respond to change quickly and efficiently Drivers of change are diverse and unpredictable System/service requirements Temporal changes Operational models Political changes Disruptive technologies Failures Employs the principles of self-service and automationStandardization Balance between uniformity and flexibility It’s about delivering value to customers Emphasizes the qualities of the service The development platform codifies the patterns Enticing and commendable Keep efficiencies high and operating costs lowVendor Independence Mitigate the risk of vendor lock-in Standards Open Source Brokering Abstraction Commoditization Increased awareness of lock-in High Premium Cost-benefit analysisFront-door value Focus on line-of-business innovation Production readiness is “built-in” System complexities are factored out SLA improvement and verification Cost analysis and forecasting “Rise with the tide” Rich monitoringBack-door value Limited involvement in service delivery Automation of processes Standard hardware/software patterns Five 9 service on three 9 infrastructure Production readiness compliance
  • Infrastructure Monitoring What Physical resources Compute Storage Network Environment Power Utilization Resource Utilization Hypervisor Etc. Why Infrastructure SLA Minimize downtime Maximize satisfaction Capacity Planning Resource for forecasting Minimize operating costs Power Management Usage optimizationApplicationMonitoring What Local host resources Compute Storage Network Application/Service behavior Error rates Response times/latency Component performance System trends Logs Why Development/Planning Increased insight Increased performance Fast problem resolution Roadmaps Application SLA Autonomic responses Horizontal scaling Failure recovery Continuous Delivery Rolling or B/G deploy System definitionsBusiness Monitoring What Process characteristics End-to-end responsiveness Downtime Error rates Quality of service Usage characteristics User return rates Length of time on a page Time to completion Patterns Why Business SLA Customer satisfaction Increase usability Business process System I/O management Process visibility Process improvement Process control
  • Request Analysis Distribution across resources Response timeLoad Analysis CPU Network Memory StorageError Analysis New errors Increased error rates
  • Transcript

    • 1. When Big Data Meets SOA Bob Hartley (@hartleybob)Development Manager, FamilySearch
    • 2. Introduction• Bob Hartley – 14 years developing Large Web Apps – C++ since 1992 & Java since 1998 – Consult with application teams on best practices and behaviors• My Team – 6 Developers – Platform & Tools to operate Prod Apps• My Company FamilySearch – Founded in 1894 – Largest genealogy organization in the world – 500 employees + 1000’s volunteers – Launched familysearch.org in 1999
    • 3. Understanding the Past In 1975 visitors to a genealogical library would need to search through millions of binders stacked floor to ceiling and hand copy information from typewritten pages.Visitors wanting to research newinformation would need to requestmicrofiche rolls stored in granite vaults ortake an expensive trip to visit recordkeepers in distant cities or countries.
    • 4. FamilySearch.orgIn 1999 familySearch.org provided search capability for several online electronicdatabases containing billions of records and shortly thereafter started digitizingrecords from archives all over the world to create the largest collection of onlinegenealogical data.
    • 5. Few Interesting Facts• 2.8 Billion Names – 300 Million added per year• 560 Million digital images – 300 Million added per year• 10 million hits per day – ~7,000 trx/min
    • 6. SOA Meets Big Data EJB Struts Spring Spring/Node.js Monolithic Monolithic SOA SOA/Big Data
    • 7. Our Application Architecture Today Insert AppDynamics Application Flow Map of your environment Screenshot
    • 8. Our Big Data Architecture• Search Queries – Compute clusters of Apache Lucene, SOLR and custom map- reduce – Single cluster cannot efficiently answer all types of queries• Image Data – Home-grown object store backed by both spinning disk and tape• Long term storage – Oracle, MySQL, Postgres• Presentation – Node.JS and Java Front-end services – Correlate data from clusters and present to the user
    • 9. Our Big Data Challenges• Size and number of compute clusters• Cost and speed of storage• Longevity of URL’s• Multi-regional distribution of systems• Multiple infrastructure providers• Long-term preservation of data• Ability to respond quickly
    • 10. My Team’s mission• Enable Business Agility Thru: – Visibility – Responsiveness – Standardization – Vendor Independence
    • 11. My Team’s other Mission• Provide Joy to Customers and Stakeholders – Deliver Features that matter Faster – Software is always production ready – Releases tied to Business Needs NOT Operational Constraints – Hyper-Responsive to System Failure
    • 12. My Team’s principles• Repeatable and Reliable processes• Automate everything!• 100% version controlled• Everything is Monitored
    • 13. SOA+Big Data+Agile = Challenging “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Principles behind the Agile Manifesto
    • 14. Software Engineering at FamilySearch40+ independentdistributed teams + Supporting single 280 Developers = SOA application +Java and Node.js
    • 15. The Industry Standard “How long would it take your organization to deploy a change that involves just a single line of code? “Do you do this on a repeatable, reliable basis?” Tom and Mary Poppendieck Authors – Lean Software Development: An Agile Toolkit
    • 16. Being Agile thru Continuous Delivery• Challenges – Previously had 3+ month release cycle – Push new features & bugs to familysearch.org• Objectives – Continuous Delivery Model – Development to production within 40 minutes
    • 17. Hard Problems to Solve• Monolithic Code Changes – Rethink to employ branch by abstraction – Experiment with changes in production – Replace monolithic code with single purpose components/services• Database Changes – Capture each change independently – Store metadata to audit change – Decouple changes from deployment – Understand impact of unused data or columns
    • 18. Aha… Release != Deploy
    • 19. Teasing a Release All Users … % of Users Key Users All Employees Test Team• Experiment features across samples• Feature isn’t complete until its available to all users and proven to work and add value.
    • 20. Monitoring is Key to Continuous Delivery • Metrics • Baselining • Data Gathering • Trending • Aggregation • History • Event Correlation Continuous Delivery • Profiling • Alerts • Root cause • Notifications analysis Development Operations • Web Console • Role-based Access • Reporting
    • 21. Monitoring a Deployment • Business SLA • Usability • Product Mgmt. • Business Processes Business • UX • BP Owner • App/Service SLA • App Operations • Continuous Delivery Application • App Development • Planning • CD System• Infrastructure SLA • Infrastructure• Capacity planning Infrastructure Operations• Operating Costs
    • 22. Detecting Problems• Establish Baselines, Identify Outliers – Request Analysis – Load Analysis – Error Analysis
    • 23. Fixing Problems• Resolve two ways – Rollback – Push a fix• Diagnosis of problem is key – Rapid triage to find root cause – History of behavior• Right Tool for Right People – Developers are best at finding and fixing – Operations are best at detecting and verifying resolution
    • 24. AppDynamics• Met our critical requirements – Automation of delivery – Dynamic Discovery & Configuration – Baselining, Trending & Alerting• Evaluated 20+ vendors over 6 month periods – “AppDynamics gave us valuable performance data in less than one day. The closest competitor took over 2 weeks just to install their tools.”
    • 25. Our AppDynamics Controller• Deployed in March 2012• 4 Controllers – HA Mode – Pair for production – Pair for development and testing – 32 core, 256GB RAM, 4TB storage• Built to scale – 5,000 Servers – 40+ Application Services
    • 26. Results with AppDynamics• Found dozens of problems we’ve had for years• Increased performance & Scale by 10x – Without increasing resource• Reduced MTTD from days to minutes• Reduced MTTR from months to hours/mins
    • 27. Lessons Learned for SOA, Big Data and Agile• Keep architecture simple – Use small well-defined single purposed independent services• Speed of delivery is essential• Systems will eventually fail – Almost never in the way you expected• Adding more memory can solve most problems• Working with SOA, Big Data, and Agile is hard
    • 28. The Future for FamilySearch• Evaluate MongoDB and Redis• Data partioning – Reduce individual cluster sizes and support regional distribution of data• Hierarchical storage with algorithms – Predict which data will be needed next• Auto-scaling of compute resources• PaaS across multiple infrastructure providers
    • 29. Questions ?

    ×