News events led to dramatically increased traffic, causing the ACLU’s donation platform to go down under load, impacting revenue and supporter engagement at a critical time for the organization. Performance tuning under normal circumstances is difficult, but even more so while under extreme load and experiencing downtime with millions of dollars being lost by the hour.
The ACLU called on Tag1’s Technical Architecture and Leadership to perform emergency support and rescue work to get the ACLU Action website back online as quickly as possible and to help it withstand even bigger traffic spikes in the future. The results of Tag1’s efforts were 3,000% increase in donations from a yearly average of $4mm to $120mm, $24mm in donations on a single weekend, 57% faster database response times, 900% throughput increase in requests per minute. In addition, systems stay online and perform quickly under extreme loads.
The Wikimedia Foundation is a non-profit and charitable organization driven by a vision of a world where every human can freely share in the sum of all knowledge. Each month Wikimedia sites serve over 18 billion page views to 500 million unique visitors around the world.
Among the many resources offered by Wikimedia is a public-facing API that provides low-latency, programmatic access to full-history content and meta-data, in a variety of formats. Commonly, results from this system are the product of computationally intensive transformations, and must be pre-generated and persisted to meet latency expectations. Unsurprisingly, there are numerous challenges to providing low-latency storage of such a massive data-set, in a demanding, globally distributed environment.
This talk covers Wikimedia Content API, and it's use of Apache Cassandra, a massively-scalable distributed database, as storage for a diverse and growing set of use-cases. Trials, tribulations, and triumphs, of both a development and operational nature are discussed.
Short overview of data infrastructure at Bazaarvoice. We use a combination of many different data stores such as MySQL, SOLR, Infobright, MongoDB and Hadoop.
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk, offering over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python and R shells. Spark powers a stack of high-level tools including Spark SQL, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.
Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3.
Aram is a Senior Software Engineer at PicsArt's Analytics team. Aram has about 6 years of experience in software development. His interests are Scala, Big Data Engineering and playing guitar
The Wikimedia Foundation is a non-profit and charitable organization driven by a vision of a world where every human can freely share in the sum of all knowledge. Each month Wikimedia sites serve over 18 billion page views to 500 million unique visitors around the world.
Among the many resources offered by Wikimedia is a public-facing API that provides low-latency, programmatic access to full-history content and meta-data, in a variety of formats. Commonly, results from this system are the product of computationally intensive transformations, and must be pre-generated and persisted to meet latency expectations. Unsurprisingly, there are numerous challenges to providing low-latency storage of such a massive data-set, in a demanding, globally distributed environment.
This talk covers Wikimedia Content API, and it's use of Apache Cassandra, a massively-scalable distributed database, as storage for a diverse and growing set of use-cases. Trials, tribulations, and triumphs, of both a development and operational nature are discussed.
Short overview of data infrastructure at Bazaarvoice. We use a combination of many different data stores such as MySQL, SOLR, Infobright, MongoDB and Hadoop.
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk, offering over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python and R shells. Spark powers a stack of high-level tools including Spark SQL, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.
Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3.
Aram is a Senior Software Engineer at PicsArt's Analytics team. Aram has about 6 years of experience in software development. His interests are Scala, Big Data Engineering and playing guitar
Saratov open it teach talk.
Дамир Яраев:
Введение в Apache Cassandra (В ходе презентации Дамир расскажет, когда и почему стоит переходить с проверенных временем реляционных баз данных на ставшие модными в последнее время решения на базе NoSQL. В качестве примера рассмотрит колоночную NoSQL базу данных Apache Cassandra)
Drupal 7 will use RDFa markup in core, in this session I will:
-explain what the implications are of this and why this matters
-give a short introduction to the Semantic web, RDF, RDFa and SPARQL in human language
-give a short overview of the RDF modules that are available in contrib
-talk about some of the potential use cases of all these magical technologies
This is a talk from the Drupal track at Fosdem 2010.
How to build and run a big data platform in the 21st centuryAli Dasdan
This tutorial was presented in the IEEE Big Data Conference in 2019. It shows that building and running a big data platform for both real-time streaming and batch data processing for all kinds of applications involving analytics, data science, reporting, and the like in today’s world can be as easy as following a checklist. We live in a fortunate time that many of the components needed are already available in the open source or as a service from commercial vendors. This tutorial shows how to put these components together in multiple sophistication levels to cover the spectrum from a basic reporting need to a full fledged operation across geographically distributed regions with business continuity measures in place. This tutorial provides enough information and checklists to the audience that it can also serve as a goto reference in the actual process of building and running.
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu.
Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com..
Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.
Making Round the Clock work for your Database AdministrationDatavail
This session brings together three Oracle Database Administrators from a Fortune 500 company who team together to manage the databases 24x7 for a data-intensive company. The team will discuss the best practices, communications and processes they follow to have a truly seamless, “follow the sun” support of major databases, enhancing the uptime, availability and performance of the databases. They will share examples of how they are able to more effectively and efficiently support their company’s databases, using both planned and ad-hoc communications, knowledge sharing, escalation and education processes, and reporting tools.
This presentation was given in one of the DSATL Mettups in March 2018 in partnership with Southern Data Science Conference 2018 (www.southerndatascience.com)
Elasticsearch : petit déjeuner du 13 mars 2014ALTER WAY
Elasticsearch est un moteur de recherche Open Source très puissant basé sur
Apache Lucene. Il permet l'indexation de millions de données, leur recherche et leur
analyse en temps réel. Les outils Elascticsearch sont déjà utilisés par des acteurs de
référence tels que FourSquare, GitHub, OpenDataSoft ou encore Dailymotion.
Alter Way et Elasticsearch vous convient à venir découvrir la suite Elasticsearch
enfin disponible en version 1.0 et prête pour la production !
From determining the most convenient rider pickup points to predicting the fastest routes, Uber aims to use data-driven analytics to create seamless trip experiences. Within engineering, analytics inform decision-making processes across the board.
One of the distinct challenges for Uber is analyzing geospatial big data. City locations, trips, and event information, for instance, provide insights that can improve business decisions and better serve users. Geospatial data analysis is particularly challenging, especially in a big data scenario, such as computing how many rides start at a transit location, how many drivers are crossing state lines, and so on. For these analytical requests, we must achieve efficiency, usability, and scalability in order to meet user needs and business requirements.
To accomplish this, we use Hadoop, Hive, and Presto in our production environment to process the big data powering our interactive SQL engine. In this talk, we discuss our engineering effort to optimize geospatial queries in the whole Hadoop stack.
Speakers
Zhenxiao Luo, Engineering Manager, Uber
Lu Niu, Sr Software Engineer, Uber
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB
Moving to a new home is daunting. Packing up all your things, getting a vehicle to move it all, unpacking it, updating your mailing address, and making sure you did not leave anything behind. Well, the move to MongoDB Atlas is similar, but all the logistics are already figured out for you by MongoDB.
Many NoSQL DBaaS vendors limit what cloud platform you can run on, the size of the data you can run and require you to over-provision cloud infrastructure resources while failing to deliver performance and low latency at scale.
In this session, we will compare the performance and Total Cost of Ownership (TCO) of competing NoSQL DBaaS offerings. We will also review how to migrate to Scylla Cloud, our fully managed database service.
You will learn:
- The true cost of ownership for selected NoSQL DBaaS offerings
- The 8 essentials for selecting a NoSQL DBaaS
- Migration options from Apache Cassandra, DynamoDB and other databases
5 facets of cloud computing - Presentation to AGBCRaymond Gao
My presentation to AGBC (American German Business Club) on Cloud Computing and Social Causes. How doing non-profit work helps finding and validates Use Cases, the heart of any application, business venture, etc.
Saratov open it teach talk.
Дамир Яраев:
Введение в Apache Cassandra (В ходе презентации Дамир расскажет, когда и почему стоит переходить с проверенных временем реляционных баз данных на ставшие модными в последнее время решения на базе NoSQL. В качестве примера рассмотрит колоночную NoSQL базу данных Apache Cassandra)
Drupal 7 will use RDFa markup in core, in this session I will:
-explain what the implications are of this and why this matters
-give a short introduction to the Semantic web, RDF, RDFa and SPARQL in human language
-give a short overview of the RDF modules that are available in contrib
-talk about some of the potential use cases of all these magical technologies
This is a talk from the Drupal track at Fosdem 2010.
How to build and run a big data platform in the 21st centuryAli Dasdan
This tutorial was presented in the IEEE Big Data Conference in 2019. It shows that building and running a big data platform for both real-time streaming and batch data processing for all kinds of applications involving analytics, data science, reporting, and the like in today’s world can be as easy as following a checklist. We live in a fortunate time that many of the components needed are already available in the open source or as a service from commercial vendors. This tutorial shows how to put these components together in multiple sophistication levels to cover the spectrum from a basic reporting need to a full fledged operation across geographically distributed regions with business continuity measures in place. This tutorial provides enough information and checklists to the audience that it can also serve as a goto reference in the actual process of building and running.
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu.
Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com..
Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.
Making Round the Clock work for your Database AdministrationDatavail
This session brings together three Oracle Database Administrators from a Fortune 500 company who team together to manage the databases 24x7 for a data-intensive company. The team will discuss the best practices, communications and processes they follow to have a truly seamless, “follow the sun” support of major databases, enhancing the uptime, availability and performance of the databases. They will share examples of how they are able to more effectively and efficiently support their company’s databases, using both planned and ad-hoc communications, knowledge sharing, escalation and education processes, and reporting tools.
This presentation was given in one of the DSATL Mettups in March 2018 in partnership with Southern Data Science Conference 2018 (www.southerndatascience.com)
Elasticsearch : petit déjeuner du 13 mars 2014ALTER WAY
Elasticsearch est un moteur de recherche Open Source très puissant basé sur
Apache Lucene. Il permet l'indexation de millions de données, leur recherche et leur
analyse en temps réel. Les outils Elascticsearch sont déjà utilisés par des acteurs de
référence tels que FourSquare, GitHub, OpenDataSoft ou encore Dailymotion.
Alter Way et Elasticsearch vous convient à venir découvrir la suite Elasticsearch
enfin disponible en version 1.0 et prête pour la production !
From determining the most convenient rider pickup points to predicting the fastest routes, Uber aims to use data-driven analytics to create seamless trip experiences. Within engineering, analytics inform decision-making processes across the board.
One of the distinct challenges for Uber is analyzing geospatial big data. City locations, trips, and event information, for instance, provide insights that can improve business decisions and better serve users. Geospatial data analysis is particularly challenging, especially in a big data scenario, such as computing how many rides start at a transit location, how many drivers are crossing state lines, and so on. For these analytical requests, we must achieve efficiency, usability, and scalability in order to meet user needs and business requirements.
To accomplish this, we use Hadoop, Hive, and Presto in our production environment to process the big data powering our interactive SQL engine. In this talk, we discuss our engineering effort to optimize geospatial queries in the whole Hadoop stack.
Speakers
Zhenxiao Luo, Engineering Manager, Uber
Lu Niu, Sr Software Engineer, Uber
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB
Moving to a new home is daunting. Packing up all your things, getting a vehicle to move it all, unpacking it, updating your mailing address, and making sure you did not leave anything behind. Well, the move to MongoDB Atlas is similar, but all the logistics are already figured out for you by MongoDB.
Many NoSQL DBaaS vendors limit what cloud platform you can run on, the size of the data you can run and require you to over-provision cloud infrastructure resources while failing to deliver performance and low latency at scale.
In this session, we will compare the performance and Total Cost of Ownership (TCO) of competing NoSQL DBaaS offerings. We will also review how to migrate to Scylla Cloud, our fully managed database service.
You will learn:
- The true cost of ownership for selected NoSQL DBaaS offerings
- The 8 essentials for selecting a NoSQL DBaaS
- Migration options from Apache Cassandra, DynamoDB and other databases
5 facets of cloud computing - Presentation to AGBCRaymond Gao
My presentation to AGBC (American German Business Club) on Cloud Computing and Social Causes. How doing non-profit work helps finding and validates Use Cases, the heart of any application, business venture, etc.
Monolith to serverless service based architectures in the enterpriseSameh Deabes
Brief introductions about the following topics and how they relate to each other: SOA, ESB, Cloud, Cloud Native Architecture, Microservices, Multigrained Services, NoESB, API-First, Full Lifecycle API Management, etc.
The session will focus mainly on microservices pitfalls and what to do about it.
ContainerCon 2016: Finding (and Fixing!) Performance Anomalies in Large Scale...Victor Marmol
Borg provides a common runtime layer for Containers at Google. We try to guarantee a performance baseline for each class of tasks without looking into the task's runtime details or any metric from the application itself. This talk will cover the methodology we use to collect black-box performance monitoring information from Containers and presents case studies of interesting performance problems we detect and ways to mitigate them.
A Day in the Life of a Druid Implementor and Druid's RoadmapItai Yaffe
Benjamin Hopp (Solutions Architect) @ Imply:
Druid is an emerging standard in the data infrastructure world, designed for high-performance slice-and-dice analytics (“OLAP”-style) on large data sets.
This talk is for you if you’re interested in learning more about pushing Druid’s analytical performance to the limit.
Perhaps you’re already running Druid and are looking to speed up your deployment, or perhaps you aren’t familiar with Druid and are interested in learning the basics.
Some of the tips in this talk are Druid-specific, but many of them will apply to any operational analytics technology stack.
The most important contributor to a fast analytical setup is getting the data model right.
The talk will center around various choices you can make to prepare your data to get best possible query performance.
We’ll look at some general best practices to model your data before ingestion such as OLAP dimensional modeling (called “roll-up” in Druid), data partitioning, and tips for choosing column types and indexes.
We’ll also look at how more can be less: often, storing copies of your data partitioned, sorted, or aggregated in different ways can speed up queries by reducing the amount of computation needed.
We’ll also look at Druid-specific optimizations that take advantage of approximations; where you can trade accuracy for performance and reduced storage.
You’ll get introduced to Druid’s features for approximate counting, set operations, ranking, quantiles, and more.
And we will finish with the latest and greatest Druid news, including details about the latest roadmap and releases.
What is a data platform? Why do we need one? And how to build one in the cloud? This talk covers the essential engineering facets of a data platform: flows, persistence, access, standardization and data processing. How these facets combine into a unified platform and how and what cloud technologies as managed services and serverless help/challenge us to build it into a powerful business tool.
These are slides from a presentation from a "code naturally" meetup we held on 30/4 2018.
Similar to ACLU Partners with Tag1 to Raise Most-Ever $120M in Donations at Mission-Critical Moments (20)
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
ACLU Partners with Tag1 to Raise Most-Ever $120M in Donations at Mission-Critical Moments
1. ACLU.org in 2017
Patrick Jensen (ACLU), Narayan Newton (Tag1 Consulting), & Matthew Cheney (Pantheon)
Handling a Big Year
2. ACLU
● Nonprofit founded 1920 with over 3
million supporters
● Defend individual rights and liberties
● Famous cases
○ Led fight against Japanese-American
internment camps
○ 1996 Communications Decency Act
○ Marriage equality
image
3. ACLU Action Website
● Act
○ Sign petitions
○ Send messages
○ Request legal aid
● Support
○ Donate
○ Sign up to volunteer
● Accomplished via form submissions
● Drupal 6 (now Drupal 7)
image
4. Before Pantheon
Instability and Uncertainty
2013
● Database Strain
○ Using core Drupal search
● Hardware upgrades took weeks
● Maintenance was onerous
○ test and development environments
○ infrastructure (e.g. varnish)
5. Hosting Websites is Hard Work
image
● Need to Know Lots of Technology
○ Linux, LXC, NGINX, MariaDB, PHP,
Redis, Solr, Git, Varnish, New Relic
● Need to Do Lots of Things
○ Workflow, Branches, Backups,
Scalability, Performance, Security
● 24 hours a day, 7 days a week
7. Putting Organizational Mission at Top of Stack
There is already so much to do!
● The World is Already Full of Challenges
● Don’t be “ambitious” about a backup
system or your load balancers
● Leverage the Experience of Others
● Be the Pyramidion you want to be in
the world!
8. That Is Why Folks Like the ACLU Use Drupal
Stand on the Shoulders of Giants
● Leverage the Expertise of Others
○ Drupal Core
○ Contrib Modules
○ External Libraries
● Benefit from Community of Practice
○ Best Practices, Security Process,
Performance, Documentation
9. And Why Folks Use Managed Cloud Services
Free up Time & Resources to Focus
● Drupal is Getting More Complicated &
The Web is Getting More Ambitious
● Leverage Pre-Built Feature Sets
○ Redis (Object Caching), Solr (Search
Indexing, Dev->Test->Live (Workflow)
● Use Best In Class Security Processes +
Performance/Scalability Tooling
10. And Be Prepared. Now and in the Future.
Behold the Power of Containerization!
11. And Be Prepared. Now and in the Future.
Behold the Power of Containerization!
12. And Be Prepared. Now and in the Future.
Behold the Power of Containerization!
13. Be Prepared. You Never Know
What Is Going to Happen
Andrew Lowery
“ “
14. Donald Trump Elected
● Donations in the 5 days after election
■ 2012: $25,000
■ 2016: $7,200,000
● Page views Nov. 9 - 13
■ 2015: 400,000
■ 2016: 4,250,000
15. Nov 16, 2016: The wake-up call
Site outage
Formsubmissionsperminute
18. Outage Review
Tag1 Consulting brought in to review outage after
Rachel Maddow interview
Specifically --
● Fabian Franz (d.o.: fabianx)
● Narayan Newton (d.o.: nnewton)
● Jeremy Andrews (d.o.: Jeremy)
Overall issue was clear and was somewhat on-going.
Immediately transitioned into developing and
deploying fixes.
image
19. Example Query Fix
+------+-------------+-------+--------+----------------------+---------+---------+---------------------+--------++
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+----------------------+---------+---------+---------------------+--------++
| 1 | SIMPLE | fo | ALL | NULL | NULL | NULL | NULL | 282880 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | o | eq_ref | PRIMARY,order_status | PRIMARY | 4 | aclu.fo.oid | 1 | Using where |
| 1 | SIMPLE | os | eq_ref | PRIMARY | PRIMARY | 98 | aclu.o.order_status| 1 | |
+------+-------------+-------+--------+----------------------+---------+---------+---------------------+--------++
SELECT o.order_id, o.uid, o.billing_first_name,
o.billing_last_name, o.order_total, o.order_status, o.created, os.title
FROM uc_orders o INNER JOIN fundraiser_og fo ON fo.oid = o.order_id AND
fo.gid IN (8888,9999) LEFT JOIN uc_order_statuses os ON o.order_status =
os.order_status_id WHERE o.order_status IN ('refunded', 'pending', 'processing',
'payment_received', 'completed') ORDER BY o.order_id DESC LIMIT 0, 30;
20. Index Solution
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+----------------------+--------------+---------+---------------------+------++
| 1 | SIMPLE | o | range | PRIMARY,order_status | order_status | 98 | NULL | 76 | Using index condition; Using filesort |
| 1 | SIMPLE | os | eq_ref | PRIMARY | PRIMARY | 98 | aclu.o.order_status | 1 | |
| 1 | SIMPLE | fo | ref | test | test | 4 | aclu.o.order_id | 1 | Using where; Using index |
+------+-------------+-------+--------+----------------------+--------------+---------+---------------------+------++
+ db_add_primary_key($ret, 'fundraiser_og', array('oid', 'gid', 'nid'));
+ db_add_index($ret, 'fundraiser_og', 'idx_gid', array('gid'));
+ db_add_index($ret, 'fundraiser_og', 'idx_nid', array('nid'));
ALTER TABLE fundraiser_og ADD INDEX test (oid,gid,nid);
23. It Works on My Local (cluster)
Performance Testing For Complex Sites
● Performance Testing is Complicated
○ Varnish/CDN
○ Redis/APC
○ PHP, MariaDB
● Production Parity Testing!
● But Replicating a Cluster is Hard Work
○ Nobody has time for that!
24. Let the Robots Do the Work!
They already do so much. What’s a little more SysAdmin?
33. Payment Gateway
Toolkit
● curl_log
○ Adding verbose logging to the curl requests
○ Logging to a table in the DB
○ In-flight sanitization of user information
● curl_loadbalance
○ Decaying ticket-based curl endpoints load balancer
○ Removes failing endpoints for a window of time after X failures
○ Specifically designed to always have at least one endpoint
34. Performance Next Steps
● query_cache
○ Caching “shim” to adding db_query caching to
contrib modules without patching them
○ Ability to map queries to a single base query
○ Moves read-only traffic from the DB to the object
cache
● rate_limit
○ An in-drupal solution to rate limiting specific types of
requests
○ Webform protection
○ Search protection
41. Join us for
contribution sprints
Friday, April 13, 2018
9:00-12:00
Room: Stolz 2
Mentored
Core sprint
First time
sprinter workshop
General
sprint
#drupalsprint
9:00-12:00
Room: Stolz 2
9:00-12:00
Room: Stolz 2
42. What did you think?
Locate this session at the DrupalCon Nashville website:
http://nashville2018.drupal.org/schedule
Take the Survey!
https://www.surveymonkey.com/r/DrupalConNashville
Editor's Notes
Narayan Newton, Lead systems engineer at Tag1 Consulting
Matthew Cheney, Chaos Wizard at Pantheon
A non-profit founded almost 100 years ago and we have over 3 million supporters
Our mission is to defend and preserve the individual rights and liberties guaranteed by the Constitution and laws of the United States.
To put it succinctly, we consider ourselves to be the first responders for the Constitution
We take on issues like:
Voting rights
Reproductive Freedom
the intersection of privacy and technology
For example,
Led fight against Japanese-American internment camps during WWII
took on and defeated 1996 Communications Decency Act, which censored the Internet by banning "indecent" speech
Marriage equality - We brought the first lawsuit in the country seeking the freedom to marry for same-sex couples in 1970.
We appear before the Supreme Court more than any other organization except the Department of Justice.
We maintain about 40 Drupal websites at the ACLU
but today we’re going to talk about just one really important website: action.aclu.org
Take action online
Sign an online petition
Send a letter to an elected official
Request legal aid from the ACLU
Where our members can go to support us
Fundraising
Sign up to volunteer
action.aclu.org is currently on Drupal 7 but for the time period we’re talking about Drupal 6
Critical for our organization that our websites are available and performant.
Before Pantheon in like 2013 on dedicated hosting
There was an initiative at the ACLU to build our online presence
But we found our infrastructure wasn’t quite up to the task of handling the increased traffic
site slowness
some site outages
Problems with our infra
Database strain (using core drupal search bc Solr wasn’t set up)
hardware upgrades took weeks and weeks
maintaining test and development environments and varnish involved a lot of developer time
ACLU CTO, Marco Carbone who is an old-school drupal dev heard about Pantheon by attending DrupalCon event
We did our research and decided they’d be a great host for us. Matt’s going to tell you why.
-- This may not be surprising, but hosting websites is hard work.
-- Not as hard as hard as resisting executive overreach through constitutional law of course.
-- Need to Use Lots of Technologies and Do Lots of Things 365 days a year
-- Plus you need to keep it all up to date and adopt NEW stuff when it come
-- What does knowing Git have to do with civil rights?
-- Its about as necessary as this guinea pig wearing sunglasses.
-- I mean its great to know how Git works, but its not necessary
-- The world is full of challenges, why add to things you need to do!
-- Things move quickly. Organizations need to be able to respond.
-- Time/Resources need to be focused on organiational goals.
-- Even more true with “Ambitious Digital Experiences”.
-- Be the Pyramidion you want to be in the world
-- Leverage the expertise of others through reusable modules/libraries
-- Benefit from a community of practice around web development
-- Leveraging the expertise of others is why people use CLOUD
-- Drupal is getting more complciarfed. Web is getting more ambitious.
-- Features You Need Require Spercialized Knowledge to Make. Even More to Maintain.
-- Security is Ongoing Challenge Requiring Lots of Knowledable People
-- Performance/Scalability Takesa a Village
-- Horizontally Scaling PHP is Hard Work
-- Hosting Platforms That Have This Tech Work Really Hard To Make it Awesome
-- It Wont Solve All Your Performance Problems
-- But It will Provide you a SOLID Starting Point
-- Be preapred, you never know what is going to happen
-- Its Not About Having all The Answers, It’s About Having the Right Tools
Pat:
After switching to Pantheon, our site was quite stable… until Nov. 8th 2016
After switching to Pantheon, our site was quite stable… until Nov. 8th 2016
We received $7.2 million in the 5 days after the 2016 election.
Compare that to the $25k in donations we received in the 5 days after the 2012 election
In the 5 days after the election our websites saw over 4 million page view
Compare that to 400,000 page views the year before
Our web traffic increased to more than 10x what we were used to seeing in the days after the election, essentially overnight
This was a great outpouring of support for our organizaion
but we started seeing small performance issues
Those small performance issues turned into a really big performance issue on Nov 16, 2016
The ACLU’s executive director appeared on the Rachel Maddow show.
Rachel Maddow Appearance Nov. 16 2016
500 peak form submissions per minute
~15 minutes site outage
Only able to sustain ~300 submissions per minute
This graph shows the spike in HTTP 500 errors our site was returning during the Maddow appearance
Huge missed opportunity for us.
Supporters were trying to donate to us, send letters to their elected officials via our site and sign up for our email lists, but they were being met by errors
Luckily ACLU mgt realized this wouldn’t be a one-time spike
They realized the Trump era meant that we’d be seeing spikes like this on the the regular for the next 4 - 8 years
But we didn’t have 4 - 8 years to fix these performance issues
The next spike could come at ANY time
so we called in Tag1 to do emergency weekend
Tag1 Brought in to look at outage period
Issue was clearly that we were DB bound, brought in 3 engineers including myself to review new relic traces
Developed indexes, fixed queries, worked in concert with the ACLU team to deploy fixes.
An example of what type of thing we were doing.
This is a fairly typical ubercart-esqe query, with the addition of an og table.
An interesting quirk of this additional table is that it lacks all indexes. This is more common than you might think.
Looked at the table to find the datasets natural key and pushed a primary key and some additional keys for filtering and joining.
Note, we have a key on oid, gid, nid but then I have indexes on specifically gid and nid. Why? Because of the order of gid and nid in the primary index
As you can see, we went from 200k rows to 76.
And here is the result of just that change. You can see the green query being marked fundraiser_og, that is this query and you can see it basically dropping out of the graph.
Put together our fixes as a patchset, tested against multidev
at this point wanted to ensure that the ACLU site would survive larger traffic spikes and find other issues
Turned to pantheon to setup a production-alike environment to enable testing at that capacity
-- performasnce testing is complicated. just ask narayan.
-- important to test in as “close a production parity” as possible
-- but setting all this stuff up is hard!
-- robots will drive our cars. raise our children on ipads. tell us what to believe politically
-- is it really too much to ask that they can create production parity developkment environments on demand?
-- on demand environments are the answer
-- at pantheon we call this “Multidev”, but its basically ONE ENVIRONMENT PER GIT BRANCH
---- integrated with new relic, production parity
-- made possible by Containers and Robots
-- Allowed Tag1 and ACLU to quickly iterate and test features
The emergency improvements Tag1 put in over that weekend in late Nov 2016 were very effective.
Made it through:
Giving Tuesday 2016
end of year fundraising pushes
received 15x more donations in our end of year fundraising than previous year (20,548 gifts)
But we weren’t out of the woods yet
Jan 27 2017, issue Executive Order 13769 (AKA Muslim travel ban)
Barred people from 7 Muslim-majority countries from entering the US
Thousands protested the executive order at airports across the country
The ACLU fulfilled our reputation as first responders for the Constitution
Within hours, the ACLU—and partnering organizations nationwide—obtained the first injunction to block the order
When news broke of what the ACLU had accomplished
People rushed to our websites
That top line on the graph there shows page views in the page views before during and after the executive order
The line at the bottom of the screen shows the same dates from the previous year
The big spike is at almost 4 million hits, on the same day the previous year is at 44,000
85x traffic spike… almost 2 orders of magnitude
Donations over the weekend after the executive order were six times the organization’s yearly average
So how did our websites hold up during this crazy post-executive order weekend?
Rachel Maddow Appearance
Able to sustain 300 submissions per minute
~15 minutes site outage
Executive Order
900 peak form submissions per minute
Sustained 500 submissions per minute for ~8 hours
We did have a 10 minute ‘site outage’
We did 2 smart things to mitigate this outage
New Relic alerts when traffic got high or response times increased
Static CDN-hosted donation page
After the dust settled, we took some time and confirmed what we previously suspected
slow responses from one of our payment gateways was the root cause of the site outage
we still had some issues with database performance to address
Once again, we handed the reins over to Tag1
So at this point we know things are better, but that we are still having issues at very high load. We are past the easy fixes you can detect at low load situations and need actual traffic
I build a botnet
initial results of the patchset
starting seeing issues with DB and external request/curl requests to the payment gateway
We took a two pronged approach to fix these issues
First, we starting look at the external requests. It was very unclear what was actually happening with our payment gateways
Developed curl_log to log the actual responses from the gateway, but also to sanitize
Finally found that there was an issue with CDN
curl_load balance was developed
Turned towards DB issues, which were more over-all load and less bad queries specifically
Legacy deployment, don’t want to patch every module. Fabian developed query_cache
We also developed rate_limit, which is sort of a performance tool and sort of a security tool. It allows us to rate_limit specific actions in Drupal itself.
How well did this second and final round of changes serve us?
We got a chance to find out when the Trump administration’s FCC repealed 'Net Neutrality' rules for internet providers in mid-December 2017
The internet reacted with outrage and once again the action.aclu.org website was a conduit for that outrage
This time, we nailed it.
Nov 2016 Before changes
Rachel Maddow Appearance
Maxed out at 300 form submissions per minute
~15 minute outage
January 2017 After first round of changes
Executive Order
900 peak form submissions per minute
~10 minute mitigated outage
After the second round of changes
we were able to hit a peak of 1,900 form submissions per minute
and easily sustained 500 submissions per minute for 10 hours (probably indefinitely)
This was a big victory for us… 100s of thousands
In his Nov 2016 appearance on the Maddow show our exec dir said about Pres. Trump’s election
While the rest of the organization was ready, the website wasn’t quite prepared.
But after a year of:
work by the developers and management at the ACLU
leveraging Tag1’s expertise
and having Pantheon’s infrastructure having our backs
We’re now confident that our websites are really ready to be used in their full capacity to defend civil liberties in the US