Tonight's Meetup will discuss testing big data in AWS. The presenter Michael Sexton will cover what big data is in AWS, types of testing including functional and performance testing, and challenges of testing large varied datasets in the cloud. Attendees will learn about services AWS provides for big data, how to test applications that handle data, and tools for automating testing including AWS services and Python scripts. There will be time for questions at the end.

1. Tonight’s Meetup
Michael Sexton
"Testing Big Data in AWS"
23rd September 2021

2. Travel Plan
What is Big Data in AWS
What is Testing Big Data in AWS Cloud
Types of Testing
Deployment Testing and Deployments
Tools & Automation
Challenges
Conclusion
Q & A

4. What is Big Data?
- IoT, Netflix,
Cybersecurity,
Social Media,
EuroVision,
Traffic on
GoogleMaps,
Machine
Learning, Video,
Communications,
etc.
https://bigdatapath.wordpress.com/

5. What Work is There?
Senior QA Engineer - Data Analytics Software
- Dublin - Selenium/Python - 65k - 75k
Senior SDET - Big Data/Analytics -
Dublin/Remote - Python, System Testing,
Performance Testing - 80k - 90k
Lead SDET - Big Data/Analytics -
Dublin/Remote - Python, System Testing,
Performance Testing - 90k - 100k
Senior QA Automation Engineer - Data
Management Software - Galway/Remote -
Python/Selenium 60k - 70k Reperio
Human Capital

6. Who Are the Cloud Providers?
- Various cloud providers: AWS, Azure, Google
Cloud, Alibaba, IBM, Dell, Tencent

7. What is Big Data in AWS?

8. Services Provided By AWS

9. What is Testing Big Data in AWS?
- Testing application that carries data works
well (no anomalies)
- Functional, Performance testing
- Testing of migration from on-prem to the
cloud
- Verifying resultant big data analysis is
correct

10. Functional Testing
- Testing with varied valid and invalid input
- Boundary cases, Calculations
- Scripts - Latin, Sanskrit, Arabic, encoded
- Testing against existing on-prem results
- Failure cases

11. Performance and SEcurity Testing
- Data ingestion (many different sources of
data e.g. v1 and v3)
- Data processing and throughput - soak
- Sub-component performance
- Security of pipeline and stored data
- Robustness (no data arrives - what then?)

12. Pythagorean Cup

13. Deployment Testing and DEployments
- Deployment Testing (upgrades, timings)
- Code merged to Master branch, deploy
scripts & documentation written & tested
- Are go-to person for devops during
deployment and upgrades
- Staging Environment, Production Environment
- Integration testing with other teams.

14. Tools & Automation
- AWS EC2 machine with linux/python scripts
- Pytest/Robotframework for automation &
regression testing
- AWS eco system (Athena, CloudWatch, X-Ray,
QuickSight)
- EXCEL (max=1048576 rows) & VLOOKUP
- Pyspark

15. Challenges
- Large varied dataset
- Automation and scripting skills
- Costs & AWS Knowledge
- Knowing the big picture
- Deployments to staging and production
- Communicating with other teams

16. Conclusion
- What is Big Data … in AWS?
- What is Big Data Testing in AWS
- Types of Testing (Functional & Performance)
- Deployment Testing & Deployments
- Tools & Automation
- Challenges

17. Any Questions?

18. Thanks
Ministry of Testing
Poppulo
Twitter: @MinistryCork