1. The document discusses various data analysis, storage, and processing solutions including data analysis, data analytics, data lakes, data warehouses, data marts, batch processing, and stream processing. 2. It describes challenges of data analytics including volume, velocity, and variety and recommends solutions like Amazon S3, Redshift, EMR, and Kinesis to address these challenges at scale. 3. The key aspects covered are data storage methods like data lakes, data warehouses, and data marts and data processing methods including batch processing using EMR and stream processing using Kinesis.

1. Introduction To
Data Analysis, Storage, & Processing Solutions

2. Data analysis and data analytics solutions
Generally, Analysis is a examination of something in order to understand
its nature or determine its essential features.
Specifically, Data Analysis is a process of compiling, processing, and
analyzing data so that you can use it to make proper decisions.
Analytics is systematic analysis of data.
Data analytics is the specific analytical process being applied for analysis.

3. Why use data analytics? 🤔
It’s simple,to stop making business or any such decisions not just based on
intuitions but based on data. Other specific use cases are:
● Customer Personalization
● Fraud Detection
● Security Threat Detection
● User Behaviour
● Financial modeling and forecasting
many more...

4. Components of data analytics solution

5. Steps of a data analysis solution
1. Get the data [Collect, Store]: Know where your data comes
from.
2. Discover and analyze your data[Analyze/Process]: Know the
options for processing your data
3. Visualize and learn from your data[Consume/Visualize]: Know
what you need to learn from data

6. Challenges
of data
analytics
😑

7. Knowledge check
Scenario
My business has a set of 15 JSON data files that are
each about 2.5 GB in size. They are placed on a file
server once an hour. They must be ingested as soon
as they arrive in this location. This data must be
combined with all transactions from the financial
dashboard for this same period, then compared to the
recommendations from the marketing engine. All data
is fully cleansed. The results from this time period
must be made available to decision makers by 10
minutes after the hour in the form of financial
dashboards.
Based on the scenario, which of
the following Vs pose a
challenge for this business?
● Volume
● Velocity
● Variety
● Veracity
● Value

8. Volume - data storage
When businesses have more data than they are able to process and analyze,
they have a volume problem.
Classification of data source types:
● Structured data
● Semistructured data
● Unstructured data

9. Unstructured data is every file we store, every picture we take, and email we send.

10. Introduction to
Amazon S3
Amazon S3 is object
storage built to store and
retrieve any amount of
data from anywhere.
It is the perfect place to
store your semi-structured
and unstructured data in
the internet.

11. Amazon S3 concepts
How does S3 store your data?
- Amazon S3 stores data as objects within buckets.
How to access your content?
- Through object key

12. Data Analysis solution on Amazon S3
● Decoupling of storage from compute and data processing
● Centralized data architecture
● Integration with clusterless and serverless AWS services
● Standardized Application Programming Interfaces (APIs)

13. Knowledge Check
Which of the following elements does an Amazon S3 object URL contain?
● Object key
● Bucket
● User key
● Access token

14. Introduction to
data lakes
● A data lake is a
centralized
repository that
allows you to store
structured,
semistructured, and
unstructured data at
any scale.

15. Benefits of data lakes
● Single source of Truth
● Store any type of data, regardless of structure
● Can be analyzed using Artificial Intelligence and Machine
Learning

16. Introduction to data storage methods
- Data Lakes
- Data Warehouse

17. Data WareHouse
A data warehouse is a central repository of structured data from many data
sources. This data is transformed, aggregated, and prepared for business
reporting and analysis.

18. Data Marts

19. Traditional data warehousing: pros and cons
Pros Cons
Fast data retrieval Costly to implement
Curated data sets Maintenance can be challenging
Centralized storage Security concerns
Better business intelligence Hard to scale to meet demand

20. Amazon Redshift
It is a cloud-based, scalable, secure environment for your data warehouse
Benefits of Amazon Redshift
Faster performance
10x faster than other data warehouses
Easy to set up, deploy, and manage
Secure
Scales quickly to meet your needs

21. Data storage on Big Scale
We have discussed several recommendations for storing data:
● When storing individual objects or files, AWS recommends Amazon S3.
● When storing massive volumes of data, both semistructured and
unstructured, AWS recommends building a data lake on Amazon S3.
● When storing massive amounts of structured data for complex analysis,
AWS recommends storing your data in Amazon Redshift.

22. Apache Hadoop
Hadoop uses a distributed processing architecture, in which a task is mapped to a cluster of
commodity servers for processing.

23. Velocity- data processing
When businesses need rapid insights from the data they are collecting, but the systems
in place simply cannot meet the need, there's a velocity problem.
Data processing means the collection and manipulation of data to produce meaningful
information. Data processing is divided into two parts:

24. Introduction to data processing methods

25. Introduction to batch data processing
Batch processing is the execution of a series of programs, or jobs, on one or
more computers without manual intervention.

26. Batch processing architecture
● AWS EMR - It is used for processing vast amounts of data. It does ETL
operations.
● AWS Glue - It is used for processing vast amounts of data. It helps us with
data discovery, conversion, mapping and job scheduling.
● AWS Lambda - It is a serverless compute service that runs your code in
response to events and automatically manages the underlying compute
resources for you.

27. Batch processing architecture
Basic Batch processing architecture using Amazon EMR Basic Batch processing architecture using AWS Glue

28. Introduction to stream data processing
Stream processing is the collection and processing of a constant stream of
data.

29. Benefits of stream processing

30. Stream processing architecture
1. Amazon Kinesis - It makes it easy to collect, process, and analyze real-
time, streaming data so you can get timely insights and react quickly to
new information.
2. Amazon Athena - It is used for querying data directly in Amazon S3
3. Amazon Quicksight - It is used to produce insightful dashboards and
reports

31. How does Stream processing takes place?

32. Stream processing architecture

33. Combined processing architecture

34. Thank you!!!