Download to read offline
Uploaded bySubramanyam Vemala
Aws centralized logs
The document discusses various approaches for centralized logging, data analytics, visualization, and ETL services using AWS tools. It covers several AWS services, including Amazon Elasticsearch, AWS Glue, AWS Data Pipeline, Amazon EMR, and Amazon Kinesis, highlighting their functionalities and architectures for handling data processing and analytics. Each approach is tailored to specific use cases, emphasizing the need to select appropriate tools based on organizational requirements.
More Related Content
Aws centralized logs
- 1. Centralized Logs/ Data,Data Analytics, Visualization and ETL (Extract, Transform and Load) services. Mr. Subramanyam Tirumani Vemala subramanyam.vemala@capgemini.com
- 2. Centralized Logs, DataAnalytics and Visualization: In this PPT, have discussed with multiple approaches for processing 1. Centralized Logs, 2. Data Analytics, 3. Visualization and 4. ETL (Extract, Transform and Load) We can decide on the type of approach, that we need to implement is based the exact requirements.
- 3. Approach-1: Elasticsearch andKibana: (Third party tools)
- 4. Approach-2: Amazon Elasticsearch+ Inbuilt Kibana (AWS services)
- 5. Amazon Elasticsearch –A sample Architecture:
- 6. Approach-3: AWS Glue(An AWS ETL service) • AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. • You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. • Once cataloged, your data is immediately searchable, queryable, and available for ETL.
- 7. AWS Glue –an AWS ETL service. (Extract, Transforn and Load) – A sample Architecture:
- 8. AWS Glue –Anothersample Architecture:
- 9. Approach-4: AWS DataPipeline – ETL (Extract, Transform and Load) • AWS Data Pipeline helps you move, integrate, and process data across AWS compute and storage resources, as well as your on-premises resources. • AWS Data Pipeline supports integration of data and activities across multiple AWS regions. • With AWS Data Pipeline, you can regularly access your data where it’s stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR.
- 10. AWS Data Pipeline- A sample Architecture:
- 11. Approach-5: AWS LakeFormation • AWS Lake Formation is a service that makes it easy to set up a secure data lake in days. A data lake is a centralized, curated, and secured repository that stores all your data, both in its original form and prepared for analysis. • A data lake enables you to break down data silos and combine different types of analytics to gain insights and guide better business decisions.
- 12. AWS Data Lake– A sample Architecture:
- 13. Lake Formation andData Lake storage - A sample Architecture:
- 14. Lake Formation –Security:
- 15. Approach-6: Amazon EMR– Elastic MapReduce - Managed Hadoop Framework. (Big Data) • Amazon Elastic MapReduce (Amazon EMR) is a web service that enables businesses, researchers, data analysts, and developers to easily and cost- effectively process vast amounts of data. • Amazon EMR is the industry-leading cloud big data platform for processing vast amounts of data using open source tools such as Apache Spark, Apache Hive, Apache HBase, Apache Flink, Apache Hudi, and Presto. • With EMR you can run Petabyte-scale analysis at less than half of the cost of traditional on-premises solutions and over 3x faster than standard Apache Spark. • For short-running jobs, you can spin up and spin down clusters and pay per second for the instances used. For long-running workloads, you can create highly available clusters that automatically scale to meet demand.
- 16. How the EMR,Elastic MapReduce Works?
- 17. Approach-7: Amazon Kinesis- Easily collect, process, and analyze video and data streams in real time. • Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. • Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. • With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. • Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.
- 18. Amazon Kinesis –Data Analytics:
- 19. Amazon Kinesis –Data Streams:
- 20. Amazon Kinesis –Video Streams:
- 21. Amazon Kinesis –Data Firehose:
- 22. Some slides formore information:
- 23. AWS Analytics services:We need to choose the exact AWS service depends on the use cases.
- 24. AWS Data Lake:Usage of different data storage services depends on the type of input Data.
- 25. AWS Data movement:This can be used if live data streaming is needed – May be MSK or Amazon Kinesis.
- 26. Visualization Tools: Weneed to opt the right one based on the approaches that we have chosen above. 1. AWS QuickSight – From Amazon. 2. Kibana - Kibana is a popular open source visualization tool designed to work with Elasticsearch.
- 27. AWS Analytics Services:The type of service will be chosen based the exact need. 1. Amazon ElasticSearch service 2. Athena 3. RedShift 4. EMR 5. Kinesis 6. ElasticSearch (https://www.elastic.co/)
- 28.
- 29.