Querying Data in Place with AWS Object Storage Features and Analytics Tools (STG378-R2) - AWS re:Invent 2018

•

0 likes•309 views

AWS offers tools and services that make analyzing and processing petabytes of data in the cloud faster, simpler, and more cost effective. In this chalk talk, AWS experts provide an overview of our querying data-in-place services, such as Amazon S3 Select, Amazon Glacier Select, Amazon Athena, and Amazon Redshift Spectrum. We explore best practices around using them with other analytics services (like Amazon EMR and AWS Glue) and third-party tools to build data lakes in Amazon S3 and Amazon Glacier and deploy other analytics solutions. Our AWS experts also provide sample use cases.

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Querying Data in Place with AWS Object
Storage Features and Analytics Tools
Damon Cortesi
Big Data Architect
AWS
S T G 3 7 8 - R
Tim Harris
Principal Engineer
AWS
Mert Hocanin
Big Data Architect
AWS
Ippokratis Pandis
Principal Engineer
AWS

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data lakes in Amazon Simple Storage Service
(Amazon S3)

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a data lake on Amazon S3
Search, access, and visualize
Protect and secure
Amazon Kinesis
Data Firehose
Amazon
CloudWatch
AWS CloudTrailAWS IoT
Amazon Redshift
Spectrum
Amazon
Athena
Amazon
EMR
Amazon
S3 Select
Analyze
AWS Glue
ETL and data
catalog
Ingest
Amazon S3

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Glue

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EMR

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift Spectrum

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Athena

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3 Select

Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Damon Cortesi
dcortesi@amazon.com
Tim Harris
tlh@amazon.com
Mert Hocanin
hocanint@amazon.com
Ippokratis Pandis
ippo@amazon.com

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Frequently asked questions
Amazon S3 Select and Amazon Athena both let me run serverless queries
over data at rest in S3 – What service should I use, and for what use
cases?

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Frequently asked questions
How should I name the objects I use in a data lake stored in Amazon
S3? I heard that I ought to include hashes at the start of the names – Is
that correct?

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Frequently asked questions
How should I optimize data for querying? What file formats should I use?
How should I split data across multiple files, and what should I aim for in
file size or number of files? Should I use Hive-style partitioning or
buckets?

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Frequently asked questions
I read that Amazon EMR lets me run part of my query on Amazon S3
Select. When should I enable that?

Amazon EMR helps you process all your data for analytics, but with great scale comes great responsibility—you need to make sure that data is secured by design. In this chalk talk, we walk through how to configure your environment to take full advantage of comprehensive security controls: including identifying sensitive data, encrypting data and managing keys, authenticating and authorizing users, utilizing fine-grained access controls, and using audit logs to demonstrate compliance.

Best Practices for Building Multi-Region, Active-Active Serverless Applicatio...

Amazon Web Services

[NEW LAUNCH!] Deep Dive on Amazon FSx for Windows File Server (STG322-R1) - A...

Amazon Web Services

In this session, we explore the new fully managed service that makes it easy to launch and use shared file storage for Windows applications. FSx for Windows provides fully managed file storage built on Microsoft Windows Server, providing native support for Windows file system features and for the Server Message Block (SMB) protocol. Learn about Amazon FSx for Windows File Server features and benefits, and details about its performance and security models. The target audience is storage administrators, application developers, and applications owners and infrastructure operations personnel who build or operate file-based Windows applications or NAS.

Building Your Geospatial Data Lake (WPS324) - AWS re:Invent 2018

Amazon Web Services

In this chalk talk, we discuss how to design a data lake, and how to permission different groups and applications to access and analyze datasets. Learn from subject-matter experts about a variety of AWS technologies for populating your data lake, monitoring new ingestion, and processing data for meaningful analysis. We examine considerations for structured data, such as relevant database engines with geospatial support, as well as considerations for unstructured data in the form of object storage. In addition, we address how to protect and secure data based on an organization’s needs.

Enterprise Data Protection with Veritas NetBackup on AWS (STG347) - AWS re:In...

Amazon Web Services

Veritas Technologies is an AWS Partner Network (APN) Advanced Technology Partner that addresses information management challenges, including backup and recovery, business continuity, software-defined storage, and information governance to simplify backup and improve efficiency. In this chalk talk, we do a deep dive on Veritas NetBackup, an industry-leading backup software that protects on-premises and Amazon EC2 workloads and provides support for Amazon S3 Standard-Infrequent Access and Amazon Glacier. We also hear from Bell Media, who is utilizing Veritas NetBackup to retain backups long-term on Amazon S3.

Customizing Data Lakes to Work for Your Enterprise with Sysco (STG340) - AWS ...

Amazon Web Services

Data lakes are helping enterprises of all sizes and industries make the most of their data. However, building a data lake requires consideration of your goals and an understanding of data lakes, including data ingestion, data consumption, and usability layers. In this chalk talk, AWS experts and representatives from Sysco, a Fortune 50 company and leader in food distribution and marketing, discuss parts of a data lake, design considerations, and the pros and cons of different architectural designs. They share guidance around data tracking, costs, user access, synchronization, and data integrity so that your data lake complies with governance requirements and works towards your data goals. Sysco representatives share their data lake experiences, best practices, and lessons learned. We highlight Amazon S3 and S3 Select, Amazon Athena, Amazon EMR, Amazon EC2, and Amazon Redshift Spectrum.

Amazon EMR was built for agility, enabling you to spin up and down resources for big data processing and analytics on demand, and realize the flexible potential of cloud. In this chalk talk, we discuss in detail how to efficiently start, stop, and resize your clusters for Apache Spark and Hadoop, reducing your costs, and accelerating your time to completion (TTC) for jobs. Join us to hear expert advice on how to optimize your "one-and-done" workloads.

Post-Production Media Delivery at Scale with AWS (STG391) - AWS re:Invent 2018

Amazon Web Services

Netflix is using AWS Snowball Edge to deliver post-production content to our asset management system, called Content Hub, in the AWS Cloud. Production companies have been historically using LTO tapes to move data around, and that has well-known complications. In order to accelerate and secure our media workflows Netflix has shifted to using Snowball Edge devices for data migration. Please join us to learn how Netflix is using the Snowball Edge service at scale.

Migrating Data to the Cloud: Exploring Your Options from AWS (STG205-R1) - AW...

Amazon Web Services

The document discusses various options for migrating data to the AWS cloud, including AWS Direct Connect for private connectivity, AWS DataSync for online data transfer, the AWS Snowball and Snowball Edge devices for offline data transfer of large volumes, AWS Storage Gateway for hybrid storage, AWS Transfer for SFTP, and Amazon S3 Transfer Acceleration. It provides overviews and use cases for each service and how they can help with migrating and managing data in hybrid cloud environments.

Implementing Multi-Region AWS IoT, ft. Analog Devices (IOT401) - AWS re:Inven...

Amazon Web Services

This document discusses implementing multi-region architectures for AWS IoT. It begins by explaining why a multi-region approach is important for IoT applications. It then covers foundational aspects like account structure, device bootstrapping and configuration, and building single region resiliency. The document also presents variations on multi-region architectures like active-passive and active-active models. Finally, it discusses specific examples and considerations for Analog Devices' machine health monitoring solution.

Got Files? We Got You Covered! Deploy Your File Workloads Quickly & Easily wi...

Amazon Web Services

AWS offers fully managed file system services that enable you to quickly and simply lift and shift or build new applications that access file data in the AWS Cloud. In this session, join Wayne Duso, the leader of file storage, hybrid-edge storage, and data transport services, to learn about our full set of file services and latest launches. Learn all about file storage, and get firsthand input on how you can accelerate your journey to AWS as you move from on-premises or do-it-yourself implementations to fully managed file storage solutions. Hear how AWS file storage solutions enabled LoanLogics to migrate its applications to the cloud, enabling better scalability, performance, and availability. Also discover why customers choose AWS file systems for migrating their mission-critical enterprise applications and compute-intensive workloads to deliver the performance they need in a cost-effective way, saving time and money.

Cost Optimization Tooling (ARC301) - AWS re:Invent 2018

Amazon Web Services

Introduction to GraphQL (MOB316-R1) - AWS re:Invent 2018

Amazon Web Services

Kickstart Your All-In Move to the AWS Cloud Using AWS Storage Gateway and Ama...

Amazon Web Services

When you move your data center to the AWS Cloud, you need an innovative and cost-effective approach to ensure a successful transition. In this chalk talk, we discuss how to efficiently and securely migrate all of your data center workloads to the AWS Cloud. Hear how one customer, LoanLogics, started with AWS Storage Gateway and Amazon EFS to solve an immediate and urgent expansion challenge and used that as a starting point for their journey to the AWS Cloud. Hear how they took advantage of the agility of the cloud to transition their storage and database workloads to Amazon Aurora and Amazon RDS. Learn tips to ease your own transition to the cloud. Bring your questions, prepare to discuss your implementation, and expect to have a deep discussion on how you can optimize your workflows.

Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...

Amazon Web Services

Amazon Relational Database Service (Amazon RDS) provides a managed service to run SQL Server databases in AWS. While Amazon RDS handles provisioning and maintaining the SQL Server instance, there are things you can do to ensure that the SQL Server instance is healthy. We'll review some best practices involved in configuring the Amazon RDS SQL Server instance, focusing on availability, security and migration. We'll also hear from our customer Allstate, sharing details about their use of Amazon RDS.

Save up to 90% on Big Data and Machine Learning Workloads with Spot Instances...

Amazon Web Services

The document discusses how to use Amazon EC2 Spot Instances to save up to 90% on big data and machine learning workloads. It explains that Spot Instances provide access to unused Amazon EC2 computing capacity at steep discounts compared to on-demand prices. It then provides guidance on how to best utilize Spot Instances for both training machine learning models and deploying trained models, including using checkpointing and Auto Scaling to handle potential interruptions of Spot Instance availability.

Reducing Branch Office Storage with AWS Storage Gateway (STG332) - AWS re:Inv...

Amazon Web Services

Many IT organizations must support distributed remote offices that have local storage needs far from central data centers. Providing cost effective, scalable storage and data protection for these branch locations can be a challenge for operations teams. In this chalk talk, an AWS customer explains how he has used AWS Storage Gateway cached and stored volumes and file shares for low-latency cloud-backed storage for on-premises applications, including local user file services. Learn from his experiences to understand your recovery and data migration options using volume clones and Amazon EBS Snapshot.

Resolving NLP Problems Using Amazon SageMaker Algorithms (GPSCT305) - AWS re:...

Amazon Web Services

Natural language or text is regularly voted as the most important data type after structured data. This is because text can carry an enormous amount of hidden insight. In this session, we show you how to get started with the natural language processing (NLP) techniques using Amazon SageMaker, a platform to easily build, train, and deploy models at scale. We review all of the NLP-related algorithms available on Amazon SageMaker, including Amazon algorithms (Latent Dirichlet Allocation (LDA), Neural Topic Model (NTM), and Sequence2Sequence (seq2seq), and BlazingText). We dive into the details, and we run through an example Jupyter Notebook that uses these algorithms.

Build Your Own Log Analytics Solutions on AWS (ANT323-R) - AWS re:Invent 2018

Amazon Web Services

With Amazon Elasticsearch Service's simplicity comes a multitude of opportunity to use it as a back end for real-time application and infrastructure monitoring. With this wealth of opportunities comes sprawl - developers in your organization are deploying Amazon Elasticsearch Service for many different workloads and many different purposes. Should you centralize into one Amazon Elasticsearch Service domain? What are the tradeoffs in scale and cost? How do you control access to the data and dashboards? How do you structure your indexes - single tenant or multi-tenant? In this session, we'll explore whether, when, and how to centralize logging across your organization to minimize cost and maximize value and learn how Autodesk has built a unified log analytics solution using Amazon Elasticsearch Service.

Architecting Digital Media Archive Migrations with AWS (STG357) - AWS re:Inve...

Amazon Web Services

Build a Searchable Media Library & Moderate Content at Scale Using Machine Le...

Amazon Web Services

Companies have to process, analyze, and extract meaning from ever-growing volumes of audio, image, and video data. Automating media workflows, such as image and video indexing or manual transcription for closed captions, can help you scale the growth of your media library and save time from manual, error-prone work. In this workshop, you learn how to automate workflows using the Media Analysis Solution, which includes Amazon Rekognition, Amazon Transcribe, and Amazon Comprehend. You learn how to extract metadata from media files and create a searchable library of metadata.

Build a High-Performance, Cloud-Native, Open-Source Platform on AWS & Save Mi...

Amazon Web Services

As a leader in agriculture technologies and services, Bayer is using technologies such as unmanned aerial vehicles (UAV), satellite imagery, and sensor data from multiple sources to generate real time insights. Over 300 data sources are ingested into their open source HPC geospatial platform to generate on average 100M API calls per day. The platform is used to provide real-time visualization and computational analysis to Bayer’s internal research community, partners, and is licensed to third-party applications to provide insights relevant to high-yield production of crops. In this session, Mendez-Costabel discusses how Bayer transitioned from on-premises packaged software architecture to open-source software and cloud services from AWS to build a modern, scalable, high-performance, open-source app on AWS. Learn about the open-source application architecture and AWS services used. Learn how the computing environment has changed the way that Bayer is performing R&D projects, and how the move to a modern architecture has enabled Bayer’s customers to gain insights that are transforming their businesses.

AWS reInvent 2018 recap edition

Amazon Web Services

AWS re:Invent è l’annuale conferenza globale di Amazon Web Services. Ogni anno presentiamo più di 1000 sessioni tecniche, workshops e hackathon che coprono argomenti chiave inerenti a AWS e che illustrano le tecnologie che AWS sviluppa e introduce. In questo webinar vedremo un riepilogo degli annunci e delle novità presentate a Las Vegas e diversi casi d’uso per i principali servizi introdotti.

Bridge the Storage Gap: Hybrid Media Workflows with AWS Storage Gateway (STG3...

Amazon Web Services

Media and entertainment companies are navigating their own digital transformation strategies, and they want to make their editors, artists, and VFX compositing teams comfortable with using the cloud for their production workflows. Companies such as Disney and Netflix are on track to spend over $20B in original content this year and they need ways to get their content in front of audiences as fast as possible. In this chalk talk, learn how AWS Storage Gateway can help studios and other media and entertainment firms bridge the storage gap between the cloud and their on-premises editorial environments.

Tape Is a Four Letter Word: Back Up to the Cloud in Under an Hour (STG201) - ...

Amazon Web Services

Tape backups. Yes, they're still a thing. If you want to stop using tapes but need to store immutable backups for compliance or operational reasons, attend this session to learn how to make an easy switch to a cloud-based virtual tape library (VTL). AWS Storage Gateway provides a seamless drop-in replacement for tape backups with its Tape Gateway. It works with the major backup software products, so you simply change the target for your backups, and they go to a VTL that stores virtual tapes on Amazon S3 and Amazon Glacier. Come see how it works.

PreReLa #1 (PRL1-R1) - AWS re:Invent 2018

Amazon Web Services

The document discusses Amazon FSx for Windows File Server, a fully managed native Windows file system for AWS. It provides concise summaries of key features: - FSx for Windows File Server allows users to lift and shift Windows file storage to AWS with fully managed Windows file servers that provide native Windows compatibility, fast performance, broad accessibility, and are enterprise-ready. - Key capabilities include high throughput and IOPS, integration with Active Directory and DFS, automated backups to S3, encryption at rest and in transit, and support for a wide range of use cases. - The service aims to simplify management of Windows file storage in AWS by automatically handling tasks like backups, software updates, and provisioning/man

Preparing Data for the Lake

Amazon Web Services

AWS Storage Leadership Session: What's New in Amazon S3, Amazon EFS, Amazon E...

Amazon Web Services

Mai-Lan Tomsen Bukovec, VP of Amazon S3, introduces the latest innovations across all AWS storage services. In this keynote address, we announce new storage capabilities, and we talk about features and services that make AWS storage unique. We focus on new innovations in object storage, file storage, block storage, and data transfer services. You also hear from executives from companies that are major AWS storage customers, Sony and Expedia, about how they're using AWS storage to create a competitive advantage in their businesses.

What's hot

Amazon EMR: Optimize Transient Clusters for Data Processing & ETL (ANT341) - ...

Amazon Web Services

Post-Production Media Delivery at Scale with AWS (STG391) - AWS re:Invent 2018

Amazon Web Services

Migrating Data to the Cloud: Exploring Your Options from AWS (STG205-R1) - AW...

Amazon Web Services

Implementing Multi-Region AWS IoT, ft. Analog Devices (IOT401) - AWS re:Inven...

Amazon Web Services

Got Files? We Got You Covered! Deploy Your File Workloads Quickly & Easily wi...

Amazon Web Services

Cost Optimization Tooling (ARC301) - AWS re:Invent 2018

Amazon Web Services

Introduction to GraphQL (MOB316-R1) - AWS re:Invent 2018

Amazon Web Services

Kickstart Your All-In Move to the AWS Cloud Using AWS Storage Gateway and Ama...

Amazon Web Services

Best Practices for Running SQL Server on Amazon RDS (DAT323) - AWS re:Invent ...

Amazon Web Services

Save up to 90% on Big Data and Machine Learning Workloads with Spot Instances...

Amazon Web Services

Reducing Branch Office Storage with AWS Storage Gateway (STG332) - AWS re:Inv...

Amazon Web Services

Resolving NLP Problems Using Amazon SageMaker Algorithms (GPSCT305) - AWS re:...

Amazon Web Services

Build Your Own Log Analytics Solutions on AWS (ANT323-R) - AWS re:Invent 2018

Amazon Web Services

Architecting Digital Media Archive Migrations with AWS (STG357) - AWS re:Inve...

Amazon Web Services

Build a Searchable Media Library & Moderate Content at Scale Using Machine Le...

Amazon Web Services

Build a High-Performance, Cloud-Native, Open-Source Platform on AWS & Save Mi...

Amazon Web Services

AWS reInvent 2018 recap edition

Amazon Web Services

Bridge the Storage Gap: Hybrid Media Workflows with AWS Storage Gateway (STG3...

Amazon Web Services

Tape Is a Four Letter Word: Back Up to the Cloud in Under an Hour (STG201) - ...

Amazon Web Services

PreReLa #1 (PRL1-R1) - AWS re:Invent 2018

Amazon Web Services

What's hot (20)