Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS 기반 데이터 레이크(Datalake) 구축 및 분석 - 김민성 (AWS 솔루션즈아키텍트) : 8월 온라인 세미나

4,455 views

Published on

..

Published in: Technology
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ..................................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ..................................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ..................................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

AWS 기반 데이터 레이크(Datalake) 구축 및 분석 - 김민성 (AWS 솔루션즈아키텍트) : 8월 온라인 세미나

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 김민성, 솔루션스 아키텍트 2017년 8월 29일 AWS 기반 데이터 레이크(Datalake) 구축 및 분석
  2. 2. 강연 중 질문하는 법 자신이 질문한 내역이 표시되며, 전체 공개로 답변된 내용은 검은색, 질문자 본인에게만 공개로 답변된 내용은 붉은 색으로 돌아옵니다.
  3. 3. 본 세션의 주요 주제 • 빅데이터 • Data Lake의 정의 및 조건 • AWS에서의 Data Lake
  4. 4. 왜? 빅데이터인가?
  5. 5. 빅 데이터의 도전과 과제 Source: Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012. Base: 634 business intelligence users and planners Unstructured 50TB Semi-structured 2 TB Structured 12 TB 12% 기업별 평균 데이터 볼륨의 크기 9 TB 75 TB 0.6 TB 5 TB 4 TB 50 TB 중소기업 대기업
  6. 6. 빅데이터의 종류 통찰 그리고 변화의 기회
  7. 7. 빅데이터 분석 도구 Amazon Kinesis Amazon Glacier S3 DynamoDB RDS EMR Amazon Redshift Data Pipeline Amazon Kinesis Streams app Lambda Amazon ML SQS ElastiCache DynamoDB Streams Amazon Kinesis Analytics Amazon Elasticsearch Service
  8. 8. 데이터 아키텍처 Hadoop Cluster SQL Database Data Warehouse Appliance
  9. 9. 모놀릭 기반의 빅데이터 분석 아키텍처 CPU Memory HDFS Storage CPU Memory HDFS Storage CPU Memory HDFS Storage Hadoop Master Node Multiple layers of functionality all on a single cluster
  10. 10. 어떻게 극복할 것인가? (S3 데이터 레이크 기반의 EMR클러스터)
  11. 11. EMR EMR ClusterS3 1. 데이터를 S3에 저장 2. 하둡 클러스터, 노드 수, 노드 type, Hive/Pig/Hbase와 같은 Hadoop 툴 선택 4. S3에서 추출 3. EMR 콘솔, CLI, SDK, 혹은 API를 통해 클러스터 시작
  12. 12. EMR EMR Cluster S3 쉽게 클러스터의 크기 조정 같은 데이터를 사용하는 다른 클러스터 생성
  13. 13. 클러스터 스케일 조정 인스턴스 용량 설정 YARN 리소스 사용기반 설정 자동확장/축소선택
  14. 14. EMR EMR Cluster S3 스팟 노드를 사용하여 비용 절감
  15. 15. EMR Instance Fleets 활용 최대 5개까지 서로 다른 타입 선택 Spot인스턴스 중단시 온디맨드로 자동 전환
  16. 16. EMR ClusterS3 모든 작업이 끝나면 클러스터 종료 (따라서 과금 중단!)
  17. 17. 메타데이터 관리 [ { "Classification": "hive-site", "Properties": { "javax.jdo.option.ConnectionURL":"jdbc:mysql://RDS- endpoint:3306/hive?createDatabaseIfNotExist=true", "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver", "javax.jdo.option.ConnectionUserName": "username", "javax.jdo.option.ConnectionPassword": "password" } } ] aws emr create-cluster --release-label emr-5.4.0 --instance-type m3.xlarge --instance-count 2 --applications Name=Hive -- configurations hivemetadata.json --use-default-roles
  18. 18. 부트스트랩 활용 S3를 스크립트 저장소로 활용
  19. 19. 사례 연구: 삼성 빅데이터 분석 S3 : 원천 데이터의 적제 및 분석 시스템간의 BUS 역할 제공
  20. 20. Unlocking Data (Data Lake)
  21. 21. Data Lake – Unlocking Data 대부분의 회사와 조직은 데이터 잠금 해제를 위해 혁신 이니셔티브에 착수함 데이터가 이미 있지만 사용되지 않거나 격리 된 데이터가 사용되지 않고 잠겨있음.
  22. 22. 22 Data Lake의 특징과 장점 Store and analyze all of your data, from all of your sources, in one centralized location. “Why is the data distributed in many locations? Where is the single source of truth ?” 1. 모든 데이터를 한곳에
  23. 23. 23 Data Lake의 특징과 장점 Quickly ingest data without needing to force it into a pre-defined schema. “How can I collect data quickly from various sources and store it efficiently?” 2. 신속한 데이터 추출 및 저장
  24. 24. 24 Data Lake의 특징과 장점 Separating your storage and compute allows you to scale each component as required “How can I scale up with the volume of data being generated?” 3. 데이터 저장과 처리를 분리
  25. 25. 25 Data Lake의 특징과 장점 4. 구조화 없이 분석 처리 (Schema on Read) “Is there a way I can apply multiple analytics and processing frameworks to the same data?” A Data Lake enables ad-hoc analysis by applying schemas on read, not write.
  26. 26. AWS Approach to Data Lake
  27. 27. Data Lake 로써의 S3
  28. 28. S3 Data Lake Fixed Cluster Data Lake AWS S3 Data Lake  클러스터에 포함 된 단일 도구 (예 : Hadoop 또는 데이터웨어 하우스 또는 Cassandra 등)로만 제한되고, 유스 케이스 및 생태계 도구가 빠르게 변함  스토리지 용량을 추가하기 위해 노드를 추가하는 데 비용이 많이 증가  노드 손실에 대한 데이터를 복제하는 데 고비용 구조  로컬 스토리지 용량 확장의 복잡성  추가 저장 장치를 추가하고 적용하는 많은 데이터 이행 기간 필요  고정된 클러스터가 아닌 다양한 데이터 객체를 지원하는 S3 저장소를 기반으로 컴퓨팅 처리 자원과 분리  데이터 관련 모든 생태계의 도구를 사용할 수있는 유연성과 적합성을 제공함  미래 지향적으로 검증된 아키텍처로 새로운 활요 사례나 새로운 도구를 간편하게 지원  현재의 최상의 제품을 플러그 앤 플레이(Plug and Play)로 활용
  29. 29. Data Lake로써의 S3 Designed for 11 9s of durability Designed for 99.99% availability Durable Available High performance  Multiple upload  Range GET  Store as much as you need  Scale storage and compute independently  No minimum usage commitments Scalable  Amazon EMR  Amazon Redshift  Amazon DynamoDB Integrated  Simple REST API  AWS SDKs  Event notification  Lifecycle Management Easy to use
  30. 30. 만약 빅데이터 분석을 처음 시작하신다면…
  31. 31. Data Lake Amazon EMR Web Mobile Application LOG 데이터 Logstash Crewing Amazon Kinesis 실시간 분석 데이터 변환 원천 데이터 수집 실시간 예측 Amazon ML Amazon EMR Amazon Elasticache Amazon DDB Amazon Elasticsearch Amazon ML Amazon Athena 다양한 목적에 따른 분석 도구 “수많은 원천데이터를 실시간으로 수집 변환 하고” “실시간으로 분석 하고” “실시간으로 예측하며” “분석에 목적에 맞춰 다양한 도구를 기반으로 분석 역량의 확장”
  32. 32. 사례: 실 시간 로그 데이터 행동 분석
  33. 33. S3 Data Lake의 구축 Amazon Redshift Amazon Elastic MapReduce Data Warehouse Semi-structured Amazon GlacierAmazon Simple Storage Service Data Storage Archive Amazon DynamoDB Amazon Machine Learning Amazon Kinesis NoSQL Predictive Models Other AppsStreaming Internet S3 Endpoint
  34. 34. S3의 혁신
  35. 35. Data Lake 로써의 S3
  36. 36. 감사합니다.
  37. 37. 질문에 대한 답변 드립니다. 발표자료/녹화영상 제공합니다. http://bit.ly/awskr-webinar 더 나은 세미나를 위해 여러분의 의견을 남겨 주세요!

×