Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS를 활용한 Big Data 실전 배치 사례 :: 이한주 :: AWS Summit Seoul 2016

2,426 views

Published on

5월 17일 서울COEX에서 열린 AWS Summit Seoul 2016에서 Bespin Global 이한주님이 발표하신 "AWS를 활용한 Big Data 실전 배치 사례" 발표자료입니다.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

AWS를 활용한 Big Data 실전 배치 사례 :: 이한주 :: AWS Summit Seoul 2016

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 베스핀글로벌 이한주 대표 2016.5.17 AWS를 활용한 Big Data 실전 배치 사례 모니터링, IOT의 Big Data 배치 및 Data Visualization 사례
  2. 2. About HanJoo Lee CEO & Cofounder , BESPIN GLOBAL GeneralPartner, 1998 Cofounded IT Infra entrepreneur
  3. 3. Built and Operated 12 Datacenters 100,000+ 서버 운영 11 Countries with 12 Datacenters around the world: 1998년 시작 Australia Belgium Canada France Germany Korea Netherlands USA UK India Rumania Sydney Antwerp Vancouver Paris Frankfurt Seoul Amsterdam Chicago London Mumbai Bucharest Hannover Tampa / Austin Fort Lauderdale
  4. 4. About Bespin Global 베스핀 글로벌 Managed Service Provider 한국에서 최초로 AWS MSP 인증 받은 회사 한국에는 나와있는 3개 MSP 인증 중 2개가 Bespin • Cloud 전략 수립 • Cloud Architecture • Cloud Migration • Cloud 운영 • Hybrid IT Management • 한국과 중국에서 140명
  5. 5. About Bespin Global Cloud의 길잡이
  6. 6. Data lifecycle in Cloud Infra 클라우드 인프라 데이터 애플리케이션 데이터 입력 가공 분석 활용 • IoT를 통해 세상의 모든 것들을 디지털 데이터 로 수집 • 빅데이터 기술을 통해 과거의 텍스트 중심의 정형 데이터뿐만 아니 라 동화상, 음성 등 비 정형 데이터까지 처리 • 대규모 데이터를 실시 간 분석해 결과를 추출 하고 의사결정 진행 • 디지털 골든크로스로서 인간의 두뇌 이상의 의 사결정과 작업을 컴퓨 터가 스스로 수행 머신러닝 빅데이터 IoT
  7. 7. Bespin Global 김성수 상무 Building Big Data Backend Using Native AWS Services WhaTap 김성조 CTO Building Big Data System Using Proprietary File System while Using AWS N3N 김호민 대표 Building Big Data Visualization Using Splunk and AWS Big Data for SaaS based Monitoring Services
  8. 8. Bespin Global: How to build and Operate Big Data System on AWS?
  9. 9. Avoid unplanned Downtime Monitoring Process Server DBMS Application MonitoringTarget 분석 DB Basic Process Advanced Process (사후 처리) Metric Data 수집 및 모니터링 Big Data Processing ※ Ca사 모니터링 솔루션 UIM 적용 Threshold Alarm 사전 예방 활동 Scale up Scale out (Time Series, Correlation, …) 알람 or 장애 조치 Monitoring Dashboard
  10. 10. Building Big Data System on On-Premise R Analysis ( Time Series / Correlation ) Pre detection DatabaseDistributed Messaging System ….. M N1 N2 Nn Cluster Computing Framework Yarn ….. HDFS Time Series Database N2 Nn…..M N1 Analysis Platform …… Data Sources Application Server DBMS LogAgent … Applications BI Marketing Advertisement … Limited Scalability Internal IT Resources to Manage Cluster (Tuning and monitoring, etc…) Upfront capital expense NoSQL … 사전 진단/예방 Producer Stream Collector Consumer Stream
  11. 11. Building Big Data System on AWS Data Sources Application Server DBMS LogAgent … Producer Stream Kinesis Streaming Data Platform EMR AWS ElasticSearch Time Series Database RDS Pre detection KCL Kinesis S3 Connector Consumer S3 Lambda Archiving Data EMR Redshift RDS Analysis(BISolutions) Application Upsell Analysis Marketing Advertisement … Elastic and Highly Scalable Don’t Manage Cluster (and AWS has tuned Services) Easy to Use and Deploy to Multiple Locations No upfront capital expense – Pay as you go Cluster monitoring Cluster monitoring R Analysis ( Time Series / Correlation ) R on EMR Consumer 사전 진단/예방
  12. 12. On-Premise vs AWS R Analysis ( Time Series / Correlation ) Database Streaming Data Platform Kinesis Time Series Database Elastic Search Easy to Use and Managed Console RDS Database Complex to Use Managed Cost+ Cluster Managed Cost (Tuning and monitoring, etc…)+ Elastic Scalability Don’t Managed Cluster (and AWS has tuned Services) Upfront capital expense Pay as you go Distributed Messaging System ….. M N1 N2 Nn Cluster Computing Framework Yarn ….. Time Series Database N2 Nn…..M N1 Limited Scalability EMR R Analysis ( Time Series / Correlation ) R on EMR
  13. 13. WhaTap: Running Cloud Based Monitoring Service on AWS
  14. 14. Monitoring Issue Changes Testing Open Stable Application Physical SystemNeeds Unknown bottlenecks Known issues
  15. 15. Monitoring Solutions Needs Testing Open Stable APM (분석, 문제 해결, 실시간) SMS (관제, 경고 이벤트)
  16. 16. The differences on Cloud Dedicated Unix Infra 개별 서버 성능 서버 관리 유연성 서버(노드) 수 시스템 처리 성능 부분 오류 가능성 Cloud Infra 서비스 안정성
  17. 17. Monitoring Service(SaaS based) Master security group security 1 2 200 Slave Monitoring Service
  18. 18. APM on Premise vs on Cloud • Data collection per several seconds • Active Transaction Analysis • All Transaction Profiling On Premise Solution • Data collection per 10 seconds • Active Transaction Analysis • Selective Profiling & Integrative Analysis On Cloud Service
  19. 19. 사용자 접속 정보, 트랜잭션, 자원, 튜닝 정보를 하나의 관점으로 비교 분석할 수 있어야 합니다. Archiving Tech User (browser) Transaction IP, 도시/국가, 접속 매체, OS, 최근 방문자, 액티브 사용자 TPS, Response time, Error time, URL, SQL Resource CPU, Heap, Disk, GC Tuning History 비교분석 Active Stack 분석 Hit Map 분석 Integrated analysis of perf. data
  20. 20. EC2 instance Data Server EBS Backup (S3) Staticstics RDS (MySql) Scalable APM (AWS based) EC2 instance Data Server EBS EC2 instance Data Server EBS EC2 instance Data Server EBS Scalable!!
  21. 21. Elastic Load Balancing Amazon Route 53 Auto Scaling group EC2 instance Data Server EBS Backup (S3) Project/Tenant Mgmt RDS (MySql) Staticstics RDS (MySql) ElastiCache (REDIS) Elastic Load Balancing Amazon Route 53 Auto Scaling group Region (Tokyo) Elastic Load Balancing Auto Scaling group EC2 instance Data Server EBS Backup (S3) Staticstics RDS (MySql) Region (US.West) ElastiCache (REDIS) Elastic Load Balancing Auto Scaling group Region (Tokyo) Region (US.West) Elastic Search Scalable APM (AWS based)
  22. 22. APM to SMS on Cloud Needs Testing Open Stable APM • 분석, 문제 해결 • 실시간 • 대용량 데이터 SMS • 관제, 문제 인지 • 경고 이벤트 • 소규모 데이터
  23. 23. SUPPORTS MOBILE LOWER COST SHORTER IMPLEMENTATION PERIOD SMS on Cloud 1 Month EXISTING MONITORING CLOUD MONITORING 5 Minutes $10000 + α EXISTING MONITORING CLOUD MONITORING $100/mon EXISTING MONITORING CLOUD MONITORING In Office Anywhere
  24. 24. WhaTap TokyoWhaTap Virginia WhaTap Oregon 출처 : https://rctom.hbs.org/ WhaTap can service to global customers
  25. 25. Big Data Visualization on AWS
  26. 26. N3N: What We Do N3N provides IOT Visualization for Fortune 500 Companies and Cities Governments.
  27. 27. N3N - Physical Visualization Global Asia Korea Seoul Suwon Z: Hierarchies Daejeon X: Dependencies Y: Relationships
  28. 28. N3N - Logical Visualization Business Unit Service Application X: Dependencies Y: Relationships ApplicationData Bases Z: Hierarchies
  29. 29. N3N - Business Impact QuantitativeBenefits KPI Before After Result Remarks MTTD, MTTR 2 days 1.5 days -0.5 day CISCO Stat, Splunk .conf 2014 Big Data Solution Usage Rate 5% 100% +95% Improvements in Stability & availability +25% Reduction in Operational Cost +10% 25% 95% 25% 10%
  30. 30. Wrap Up
  31. 31. Visualization Big data for Biz Decision Monitoring Log IOT event Log System Event Log AWS event Log Visualized Analysis On AWS
  32. 32. Wrap Up - Big Data on AWS – Monitoring Use Case - Not only for Monitoring - Merchandise, Logistics, Turnover, Marketing, etc. - Best Practice & Consulting from Bespin Global
  33. 33. WE ARE HIRING!!!! • KOREA • CHINA • JAPAN • USA PLEASE JOIN US
  34. 34. 감사합니다! HELPING YOU ADOPT CLOUD.

×