韓国オンラインゲームから学ぶアドホックなビックデータ分析
Data Analytics in Gaming
with Azure Data Explorer (Kusto)
Jisun Kim
Global Black Belt, Microsoft
Daisuke Masubuchi
Global Black Belt, Microsoft
Whoweare?
Global Black Belt, Gaming Technology Sales Professional is technical sales role within our
enterprise sales organization with a special focus on advanced IT workloads in gaming industry.
We're passionate about helping our gaming customers and partners in China, Japan, Korea and
other Asian countries to start Azure journey with successful launch
ジサン
• データ分析
• 車の運転
• MMORPG
• ヨガ
• 日本食!
ぶっちーさん
• ビール
• ブロスタ
• カレー
• Webサーバー
• Bot開発
GamingIndustryinKorea
韓国ゲーム業界の特徴
MMORPG
Free to Play
Online Service Oriented
Huge Log Data
PC games
Mobile games Massive CCU
Hardcore Gamers
BigDataAnalyticsinGaming
オンラインゲームのデータ分析の役目
Data Analyst
AzureDataExplorerinGamingAnalytics
Fast & Low latency
早い。低レイテンシー
データの収集(インジェスチョン)に最適化
今年注目のAzure新サービス「AzureDataExplorer」
General Availability: Azure Data Explorer
Posted on Thursday, February 7, 2019
Ask questions and iteratively explore data on the fly to improve p
roducts, enhance customer experiences, monitor devices, and bo
ost operations.
Quickly identify patterns, anomalies, and trends in your data. Exp
lore new questions and get answers in minutes. Run as many que
ries as you need, thanks to the optimized cost structure.
2⽉に⼀般提供開始したAzure Data Explorer
質問によってすばやく反復的にデータを調査
製品の改良、カスタマー エクスペリエンスの強化、
デバイスの監視、操作の向上を実現できます。
データのパターン、異常、および傾向を特定する
新たな問題を調査し、数分で回答を得る
AzureDataExplorer
Fastandfullymanagedbigdataanalyticsservice
フルマネージド
Fully managed for efficiency
Focus on insights, not the infra-
structure for fast time to value
No infrastructure to manage;
provision the service, choose the
SKU for your workload, and create
database.
ストリーミングデータ
Optimized for streaming data
Get near-instant insights
from fast-flowing data
Scale linearly up to 200 MB per second
per node with highly performant, low
latency ingestion.
ノードごとに200MB/Sec
リニアにスケールアウト可能
フレキシブルなBigデータ分析
Designed for data exploration
Run ad-hoc queries using the
intuitive query language
Returns results from 1 Billion records <
1 second without modifying the data
or metadata
10億レコードを対象にしたクエリーも、
1秒以内に応答します。(Demoあり)
Telemetry Analytics for internal Analytics Data Platform for products
AI IOT
Interactive Analytics Big Data Platform
2015 - 2016
Starting with 1st party validation
Building modern analytics
Vision of analytics platform for MSFT
2019
Analytics engine for 3rd party offers
Unified platform across OMS/AI
Expanded scenarios for IOT timeseries
Bridged across client/server security
2017
GA - February 2019
AzureDataExplorer-Evolution
OMS ASC Defender
5 HB
Data ingested per year
13.1 B
Total queries
600 K
Cores
3.7 K
Cluster
460 PB
Total data size
46
Regions in Azure
42 K
Nodes
265 K
Databases
22.6K
Explorer distinct users
AzureDataExplorer-BytheNumbers
© Microsoft Corporation
AzureDataExploreroverview
1. Capability for many data types,
formats, and sources
Structured (numbers), semi-structured (JSONXML),
and free text
2. Batch or streaming ingestion
Use managed ingestion pipeline or
queue a request for pull ingestion
3. Compute and storage isolation
• Independent scale out / scale in
• Persistent data in Azure Blob Storage
• Caching for low-latency on compute
4. Multiple options to support
data consumption
Use out-of-the box tools such as Kusto Explorer and
connectors
or use APIs/SDKs for custom solution
Data Lake
/ Blob
IoT
Ingested Data
Engine
Data
Management
Azure Data Explorer
Azure Storage
Event Hub
IoT Hub
Customer Data
Lake
Kafka Sync
Logstash Plugin
Event Grid
Azure Portal
Power BI
ADX Web UI
ODBC / JDBC Apps
Apps (Via API)
Logstash Plugin
Apps (Via API)
Create,
Manage
Stream
Batch
Grafana
Query,
Control Commands
Azure OSS Applications
Active Data
Connections
Kusto Explorer
© Microsoft Corporation
Intuitivequerying
Simple and powerful
• Rich rational query language (filter, aggregate, join,
calculated columns, and more)
• Built-in full-text search, time series, user analytics, and
machine learning operators
• Out-of-the box visualization (render)
• Easy-to-use syntax + Microsoft IntelliSense
• Highly recognizable hierarchical schema entities
Comprehensive
• Built for querying over structured, semi-structured and
unstructured data simultaneously
Extensible
• In-line Python
• SQL
[Requirement]
• Stream & batch ingestion with Logstash & Azure Log Analytics
• Send alert for anomaly to Slack in timely manner
CustomerStory:CaseA
[Requirement]
• Experience overall Azure Data Service
• ETL Process(orchestration) is the highest priority
• Initial data ingestion & daily batch process
• Comfortable for co-working with business dept.
[Compete]
• AWS, Google
CustomerStory:CaseB
ConnecttoAzureDataExplorer
Link to our WebSite
Azure Data Explorer
© Microsoft Corporation
Kusto は Azure内部や Windowsで使われてきたクエリー言語
KQL
• The Kusto Query Language
• Kusto Engine :Kusto を処理する機構
Where to use Kusto
• Azure Log Analytics
• Azure Application Insight
• Windows Defender Advanced Thread Protection
• Azure Security Center
サンプルですぐに試せるます!
ぜひチェックしてみてください。
• Log Analytics https://aka.ms/LADemo
• Application Insights https://aka.ms/AIAnalyticsDemo
• ATP https://aka.ms/WinDefATP
© Microsoft Corporation
画面の操作イメージ
スキーマから
DB, TBL, 項目
などを選択
クエリーを記述して実行
データを展開 実行結果を表示
© Microsoft Corporation
他のソリューションとの組み合わせ例
VisualizationExploration ModelingAnalyticsIntegration /
Ingestion
Storage
Apache Kafka for
HDInsight
Azure Data Factory
Azure SQL Data
Warehouse
Azure Analysis
Services
Power BI
Azure Data Lake Storage
Azure Databricks
SparkRAzure Data Explorer
© Microsoft Corporation
Azure Data Lake Storage Gen2
ADLS Gen2 adds a high performance HDFS Endpoint to Azure Blob Storage and inherits the rich feature set of Azure
Blob Storage *
Object Tiering and Lifecycle Polic
y Management
AAD Integration, RBAC, Storage A
ccount Security
HA/DR support through ZRS and
RA-GRS
Common Blob Storage Fo
undation
Blob API Gen2 API
Server Backups, Archive Storag
e, Semi-structured Data
Unstructured O
bject Data
Hadoop File System, File and F
older Hierarchy, Granular ACLS
Atomic File Transactions
File Data
AzureDataExplorerコスト試算ツール
(http://aka.ms/adx.cost)

韓国オンラインゲームから学ぶアドホックなビックデータ分析

  • 1.
    韓国オンラインゲームから学ぶアドホックなビックデータ分析 Data Analytics inGaming with Azure Data Explorer (Kusto) Jisun Kim Global Black Belt, Microsoft Daisuke Masubuchi Global Black Belt, Microsoft
  • 2.
    Whoweare? Global Black Belt,Gaming Technology Sales Professional is technical sales role within our enterprise sales organization with a special focus on advanced IT workloads in gaming industry. We're passionate about helping our gaming customers and partners in China, Japan, Korea and other Asian countries to start Azure journey with successful launch ジサン • データ分析 • 車の運転 • MMORPG • ヨガ • 日本食! ぶっちーさん • ビール • ブロスタ • カレー • Webサーバー • Bot開発
  • 3.
    GamingIndustryinKorea 韓国ゲーム業界の特徴 MMORPG Free to Play OnlineService Oriented Huge Log Data PC games Mobile games Massive CCU Hardcore Gamers
  • 4.
  • 5.
    AzureDataExplorerinGamingAnalytics Fast & Lowlatency 早い。低レイテンシー データの収集(インジェスチョン)に最適化
  • 6.
    今年注目のAzure新サービス「AzureDataExplorer」 General Availability: AzureData Explorer Posted on Thursday, February 7, 2019 Ask questions and iteratively explore data on the fly to improve p roducts, enhance customer experiences, monitor devices, and bo ost operations. Quickly identify patterns, anomalies, and trends in your data. Exp lore new questions and get answers in minutes. Run as many que ries as you need, thanks to the optimized cost structure. 2⽉に⼀般提供開始したAzure Data Explorer 質問によってすばやく反復的にデータを調査 製品の改良、カスタマー エクスペリエンスの強化、 デバイスの監視、操作の向上を実現できます。 データのパターン、異常、および傾向を特定する 新たな問題を調査し、数分で回答を得る
  • 7.
    AzureDataExplorer Fastandfullymanagedbigdataanalyticsservice フルマネージド Fully managed forefficiency Focus on insights, not the infra- structure for fast time to value No infrastructure to manage; provision the service, choose the SKU for your workload, and create database. ストリーミングデータ Optimized for streaming data Get near-instant insights from fast-flowing data Scale linearly up to 200 MB per second per node with highly performant, low latency ingestion. ノードごとに200MB/Sec リニアにスケールアウト可能 フレキシブルなBigデータ分析 Designed for data exploration Run ad-hoc queries using the intuitive query language Returns results from 1 Billion records < 1 second without modifying the data or metadata 10億レコードを対象にしたクエリーも、 1秒以内に応答します。(Demoあり)
  • 8.
    Telemetry Analytics forinternal Analytics Data Platform for products AI IOT Interactive Analytics Big Data Platform 2015 - 2016 Starting with 1st party validation Building modern analytics Vision of analytics platform for MSFT 2019 Analytics engine for 3rd party offers Unified platform across OMS/AI Expanded scenarios for IOT timeseries Bridged across client/server security 2017 GA - February 2019 AzureDataExplorer-Evolution OMS ASC Defender
  • 9.
    5 HB Data ingestedper year 13.1 B Total queries 600 K Cores 3.7 K Cluster 460 PB Total data size 46 Regions in Azure 42 K Nodes 265 K Databases 22.6K Explorer distinct users AzureDataExplorer-BytheNumbers
  • 10.
    © Microsoft Corporation AzureDataExploreroverview 1.Capability for many data types, formats, and sources Structured (numbers), semi-structured (JSONXML), and free text 2. Batch or streaming ingestion Use managed ingestion pipeline or queue a request for pull ingestion 3. Compute and storage isolation • Independent scale out / scale in • Persistent data in Azure Blob Storage • Caching for low-latency on compute 4. Multiple options to support data consumption Use out-of-the box tools such as Kusto Explorer and connectors or use APIs/SDKs for custom solution Data Lake / Blob IoT Ingested Data Engine Data Management Azure Data Explorer Azure Storage Event Hub IoT Hub Customer Data Lake Kafka Sync Logstash Plugin Event Grid Azure Portal Power BI ADX Web UI ODBC / JDBC Apps Apps (Via API) Logstash Plugin Apps (Via API) Create, Manage Stream Batch Grafana Query, Control Commands Azure OSS Applications Active Data Connections Kusto Explorer
  • 11.
    © Microsoft Corporation Intuitivequerying Simpleand powerful • Rich rational query language (filter, aggregate, join, calculated columns, and more) • Built-in full-text search, time series, user analytics, and machine learning operators • Out-of-the box visualization (render) • Easy-to-use syntax + Microsoft IntelliSense • Highly recognizable hierarchical schema entities Comprehensive • Built for querying over structured, semi-structured and unstructured data simultaneously Extensible • In-line Python • SQL
  • 13.
    [Requirement] • Stream &batch ingestion with Logstash & Azure Log Analytics • Send alert for anomaly to Slack in timely manner CustomerStory:CaseA
  • 14.
    [Requirement] • Experience overallAzure Data Service • ETL Process(orchestration) is the highest priority • Initial data ingestion & daily batch process • Comfortable for co-working with business dept. [Compete] • AWS, Google CustomerStory:CaseB
  • 15.
    ConnecttoAzureDataExplorer Link to ourWebSite Azure Data Explorer
  • 17.
    © Microsoft Corporation Kustoは Azure内部や Windowsで使われてきたクエリー言語 KQL • The Kusto Query Language • Kusto Engine :Kusto を処理する機構 Where to use Kusto • Azure Log Analytics • Azure Application Insight • Windows Defender Advanced Thread Protection • Azure Security Center サンプルですぐに試せるます! ぜひチェックしてみてください。 • Log Analytics https://aka.ms/LADemo • Application Insights https://aka.ms/AIAnalyticsDemo • ATP https://aka.ms/WinDefATP
  • 18.
    © Microsoft Corporation 画面の操作イメージ スキーマから DB,TBL, 項目 などを選択 クエリーを記述して実行 データを展開 実行結果を表示
  • 19.
    © Microsoft Corporation 他のソリューションとの組み合わせ例 VisualizationExplorationModelingAnalyticsIntegration / Ingestion Storage Apache Kafka for HDInsight Azure Data Factory Azure SQL Data Warehouse Azure Analysis Services Power BI Azure Data Lake Storage Azure Databricks SparkRAzure Data Explorer
  • 20.
    © Microsoft Corporation AzureData Lake Storage Gen2 ADLS Gen2 adds a high performance HDFS Endpoint to Azure Blob Storage and inherits the rich feature set of Azure Blob Storage * Object Tiering and Lifecycle Polic y Management AAD Integration, RBAC, Storage A ccount Security HA/DR support through ZRS and RA-GRS Common Blob Storage Fo undation Blob API Gen2 API Server Backups, Archive Storag e, Semi-structured Data Unstructured O bject Data Hadoop File System, File and F older Hierarchy, Granular ACLS Atomic File Transactions File Data
  • 21.