TechEd 2012 - Сценарии хранения и обработки данных в windows azure
Upcoming SlideShare
Loading in...5
×
 

TechEd 2012 - Сценарии хранения и обработки данных в windows azure

on

  • 1,009 views

 

Statistics

Views

Total Views
1,009
Views on SlideShare
856
Embed Views
153

Actions

Likes
0
Downloads
3
Comments
0

4 Embeds 153

http://reznik.uneta.com.ua 149
https://twitter.com 2
https://si0.twimg.com 1
http://dev.net.ua 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Netflix accounts for 33%of peak traffic today, doubling Google Inc.'s YouTubeAll of this needs to be encoded, hosted & delivered at a minimum which directly translates to Compute, Storage & Bandwidth & hence we care about media on Azure According to the NPD Group's Retail Tracking Service, “In the U.S., the Xbox 360 not only sold more than any other game console in 2011, it also outsold other individual TV-connected devices like DVD players, as well as digital media receivers and home theater systems."Comcast moving to adopt our video platform for OTTPortugal Telecom, Turkish Telecom, Telecom Italia, Swisscom, Telstra, Cable & Wireless, etc.Broadcasters moving to IP video platformNBC, CTV, CBS: Want to do more events onlineCanal+, BSkyB, NRK, RAI, TV2 Norway, France24: Already launched subscription services on our platform
  • By “Media Services”, we mean a collection of components and technologies from Microsoft and third parties that are curated, ready-to-use, and integrated into a single platform to simplify the creation of end-to-end media solutions.

TechEd 2012 - Сценарии хранения и обработки данных в windows azure TechEd 2012 - Сценарии хранения и обработки данных в windows azure Presentation Transcript

  • Windows Azure Keynote Знакомство Создание приложений Решение задач с Windows Azure Windows Azure с Windows AzureWindows Azure Internals Сценарии хранения Media Services - обработка медиа и обработки данныхWindows Azure Q&A в Windows Azure Миграция имеющихся Использование Azure ACS приложений в Windows AzureБезопасность для решения задачWindows Azure аутентификации Service Bus – связь облака пользователей с сервисами предприятияЛабораторные работыВведение в Windows Azure Virtual Machines Продвинутые техники выявления проблемРазвертывание Active Directory в Windows Azure для решений на базе Windows AzureМиграция приложения ASP.NET на Windows Azure Развертывание фермы SharePoint в Windows Azure
  • 4.5 4.03Кол-во объектов в триллионах 4 3.5 3 2.89 2.5 2 1.7 1.5 1.33 0.94 1 0.5 0 Axis Title Источник: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/20/windows-azure-storage-4-trillion-objects-and-counting.aspx
  • average peak 1000000Кол-во запросов в секунду 900000 800000 700000 600000 500000 400000 300000 200000 100000 0 Источник: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/20/windows-azure-storage-4-trillion-objects-and-counting.aspx
  • Некоторые клиенты Windows Azure
  • West US East US South Central US
  • Container Blobs https://<account>.blob.core.windows.net/<container>Account Table Entities https://<account>.table.core.windows.net/<table> Queue Messages https://<account>.queue.core.windows.net/<queue>
  • http://myaccount.blob.core.windows.net/mycontainer/myblob http://myaccount.blob.core.windows.net/$root/myblob http://myaccount.blob.core.windows.net/myblobhttp://myaccount.blob.core.windows.net/mycontainer/path/myblob
  • string storageConnectionString = CloudConfigurationManager.GetSetting("StorageConnectionString");CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();CloudBlobContainer container = blobClient.GetContainerReference("files");container.CreateIfNotExists();BlobContainerPermissions publicPermissions = new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Blob};container.SetPermissions(publicPermissions);
  • using (Stream stream = System.IO.File.OpenRead(@"D:blob.txt")){ CloudBlockBlob blob = container.GetBlockBlobReference("blob.txt"); blob.UploadFromStream(stream); CloudBlockBlob blob2 = container.GetBlockBlobReference("demo/blob2.txt"); blob2.UploadFromStream(stream); CloudBlockBlob blob3 = container.GetBlockBlobReference("demo/blob3.txt"); blob3.UploadFromStream(stream);}foreach (IListBlobItem item in container.ListBlobs("demo/")){ // Do some action}
  • Прогресс обработки периодически сохраняется в сообщении Увеличение visibility timeout на 5 минут7:04 AM7:007:09 AM7:07 Получение сообщения с 5 минутным visibility timeout Expires @ 7:05AM Expires @ 7:09AM Azure Queue 7:09 7:05 7:14 Получение прогресса и продолжение обработки
  • CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();CloudQueue queue = queueClient.GetQueueReference("booking");queue.CreateIfNotExists();CloudQueueMessage message = new CloudQueueMessage("Flight: Kharkov - Moskow");queue.AddMessage(message);...CloudQueueMessage message = queue.GetMessage();queue.DeleteMessage(message);
  • CloudTableClient tableClient = storageAccount.CreateCloudTableClient();CloudTable table = tableClient.GetTableReference("people");TableQuery<CustomerEntity> query = new TableQuery<CustomerEntity>() .Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "Smith"));foreach (CustomerEntity entity in table.ExecuteQuery(query)){ Console.WriteLine("{0}, {1}t{2}t{3}", entity.PartitionKey, entity.RowKey, entity.Email, entity.PhoneNumber);}
  • «In "Avatar," every blade of grass, every cloud in the sky, every vinein the jungle existed digitally and had to be stored somewhere.» Jon Landau, Producer Источник: http://www.microsoft.com/microsoftservices/en/uk/article_Microsoft_Role_In_Avatar.aspx
  • Более миллиона уникальных пользователей посетили сайт компании в период проката фильма в 2010 году. «We developed an interactive game where users try to navigate a course maze in a Volvo car in the fastest time and using the shortest route.» David Holecek, Digital Strategy Manager at Volvo Car Corporation Около 400 000 активных пользователей игрыИсточник: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000008490
  • Microsoft Zune MediaStorage and Delivery
  • Язык запросов HiveQLCREATE TABLE logs(t1 string, t2 string, t3 string, t4 string, t5 string, t6string, t7 string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ;LOAD DATA LOCAL INPATH sample.log OVERWRITE INTO TABLE logs;SELECT t5 AS sev, COUNT(*) AS cnt FROM logsWHERE t5 LIKE [%GROUP BY t5;
  • “We take in raw data and make it into something that isactionable for our consumers, brands, and partners.” David Mariani Vice President of Engineering at Klout “Klout chose the best of both worlds by marrying the Microsoft BI platform with Hadoop and Hive.” David Mariani Vice President of Engineering at Klout Источник: http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2012-Enterprise/Klout/Data-Services-Firm-Uses-Microsoft-BI-and- Hadoop-to-Boost-Insight-into-Big-Data/710000000129
  • Account Server Database• Работа с • Информация о • Аналог SQL порталом базах данных Server базы• Инструмент • Аутентификация данных оплаты и безопасность • Содержит таблицы, процедуры, индексы, и т.п.
  • SQL Azure Network Applications use standard SQL client Topology Application libraries: ODBC, ADO.Net, PHP, … Internet Azure Cloud TDS (tcp) Security Boundary Load balancer forwards „sticky‟ sessions to LB TDS protocol tier TDS (tcp) Gateway Gateway Gateway Gateway Gateway Gateway Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB TDS (tcp)L SQL SQL SQL SQL Scalability and Availability: Fabric, Failover, Replication, and Load balancing
  • Запрос к базе SQL Azure 1 TDS Gateway Front-end Node TDS Session Protocol Parser 2 3 Gateway Logic Master Cluster Master Node 4 Master Node Components 7 5 8 6 Backend Node 1 Backend Node 2 Backend Node 3 SQL Instance SQL Instance SQL Instance SQL DB SQL DB SQL DB Scalability and and Availability: Fabric,Failover,Replication, and Load balancing Scalability Availability: Fabric, Failover, Replication, and Load balancing
  • FederationsSalesDB CustomerFederationeration CustomerFederationeration CustomerFederation Federation Root Federation Members
  • Federations member: Range [1000, 2000)SalesDB AU AU AU CustomerFederationeration PK=5 PK=25 PK=35 CustomerFederationeration AU AU AU CustomerFederation PK=5 PK=25 PK=35 AU AU AU PK=1005 PK=1025 PK=1035 Federation Root Federation Members Atomic Units
  • Central Tables FederationsSalesDB Federated Tables CustomerFederationeration CustomerFederationeration CustomerFederation Reference Tables User Database and Federation root Federation members
  • USE FEDERATION CustomerFederation(customer_id = 5075) … SalesDB CustomerFederationeration CustomerFederationeration CustomerFederation Range Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]
  • USE FEDERATION CustomerFederation(cid = 55) WITH RESET, FILTERING=OFF GOFILTERING=OFFРаботаем с данными всего Federation member Неограниченный доступ к данным базы: Всѐ равно что соединиться по имени БД DDL, DML и доступ ко всем Atomic Units внутри Federation MemberХорошо для… Management Tasks: Обновление схемы данных Fan-out Querying – получение данных нескольких atomic units member: Range [100,200) Products (referece) Customer_id=55SalesDB CustomerFederation CustomerFederation Customers Orders (federated) (federated) Microsoft Confidential Microsoft Confidential
  • USE FEDERATION CustomerFederation(cid = 55) WITH RESET, FILTERING=ON GOFILTERING=ONРаботаем с данными одного Atomic Unit Полностью доступны данные Reference Tables Запрещены любые изменения глобального состояния Federation MemberХорошо для… Безопасной модели разработки Предотвращения утечки данных member: Range [100,200) Products (referece) Customer_id=55SalesDB CustomerFederation CustomerFederation Customers Orders (federated) Customer_id=55 (federated) Microsoft Confidential
  • ALTER FEDERATION CustomerFederation SPLIT AT (tenant_id=7500) SalesDB Orders_federation CustomerFederationeration CustomerFederationRange Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]
  • App member: Range [100,200) PostalCodeSalesDB (reference) CustomerFederationeration CustomerFederationeration CustomerFederation Customer CustomerAddress (federated) (federated) Member: Range [100,150) Member: Range [150,200)
  • “We can be selling a few hundred tickets an hour for most of the week, but when a big event goes on sale at 9:00 in the morning, we get an enormous spike in load against our application.” Matthew Davey, Founder, TicketDirectИсточник: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000005890
  • Windows Azure Roles Client Applications Service Bus Castellan.Azure • Box Office sales • Ticket Printing • System Administration • Venue/Eventhttp:// TicketDirect .* Dynamic Worker Distributed Partitioner WCF Management (tasks uploaded Cache Worker Worker • Partitioning as blobs)Windows Azure Storage On Premises Blobs to -- -- Tables to -- Queues for SQL Server store web and worker --- record --- server & communication between clients role resources ---- partition information and roles Castellan VenueSQL Azure Castellan Castellan Castellan Castellan Castellan Castellan Castellan Venue DB 1 Venue Venue Venue DB 2 Venue Venue Venue DB N Venue Venue ... Partition(s) DB‟s Partition(s) DB‟s Partition(s) DB‟s Castellan.old (VB6)One application DB, many venue DB‟s – each partitioned in to many parts (40+)
  • Monitor/Comm [SLA & Usage] Monitor/Comm [Performance][Traces &[SLA & Usage] Logs] [Performance] [Traces & Logs]RS Engine RS Engine
  • Источник: http://weblogs.asp.net/scottgu/archive/2012/08/21/windows-azure-media-services-and-the-london-2012-olympics.aspx
  • http://sigops.org/sosp/sosp11/current/2011-Cascais/printable/11-calder.pdfhttp://research.google.com/archive/mapreduce-osdi04.pdfhttp://social.technet.microsoft.com/wiki/contents/articles/1695.inside-windows-azure-sql-database.aspxhttp://www.microsoft.com/casestudies
  • denisreznik@live.ruhttp://reznik.uneta.com.ua