TechEd 2012 - Сценарии хранения и обработки данных в windows azure


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Netflix accounts for 33%of peak traffic today, doubling Google Inc.'s YouTubeAll of this needs to be encoded, hosted & delivered at a minimum which directly translates to Compute, Storage & Bandwidth & hence we care about media on Azure According to the NPD Group's Retail Tracking Service, “In the U.S., the Xbox 360 not only sold more than any other game console in 2011, it also outsold other individual TV-connected devices like DVD players, as well as digital media receivers and home theater systems."Comcast moving to adopt our video platform for OTTPortugal Telecom, Turkish Telecom, Telecom Italia, Swisscom, Telstra, Cable & Wireless, etc.Broadcasters moving to IP video platformNBC, CTV, CBS: Want to do more events onlineCanal+, BSkyB, NRK, RAI, TV2 Norway, France24: Already launched subscription services on our platform
  • By “Media Services”, we mean a collection of components and technologies from Microsoft and third parties that are curated, ready-to-use, and integrated into a single platform to simplify the creation of end-to-end media solutions.
  • TechEd 2012 - Сценарии хранения и обработки данных в windows azure

    1. 1. Windows Azure Keynote Знакомство Создание приложений Решение задач с Windows Azure Windows Azure с Windows AzureWindows Azure Internals Сценарии хранения Media Services - обработка медиа и обработки данныхWindows Azure Q&A в Windows Azure Миграция имеющихся Использование Azure ACS приложений в Windows AzureБезопасность для решения задачWindows Azure аутентификации Service Bus – связь облака пользователей с сервисами предприятияЛабораторные работыВведение в Windows Azure Virtual Machines Продвинутые техники выявления проблемРазвертывание Active Directory в Windows Azure для решений на базе Windows AzureМиграция приложения ASP.NET на Windows Azure Развертывание фермы SharePoint в Windows Azure
    2. 2. 4.5 4.03Кол-во объектов в триллионах 4 3.5 3 2.89 2.5 2 1.7 1.5 1.33 0.94 1 0.5 0 Axis Title Источник:
    3. 3. average peak 1000000Кол-во запросов в секунду 900000 800000 700000 600000 500000 400000 300000 200000 100000 0 Источник:
    4. 4. Некоторые клиенты Windows Azure
    5. 5. West US East US South Central US
    6. 6. Container Blobs https://<account><container>Account Table Entities https://<account><table> Queue Messages https://<account><queue>
    7. 7.$root/myblob
    8. 8. string storageConnectionString = CloudConfigurationManager.GetSetting("StorageConnectionString");CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();CloudBlobContainer container = blobClient.GetContainerReference("files");container.CreateIfNotExists();BlobContainerPermissions publicPermissions = new BlobContainerPermissions{ PublicAccess = BlobContainerPublicAccessType.Blob};container.SetPermissions(publicPermissions);
    9. 9. using (Stream stream = System.IO.File.OpenRead(@"D:blob.txt")){ CloudBlockBlob blob = container.GetBlockBlobReference("blob.txt"); blob.UploadFromStream(stream); CloudBlockBlob blob2 = container.GetBlockBlobReference("demo/blob2.txt"); blob2.UploadFromStream(stream); CloudBlockBlob blob3 = container.GetBlockBlobReference("demo/blob3.txt"); blob3.UploadFromStream(stream);}foreach (IListBlobItem item in container.ListBlobs("demo/")){ // Do some action}
    10. 10. Прогресс обработки периодически сохраняется в сообщении Увеличение visibility timeout на 5 минут7:04 AM7:007:09 AM7:07 Получение сообщения с 5 минутным visibility timeout Expires @ 7:05AM Expires @ 7:09AM Azure Queue 7:09 7:05 7:14 Получение прогресса и продолжение обработки
    11. 11. CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();CloudQueue queue = queueClient.GetQueueReference("booking");queue.CreateIfNotExists();CloudQueueMessage message = new CloudQueueMessage("Flight: Kharkov - Moskow");queue.AddMessage(message);...CloudQueueMessage message = queue.GetMessage();queue.DeleteMessage(message);
    12. 12. CloudTableClient tableClient = storageAccount.CreateCloudTableClient();CloudTable table = tableClient.GetTableReference("people");TableQuery<CustomerEntity> query = new TableQuery<CustomerEntity>() .Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "Smith"));foreach (CustomerEntity entity in table.ExecuteQuery(query)){ Console.WriteLine("{0}, {1}t{2}t{3}", entity.PartitionKey, entity.RowKey, entity.Email, entity.PhoneNumber);}
    13. 13. «In "Avatar," every blade of grass, every cloud in the sky, every vinein the jungle existed digitally and had to be stored somewhere.» Jon Landau, Producer Источник:
    14. 14. Более миллиона уникальных пользователей посетили сайт компании в период проката фильма в 2010 году. «We developed an interactive game where users try to navigate a course maze in a Volvo car in the fastest time and using the shortest route.» David Holecek, Digital Strategy Manager at Volvo Car Corporation Около 400 000 активных пользователей игрыИсточник:
    15. 15. Microsoft Zune MediaStorage and Delivery
    16. 16. Язык запросов HiveQLCREATE TABLE logs(t1 string, t2 string, t3 string, t4 string, t5 string, t6string, t7 string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ;LOAD DATA LOCAL INPATH sample.log OVERWRITE INTO TABLE logs;SELECT t5 AS sev, COUNT(*) AS cnt FROM logsWHERE t5 LIKE [%GROUP BY t5;
    17. 17. “We take in raw data and make it into something that isactionable for our consumers, brands, and partners.” David Mariani Vice President of Engineering at Klout “Klout chose the best of both worlds by marrying the Microsoft BI platform with Hadoop and Hive.” David Mariani Vice President of Engineering at Klout Источник: Hadoop-to-Boost-Insight-into-Big-Data/710000000129
    18. 18. Account Server Database• Работа с • Информация о • Аналог SQL порталом базах данных Server базы• Инструмент • Аутентификация данных оплаты и безопасность • Содержит таблицы, процедуры, индексы, и т.п.
    19. 19. SQL Azure Network Applications use standard SQL client Topology Application libraries: ODBC, ADO.Net, PHP, … Internet Azure Cloud TDS (tcp) Security Boundary Load balancer forwards „sticky‟ sessions to LB TDS protocol tier TDS (tcp) Gateway Gateway Gateway Gateway Gateway Gateway Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB TDS (tcp)L SQL SQL SQL SQL Scalability and Availability: Fabric, Failover, Replication, and Load balancing
    20. 20. Запрос к базе SQL Azure 1 TDS Gateway Front-end Node TDS Session Protocol Parser 2 3 Gateway Logic Master Cluster Master Node 4 Master Node Components 7 5 8 6 Backend Node 1 Backend Node 2 Backend Node 3 SQL Instance SQL Instance SQL Instance SQL DB SQL DB SQL DB Scalability and and Availability: Fabric,Failover,Replication, and Load balancing Scalability Availability: Fabric, Failover, Replication, and Load balancing
    21. 21. FederationsSalesDB CustomerFederationeration CustomerFederationeration CustomerFederation Federation Root Federation Members
    22. 22. Federations member: Range [1000, 2000)SalesDB AU AU AU CustomerFederationeration PK=5 PK=25 PK=35 CustomerFederationeration AU AU AU CustomerFederation PK=5 PK=25 PK=35 AU AU AU PK=1005 PK=1025 PK=1035 Federation Root Federation Members Atomic Units
    23. 23. Central Tables FederationsSalesDB Federated Tables CustomerFederationeration CustomerFederationeration CustomerFederation Reference Tables User Database and Federation root Federation members
    24. 24. USE FEDERATION CustomerFederation(customer_id = 5075) … SalesDB CustomerFederationeration CustomerFederationeration CustomerFederation Range Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]
    25. 25. USE FEDERATION CustomerFederation(cid = 55) WITH RESET, FILTERING=OFF GOFILTERING=OFFРаботаем с данными всего Federation member Неограниченный доступ к данным базы: Всѐ равно что соединиться по имени БД DDL, DML и доступ ко всем Atomic Units внутри Federation MemberХорошо для… Management Tasks: Обновление схемы данных Fan-out Querying – получение данных нескольких atomic units member: Range [100,200) Products (referece) Customer_id=55SalesDB CustomerFederation CustomerFederation Customers Orders (federated) (federated) Microsoft Confidential Microsoft Confidential
    26. 26. USE FEDERATION CustomerFederation(cid = 55) WITH RESET, FILTERING=ON GOFILTERING=ONРаботаем с данными одного Atomic Unit Полностью доступны данные Reference Tables Запрещены любые изменения глобального состояния Federation MemberХорошо для… Безопасной модели разработки Предотвращения утечки данных member: Range [100,200) Products (referece) Customer_id=55SalesDB CustomerFederation CustomerFederation Customers Orders (federated) Customer_id=55 (federated) Microsoft Confidential
    27. 27. ALTER FEDERATION CustomerFederation SPLIT AT (tenant_id=7500) SalesDB Orders_federation CustomerFederationeration CustomerFederationRange Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]
    28. 28. App member: Range [100,200) PostalCodeSalesDB (reference) CustomerFederationeration CustomerFederationeration CustomerFederation Customer CustomerAddress (federated) (federated) Member: Range [100,150) Member: Range [150,200)
    29. 29. “We can be selling a few hundred tickets an hour for most of the week, but when a big event goes on sale at 9:00 in the morning, we get an enormous spike in load against our application.” Matthew Davey, Founder, TicketDirectИсточник:
    30. 30. Windows Azure Roles Client Applications Service Bus Castellan.Azure • Box Office sales • Ticket Printing • System Administration • Venue/Eventhttp:// TicketDirect .* Dynamic Worker Distributed Partitioner WCF Management (tasks uploaded Cache Worker Worker • Partitioning as blobs)Windows Azure Storage On Premises Blobs to -- -- Tables to -- Queues for SQL Server store web and worker --- record --- server & communication between clients role resources ---- partition information and roles Castellan VenueSQL Azure Castellan Castellan Castellan Castellan Castellan Castellan Castellan Venue DB 1 Venue Venue Venue DB 2 Venue Venue Venue DB N Venue Venue ... Partition(s) DB‟s Partition(s) DB‟s Partition(s) DB‟s Castellan.old (VB6)One application DB, many venue DB‟s – each partitioned in to many parts (40+)
    31. 31. Monitor/Comm [SLA & Usage] Monitor/Comm [Performance][Traces &[SLA & Usage] Logs] [Performance] [Traces & Logs]RS Engine RS Engine
    32. 32. Источник:
    33. 33.
    34. 34. denisreznik@live.ru