0
Nik Kalyani<br />DotNetNuke Corporation<br />Windows Azure: Building a Twitter Clone<br />Session Code: NE.09<br />
About<br />Nik Kalyani<br /><ul><li>Co-founder, DotNetNuke Corporation
Technology Entrepreneur, 15 years
Microsoft MVP (2005-2009)</li></ul>Online  <br /><ul><li>Email: nik@kalyani.com
Twitter: @techbubble
Blog: http://www.techbubble.net</li></li></ul><li>Session Objective<br />Provide insights on how to architect a large-scal...
Agenda<br />Azure Architecture<br />Storage Concepts<br />Micro-blogging Application <br />Requirements<br />Architecture<...
Azure Architecture<br />Your Applications<br />…<br />ServiceBus<br />Workflow<br />Database<br />Analytics<br />Identity<...
Azure Application Architecture<br />n<br />m<br />Worker Role<br />Web Role<br />LB<br />Cloud Storage (blob, table, queue...
Azure Durable Storage<br />Durable, scalable, available store<br />Simple interface<br />REST<br />ADO.NET Data Services<b...
Azure Data Storage Concepts<br /> Container<br />Blobs<br />http://&lt;account&gt;.blob.core.windows.net/&lt;container&gt;...
Block<br />Blob<br />Container<br />Account<br />Blob Storage Concepts<br />IMG001.JPG<br />pictures<br />IMG002.JPG<br />...
Blobs: Features and Functions<br />Store large objects <br />Associate metadata with blob<br />Metadata is &lt;name, value...
Table Storage Concepts<br />Entity<br />Table<br />Account<br />Name =…<br />Email = …<br />users<br />Name =…<br />Email ...
Tables: Features and Functions<br />Massively scalable, highly available and durable structured storage<br />Not RDMS -- n...
Queue Storage Concepts<br />Message<br />Queue<br />Account<br />128x128, http://…<br />thumbnail jobs<br />256x256, http:...
Queues: Features and Functions<br />An account can create many queues..<br />Queue Name is scoped by the account<br />A Qu...
Micro-blogging ApplicationRequirements<br />Relationships<br />Any user can “follow” any other user<br />A user can “unfol...
Micro-blogging ApplicationRequirements<br />Micro-blogging<br />Any user can post a message of up to 140 characters in len...
Micro-blogging ApplicationRequirements<br />Timeline<br />A public timeline of all posted messages is available to anybody...
Micro-blogging ApplicationArchitectural Considerations<br />Micro-blogging<br />The number of users will determine volume ...
Micro-blogging ApplicationArchitectural Considerations<br />Timelines<br />The public timeline is transient and will have ...
Micro-blogging ApplicationProfile, Messages and Relationships Storage<br />Table = User<br />PartitionKey = Username<br />...
Micro-blogging ApplicationFollow<br />User B Partition<br />User A follows User B<br />Following<br />[User D]<br />Messag...
Micro-blogging ApplicationFollow<br />User B Partition<br />Follower<br />[User A]<br />Following<br />[User D]<br />Messa...
Micro-blogging ApplicationFollow<br />User B Partition<br />User B follows User A<br />Following<br />[User D]<br />Follow...
Micro-blogging ApplicationFollow<br />User B Partition<br />Follower<br />[User A]<br />Following<br />[User A]<br />Follo...
Micro-blogging ApplicationMessage Posting<br />User B Partition<br />User A posts a message<br />Follower<br />[User A]<br...
Micro-blogging ApplicationMessage Posting<br />User B Partition<br />Message added to Message Queue<br />Follower<br />[Us...
Micro-blogging ApplicationMessage Posting<br />User B Partition<br />After Message Worker Role processes…<br />Follower<br...
Micro-blogging ApplicationMessage Posting<br />User B Partition<br />After Maintenance Worker Role processes…<br />Followe...
Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />User A posts a message<br />Follower<br />[User A...
Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />Message added to Message Queue<br />Follower<br /...
Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />After Message Worker Role processes…<br />Followe...
Upcoming SlideShare
Loading in...5
×

Hands On Windows Azure Building A Twitter Clone

2,120

Published on

Slide deck from my talk at the SDN Conference in Amsterdam in Fall 2009

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,120
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Hands On Windows Azure Building A Twitter Clone"

  1. 1. Nik Kalyani<br />DotNetNuke Corporation<br />Windows Azure: Building a Twitter Clone<br />Session Code: NE.09<br />
  2. 2. About<br />Nik Kalyani<br /><ul><li>Co-founder, DotNetNuke Corporation
  3. 3. Technology Entrepreneur, 15 years
  4. 4. Microsoft MVP (2005-2009)</li></ul>Online <br /><ul><li>Email: nik@kalyani.com
  5. 5. Twitter: @techbubble
  6. 6. Blog: http://www.techbubble.net</li></li></ul><li>Session Objective<br />Provide insights on how to architect a large-scale, asynchronous application with the same capabilities as Twitter on Azure while leveraging its non-RDMS capabilities<br />
  7. 7. Agenda<br />Azure Architecture<br />Storage Concepts<br />Micro-blogging Application <br />Requirements<br />Architecture<br />Processing<br />
  8. 8. Azure Architecture<br />Your Applications<br />…<br />ServiceBus<br />Workflow<br />Database<br />Analytics<br />Identity<br />Contacts<br />AccessControl<br />…<br />Reporting<br />…<br />Devices<br />…<br />Compute<br />Storage<br />Manage<br />…<br />
  9. 9. Azure Application Architecture<br />n<br />m<br />Worker Role<br />Web Role<br />LB<br />Cloud Storage (blob, table, queue)<br />
  10. 10. Azure Durable Storage<br />Durable, scalable, available store<br />Simple interface<br />REST<br />ADO.NET Data Services<br />Simple abstractions<br />Blobs<br />Tables<br />Queues<br />…<br />
  11. 11. Azure Data Storage Concepts<br /> Container<br />Blobs<br />http://&lt;account&gt;.blob.core.windows.net/&lt;container&gt;<br />Table<br />Entities<br />Account<br />http://&lt;account&gt;.table.core.windows.net/&lt;table&gt;<br />Queue<br />Messages<br />http://&lt;account&gt;.queue.core.windows.net/&lt;queue&gt;<br />
  12. 12. Block<br />Blob<br />Container<br />Account<br />Blob Storage Concepts<br />IMG001.JPG<br />pictures<br />IMG002.JPG<br />erica<br />Block 1<br />movies<br />MOV1.AVI<br />Block 2<br />Block 3<br />
  13. 13. Blobs: Features and Functions<br />Store large objects <br />Associate metadata with blob<br />Metadata is &lt;name, value&gt; pairs<br />Set/Get with or separate from blob data bits<br />Standard REST Interface<br />Limitations<br />Block Id - 64 Bytes Per Block<br />PutBlob - 64 MB Total<br />Blob MetaData - 8 KB Per Blob<br />PutBlock / PutBlockList - 50 GB Each<br />Blocks - 4 MB Each<br />
  14. 14. Table Storage Concepts<br />Entity<br />Table<br />Account<br />Name =…<br />Email = …<br />users<br />Name =…<br />Email = …<br />erica<br />Photo ID =…<br />Date =…<br />photo index<br />Photo ID =…<br />Date =…<br />
  15. 15. Tables: Features and Functions<br />Massively scalable, highly available and durable structured storage<br />Not RDMS -- no joins, no maintenance of foreign keys<br />LINQ and REST interfaces via ADO.Net Data Services<br />Partition key is exposed to the application<br />Limitations<br />Up to 255 Entity Properties (including PartitionKey, RowKey, and Timestamp)<br />1 MB Max for Combined Size of All Data in an Entity&apos;s Properties<br />
  16. 16. Queue Storage Concepts<br />Message<br />Queue<br />Account<br />128x128, http://…<br />thumbnail jobs<br />256x256, http://…<br />erica<br />http://…<br />photo processing jobs<br />http://…<br />
  17. 17. Queues: Features and Functions<br />An account can create many queues..<br />Queue Name is scoped by the account<br />A Queue contains messages..<br />No limit on number of messages stored in a queue<br />A message is stored for at most a week in a queue<br />http://&lt;Account&gt;.queue.core.windows.net/&lt;QueueName&gt;<br />Messages..<br />Message size &lt;= 8 KB<br />To store larger data, store data in blob/entity storage, and the blob/entity name in the message<br />
  18. 18. Micro-blogging ApplicationRequirements<br />Relationships<br />Any user can “follow” any other user<br />A user can “unfollow” another user he/she is “following”<br />
  19. 19. Micro-blogging ApplicationRequirements<br />Micro-blogging<br />Any user can post a message of up to 140 characters in length<br />All message posts are public<br />
  20. 20. Micro-blogging ApplicationRequirements<br />Timeline<br />A public timeline of all posted messages is available to anybody for up to 24 hours from the time the message was first posted<br />A user timeline of all messages posted by a user is available to anybody forever<br />A “following” timeline of all messages posted by all users that a user is following in the prior 24 hours is available to each user<br />
  21. 21. Micro-blogging ApplicationArchitectural Considerations<br />Micro-blogging<br />The number of users will determine volume of messages<br />However, common situation is that a small number of users will post majority of the messages<br />
  22. 22. Micro-blogging ApplicationArchitectural Considerations<br />Timelines<br />The public timeline is transient and will have a high frequency of reads and writes<br />A user timeline is permanent and will have a small number of writes and a large number of reads<br />A “following” timeline is transient and will have a large number of irregularly timed writes and a small number of reads<br />
  23. 23. Micro-blogging ApplicationProfile, Messages and Relationships Storage<br />Table = User<br />PartitionKey = Username<br />RowKey=string.Format(&quot;{0:D19}&quot;, <br />DateTime.MaxValue.Ticks – <br />DateTime.UtcNow.Ticks);<br />Use Table storage sharded by user<br />Entities with the same PartitionKey are stored on the same storage node<br />Obtaining recent messages, followers or following list require only a single query for each<br />Adding or Deleting Follower or Following requires two writes (i.e. one for each user)<br />
  24. 24. Micro-blogging ApplicationFollow<br />User B Partition<br />User A follows User B<br />Following<br />[User D]<br />Message Queue<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User C]<br />
  25. 25. Micro-blogging ApplicationFollow<br />User B Partition<br />Follower<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User C]<br />Following<br />[User B]<br />
  26. 26. Micro-blogging ApplicationFollow<br />User B Partition<br />User B follows User A<br />Following<br />[User D]<br />Follower<br />[User A]<br />Message Queue<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User C]<br />Following<br />[User B]<br />
  27. 27. Micro-blogging ApplicationFollow<br />User B Partition<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />
  28. 28. Micro-blogging ApplicationMessage Posting<br />User B Partition<br />User A posts a message<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />
  29. 29. Micro-blogging ApplicationMessage Posting<br />User B Partition<br />Message added to Message Queue<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />User A<br />Message<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />
  30. 30. Micro-blogging ApplicationMessage Posting<br />User B Partition<br />After Message Worker Role processes…<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Msg (A)<br />Maintenance Queue<br />User B<br />User C<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (A)<br />
  31. 31. Micro-blogging ApplicationMessage Posting<br />User B Partition<br />After Maintenance Worker Role processes…<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Msg (A)<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (A)<br />
  32. 32. Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />User A posts a message<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Msg (A)<br />Msg (D)<br />Msg (D)<br />25 hours<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (X)<br />Msg (A)<br />
  33. 33. Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />Message added to Message Queue<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />User A<br />Message<br />Msg (A)<br />Msg (D)<br />Msg (D)<br />25 hours<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (X)<br />Msg (A)<br />
  34. 34. Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />After Message Worker Role processes…<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Msg (A)<br />Msg (D)<br />Msg (D)<br />25 hours<br />Msg (A)<br />Maintenance Queue<br />User B<br />User C<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (X)<br />Msg (A)<br />Msg (A)<br />Msg (A)<br />
  35. 35. Micro-blogging ApplicationMessage Maintenance<br />User B Partition<br />After Maintenance Worker Role processes…<br />Follower<br />[User A]<br />Following<br />[User A]<br />Following<br />[User D]<br />Message Queue<br />Msg (A)<br />Msg (D)<br />Msg (A)<br />Maintenance Queue<br />User C Partition<br />User A Partition<br />Follower<br />[User Y]<br />Follower<br />[User Z]<br />Following<br />[User A]<br />Following<br />[User X]<br />Follower<br />[User B]<br />Follower<br />[User C]<br />Following<br />[User B]<br />Msg (A)<br />Msg (X)<br />Msg (A)<br />Msg (A)<br />Msg (A)<br />
  36. 36. Micro-blogging ApplicationTakeaways<br />Cloud app storage significantly different from RDMS…requires shift in thinking<br />Queues make apps efficient … processing is done asynchronously<br />Duplication of data is often required and this is OK<br />Sharding is an important consideration for the scalability of your cloud application<br />Absence of join, sort make application design challenging but with planning this is manageable<br />
  37. 37. Evaluation form<br />Vul je evaluatieformulier in en maak kans op een van de prachtige prijzen!!<br />Fill out your evaluation form and win one of the great prizes!!<br />Session Code: NE.09<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×