Hands On Windows Azure Building A Twitter Clone

  • 2,048 views
Uploaded on

Slide deck from my talk at the SDN Conference in Amsterdam in Fall 2009

Slide deck from my talk at the SDN Conference in Amsterdam in Fall 2009

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,048
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
22
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Nik Kalyani
    DotNetNuke Corporation
    Windows Azure: Building a Twitter Clone
    Session Code: NE.09
  • 2. About
    Nik Kalyani
    • Co-founder, DotNetNuke Corporation
    • 3. Technology Entrepreneur, 15 years
    • 4. Microsoft MVP (2005-2009)
    Online
    • Email: nik@kalyani.com
    • 5. Twitter: @techbubble
    • 6. Blog: http://www.techbubble.net
  • Session Objective
    Provide insights on how to architect a large-scale, asynchronous application with the same capabilities as Twitter on Azure while leveraging its non-RDMS capabilities
  • 7. Agenda
    Azure Architecture
    Storage Concepts
    Micro-blogging Application
    Requirements
    Architecture
    Processing
  • 8. Azure Architecture
    Your Applications

    ServiceBus
    Workflow
    Database
    Analytics
    Identity
    Contacts
    AccessControl

    Reporting

    Devices

    Compute
    Storage
    Manage

  • 9. Azure Application Architecture
    n
    m
    Worker Role
    Web Role
    LB
    Cloud Storage (blob, table, queue)
  • 10. Azure Durable Storage
    Durable, scalable, available store
    Simple interface
    REST
    ADO.NET Data Services
    Simple abstractions
    Blobs
    Tables
    Queues

  • 11. Azure Data Storage Concepts
    Container
    Blobs
    http://<account>.blob.core.windows.net/<container>
    Table
    Entities
    Account
    http://<account>.table.core.windows.net/<table>
    Queue
    Messages
    http://<account>.queue.core.windows.net/<queue>
  • 12. Block
    Blob
    Container
    Account
    Blob Storage Concepts
    IMG001.JPG
    pictures
    IMG002.JPG
    erica
    Block 1
    movies
    MOV1.AVI
    Block 2
    Block 3
  • 13. Blobs: Features and Functions
    Store large objects
    Associate metadata with blob
    Metadata is <name, value> pairs
    Set/Get with or separate from blob data bits
    Standard REST Interface
    Limitations
    Block Id - 64 Bytes Per Block
    PutBlob - 64 MB Total
    Blob MetaData - 8 KB Per Blob
    PutBlock / PutBlockList - 50 GB Each
    Blocks - 4 MB Each
  • 14. Table Storage Concepts
    Entity
    Table
    Account
    Name =…
    Email = …
    users
    Name =…
    Email = …
    erica
    Photo ID =…
    Date =…
    photo index
    Photo ID =…
    Date =…
  • 15. Tables: Features and Functions
    Massively scalable, highly available and durable structured storage
    Not RDMS -- no joins, no maintenance of foreign keys
    LINQ and REST interfaces via ADO.Net Data Services
    Partition key is exposed to the application
    Limitations
    Up to 255 Entity Properties (including PartitionKey, RowKey, and Timestamp)
    1 MB Max for Combined Size of All Data in an Entity's Properties
  • 16. Queue Storage Concepts
    Message
    Queue
    Account
    128x128, http://…
    thumbnail jobs
    256x256, http://…
    erica
    http://…
    photo processing jobs
    http://…
  • 17. Queues: Features and Functions
    An account can create many queues..
    Queue Name is scoped by the account
    A Queue contains messages..
    No limit on number of messages stored in a queue
    A message is stored for at most a week in a queue
    http://<Account>.queue.core.windows.net/<QueueName>
    Messages..
    Message size <= 8 KB
    To store larger data, store data in blob/entity storage, and the blob/entity name in the message
  • 18. Micro-blogging ApplicationRequirements
    Relationships
    Any user can “follow” any other user
    A user can “unfollow” another user he/she is “following”
  • 19. Micro-blogging ApplicationRequirements
    Micro-blogging
    Any user can post a message of up to 140 characters in length
    All message posts are public
  • 20. Micro-blogging ApplicationRequirements
    Timeline
    A public timeline of all posted messages is available to anybody for up to 24 hours from the time the message was first posted
    A user timeline of all messages posted by a user is available to anybody forever
    A “following” timeline of all messages posted by all users that a user is following in the prior 24 hours is available to each user
  • 21. Micro-blogging ApplicationArchitectural Considerations
    Micro-blogging
    The number of users will determine volume of messages
    However, common situation is that a small number of users will post majority of the messages
  • 22. Micro-blogging ApplicationArchitectural Considerations
    Timelines
    The public timeline is transient and will have a high frequency of reads and writes
    A user timeline is permanent and will have a small number of writes and a large number of reads
    A “following” timeline is transient and will have a large number of irregularly timed writes and a small number of reads
  • 23. Micro-blogging ApplicationProfile, Messages and Relationships Storage
    Table = User
    PartitionKey = Username
    RowKey=string.Format("{0:D19}",
    DateTime.MaxValue.Ticks –
    DateTime.UtcNow.Ticks);
    Use Table storage sharded by user
    Entities with the same PartitionKey are stored on the same storage node
    Obtaining recent messages, followers or following list require only a single query for each
    Adding or Deleting Follower or Following requires two writes (i.e. one for each user)
  • 24. Micro-blogging ApplicationFollow
    User B Partition
    User A follows User B
    Following
    [User D]
    Message Queue
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User C]
  • 25. Micro-blogging ApplicationFollow
    User B Partition
    Follower
    [User A]
    Following
    [User D]
    Message Queue
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User C]
    Following
    [User B]
  • 26. Micro-blogging ApplicationFollow
    User B Partition
    User B follows User A
    Following
    [User D]
    Follower
    [User A]
    Message Queue
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User C]
    Following
    [User B]
  • 27. Micro-blogging ApplicationFollow
    User B Partition
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
  • 28. Micro-blogging ApplicationMessage Posting
    User B Partition
    User A posts a message
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
  • 29. Micro-blogging ApplicationMessage Posting
    User B Partition
    Message added to Message Queue
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    User A
    Message
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
  • 30. Micro-blogging ApplicationMessage Posting
    User B Partition
    After Message Worker Role processes…
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Msg (A)
    Maintenance Queue
    User B
    User C
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (A)
  • 31. Micro-blogging ApplicationMessage Posting
    User B Partition
    After Maintenance Worker Role processes…
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Msg (A)
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (A)
  • 32. Micro-blogging ApplicationMessage Maintenance
    User B Partition
    User A posts a message
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Msg (A)
    Msg (D)
    Msg (D)
    25 hours
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (X)
    Msg (A)
  • 33. Micro-blogging ApplicationMessage Maintenance
    User B Partition
    Message added to Message Queue
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    User A
    Message
    Msg (A)
    Msg (D)
    Msg (D)
    25 hours
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (X)
    Msg (A)
  • 34. Micro-blogging ApplicationMessage Maintenance
    User B Partition
    After Message Worker Role processes…
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Msg (A)
    Msg (D)
    Msg (D)
    25 hours
    Msg (A)
    Maintenance Queue
    User B
    User C
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (X)
    Msg (A)
    Msg (A)
    Msg (A)
  • 35. Micro-blogging ApplicationMessage Maintenance
    User B Partition
    After Maintenance Worker Role processes…
    Follower
    [User A]
    Following
    [User A]
    Following
    [User D]
    Message Queue
    Msg (A)
    Msg (D)
    Msg (A)
    Maintenance Queue
    User C Partition
    User A Partition
    Follower
    [User Y]
    Follower
    [User Z]
    Following
    [User A]
    Following
    [User X]
    Follower
    [User B]
    Follower
    [User C]
    Following
    [User B]
    Msg (A)
    Msg (X)
    Msg (A)
    Msg (A)
    Msg (A)
  • 36. Micro-blogging ApplicationTakeaways
    Cloud app storage significantly different from RDMS…requires shift in thinking
    Queues make apps efficient … processing is done asynchronously
    Duplication of data is often required and this is OK
    Sharding is an important consideration for the scalability of your cloud application
    Absence of join, sort make application design challenging but with planning this is manageable
  • 37. Evaluation form
    Vul je evaluatieformulier in en maak kans op een van de prachtige prijzen!!
    Fill out your evaluation form and win one of the great prizes!!
    Session Code: NE.09