Hands On Windows Azure Building A Twitter Clone

Nik Kalyani
DotNetNuke Corporation
Windows Azure: Building a Twitter Clone
Session Code: NE.09

About
Nik Kalyani
Co-founder, DotNetNuke Corporation
Technology Entrepreneur, 15 years
Microsoft MVP (2005-2009)
Online
Email: nik@kalyani.com
Twitter: @techbubble
Blog: http://www.techbubble.net

Session Objective
Provide insights on how to architect a large-scale, asynchronous application with the same capabilities as Twitter on Azure while leveraging its non-RDMS capabilities

Agenda
Azure Architecture
Storage Concepts
Micro-blogging Application
Requirements
Architecture
Processing

Azure Architecture
Your Applications
…
ServiceBus
Workflow
Database
Analytics
Identity
Contacts
AccessControl
…
Reporting
…
Devices
…
Compute
Storage
Manage
…

Azure Application Architecture
n
m
Worker Role
Web Role
LB
Cloud Storage (blob, table, queue)

Azure Durable Storage
Durable, scalable, available store
Simple interface
REST
ADO.NET Data Services
Simple abstractions
Blobs
Tables
Queues
…

Azure Data Storage Concepts
Container
Blobs
http://<account>.blob.core.windows.net/<container>
Table
Entities
Account
http://<account>.table.core.windows.net/<table>
Queue
Messages
http://<account>.queue.core.windows.net/<queue>

Block
Blob
Container
Account
Blob Storage Concepts
IMG001.JPG
pictures
IMG002.JPG
erica
Block 1
movies
MOV1.AVI
Block 2
Block 3

Blobs: Features and Functions
Store large objects
Associate metadata with blob
Metadata is <name, value> pairs
Set/Get with or separate from blob data bits
Standard REST Interface
Limitations
Block Id - 64 Bytes Per Block
PutBlob - 64 MB Total
Blob MetaData - 8 KB Per Blob
PutBlock / PutBlockList - 50 GB Each
Blocks - 4 MB Each

Table Storage Concepts
Entity
Table
Account
Name =…
Email = …
users
Name =…
Email = …
erica
Photo ID =…
Date =…
photo index
Photo ID =…
Date =…

Tables: Features and Functions
Massively scalable, highly available and durable structured storage
Not RDMS -- no joins, no maintenance of foreign keys
LINQ and REST interfaces via ADO.Net Data Services
Partition key is exposed to the application
Limitations
Up to 255 Entity Properties (including PartitionKey, RowKey, and Timestamp)
1 MB Max for Combined Size of All Data in an Entity's Properties

Queue Storage Concepts
Message
Queue
Account
128x128, http://…
thumbnail jobs
256x256, http://…
erica
http://…
photo processing jobs
http://…

Queues: Features and Functions
An account can create many queues..
Queue Name is scoped by the account
A Queue contains messages..
No limit on number of messages stored in a queue
A message is stored for at most a week in a queue
http://<Account>.queue.core.windows.net/<QueueName>
Messages..
Message size <= 8 KB
To store larger data, store data in blob/entity storage, and the blob/entity name in the message

Micro-blogging ApplicationRequirements
Relationships
Any user can “follow” any other user
A user can “unfollow” another user he/she is “following”

Micro-blogging
Any user can post a message of up to 140 characters in length
All message posts are public

Timeline
A public timeline of all posted messages is available to anybody for up to 24 hours from the time the message was first posted
A user timeline of all messages posted by a user is available to anybody forever
A “following” timeline of all messages posted by all users that a user is following in the prior 24 hours is available to each user

Micro-blogging ApplicationArchitectural Considerations
Micro-blogging
The number of users will determine volume of messages
However, common situation is that a small number of users will post majority of the messages

Micro-blogging ApplicationArchitectural Considerations
Timelines
The public timeline is transient and will have a high frequency of reads and writes
A user timeline is permanent and will have a small number of writes and a large number of reads
A “following” timeline is transient and will have a large number of irregularly timed writes and a small number of reads

Micro-blogging ApplicationProfile, Messages and Relationships Storage
Table = User
PartitionKey = Username
RowKey=string.Format("{0:D19}",
DateTime.MaxValue.Ticks –
DateTime.UtcNow.Ticks);
Use Table storage sharded by user
Entities with the same PartitionKey are stored on the same storage node
Obtaining recent messages, followers or following list require only a single query for each
Adding or Deleting Follower or Following requires two writes (i.e. one for each user)

Micro-blogging ApplicationFollow
User B Partition
User A follows User B
Following
[User D]
Message Queue
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User C]

User B Partition
Follower
[User A]
Following
[User D]
Message Queue
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User C]
Following
[User B]

User B Partition
User B follows User A
Following
[User D]
Follower
[User A]
Message Queue
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User C]
Following
[User B]

User B Partition
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]

Micro-blogging ApplicationMessage Posting
User B Partition
User A posts a message
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]

User B Partition
Message added to Message Queue
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
User A
Message
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]

User B Partition
After Message Worker Role processes…
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Msg (A)
Maintenance Queue
User B
User C
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (A)

User B Partition
After Maintenance Worker Role processes…
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Msg (A)
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (A)

Micro-blogging ApplicationMessage Maintenance
User B Partition
User A posts a message
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Msg (A)
Msg (D)
Msg (D)
25 hours
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (X)
Msg (A)

User B Partition
Message added to Message Queue
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
User A
Message
Msg (A)
Msg (D)
Msg (D)
25 hours
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (X)
Msg (A)

User B Partition
After Message Worker Role processes…
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Msg (A)
Msg (D)
Msg (D)
25 hours
Msg (A)
Maintenance Queue
User B
User C
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (X)
Msg (A)
Msg (A)
Msg (A)

User B Partition
After Maintenance Worker Role processes…
Follower
[User A]
Following
[User A]
Following
[User D]
Message Queue
Msg (A)
Msg (D)
Msg (A)
Maintenance Queue
User C Partition
User A Partition
Follower
[User Y]
Follower
[User Z]
Following
[User A]
Following
[User X]
Follower
[User B]
Follower
[User C]
Following
[User B]
Msg (A)
Msg (X)
Msg (A)
Msg (A)
Msg (A)

Micro-blogging ApplicationTakeaways
Cloud app storage significantly different from RDMS…requires shift in thinking
Queues make apps efficient … processing is done asynchronously
Duplication of data is often required and this is OK
Sharding is an important consideration for the scalability of your cloud application
Absence of join, sort make application design challenging but with planning this is manageable

Evaluation form
Vul je evaluatieformulier in en maak kans op een van de prachtige prijzen!!
Fill out your evaluation form and win one of the great prizes!!
Session Code: NE.09

http://www.kalyani.com	150
http://kalyani.com	25
http://www.slideshare.net	2
http://b.hatena.ne.jp	1
http://feeds2.feedburner.com	1

Hands On Windows Azure Building A Twitter Clone

by Nik Kalyani

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 179

Statistics