SQL under the hood


Published on

In this presentation we review some internal process inside SQL Azure.


Ing. Eduardo Castro, PhD

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SQL under the hood

  1. 1. SQL Azure DatabaseUnder the hood<br />Ing. Eduardo Castro, PhD<br />Comunidad Windows<br />ecastro@grupoasesor.net<br />http://ecastrom.blogspot.com<br />
  2. 2. Agenda<br />Service Review<br />SQL Azure Architecture & Workflows<br />Service Resilience<br />Service Monitoring <br />Attack Vectors/Security considerations<br />Wrap up<br />
  3. 3. What is “SQL Azure”?<br />
  4. 4. The Azure Services PlaformAn illustration<br />.NET Services<br />SQL Azure<br />Applications<br />Windows Azure<br />Applications<br />Windows<br />Mobile<br />Windows<br />Vista/XP<br />Windows<br />Server<br />Others<br />
  5. 5. Review – Conceptual model<br />Subscription <br />Used to map service usage to the billing instrument<br />Users may have many subscriptions<br />Logical Server<br />Akin to SQL Server Instance<br />Unit of Geo-Location & Billing<br />1:1 Subscription & server<br />User Database<br />Restricted T-SQL surface area<br />Additional catalog views provided e.g. sys.billing, sys.firewall_rules, etc<br />
  6. 6. SQL AzureA relational DB in the cloud<br />SQL Azure Database<br />Data Hub<br />Others (Future)<br />Relational database as a service<br />Highly available, automatically maintained<br />Extension of the SQL Server Data Platform<br />.NET Services<br />SQL Services<br />Applications<br />Live Services<br />Windows Azure<br />Applications<br />Windows<br />Mobile<br />Windows<br />Vista/XP<br />Windows<br />Server<br />Others<br />
  7. 7. Extending SQL Server Data Platform to the Cloud<br />Data Sync<br />Reference Data<br />Database<br />Symmetric Programming Model<br />Data Hub Aggregation<br /><ul><li>Initial services – core RDBMS capabilities with SQL Azure Database, Data Sync
  8. 8. Future Offerings
  9. 9. Additional data platform capabilities: Reporting, BI
  10. 10. New services: Reference Data</li></li></ul><li>The New SQL Data Services<br />Clear Feedback: “I want a database in the Cloud”<br />Familiar SQL Server relational model<br />Uses existing APIs & tools<br />Built for the Cloud with availability and scale<br />Accessible to all from PHP, Ruby, and Java<br />Focus on combining the best features of SQL Server running at scale with low friction<br />
  11. 11. The Evolution of SDS<br />Evolves<br />BrowserApplication<br />Application<br />Application<br />BrowserApplication<br />Application<br />ODBC, OLEDB, ADO.Net PHP, Ruby, …<br />REST Client<br />SQL Client*<br />REST Client<br />Cloud<br />Cloud<br />Windows Azure<br />REST (Astoria)<br />Web App<br />ADO.Net + EF<br />REST Client<br />HTTP+REST<br />HTTP+REST<br />HTTP<br />TDS<br />HTTP<br />Windows Azure<br />Web App<br />SQL Client*<br />Data Center<br />Data Center<br />TDS + TSQL Model<br />REST/SOAP + ACE Model<br />SDS Next<br />SDS Current<br />* Client access enabled using TDS for ODBC, ADO.Net, OLEDB, PHP-SQL, Ruby, …<br />
  12. 12. SQL Azure Network Topology<br />Applications use standard SQL client libraries: ODBC, ADO.Net, PHP, …<br />Application<br />Internet<br />Azure Cloud<br />TDS (tcp)<br />Security Boundary<br />Load balancer forwards ‘sticky’ sessions to TDS protocol tier<br />LB<br />TDS (tcp)<br />Gateway<br />Gateway<br />Gateway<br />Gateway<br />Gateway<br />Gateway<br />Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB<br />TDS (tcp)<br />SQL<br />SQL<br />SQL<br />SQL<br />SQL<br />SQL<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />
  13. 13. TDS Gateway<br />TDS Listener<br />Capability negotiation<br />TDS Packet inspection<br />Security<br />Logical->Physical mapping via metadata catalog<br />Enabler for multi-tenet capabilities<br />Isolation layer<br />
  14. 14. TDS Gateway Layering<br />Gateway Process<br />TDS Endpoint<br />AdminSvc Endpoint<br />Provisioning Endpoint<br />Protocol Parser<br />Business Logic Services<br />Connection Mgmt<br />SQL<br />SQL<br />SQL<br />SQL<br />SQL<br />SQL<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />
  15. 15. Provisioning <br />Subscription<br />Coordinated across all Azure services<br />Executed in parallel w/retries<br />Server<br />May occur between data centers<br />Point where Geo-location is established<br />Database<br />Always occurs within a single data center<br />Cross node operations executed during this process e.g. add new db to sys.databases on the master<br />
  16. 16. Server Provisioning<br />Driven by administrator Portal<br />Provision request is sent to Gateway<br />Metadata catalog entry created<br />DNS record (CNAME) created within LiveDNS service<br />Master DB created<br />On completion metadata catalog updated<br />
  17. 17. SQL Azure Server Provisioning<br />Live DNS Cluster<br />Customer Browser<br />Live DNS Svc<br />Datacenter (Sub-Region)<br />1<br />5<br />Portal LB<br />Gateway LB<br />2<br />4<br />3<br />6<br />Front-end Node<br />Front-end Node<br />Front-end Node<br />Front-end Node<br />Gateway <br />Gateway <br />Admin Portal <br />Admin Portal <br />7<br />Backend Node<br />Backend Node<br />Backend Node<br />SQL Server<br />SQL Server<br />SQL Server<br />Mgmt. Services<br />Mgmt. Services<br />Mgmt. Services<br />Fabric<br />Fabric<br />Fabric<br />
  18. 18. Database Provisioning<br />Gateway performs stateful TDS packet inspection<br />Picks out subset of messages<br />Parses out args for create database<br />Makes entry into Gateway metadata catalog<br />Unused replica set located and reserved<br />Replica set (UserDB) is prepped for use <br />Metadata catalog is updated<br />
  19. 19. SQL Azure Database provisioning<br />TDS Gateway<br />1<br />Front-end Node<br />Protocol Parser<br />TDS Session<br />2<br />3<br />Gateway Logic<br />Master Node<br />Master Cluster<br />Master Node Components<br />4<br />7<br />5<br />6<br />8<br />Backend Node 1<br />Backend Node 2<br />Backend Node 3<br />SQL Instance<br />SQL Instance<br />SQL Instance<br />SQL DB<br />SQL DB<br />SQL DB<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />
  20. 20. SQL Azure Login Process<br />Login request arrives at the Gateway<br />Gateway locates MasterDb & UserDb replica sets<br />Credentials are validated against MasterDb<br />TDS session is opened to UserDB and requests are forwarded<br />
  21. 21. SQL Azure Login Process<br />TDS Gateway<br />7<br />1<br />Front-end Node<br />Protocol Parser<br />TDS Session<br />2<br />6<br />Gateway Logic<br />Master Node<br />Global Partition Map<br />Master Node Components<br />3<br />8<br />4<br />5<br />Backend Node 1<br />Backend Node 2<br />Backend Node 3<br />SQL Instance<br />SQL Instance<br />SQL Instance<br />SQL DB<br />SQL DB<br />SQL DB<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />
  22. 22. Service Resilience<br />Provisioning<br />State machines used to coordinate activities across node (and datacenter) boundaries<br />Failed provisioning attempts cleaned automatically after 10 minutes<br />Login<br />Failovers during the login will be transparent (<30 seconds)<br />Metadata catalog refresh occurs automatically<br />Active Session<br />Surface as connection drops (due to state)<br />
  23. 23. Monitoring Service Health<br />Metrics<br />Cluster wide performance counters gather key metrics on the service<br />Used to alert Operations to issues before they become a problem<br />Early warning system<br />Code issues<br />Capacity warnings<br />Health<br />Exercises the service routinely looking for problems<br />When issues are encountered runs deep diagnostics<br />Network connectivity at the node level<br />Validate all dependent services (Live DNS, Live ID, etc)<br />Monitoring from other MSFT DC’s<br />Validates accessibility from multiple geographic locations<br />Alerts fired automatically when test jobs fail<br />
  24. 24. Security/Attack Considerations<br />Service <br />Secure channel required (SSL)<br />Denial Of Service trend tracking<br />Packet Inspection<br />Server <br />IP allow list (Firewall) <br />Idle connection culling<br />Generated server names <br />Database<br />Disallow the most commonly attacked user id’s (SA, Admin, root, guest, etc) <br />Standard SQL Authn/Authz mode <br />
  25. 25. Wrap Up<br />Reviewed SQL Azure Architecture & Workflows<br />Provisioning (Server & DB)<br />Login<br />Service Resilience & Health<br />Failure detection and correction<br />How we determine service health<br />Security considerations<br />Attack vectors and mitigations <br />Questions?<br />
  26. 26. Links<br />http://comunidadwindows.org<br />http://ecastrom.blogspot.com<br />http://www.sqlazurelabs.com<br />http://www.microsoft.com/windowsazure/<br />http://sql.azure.com/<br />
  27. 27. Q&A<br />