Amazon Aurora is a MySQL/PostgreSQL compatible managed database created by Amazon Web Services. Apart from being a managed service, it has better performance, better availability, and some nice extra features. Learn in this presentation some of the internals that make Aurora possible.
50. CLASSROOM ADMINISTRATION ACADEMIC COMMUNICATION
QUALITY TOOLS 100% CLOUD EASY LMS AND CONNECTABLE EDUCATIVE CONTENTS
The MIS for K-12: an educative E.R.P.
51. functionality
Administration Controls
• Reception Notices
• Room Booking
• Inventory Management
• Staff attendance and cover
module
• Documentation
Management
• Grades
• Payments
• EFQM & ISO quality
Services
• Extracurricular activities
• Transport and catering
• Special education
• Flipped classroom
• Individual timetables
• Student documentation
• Pupil Census
• General reporting
• Workforce census
Communication
• SMS & Email
• Centralized Communication
• Internal Messenger
• Contacts & Calendars
• Integrate with Google Mail
and Microsoft 365
• Parent Portal
• Behaviour Management
• Survey, newsletter, website
creation
LMS & Assessments
• Complete student profiles
• Collaborative Learning
• Recommendations
• Forums
• Projects
• Submission Box
• Self-Assessment Tests
• Rubrics
• Grades and assessment
• Official grade
documentation
53. Why AWS?
• Private cloud never was enough but most of the
time was so much
• Problems of free space on File System
• Huge use of data, no scalable database systems
• There were frequent errors doing things “by
hand”
• Security
• Complaints because we had to stop our servers
• It was difficult to test on real time
• No global servers
57. Advantages
• Use of CloudWatch
• Alarm System with CloudWatch
• SES use for emails
• S3 with versioning and separate files from
code
• Environments to test new versions are easy
to create
58. Other uses of AWS
•AI with the use of
Sagemaker
•Integrate voice
systems (LEX; Polly)
•Integrate Alexa &
Chime
59.
60. RDS Advantage
• 45 days of RDS snapshots
• Autoscaling of RDS with Aurora
• No stops on the new versions
• Data Encryption & access control
61. RDS evolution
PRIVATE CLOUD MYSQL
4 Cluster MASTER-SLAVE (1-1)
db.r3.2xlarge
AWS - AURORA
MASTER + 4 SLAVE
db.r3.2xlarge
PRIVATE CLOUD MYSQL
2 Clusters MASTER-SLAVE (1-1)
db.r3.2xlarge
AWS MYSQL
4 Clusters MASTER-SLAVE
db.r3.xlarge
AWS- AURORA
MASTER + 2 SLAVE IN
AUTOSCALING MODEL
db.r3.2xlarge
2013 2014 2015 2017 2019
PRIVATE CLOUD MYSQL
4 Cluster M-S (1-2)
db.r3.2xlarge + db.r3.xlarge
63. Migration
• Lots of Planning
• Partner help
• We connect an AWS RDS
as a slave to the previous
private cloud
• Low downtime, rather
because of the DNS than
to the switch from SLAVE
to MASTER
64. Aurora Migration
• Lots of Planning
• Well advised
• Hard work reviewing the platform to be sure
all was compatible but 0 issues found.
• All process tested before
• First cluster so easy to migrate (automated
with AWS service, less than 1 hour)
• Hard to find a screen to do the change for the
other 3 clusters:
– Downtime expected of 16 hours
– Had to be summer on Spain/Andorra/UK; we found
also a suitable week for LATAM (Chile & Colombia)
• Transfer from other clusters made with dumps
to be sure all was right.
– Problems with rights and users
• Finally the transfer was 8 hours for cluster
(20 sec) You know, when databases first came out, it looked something like this. Monolithic architecture in a single box.
With local storage, we were trading availability and durability to get better performance
(30 seconds) And then we added more such boxes. As you can see, it’s the same SQL stack everywhere. Nothing changed!
Moreover, we need heavy-weight distributed consensus for data replication and they perform poorly because of multiple phases, multiple rounds, sync points etc.
(2.5 minutes) With Aurora, we did two big contributions:
We pushed down Log applicator down to the storage => that allowed us to construct pages from the logs themselves. This is really cool because we don’t have to write full pages anymore. So unlike traditional databases which write both logs and pages, we just have to write logs. This means we have significantly less network IO, fundamentally less work on the engine - you don't need checkpointing anymore, you don't need flushing of pages or cache eviction any more.
Instead of heavy weight distributed consensus for data replication. We use 4/6 write Quorum & Local tracking. The reason we can avoid distributed consensus is because we exploit monotonically increasing Log Sequence Number (LSN) by the Master that allows us to order the writes. And so SN's just accept the writes. There is no voting involved.
We are going to see both these things in action. As a result, YOU get significantly better write performance
YOU get Read scale out because they share the same storage with the master.
YOU get AZ+1 failure tolerance => Aurora stores 6 copies, two copies per AZ. Even in the presence of background radiation, an entire AZ goes down on top, Aurora can handle it. No problem.
YOU get an Instant database redo recovery because we don't have to explicitly do anything at startup other than doing some math to find out the point at which we crashed.
Overall: You NO longer have to make a trade-off between performance, availability, and durability.
We continuously back your data to S3.
How does it work?
Aurora divides the database into 10GB segments. And so we basically take snapshot of those segments and stream any delta redo logs to S3. On restore, we get those snapshots and apply delta log stream on top in parallel.
This all is happening on the storage, has no performance impact on the database nodes.
In traditional databases, we have to replay logs since the last checkpoint in a single thread.
With Aurora, these redo logs are already being applied to each segment in parallel, ASYNCHRONOUSLY. For startup - we don’t have to do anything other than doing some math to find out the point at the time of the crash.
Now, there are times when we accidentally delete a table or forgot a where clause in the delete statement.
Backtrack allows to quick get the data back w/o fully restoring from backups. Relatively quick operation. Couple of minutes vs hours.
In this example, you first backtracked to t1 and let it run some trx…
note that you can actually backtrack back and forth to find out the right point, its not a destructive operation.
(1 minute) Lets see what benefits we got out of this.
Here is how the IO profile looks like for Aurora and MySQL.
1. On the left, we have MySQL on EBS. Thing to note is that it has to replicate all kinds of data. With Aurora, on the right, we ONLY have to replicate log records. As a result we do 7.7X less IO and 35X more work, despite 6X amplification due to 6 copies.
2. The other thing to note is that steps 1, 3, 4 on the left are synchronous and leads to jitter but with Aurora we use 4/6 quorum, that 's much more resilient to tail latency. And we will see that in a second, why that matters for your applications.
(45 sec) So much about OLTP, lets talk about your OLAP queries.
Here are some of the optimizations we did for OLAP...
Batched scans – the idea is to scans tuples in batches from the innodb buffer pool to avoid latching pages again/again, traversing again/again and allows to use JIT optimizations. Mainly for in-memory workloads.
Hash joins – improves equi-join performance. Build one side and scan through the other to probe the hash table. Lots of complexity with skew and duplicates – have to minimize the #passes, not to mention when to choose hash joins over other join operators like index join or nested loop joins.
Async. Key Prefetch - prefetches pages in memory for index joins using BKA, quite useful for non-equi joins or some equi joins (if one side is small and the big side has a high-cardinality index on the joining column). For Out of cache workloads.
(15 seconds) We ran TPCH like workload.
Here you can the performance improvement from all those improvements. And as you can see, roughly half the queries are >2x better, with a peak speed up of roughly 18x.
(30 sec) Processing closer to data significantly reduces the data transfer between HN and SN. As a result, there is a significant less impact on the OLTP performance because of buffer pool pollution and reduction in network traffic.
We use 150GB just to show the impact because 8xl buffer pool is around 150GB something. And so if we bring in pages for OLAP queries, it will evict pages needed by OLTP queries.
(2.5 min) With Aurora MM, there is no pessimistic locking, no explicit global ordering, no global commit-coordination. Aurora architecture is based on 3 techniques:
First, it uses optimistic-conflict-resolution in the storage. To understand this better, lets say Orange Master runs T1 and Blue Master runs T2. Now, if T1 and T2 modifies different pages, there is no conflict and hence no sync required. However, if T1 and T2 both touches P2, then one of them wins and the other one have to retry based on the quorum. And as you can see, that doesn’t require any heavy-weight consensus protocol. Again we rely on quorum and local tracking with partitioned, monotonic LSN sequencing, from individual database nodes, to order the writes.
Since logging layer is pushed down, Aurora decouples the Transaction layer from the logging layer, and this allows Aurora to separate physical conflict on the pages from the logical conflict between the transactions. Trx conflicts are handled through MVCC, and physical conflicts through OCR. Moreover, there is no direct coupling in the storage partitions or, with-in the database nodes in the cluster.
Microservices architecture – there are independent, minimal and resilient services running in the cluster that are needed for async coordination. And any of them temporarily going down doesn’t NOT impact the whole cluster.
Net-net: Aurora only coordinates when it has to coordinate. Lets see this in action.
The cost of Clickedu for a school has two parts:
Initial deployment (one-time fee)
Annual service fee (for teachers and students)
Every school is made up differently, and we work with you to create an agreement that meets the needs of your school. Ask about our ‘end of school year’ offer and related deployment fee discounts that may apply to you.