Skyscanner uses Couchbase to cache live pricing data for flights and hotels to handle 2000 queries per second and 2 billion itineraries and quotes. They needed a database that provided consistency, horizontal scaling, elasticity, persistence, speed and resilience while being simple to use. While relational databases and data warehouses were no longer suitable, NoSQL databases like Couchbase with its map reduce and views capabilities met their requirements for scaling the live pricing cache in a changing technological environment.

1. SomeSQL
Scaling in a changing world
of databases and hardware
Alistair Hann
CTO, Skyscanner

2. Buzzwords

3. Web 2.0

4. Year of Mobile

5. Big Data

6. NoSQL

7. Scaling the live pricing
cache

9. Website Native Apps APIs and White
Labels
Traditional Airlines Budget Airlines Online Travel Agencies

12. Prices +Timetables
Data Collection
Services
1) Which websites
should we show?
2) What prices do
we already have
cached?
3) Live update
what we still
need.
5) Return the
prices to the user.
4) Clean up and
save the new data
Live Pricing
Service

13. Live Pricing
Service

14. Live Pricing
Service
Cached Prices (key/value)

15. 2000 quotes per second
2 bn itineraries and quotes
270 GB table
250 GB indices

16. What we really needed
Consistency
Horizontal Scaling
Elasticity
Persistence
Speed
Resilience
Simplicity

17. Live Pricing
Service
Cached Prices (key/value)

21. Beyond key value

23. Couchbase – Map Reduce Views
{
"website": {
"published": true,
"id": "affd", ...
},
"office_id": "1",
"city_id": "AUHA",
"raw_data": [...]
"address":
"closing_time": "00:00",
"routenodeid": "9618",
"type": "office“
}

24. What about the
hardware?

25. Disk for VMs
c.f. 250,000
iops Fusion I/O
Standard $0.03 / GB
Glacier $0.01 / GB

26. Quote
Bus
staging
UK1
Thrift
long-term archive
GZIP
queryable
hierarchical
LZO
queryable
flat
filter
Loader
GZIP
Quote
Bus
UK2
Thrift
Loader

27. hierarchical
flat
Hadoop cluster
or
Elastic MapReduce
analysts
query
load
analytical tools
feed
export

28. The death of the data
warehouse

29. A distributed database…
Fluentd
Kafka reporting
Graphite
Stitched
events
Fluentd
Stitched
events
Operational
metrics
Raw JSON events Errors
Elastic
MapReduce
Raw
Events
Trigger and view materialization Indexes on the data

30. Some things don’t
change

31. CC Images courtesy of chaya760 on Flickr

32. We still face the same challenges
RAM and Disk i/o concerns
Administration
Security
Data insert and retrieval
Monitoring and alerting
Performance optimization

33. The report of my death
was an exaggeration

34. Elastic Search

35. NoSQL

36. Microsoft SQL Server

37. Relational Vs NoSQL

38. Edinburgh
Quartermile One
15 Lauriston Place
Edinburgh
EH3 9EN
Glasgow
5th floor,
151-155 St Vincent
St, Glasgow
G2 5NW
Singapore
No. 08-01&04 & 09-
04
8th floor,
Robinson Point,
39 Robinson Rd,
Singapore
Beijing
Level 19, Tower E2,
Oriental Plaza,
No. 1 East Chang An
Avenue,
Dong Cheng District,
Beijing 100738
Miami
1395 Brickell Ave,
Suite 900,
Miami,
Florida 33131
Barcelona
Torre NN,
Calle Tarragona, 157,
4a Planta,
Barcelona, 08014
thank you