1. Hecatonchire: Transparent Resource Aggregation
Israeli Networking Day 2013
Aidan Shribman; Sr. Researcher; SAP Research Israel
With the contribution of Benoit Hudzia, Roei Tell, Steve Walsh, Peter Izsak

2. In the talk today
Motivation: the SAP HANA in-
memory database.
Intro: the Hecatonchire project
Capabilities: mainly memory
aggregation.
Numbers: quicksort & SAP
HANA TCP-H benchmarks.
© 2012 SAP AG. All rights reserved. 2

3. Back story: SAP HANA In-Memory Real-Time Analytics
SAP ERP R/3 (1980s) SAP HANA In-Memory Database (2010)
Moderate hardware requirements; Is optimized for slow Big multi-core large-memory systems; high
storage devices. memory/core ratio: 128 GB; 256 GB; … 1024 GB.
OLTP (OnLine Transactioanl Processing) : mainly OLAP (OnLine Analytical Processing): on-the-fly
storing and retrieving transactions analytics on real-time data
Single-record access: Single-scan aggregation:
SELECT * FROM Sales_Orders SELECT Country, SUM(Sales) FROM Sales_Orders
WHERE Order = ‘457’ WHERE Product=‘corn’ GROUP BY Country
Order Country Product Sales
456 France corn 1000 Order Country Product Sales
456 France corn 1000
457 Italy wheat 900

457 Italy wheat 900
458 Italy corn 600 458 Italy G corn 600
459 Spain rice 800
459 Spain rice 800
Column order organization
Row order organization
© 2012 SAP AG. All rights reserved. 3

4. A “gap” in Current Cloud Offerings
Guest are constrained by host sizes Host resources are pooled together (future?)
Limited by physical constraints (AWS high- No limitation: Better scalability; Better economics;
memory is 244 GB guest atop a 256 GB host). Better performance.
Guests Guests
VM VM VM VM
VM
App VM App App App
App OS
App
OS
OS OS
VM
App
OS OS H/W
H/W
H/W
H/W OS
H/W H/W H/W
Server #1 Server #2 Server #n Server #1 Server #2 Server #n
CPUs CPUs CPUs CPUs CPUs CPUs
Memory Memory Memory Memory Memory Memory
I/O I/O I/O I/O I/O I/O
Fast RDMA Communication Fast RDMA Communication
© 2012 SAP AG. All rights reserved. 4

5. The Hecatonchire Project
Hecatonchires in Greek mythology means “Hundred Handed One”. The project addresses the
transparent aggregation of compute resources such as memory, CPU and storage
Hecatonchire aims at extending (not replacing) existing cloud and virtualization stacks
(Linux/KVM, QEMU, OpenStack) while using of commodity hardware and modern
interconnects (10 GbE or InfiniBand)
Hecatonchire is an open source project hosted on github developed by SAP Research
© 2012 SAP AG. All rights reserved. 5

6. Faster to Scale-Out …
Author: Chaim Bendalac
© 2012 SAP AG. All rights reserved. 6

7. Challenges of Resources Aggregation
CPU (future)
 State is MBs in size (can be migrated sub-second)
 Can be virtualized using hypervisor vCPU abstraction
 Difficult challenge: Implementing efficient fine-grained cache-coherency (difficult)
Memory (in-progress)
 GBs to TBs is size (accessed in page granularity 4KB – 4MB)
 Can be virtualized by extending the MMU (Main Memory Unit) system
 Challenge: achieving near-native performance while not degrading availability
Storage (mostly-done)
 TBs to PBs in size (but accessed in blocks MBs in size)
 Already virtualized using VFS; BLK; SCSI; iSCSI; virtio/vhost ;iSER; LIO.
 Continued challenge: high bandwidth and high IOPS (partially achieved)
© 2012 SAP AG. All rights reserved. 7

8. VM Memory Pooling
Fast Remote Page Access
Low latency RDMA (Remote Direct Memory Access) page transfer protocol
Supports best price/performance hardware: InfiniBand, iWARP, RoCE
Demand pre-paging (pre-fetching) mechanism
Built-in RAID-1 style Memory Mirroring
No extra single points for failure
Minimal overhead : 1st slave to answer synchronous; 2nd - asynchronous
Complimentary to VM-based High-Availability (e.g. Kemari)
Transparent for guest VMs
Integration by adding static hooks into Linux MMU (page fault, swap-out, etc.).
QEMU Binds Remote Memory to Guest Linux/KVM Virtual Machine.
© 2012 SAP AG. All rights reserved. 8

9. Multithreaded 2 GB Quicksort Benchmark
110.00%
Completion Time / Native Time
108.00%
106.00%
104.00%
102.00% Native
100.00% Remote
Mirrored
98.00%
96.00%
94.00%
75% 50% 33% 25% 20%
Local Memory Percentage
© 2012 SAP AG. All rights reserved. 9

10. SAP HANA TPC-H like OLAP Benchmark (128 GB / 40 vCPU)
102.00%
Completion Time / Native Time
101.50%
101.00%
100.50%
Native
100.00% Mirrored
99.50%
99.00%
50% 33%
Local Memory Percentage
© 2012 SAP AG. All rights reserved. 10

11. Application Memory Pooling
Required if virtualizaiton can„t be used due to
performance of usage of exotic hardware.
When using virtualization we „remotify“ the entire
guest VM address space.
Here we need to „remofity“ all processes
associated with the application.
© 2012 SAP AG. All rights reserved. 11

12. Hecatonchire key takeaways
Already provides transparent VM / application
memory aggregation
Extends existing cloud & virtualization
solutions
Released as open source / hosted on GitHub
Developed by SAP Research Technology
Infrastructure (TI) Practice
© 2012 SAP AG. All rights reserved. 12

13. Thank you!
Aidan Shribman
SAP Research Israel
aidan.shribman@sap.com
The Hecatonchire Project
http://www.hecatonchire.com
hecatonchire@liberlist.com

14. Legal Disclaimer
The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of
SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP
has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or
release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future
developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at
any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to
deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied,
including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This
document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or
omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially
from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as
of their dates, and they should not be relied upon in making purchasing decisions.
© 2012 SAP AG. All rights reserved. 14