Dataswft has been running several benchmarks at Intel Labs, Bangalore. Dataswft team is immensely thankful to the technical staff at Intel Labs Bangalore for providing access to their facilities and guidance. This presentation presents cost effective options when running large workloads on Hadoop and the benefits using Dataswft.

1. Assessing Market Risk
of an Investment Portfolio involving
15 billion
calculations
This presentation is the result of benchmarks run at Intel Innovation Labs in
Bangalore, India in 2013.
We would like to thank all the technical staff at the labs for providing us all their
facilities and guidance

2. Big Join in Hadoop
3 million positions x 5000 risk models
Each model consists 2M products.
To achieve =>6 months of historical
data to be readily available while
calculating risk.
Current Status=> Only 5 days of
prior data is immediately available,
rest in archives.

3. Business Benefit
• Allows more broader time based
risk assessment
• Solution avoids costly architectures
such as in memory JVM cache
based computing.

4. © 2012 Bizosys Technologies Pvt Ltd.
The Cluster

5. © 2012 Bizosys Technologies Pvt Ltd.
Sl. No.
Descriptio
n
Machine Name
Machine 1 Machine 2 Machine 3 Machine 4
1 Platform S4600SDP S4600SDP S4600SDP S4600SDP
2
Processor
Details
Xeon E5-4650, 2.7
GHz
Xeon E5-4650, 2.7
GHz
Xeon E5-4650, 2.7
GHz
Xeon E5-4650, 2.7
GHz
20M L3 cache, 8
Core
20M L3 cache, 8
Core
20M L3 cache, 8
Core
20M L3 cache, 8
Core
4 Memory
16 x 8GB-PC3L-
10600R
16 x 8GB-PC3L-
10600R
16 x 8GB-PC3L-
10600R
16 x 8GB-PC3L-
10600R
5 Hard disk 300GB SAS 300GB SAS 300GB SAS 300GB SAS
6 250GB SSD 250GB SSD 250GB SSD
238.5GB SSD (4 x
60GB SSD in LVM)
7 OS Details
Redhat Enterprise
Linux 6.3 x64
Redhat Enterprise
Linux 6.3 x64
Redhat Enterprise
Linux 6.3 x64
Redhat Enterprise
Linux 6.3 x64
/boot = 1GB /boot = 1GB /boot = 1GB /boot = 1GB
swap = 32GB swap = 32GB swap = 32GB swap = 32GB
/root = 100GB /root = 100GB /root = 100GB /root = 100GB
/data = 167GB /data = 167GB /data = 167GB /data = 167GB
/ssd = 250GB /ssd = 250GB /ssd = 250GB /ssd = 238.5GB
Infrastructure - Metals

6. © 2012 Bizosys Technologies Pvt Ltd.
1 Hadoop Hadoop 1.2
2 Dataswft Dataswft 0.94.4.41
3 JDK JDK 1.6.0_45
4 HDFS JDK Memory 4 GB
5 Dataswft JDK Memory 4 GB
Hadoop

7. © 2012 Bizosys Technologies Pvt Ltd.
Learning

8. First Run: 120Sec (No-Cache), 98Sec(Cache)
Setup
1250 Models / Machine with 1 SSD /Machine.
1 Dataswft instance/machine and max 64 threads/instance
Results
120 Sec with OS Cache Disabled. 98 Sec with OS Cache
Enabled.
Observation
High I/O wait and Low CPU usage.
Software bottleneck with sequential I/O reads.
Action Taken
Code modified to parallelized I/O reads

9. Second Run: 115Sec (No-Cache), 90Sec(Cache)
Setup
1250 Models / Machine with 1 SSD /Machine.
1 Dataswft instance/machine and max 64 threads/instance
Results
115 Sec with OS Cache Disabled. 90 Sec with OS Cache
Enabled.
Observation
After app log analysis we found DFSClient bottleneck.
Action Taken
Introduced 2 Dataswft Instances/Machines

10. Third Run: 70 Sec (No-Cache), 34 Sec(Cache)
Setup
1250 Models / Machine with 1 SSD /Machine.
2 Dataswft instance/machine and max 32 threads/instance
Results
70 Sec with OS Cache Disabled. 33.8 Sec with OS Cache
Enabled.
Observation (No Cache)
Average CPU Usage 32%, max 43%, Avg interrupt 17245 and
avg context switch 6365 and avg I/O wait 9.16.
Action Taken
4 SSD drives in a single machine.

11. Fourth Run: 32.8Sec (No-Cache) 30.3Sec(Cache)
Setup
1250 Models / Machine with 2 instance/machine.
4 SSDs/Machine. Max 32 threads / instance
40ms Delay on parallel thread launch
Results
32.8 Sec with OS Cache disabled. 30.3 Sec on cache enabled.
Observation (No Cache)
Average CPU Usage 75%, max 97%, Avg interrupt 48921 and
avg context switch 23376 and avg I/O wait 2.5.
Action Taken
More Delay is introduced to reduce contention.

12. Fifth Run: 32.5 Sec (No-Cache), 32Sec(Cache)
Setup
1250 Models / Machine with 2 instance/machine.
4 SSDs/Machine. Max 32 threads / instance
45ms Delay on parallel thread launch
Results
32.504 Sec with OS Cache Disabled. 32.060 Sec with OS
Cache Enabled.
Observation (No Cache)
Average CPU Usage 55%, max 82%, Avg interrupt 37564
and avg context switch 9419 and avg I/O wait 1.0.
Action Taken
None