In this presentation, Oracle ACE Brian Marshall focuses on making sure that your transition to a virtualized environment is as smooth as possible. He discusses specific settings in the virtual environment, the operating system, and Essbase to get the most out of your server.
Download the presentation to learn:
Why IT took your Essbase server
An explanation of why your performance sucks now that your server is gone
How to fix performance
How to prove your performance is better once you've fixed it
And more!
Architecture decision records - How not to get lost in the past
IT Made Me Virtualize Essbase and Performance Sucks
1.
2. 2
AGENDA Introductions1
IT Took My Essbase Server…Why?2
Performance is Awful…Why?3
How Do We Fix It?4
EssBench5
Show Me the Benchmarks!6
Q&A7
3. 3
About Brian Marshall
VP of Delivery
19+ years IT and EPM/BI Experience
14+ years with US-Analytics
100+ projects with US-Analytics
Presented at Kscope every year since 2010
Frequent blogger at HyperionEPM.com EPMMarshall.com
4. 4
About US-Analytics
Dallas-based, Hyperion-focused for over 15 years with continuous business growth
We are nimble and respond quickly to customers’ needs
Over 500 clients and over 1,000 successful Hyperion engagements
Seasoned business and technical acumen with EPM and BI initiatives
Over 65 professionals with 12+ years each of Hyperion experience and certifications
Active leaders in the Oracle community
Founder of Hyperion Professional Women's Forum, advisory board leadership, conference
presentations, webinars, EPM Speaker of the Year at Kaleidoscope in 2015 and 2014
Corporate culture of integrity with 100% customer commitment
Managed services
Managed services team is Dallas based, each with 10+ years of experience
Proven processes for all aspects of managed services
5. ABOUTUS-ANALYTICS
Managed Services
Upgrades & Migrations
Implementations
Infrastructure
Process & Advisory
Services
Big Data
Data Governance
Business Intelligence
Financial Close & Consolidation
Planning & Forecasting
Solutions
Data IntegrationTraining
Accolades
– Original Oracle Hyperion and Pillar Partner
– Oracle Hyperion Financial Management 11
– Oracle Hyperion Planning 11
– Oracle Essbase 11
– Oracle Data Relationship Management 112013, 2014, 2015
2015 Oracle TOLA
EPM Partner of the Year
6. 6
IT Took My Essbase Server…Why?
More physical servers are more difficult and costly to support
▶ Power requirements
▶ Various hardware configurations requiring various levels of support
▶ Physical servers are naturally inefficient due to the amount of overhead compared to
a virtualized environment
More difficult to rapidly backup and restore
▶ Snapshots are handy
▶ High availability can be handled entirely inside of the host environment while the
guest is completely unaware, but still fault tolerant
If you are at this point, your server is likely due to be replaced anyway
Essbase servers, unlike relational databases, are more likely to only serve one
business group
7. 7
Performance is Awful…Why?
Some reasons are pretty simple
▶ Your old server had a lot of processors…and now you don’t
▶ Your old server had a lot of memory…and now you don’t
▶ Your old server had local storage…and now you don’t
Some reasons are more complex
▶ Your IT group has put you on a host that is overprovisioned to the point of
performance degradation
▶ The storage sub-system on many hosts don’t play nice with Essbase
8. 8
How Do We Fix It?
First we have to identify the problem
▶ Processors
▶ Memory
▶ Storage It’s probably this one…
Next we have to prove it
▶ Often times when we tell IT that the problem is on their side, they automatically
come back and blame the application
▶ Clearly the problem is Essbase, right?
▶ Wrong…but we still have to prove it
9. 9
Processors
Identifying Bottlenecks
When going to a virtual platform, CPU issues will usually present in one of three
ways:
▶ A different number of logical cores than your original physical server
‒ IE: Your physical server has 16 logical cores and your new virtual machine only has 8…or 4
▶ A different speed or generation of process
‒ You may be transitioned to a virtual host that has older hardware than your Essbase server
‒ You may be transitioned to a virtual host with much slower processors
‒ The emphasis on a hypervisor (virtual host) is the number of cores, not the speed of those cores
‒ In most instances, Essbase will benefit from speed over cores, so this presents a problem
▶ Overprovisioned Host
‒ This one is a little trickier
‒ If everything else has been eliminated, ask IT to provide a graph of the utilization of the host
10. 10
Processors
Identifying Bottlenecks
When going to a virtual platform, memory issues will usually present in one
of three ways:
▶ A lower amount of memory than the physical server
‒ If you transition to a virtualized system that has half the RAM of your physical Essbase server but keep the
settings the same, you may max our your RAM
‒ Once you run out of RAM, things start to page, errors start to occur, things grind to a hault
▶ Overprovisioned Host
‒ Like processor issues on the host, this one is a little trickier
‒ Again, let’s ask IT to provide a graph of the utilization of the host
11. 11
Storage
Identifying Bottlenecks
There’s a high probability that this is the problem
Many physical Essbase servers are running direct attached storage
It is also common for that direct attached storage to be a solid state disk
(SSD)
Essbase is sensitive to three performance characteristics of a storage
system:
▶ Random Read/Write Performance (Most Important)
▶ Latency (Directly impacts Random Read/Write Performance)
▶ Sequential Read/Write Performance (Least Important)
12. 12
Storage (cont.)
Identifying Bottlenecks
Many virtualized environments run on clustered file systems
▶ Hugely beneficial for things like high availability and backups
▶ Very high latency and often challenged at random disk I/O
Single Essbase applications are not capable of operating a high queue
depths
High queue depths are generally the only way to get decent random I/O
performance out of a clustered file system
When IT tells you that the SAN is “soooo fast”, it very well may be for
certain things
Most of those things don’t matter to Essbase
14. 14
Storage (cont.)
Identifying Bottlenecks
Other things to look for:
▶ Watch your CPU utilization
‒ If you have CALCPARALLEL at 8 and you are using less than 8 cores, you either need better cache settings, or more
than likely you need better disk performance
▶ Watch you disk throughput
‒ Don’t expect disk throughput to be very, very high
‒ Most Essbase operations are done using random reads and random writes
‒ Random reads and random writes at a very low queue depth
‒ On traditional spinning disks, even with RAID, these numbers look more like 1-5MB/s
‒ On SSD’s, these will be much higher, but still lower than you might expect
15. 15
How do we prove it?
Option 1: Test your application on your old system and compare it to your new
system
▶ This presents a problem as the old system has a variety of differences, including the
version of Essbase most likely
Option 2: Test your application on an independent system and compare it to
your new system
▶ This can be done, but it sounds expensive to either borrow another companies
instance or have your consultant do this
Option 3: Test a standardized benchmark application and compare it to other
tested configurations
▶ If only something like this existed, this would be the best option, right?
16. 16
Introduction
EssBench
But wait…something like this should exist.
This presentation serves as the launch of a new standardized benchmarking
application.
Nothing helps you help IT more than identifying the problem for them.
This application will help.
This is an Essbase model with data and processes designed to make your server
say uncle.
Additionally, the EssBench.com website will contain a full database of tested
configurations for you to compare.
Again, the more information we can provide to IT, the more likely we are to
figure out a good solution.
18. 18
Essbase Application
EssBench
BSO Application
Dimensions
▶ Account (1025 members, 838 stored)
▶ Period (19 members, 14 stored)
▶ Years (6 members)
▶ Scenario (3 members)
▶ Version (4 members)
▶ Currency (3 members)
▶ Entity (8767 members, 8709 stored)
▶ Product (8639 members, 8639 stored)
Data
▶ Millions of rows of data
▶ 10+GB of .pag and .ind files
19. 19
The Benchmark
EssBench
PowerShell Scripts
▶ Creates Log File
▶ Executes MaxL Commands
MaxL Scripts
▶ Resets the cube
▶ Loads data (several rules)
▶ Aggs the cube
▶ Executes allocation
▶ Aggs the allocated data
▶ Executes currency conversion
▶ Restructures the database
Executes three times…average the results
20. 20
Introduction
Show Me the Benchmarks!
Wait, wait, wait…what are we benchmarking with?
Physical Server Specifications:
▶ 2x Intel E5-2670 Processors
‒ 8 cores, 16 threads, 2.6GHz
‒ 16 cores, 32 threads total
▶ 128GB DDR3 Memory
‒ 16x8GB DIMMs
▶ Various storage options
‒ Single Samsung 850 EVO
‒ Four Samsung 850 EVO’s in RAID 0
‒ Twelve 15,000RPM SAS Drives in RAID 1+0
‒ Single Intel P3605 NVMe SSD
‒ Network Attached Storage
21. 21
Show Me the Benchmarks!
And where do you keep all of this?
▶ In my garage of course, where do you
keep your servers?
And how are you still married?
▶ Blind luck…
▶ Oh, and an understanding wife
22. 22
Physical vs. Virtual
Show Me the Benchmarks!
Same physical server for all tests
Boot to Windows Server 2012 R2,
execute benchmark with NVMe storage
Boot to ESXi 6.5, start virtual machine,
execute benchmark with NVMe storage
Physical Virtual
Performance
Difference
Parallel Native Load 97 112 -15%
CSV Data Load Rule 296 391 -32%
Aggregation 542 654 -21%
Allocation 483 516 -7%
Aggregation 523 662 -27%
Currency Conversion 266 353 -33%
Dense Restructure 407 464 -14%
Total 2613 3151 -21%
24. 24
Essbase Native Load
Physical vs. Virtual
Eight native Essbase load files
No rule necessary
Loading in parallel
Physical Virtual
Performance
Difference
Intel P3605 97 117 -21%
4x Samsung EVO 850 in RAID 0 102 118 -16%
Samsung EVO 850 on LSI SATA 6g 107 113 -5%
iSCSI NVMe 124 165 -34%
iSCSI HDD w/ SSD Cache 113 145 -28%
12x 15,000RPM HDD in RAID 10 342 380 -11%
25. 25
Physical Virtual
Performance
Difference
Intel P3605 295 391 -33%
4x Samsung EVO 850 in RAID 0 419 559 -33%
Samsung EVO 850 on LSI SATA 6g 410 546 -33%
iSCSI NVMe 910 1392 -53%
iSCSI HDD w/ SSD Cache 954 1480 -55%
12x 15,000RPM HDD in RAID 10 0 0 0%
Essbase Load Rule
Physical vs. Virtual
Single file, single threaded
Roughly 1GB of data
Basic load rule with no
manipulation
26. 26
Physical Virtual
Performance
Difference
Intel P3605 404 561 -39%
4x Samsung EVO 850 in RAID 0 448 606 -35%
Samsung EVO 850 on LSI SATA 6g 476 616 -29%
iSCSI NVMe 523 1056 -102%
iSCSI HDD w/ SSD Cache 521 766 -47%
12x 15,000RPM HDD in RAID 10 1907 3290 -72%
Aggregation
Physical vs. Virtual
Aggregation of two sparse
dimensions (~8000 members
each)
CALCPARALLEL set to 16
27. 27
Physical Virtual
Performance
Difference
Intel P3605 478 513 -7%
4x Samsung EVO 850 in RAID 0 490 527 -8%
Samsung EVO 850 on LSI SATA 6g 493 529 -7%
iSCSI NVMe 518 653 -26%
iSCSI HDD w/ SSD Cache 533 594 -12%
12x 15,000RPM HDD in RAID 10 996 1101 -11%
Allocation
Physical vs. Virtual
Simple allocation
FIXPARALLEL set to 8
28. 28
Physical Virtual
Performance
Difference
Intel P3605 420 600 -43%
4x Samsung EVO 850 in RAID 0 493 666 -35%
Samsung EVO 850 on LSI SATA 6g 515 678 -32%
iSCSI NVMe 745 1706 -129%
iSCSI HDD w/ SSD Cache 654 1064 -63%
12x 15,000RPM HDD in RAID 10 4042 16224 -301%
Targeted Aggregation
Physical vs. Virtual
Aggregation of the allocated
account for both sparse
dimensions
FIXPARALLEL set to 16
29. 29
Physical Virtual
Performance
Difference
Intel P3605 420 600 -43%
4x Samsung EVO 850 in RAID 0 493 666 -35%
Samsung EVO 850 on LSI SATA 6g 515 678 -32%
iSCSI NVMe 745 1706 -129%
iSCSI HDD w/ SSD Cache 654 1064 -63%
12x 15,000RPM HDD in RAID 10 4042 16224 -301%
Targeted Aggregation
Physical vs. Virtual
Let’s see if the graph makes a
little more sense without our
hard drive option
30. 30
Currency Conversion
Physical vs. Virtual
Simple currency conversion
FIXPARALLEL set to 16
Physical Virtual
Performance
Difference
Intel P3605 306 422 -38%
4x Samsung EVO 850 in RAID 0 375 445 -18%
Samsung EVO 850 on LSI SATA 6g 375 448 -20%
iSCSI NVMe 1315 2036 -55%
iSCSI HDD w/ SSD Cache 597 823 -38%
12x 15,000RPM HDD in RAID 10 3814 6876 -80%
31. 31
Currency Conversion
Physical vs. Virtual
One more time without our
hard drive option
Physical Virtual
Performance
Difference
Intel P3605 306 422 -38%
4x Samsung EVO 850 in RAID 0 375 445 -18%
Samsung EVO 850 on LSI SATA 6g 375 448 -20%
iSCSI NVMe 1315 2036 -55%
iSCSI HDD w/ SSD Cache 597 823 -38%
12x 15,000RPM HDD in RAID 10 3814 6876 -80%
32. 32
Restructure
Physical vs. Virtual
Force restructure
Physical Virtual
Performance
Difference
Intel P3605 369 433 -17%
4x Samsung EVO 850 in RAID 0 384 446 -16%
Samsung EVO 850 on LSI SATA 6g 382 449 -17%
iSCSI NVMe 380 528 -39%
iSCSI HDD w/ SSD Cache 385 461 -20%
12x 15,000RPM HDD in RAID 10 421 729 -73%
33. 33
Total Time
Physical vs. Virtual
One last time without our
spinning rust for comparison
Physical Virtual
Performance
Difference
Intel P3605 2368 3037 -28%
4x Samsung EVO 850 in RAID 0 2711 3368 -24%
Samsung EVO 850 on LSI SATA 6g 2758 3378 -22%
iSCSI NVMe 4514 7537 -67%
iSCSI HDD w/ SSD Cache 3757 5333 -42%
12x 15,000RPM HDD in RAID 10 11523 28601 -148%
35. 35
Delete a few…
But I Have More Than One App
Or…with the right storage
and server, it doesn’t
matter
Essbase may not have a
high enough queue depth
to take advantage of an
NVMe drive the right
way…with one app
But with more than one,
we can force the issue
You likely have more than
one anyway
EssBch12 EssBch13 EssBch12 (P) EssBch13 (P)
Parallel Native Load 111 112 131 126
CSV Data Load Rule 404 391 418 415
Aggregation 644 654 706 698
Allocation 515 516 525 524
Aggregation 664 662 761 751
Currency Conversion 356 353 520 490
Dense Restructure 449 464 546 542
Total 3142 3151 3607 3546
36. 36
EssBch19 EssBch20 EssBch19 (P) EssBch20 (P)
Parallel Native Load 145 145 167 168
CSV Data Load Rule 1480 1480 1645 1606
Aggregation 766 766 880 864
Allocation 594 594 552 540
Aggregation 1064 1064 1234 1198
Currency Conversion 823 823 957 898
Dense Restructure 461 461 517 520
Total 5333 5333 5953 5793
An a SAN
But I Have More Than One App
Network attached storage
suffered from a very high
amount of latency…more
on that in a second
But in general, they can
offer great throughput
even on random reads and
write if we give it more
queue depth
Two applications on iSCSI:
37. 37
Let’s take a benchmark break
iSCSI has a lot of additional overhead, especially in an virtualized environment
Let’s first take a look at direct attached storage:
Physical
Direct Attached Storage
iSCSI
Virtual
38. 38
Network attached storage bring with it more complexity and as a result…more
latency
Latency killed random read and write performance in low queue depths
(which Essbase operates on)
Let’s take a look at network based storage:
Physical
Network Based Storage
iSCSI
Virtual
39. 39
Fixing the Problem with IT
Now What?
This is all great, but now what?
First, take your information to IT
Tell them which pieces are slow and see what options they have:
▶ Ask if they can provide direct attached storage to your guest
▶ Ask for more processors (if needed)
▶ Ask for more memory (if needed)
40. 40
Fixing the Problem without IT
Now What?
If you are going virtual, hopefully you are also upgrading
Take advantage of new functionality:
▶ RESTRUCTURETHREADS
▶ FIXPARALLEL
Do as much in memory as possible
Disk IO is your enemy
41. 41
IT Still Can’t Help Me
So you took your findings to IT, and they basically can’t help you
This is not all that uncommon, why?
▶ They have a finite number of resources to spend time investigating things
▶ They have a finite amount of hardware that only gets refreshed once it becomes fully
depreciated
▶ They are fighting a turf war and you might be losing (just kidding…I hope)
So what then?
42. 42
Let’s Check Out The Cloud
Let’s start with the cloud leader
(I’m not looking at you Oracle):
Amazon
Amazon released their new I3
series of servers with NVMe storage
and very fast processors with lots
of RAM:
43. 43
Cloud Options
Amazon has the I3 series among others
Microsoft and its Azure cloud has the L series of servers with direct attached
SSD storage
Google has their database-tuned servers as well
Oracle has their DenseIO series with NVMe storage
Watch out for any cloud-based application using “Block Storage”
This is shared, storage that is generally network attached…aka slow for
Essbase
But will they actually perform well compared to physical boxes?
44. 44
AWS Essbase Performance
I was fortunate enough to have a client ask me that question just in time
for Kscope17.
In a word…YES
Physical AWS
Performance
Difference
Parallel Native Load 97 85 13%
CSV Data Load Rule 295 332 -13%
Aggregation 404 383 5%
Allocation 478 422 12%
Aggregation 420 446 -6%
Currency Conversion 306 273 11%
Dense Restructure 369 282 24%
Total 2368 2220 6%
45. 45
Sample Client Performance
Fastest non-Exalytics
servers I’ve worked with
But connected to a SAN
▶ EMC Symmetrix
Mine (NVMe) Mine (iSCSI) Client (SAN)
Parallel Native Load 97 113 97
CSV Data Load Rule 295 954 814
Aggregation 404 521 554
Allocation 478 533 394
Aggregation 420 654 715
Currency Conversion 306 597 410
Dense Restructure 369 385 292
Total 2368 3757 3277
47. 47
A Few Thank You’s
My Wife for letting me build a datacenter at home
My Company (US-Analytics) for helping me out with some of the hardware
A couple of colleagues who I probably would have stalled and not completed
this project:
▶ Jake Turrell
▶ Tim German
My clients for letting me play with their hardware endlessly
48. 48
Shameless Plug
Visit my blog:
▶ EPMMarshall.com
▶ Formerly HyperionEPM.com
Visit my benchmark:
▶ EssBench.com
Connect to #orclepm on twitter: