Kickfire: Best Of All Worlds


Published on

We wanted to know how companies viewed the changing data warehousing landscape, so we surveyed 200 businesses to learn more about the issues they faced. In "Delivering the Best of All Worlds for Today's Analytics" we compare the technology, present the options, and provide findings from our survey. We also discuss the latest column store techniques and open source technology to provide both enterprise class performance and affordability.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Kickfire: Best Of All Worlds

  1. 1. Kickfire and MySQL: Delivering the Best of All Worlds for Today's Analytics A Kickfire White Paper
  2. 2. What You’ll Learn This white paper examines the issues facing companies that are evaluating their data warehousing and analytics architectures and options. It draws on analyst and market research including a survey of 200 enterprises on their attitudes towards, and concerns and issues about the adoption of newer data warehous- ing technologies such as open source databases and appliances. The Data Warehousing Landscape Today’s data growth is nothing short of explosive. Data comes from all directions, all devices and in more volume than ever before. Organizations are drowning in it but ironically they still don’t have the information they need to meet all their business goals. Just as data is proliferating faster than ever, so is user demand for access to that data. Internal users, empowered by powerful, easy-to-use PC and browser-based tools, are becoming increasingly savvy and they are frustrated by IT’s inability to keep pace with their analytics needs. And data consumers are no longer confined to internal users. Today’s customers and business partners are requiring secure query portals through which to view and analyze their business. In this context, it should come as no surprise that a 2007 Gartner Group survey1 found CIO’s identified Business Intelligence (BI) as their number one technology priority, up from number ten just four years ago. The good – and bad – news is that there are more options than ever before to help address these issues. What Are Your Data Warehousing Options? Option 1: Software Options: Traditional, Open Source and Analytic Databases Traditional Databases The issues associated with this option are known only too well, so we will not dwell on them in this white paper. The traditional database vendors, in cooperation with their hardware partners, can provide the performance, scalability and support that are required but at a premium price that is often out of alignment with the business benefits, especially on smaller projects. In addition to the direct and indirect costs associated with these solutions, there is the issue of vendor lock-in. The deeper enterprises get into the specific feature sets of these databases and the more reliant they become on the IT teams and partner resources that are tightly aligned with the vendors of these databases, the more difficult it becomes to migrate to lower-cost, more easily implemented alternatives. That said solutions from traditional database vendors remain the most commonly deployed data warehousing solutions for most enterprises. A poll conducted by Kickfire at a recent TDWI event suggests that almost 60% of enterprises still rely on either IBM or Oracle-based data warehousing solutions. However, the ascendancy of these vendors and their solutions is being increasingly challenged by new database and appliance options as we will discuss in the following sections of this white paper. Page 2
  3. 3. Open Source Databases Many enterprises view open source databases as very attractive options to traditional databases from an initial and total cost-of-ownership perspective. In fact, an April 2008 Kickfire survey of 200 such companies indicates that 66% want to be able to use open source databases, specifically MySQL, for data warehous- ing. However, many of these enterprises expressed concerns about the open source database’s ability to support their data warehousing needs. The most common of these concerns were: Figure 1 MySQL Data Warehousing Issues and Concerns 100 90 80 70 60 % Users 50 40 30 20 10 0 Performance - reports / queries too slow Scalability - won’t scale beyond 100 GB Functionality - doesn’t support ad hoc queries Tuning - will need constant tuning Hardware Build-out - scaling only possible by adding servers Although these concerns are valid, open source databases have come a long way in a very short period of time. The ability to enhance solutions quickly, supported by the huge open source ecosystem, is one of their most attractive benefits. There are thousands of open source developers around the world ranging from the IT teams in global enterprises, to developers working for software vendors to the archetypal lone-wolf open source experts. Between them, they have contributed millions of man hours to the develop- ment of these systems – far more development time than even the largest database vendor can bring to their own system. Most analysts and industry watchers are united on three key issues regarding open source databases: • There are more than 11 million active implementations and over 50,000 downloads per day of MySQL2 alone, highlighting the fact that open source databases are here to stay. page 3
  4. 4. • Open source databases are ready for the enterprise. As Noel Yuhanna, principal analyst at Forrester, stated in his July 2008 Market Update: Open Source Databases3. “Sun Microsystems’ acquisition of MySQL further validated the open source database market’s worthiness, and enterprises can now expect even more reliability and improved support in the coming years.” In an excerpt from the same report he adds, “Every enterprise should now consider open source databases as part of its overall DBMS strategy, as doing this will deliver cost savings, especially when supporting small to midsized applications.” This is not to say that open source databases will immediately replace traditional databases. The heavy financial, application and IT skills investment that most enterprises have in traditional databases ensure that they will continue to be part of the enterprise software landscape for the foreseeable future. However, we believe that where a feature/function fit can be assured between the application and an open source database, many enterprises will choose to deploy the lower cost, more rapidly implemented option. The future for most enterprises will not be choosing between traditional or open source databases, it will be developing a co-existence strategy between them and establishing guidelines for users on which database is most suitable for which type of applications and workloads. Analytics Databases Many of the brightest data warehousing designers have come to believe that traditional databases, no matter what hardware they run on and how well tuned, cannot keep pace with the enterprise’s demand for faster and cheaper analytics. To that end, several companies have launched database products that are wholly optimized for analytics applications. These products typically stand the traditional row-based data storage model on its head and drive data access more efficiently from the column rather than the row. The best of these column-based analytics databases deliver truly dazzling performance. Academic research, in the form of the Yale/MIT paper published in 2008 entitled Column-Stores vs. Row-Stores: How Different Are They Really? 4 has confirmed that, for analytics applications, it is extremely difficult for a row-based database to perform at the same level as a column store database. However, the greatest strength of this analytics-based approach is also its greatest weakness – analytics performance is optimized at the expense of transactional performance. For most enterprises, this means that the analytics database is a complementary option to their existing traditional databases not a potential replacement. Analytics databases are still expensive and can require many CPUs to build out sufficient parallelism to achieve their performance targets. Many of the enterprises that we have surveyed, are not ready or willing to take on another proprietary data warehousing vendor much less take on the substantial incremental costs associated with this complementary strategy. Option 2: Cloud Computing and Software-as-a-Service Options Two of the hottest current topics in IT are the role of Cloud computing and Software-as-a-Service (SaaS) in the enterprise. There has been a lot of press focus around these concepts due to the meteoric growth of one or two of the SaaS vendors and because household brand names have entered the Cloud computing arena. In reality, these options are more delivery-based options than technology options since the services rely on the vendor’s ability to host either software or appliance-based systems and to deliver cost-effective, managed services rather than the vendor’s ability to innovate and bring new technology to the market. page 4
  5. 5. Cloud Computing Cloud computing-based options are relatively new and, although some enterprises are piloting these solutions, there are few, if any, examples of multi-terabyte data warehouses deployed using this model. However, it is certainly possible to see how this option might play a role in proof of concept projects; those that will have a limited production life or projects that need to be deployed extremely quickly. The chief concerns around Cloud computing are performance and security. A lot of thought and planning is required before moving large volumes of sensitive and mission critical data across the public Internet to be processed on a shared storage and computing infrastructure that is processing a mixed workload of online and analytics applications. Issues may include compliance; the speed and logistics of loading terabytes of data over the Internet; and predicting performance on a mixed workload platform when so many variables are in play. Surveying the market, there appear to be no Cloud-based “pure-plays”. Thus, even those people who are trumpeting the benefits of the Cloud-based architecture are hedging their bets and supporting this as one of several deployment models. Typically, these vendors point out that this model is well-suited to trial deployments but they will push for an on-premise model when it comes to large-scale and/or production deployments. SaaS Options Several Software-as-a-Service (SaaS) vendors have recently launched in the data warehousing/Business Intelligence market. Almost all of these companies are really Business Intelligence tool vendors going to market through a SaaS model. They offer very intuitive web-based user functionality but do little, if anything, at the database level to impact performance. In addition they share many of the same issues such as security and predictable performance that apply to the Cloud-based option. Option 3: Appliance Options Just as open source databases are here to stay, so are data warehousing or analytics appliances. As James Kobielus of Forrester Research wrote in his April 2008 report Appliance Power: Crunching Data Warehousing Workloads Faster and Cheaper than Ever,5 “Appliances are taking up permanent residence in the heart of the enterprise data center – the data warehouse (DW). DW appliances – in all their bewilder- ing proliferation – are moving into the mainstream.” Data warehousing and analytics appliances have been with us for many years and are proven architec- tures in global enterprises. They are particularly well-suited to sectors such as finance, retail, consumer packaged goods and travel where the need for high-performance and massive scalability is aligned with the ability to spend hundreds of thousands, if not millions, of dollars on appliance-based solutions. These appliances have proven the concept that a purpose-built device can compete with - and outperform - the database plus commodity server solutions that have dominated the data warehousing landscape for so long. page 5
  6. 6. We can look at the evolution of data warehousing appliances in a number of different ways. However, looking back over the last 10-15 years, it is clear that here have been three waves of innovation, whether true technology innovation or marketing-led innovation. First Generation Appliances: Proprietary Appliances The pioneering vendors who developed the first wave of these appliances had to architect their solutions around proprietary hardware and software architectures in order to deliver the performance to establish themselves as a viable alternative to traditional database solutions. Moreover, because these appliance vendors were targeting the highest level of the enterprise market and were competing with database solutions that cost millions of dollars, they built pricing models very similar to those of the database vendors with entry-level price-points in the high hundreds of thousands of dollars. The first generation vendors proved that appliances could deliver superior analytics power at a lower price-point than the traditional database plus server solutions. However, their proprietary solutions were almost as expensive as traditional options and required highly specialized teams to design, develop, deploy and maintain them. Second Generation Appliances: Virtual or Bundled Appliances Seeing the success of the early innovators, a second wave of vendors came to market. Although some of these companies brought new technology innovations to market, the majority were more marketing plays based on virtual appliances or loosely coupled bundles of software and hardware components. Many of these second generation appliance vendors are hardware companies that have acquired data warehousing business units as a go-to-market mechanism for their core hardware solutions. Others are niche software players looking to benefit from the lower support costs inherent in the appliance model and the reduced total cost-of-ownership economics that they can offer to their customers. The good news is that this second generation brought significant marketing spend to the table and educated many enterprises on the benefits of an appliance-based approach to data warehousing. The competition from having multiple, similar vendors in the market also brought prices down to where entry- level price points were typically at the $100,000 level. The marketing spend and activity also piqued the interest of many VCs and entrepreneurs who saw the business opportunity for analytics appliances and believed they could bring new innovation to the market. This investment and start-up activity is now giving rise to a third generation of analytics appliances. Third Generation Appliances: Open Source Appliances As open source databases become more feature rich and better supported and as enterprises adopt such databases in increasing numbers, it is inevitable that appliances vendors will use standard or modified open source databases as one of the key building blocks of their solutions in preference to the expensive options from the traditional database vendors. However, while these various third-generation appliance vendors use the common building blocks of commodity hardware and open source software, the way in which these vendors have configured these systems is, to repeat Forrester’s word, “bewildering”. page 6
  7. 7. Kickfire: The Best of All Worlds? Notable in the third generation category is Kickfire. Kickfire’s vision is to develop a “best of all worlds” solution; an open source data warehousing solution supporting the key features of an analytics database architecture deployed on a dedicated appliance that can deliver enterprise-level performance and function- ality at an affordable price-point. It is no secret that general purpose CPUs have major bottlenecks that need to be eliminated rather than mitigated. Typically vendors and users seek to ease these bottlenecks through proven techniques such as parallelism, disk striping and advanced tuning. However, these are stop gap measures that do not address the fundamental issue that general purpose computers are not optimized to move and analyze large amounts of data in a short period of time. Kickfire’s appliance-based solution addresses this fundamental issue in a new and innovative way: it is a purpose-built appliance optimized for data warehousing performance that deploys the latest analytic database features on a standard, open source database. In line with industry trends towards open source, Kickfire selected Sun’s MySQL database as its database engine and developed a Storage Engine that plugs directly into MySQL’s core architecture. So, what’s different about Kickfire? • World’s First SQL Chip - at the core of the Kickfire Database Appliance is the world's first SQL chip that uses parallel, pipelined data flow to deliver the power of tens of high-end, general purpose CPUs on a single processor. Backed by large amounts of directly addressable memory, this provides blazing raw performance to power the Kickfire software. This architecture, called Dataflow Architecture, has been the basis of many high-performance military, scientific and research systems going back to the mid-1980s and is well-proven. • Enterprise Class Data Warehousing Software Features - at the software level, Kickfire brings to MySQL the now proven concept of storing analytics data by columns not rows. This minimizes read access times and, combined with Kickfire’s highly-efficient data compression and hardware-based search indexes, guarantees predictable and scalable performance without the traditional need for constant tuning or adding more and faster hardware. • Open Source Standards – unlike other appliances, Kickfire uses the standard Linux Operating System and runs the standard MySQL database. This means that the ever growing range of open source business intelligence tools and utilities for data loading, backup and restore can be leveraged with Kickfire. Kickfire’s appliance delivers enterprise-class performance more efficiently than any another data ware- housing architecture. Customers therefore benefit from record-breaking performance delivered with the simplicity of an appliance in a cost-effective, low TCO package and the ease of installation and manage- ment of a standard database. page 7
  8. 8. Figure 2 Kickfire: Open Source and Industry Standard Architecture SQL Chip P o w e re d by Standard Database Standard Server Standard Storage External Storage Because the Kickfire appliance is a true appliance and not a bundle of loosely-couple hardware compo- nents, it has a small form factor and needs minimal power and cooling in sharp contrast to the racks of commodity servers and disk arrays that are typical of other high-end analytics solutions. In fact, relative to the typical server configurations needed to power traditional terabyte plus data warehousing solutions, Kickfire’s appliance needs less than 10% of the rack space and consumes less than 650 watts – about the same as a typical microwave oven. Unlike many other solutions, Kickfire is targeted towards the data marts and medium-sized data ware- houses that most forward-thinking enterprises are implementing in preference to the monolithic, complex data warehouses architected to house every shred of information within the enterprise. Kickfire is targeting data sizes from tens of gigabytes to the low tens of terabytes in size and is packaged as a purpose-built appliance that is quick to deploy, requires minimal tuning and maintenance, takes up less space and power in the data center and, in many cases, pays for itself with the first project. Kickfire believes that the appliance-driven commoditization of enterprise applications is here to stay and should not be the exclusive preserve of enterprises willing and able to spend millions of dollars on data warehousing solutions. TPC-H Benchmarks: the Proof is in the Benchmarks In May 2008 Kickfire published newly audited results based on The Transaction Processing Performance Council’s TPC-H benchmarks that shocked many of the traditional data warehousing vendors. An unknown company, Kickfire, had broken the performance record6 in the non-clustered category and the price- performance record on the rigorous industry-standard TPC-H 300 GB benchmark, delivering a record- breaking 54,895 queries per hour on a 300 GB database. Not only did Kickfire set a new performance record, it did so at an unheard of cost – the Kickfire appliance that was tested cost less than $50,000, roughly a quarter the cost of traditional solutions from the database and hardware giants. page 8
  9. 9. What does this mean for my business? Business in the 21st century is driven by the web and the architectural basis for the new web economy is the LAMP7 stack. A key component of LAMP is MySQL which is emerging as the primary repository of online information worldwide. As data volumes grow, the ability to rapidly analyze this information breaks down because MySQL is architected and optimized to support transactional systems. Kickfire is the first and only analytic appliance for MySQL enabling businesses that depend on MySQL analytics to: • Improve profit margins • Deploy services faster • Offer new self-service, high-performance information applications • Consolidate servers and data to reduce cost • Achieve 10-100X analytic and reporting performance improvements • Scale operations as data volumes grow Kickfire customers are active participants in the web economy. While there are many uses of Kickfire for data analysis, two representative use cases are: Marketing Analytics The movement of services, communications and commerce online presents marketers with compelling opportunities. In the new online world, click-stream data and session history reflect actual customer behavior at a level of detail never captured before. Both B2B and B2C marketers can replace sampling techniques based on focus groups, opinion surveys and shopping observers with real data that reflects the entire population of prospective buyers. Today, customer analytics distinguishes the leaders from the laggards. Leaders leverage customer analytics to optimize campaigns, dictate contact and advertising strategies, segment markets and improve the bottom line. Retailers, marketing service providers, e-commerce companies, mobility service providers, telecommunica- tions service providers, government organizations and others are working with Kickfire to better understand their data. Kickfire has transformed hour-long queries into queries that run in less than thirty seconds for one marketing service provider vs. their hand-coded, optimized queries. With Kickfire, this marketing service provider can allow more users access to their information and deliver new chargeable services to customers confident in the performance of the query system. Network and Security Management Data Analysis Today’s network and security management tools monitor devices and generate ever increasing amounts of as-polled network data. But the systems to analyze this data fall short in both performance and data scalability. Often businesses are forced to analyze less data or buy multiple systems to scale to the desired amount of data. By removing these barriers and enabling analysis of historical data for longer periods of time, managers can more effectively evaluate application availability, plan capacity, determine appropriate thresholds to deliver users timely and relevant alerts and manage service-level agreements. Network and security management tools providers and corporations across many industries are working with Kickfire to deliver greater value from their network data. One network management company has achieved 600X query performance improvements from Kickfire enabling them to offer trend analysis on a full year of data rather than the present maximum of 30 days data. And, higher performance means less hardware is required so this same customer will be able to consolidate 5-10 network analysis systems into one Kickfire appliance when fully deployed. page 9
  10. 10. Summary In this white paper we have examined the issues and options facing companies that are evaluating their data warehousing and analytics needs. We have highlighted concerns and issues that many enterprises have towards the adoption of newer data warehousing technologies such as open source databases and appliances. The available evidence appears to support Gartner’s prediction that by 2011, at least 80% of commercial software will contain significant amounts of open source code.8 Combined wth the trend towards appli- ances as the lowest cost and most efficient vehicle for data warehousing, the case for an open source and, specifically, a MySQL-based data warehousing appliance such as Kickfire is compelling. For data warehousing users, who are interested in the cost-saving potential of open source databases but concerned over performance and scalability, Kickfire presents a “best of all worlds” option. Combined with MySQL’s low cost-of-ownership and deserved reputation for ease-of-use, Kickfire’s column-based process- ing, indexing and compression software and the raw power of the Kickfire SQL chip bring enterprise-class performance and data warehousing functionality to MySQL. For MySQL users looking at how to use MySQL for data warehousing and analytics, the case for consider- ing the Kickfire appliance is an obvious one: Kickfire brings processing power that general purpose Linux servers simply can’t match and data warehousing-specific software features not available in any other MySQL storage engine or application. As the analyst community rightly points out, data warehousing and analytics appliances are now part of the mainstream. Kickfire recognizes this and is leading the way towards making this technology simple to implement, easy to use and affordable for companies of all sizes. page 10
  11. 11. 1 Gartner Inc. press release “Gartner EXP Survey of More Than 1,400 CIOs Shows CIOs Must Create Leverage to Remain Relevant to the Business” January 23, 2008 2 There are more than 11 million active implementations and over 50,000 downloads per day of MySQL according to Sun Microsystems/MySQL. 3 Market Update: Open Source Databases by Forrester Research, Noel Yuhanna. July 2008. 4 Column-Stores vs. Row-Stores: How Different Are They Really? Daniel J. Abadi, Samuel R. Madden, Nabil Hachem. In proceedings of SIGMOD 2008. 5 Appliance Power: Crunching Data Warehousing Workloads Faster and Cheaper Than Ever, James G. Kobielus, Forrester Research. April 2008. 6 As of August 31st, 2008, the Kickfire Database Appliance Series 2400 delivers 54,895 QphH@300GB (Queries per hour on the TPC-H benchmark) propelling Kickfire to world leadership in query performance (non-clustered systems) on the 300GB TPC-H benchmark. Kickfire is also number one in price/performance at $0.89/QphH@300GB USD on the 300GB benchmark. Moreover, Kickfire delivers this record breaking performance with a 3 year total system cost of only $48,790 USD. Kickfire’s price performance metric can be found at The Kickfire Database Appliance is in beta and will be available October 14, 2008. TPCH, QphH and $/QphH are trademarks of the TPC. For additional information on the TPCH benchmark, please visit the Transaction Processing Performance Council's Web site at 7 The LAMP “stack” software bundle is the open source web platform consisting of Linux, Apache, MySQL and Perl/PHP/Python. 8 page 11