MB to modify to tune to how she likes to define the BD problem
Emerging foundation for consistent pundit/vendor viewA bit contrived, but useful, and need to know because the 3Vs and becoming increasingly prominent
When you look at some of the information feeds, including structured (logs, M2M, etc) and unstructured (social, web), along with unstructured internal sources, you find a lot of them are very ‘dilute’ when it comes to valueThis these are the areas put into the ‘too difficult’ or ‘not cost effective’ to solve categoryThis is where parallel, distributed scale-out architectures – aka Big Data come inStress that it is not just about unstructured – e.g. one of the most common starting points for IT departments cutting their teeth is systems log file analysis
But not just about Hadoop. Players like EMC, HDS, HP, IBM, Oracle and others all have offerings in the Big Data arenaZooming out, here are some of the common categories (there are many ways of categorising, this is more of a stack view)Scale out storage can be used for large traditional warehouses as well as big dataDistributed analytics can run on proprietary grids as well as things like HadoopThe point being that no element of Big data is exclusiveIndeed there is nothing exclusive about big data at all – it will live alongside existing solutions (see next slide_
Has advantages that derivative insights are a lot more accessible when extracted from Big Data environment and placed in a traditional environment – skills sets/familiarity, toolset availability, well established integration and propagation mechanismsm,etc
1. Copyright 2013 Freeform Dynamics Ltd 1Big Data, Analytics and theFuture of Data CentresWhere are we and where are we going?www.freeformdynamics.comTony Lock – Programme Directortony@freeformdynamics.comwww.freeformdynamics.comVMUG Meeting, Manchester February 12, 2013
2. Copyright 2013 Freeform Dynamics Ltd 2About Freeform Dynamics Industry analyst firm Track IT industry developments and offerings Track the evolution of IT related activity and needs in business Advise both end user organisations and suppliers Research approach IT vendor and service provider briefings Large scale studies - face to face, telephone and online Community research programme Investigate strategy, business case, architecture, best practice Vendor patronage model allows free distribution Media partnerships for both input and output
3. Copyright 2013 Freeform Dynamics Ltd 3Agenda Big data and Analytics Where are we today? The evolution of the data centre Visions of the future Will we ever reach Nirvana? Closing thoughts
4. Copyright 2013 Freeform Dynamics Ltd 4Big Data What is it? Is anyone doing it? The only game intown?
5. Copyright 2013 Freeform Dynamics Ltd 5Defining Big Data? Analogies Panning for gold Finding the needle in the hay stack Identifying a weak signal in a very noisy environment Find valuable patterns, trends, correlations, etc. in noisy,unstructured, often complex, and high volume data sets Doing analytics better / differently?
6. Copyright 2013 Freeform Dynamics Ltd 6How much do you agree or disagree with the followingstatements?0% 20% 40% 60% 80% 100%The emergence of advanced storage, access andanalytics solutions means the end of thetraditional RDBMSRegardless of substance and reality of emergingtechnologies and techniques, the term ‘bigdata’ is currently being over-hyped by ITvendors in an unhelpful wayI have a clear understanding of what the term‘big data’ means5-Totally agree 4 3 2 1-Totally disagree Unsure
7. Copyright 2013 Freeform Dynamics Ltd 7The three Vs of Big DataVolumeVarietyVelocityRule of thumbGenerally thinkof Big Datawhen two ofthese threeapplyHigh physical volumes with low valuedensityDifferent sources and formats orinformationRapid rate of data movement, generationor acquisition
8. Copyright 2013 Freeform Dynamics Ltd 8The concept of value densityTraditional search anddocument managementTraditional BI and datawarehousingHigh value densityHigh value densityLow value densityLow value densityStructuredUnstructuredStructuredUnstructuredBIGDATA
9. Copyright 2013 Freeform Dynamics Ltd 11What level of growth are you seeing in the followingtypes of data within your organisation?0% 20% 40% 60% 80% 100%5 (Extremely high growth) 4 3 2 1 (No growth)Structured data(e.g. tabular data in RDBMSs)Unstructured data(e.g. documents, messages,multimedia content, etc.)
10. Copyright 2013 Freeform Dynamics Ltd 12In what form is your organisation’s most valuable or critical dataheld (i.e. your crown jewels in information terms)?0% 10% 20% 30% 40% 50%Exclusively structuredMostly structuredEqual splitMostly unstructuredExclusively unstructured
11. Copyright 2013 Freeform Dynamics Ltd 13How is this changing?0% 20% 40% 60% 80% 100%Steadyshift(25%)Steadyshift(21%)Nochange(43%)Rapid shifttowards value inunstructureddata(4%)Rapid shifttowards value instructured data(7%)
12. Copyright 2013 Freeform Dynamics Ltd 14To what degree does your organisation exploit itsinformation assets for analysis and decision making?0% 20% 40% 60% 80% 100%5 (Fully) 4 3 2 1 (Very poorly)Structured data(e.g. tabular data in RDBMSs)Unstructured data(e.g. documents, messages,multimedia content, etc)
13. Copyright 2013 Freeform Dynamics Ltd 16Use of traditional and emerging technologies0% 20% 40% 60% 80% 100%Legacy databases and file systemsGeneral purpose RDBMS serversHigh performance RDBMS configurationsOLAP multi-dimensional database systemsWrite once read many (WORM) databasesRule-based stream processing enginesIn memory databasesScale-out storage architecturesDistributed indexing and searchDistributed data analytics engines5 (Extensive use) 4 3 2 1 (Not used at all) Unsure-60% -40% -20% 0% 20% 40% 60%Less use More useCurrent level of use Change over next 3 years
14. Copyright 2013 Freeform Dynamics Ltd 17How much do you agree or disagree with the followingstatements?0% 20% 40% 60% 80% 100%Developments in advanced storage, access and analytics are allowingus to tackle problems today that were either too hard or too…Developments in advanced storage, access and analytics are allowingus to take different and better approaches to tackling some key…Vendors and consulting firms are well geared up to providing us withthe support and services we need to take advanced storage, access…We have a clear idea of the business benefits available to us throughthe use of big data technologies and solutionsWe have a clear idea of the advanced data storage and big dataanalytic technologies that are becoming availableDatabase vendors are well geared up to support their customers withappropriate licensing and commercial arrangements as data related…5-Totally agree 4 3 2 1-Totally disagree Unsure
15. Copyright 2013 Freeform Dynamics Ltd 18Elephants in the room Access to data – what data is there and where is it? Are there governance / regulatory / legal restrictions in playconcerning certain data sets? Skill shortage? IT skills Numerical skills in user base Just what questions could ‘Big Data’ help with? How do we exploit any results we generate? Feedback into ‘mainstream systems’
16. Copyright 2013 Freeform Dynamics Ltd 19Data Centre Evolution Where are we now? ‘Perfect Visions’ Will we ever get to Nirvana?
17. Copyright 2013 Freeform Dynamics Ltd 20Fragmentation and disjointsSystemsRigid dedicated stacksTeams, processes & toolsServer, storage, networking,applications, securityFunding and governanceDepartmental budgeting,ownership and accounting
18. Copyright 2013 Freeform Dynamics Ltd 21How much have you virtualised the following elements of your ITlandscape?0% 20% 40% 60% 80% 100%Your x86 serverestateStorage infrastructureYour corporatenetworkYour desktopenvironmentTotally Extensively Partially A bit Not at all UnsureEnterprise 481 respondents
19. Copyright 2013 Freeform Dynamics Ltd 22How much have you architected your IT infrastructure in theform of shared resource pools such a private clouds?Unsure6%Not at all40%A bit19%Partially20%Extensively11%Totally4%Enterprise 481 respondents
20. Copyright 2013 Freeform Dynamics Ltd 23YOUR VISION FOR DATA CENTRE COMPUTINGFollowing charts based on this question:Putting all of the existing constraints and the currentstate of the industry to one side for a minute, howdesirable would the following be as part of your perfectIT vision?
21. Copyright 2013 Freeform Dynamics Ltd 240% 20% 40% 60% 80% 100%All/most of our ITrequirements would befulfilled via hosted cloudservicesAll/most of our own ITinfrastructure would bebased on private cloudsHighly desirable-5 4 3 2 Not at all desirable-1 Unsure or N/ATHE CLOUD HOSTING THING (Perfect IT Vision)Enterprise 481 respondents
22. Copyright 2013 Freeform Dynamics Ltd 25THE WHOLE CLOUD THING0% 20% 40% 60% 80% 100%All/most of your ITrequirements are fulfilledvia hosted cloudservicesAll/most of your in-house(or co-located) ITinfrastructure is basedon private cloudarchitectureAlready there Within 1 yr Within 3 yrs Within 5 yrsWithin 10 yrs Later Never Dont knowEnterprise 481 respondents
23. Copyright 2013 Freeform Dynamics Ltd 260% 20% 40% 60% 80% 100%We’d be able to provision workloads and new systems capacity on ourprivate clouds with a few clicks on a management consoleAutomation would mean that shared resource usage was continuallyoptimised as demands fluctuate, with no human interventionWe’d be able to migrate applications and workloads back and forthbetween public and private clouds with easeHybrid cloud management capability would allow us to take an agnosticview of resources, mixing and matching internal and external capacityfreelyWe would have end-to-end visibility across on-premise and hostedsystems for management and troubleshooting purposesWe would have a consistent/joined-up way of managing security andaccess across in-house and hosted systemsWe would have a consistent/joined-up way of managing and protectingdata across in-house and hosted systemsWe’d be taking a unified approach to operations and management acrossservers, storage, networking, security, etcHighly desirable-5 4 3 2 Not at all desirable-1 Unsure or N/AOPERATIONS & MANAGEMENT (Perfect IT Vision)Enterprise 481 respondents
24. Copyright 2013 Freeform Dynamics Ltd 27OPERATIONS AND MANAGEMENT0% 20% 40% 60% 80% 100%New workloads and systems capacity are generally provisionedwith just a few clicks on a consoleUse of key shared resources is continually optimised asdemands fluctuate, with no human interventionYou can quickly and easily move workloads back and forthbetween public and private cloudsAn agnostic view of resourcing means choices between internaland external deployment are made purely on requirements andfitness for purposeYou have end-to-end operational visibility across on-premise andhosted systemsYou have a consistent/joined-up way of managing security acrossin-house and hosted systemsYou have a consistent/joined-up way of managing data across in-house and hosted systemsAlready there Within 1 yr Within 3 yrs Within 5 yrsWithin 10 yrs Later Never Dont knowEnterprise 481 respondents
25. Copyright 2013 Freeform Dynamics Ltd 280% 20% 40% 60% 80% 100%Server, storage, networking, security and other specialistswould be working together seamlessly as part of a fullyintegrated ops teamOther IT teams (developers, testers, support staff,departmental IT, etc) would have self-service provisioningcapability to obtain IT resourcesNon-technical users would have self-service provisioningcapability for new applications and servicesIT activity and investment would revolve around the concept ofbusiness services rather than systemsWe would be able to easily and accurately charge or report ITcosts back to the business based on activity or consumptionHighly desirable-5 4 3 2 Not at all desirable-1 Unsure or N/AORGANISATION & SERVICES (Perfect IT Vision)Enterprise 481 respondents
26. Copyright 2013 Freeform Dynamics Ltd 29ORGANISATION AND SERVICES0% 20% 40% 60% 80% 100%Self-service provisioning is inplace for IT teamsSelf-service provisioning is inplace for end usersIT activity and investmentrevolves around the conceptof business services ratherthan systemsYou can easily and accuratelycharge or report IT costs backto the business based onconsumptionAlready there Within 1 yr Within 3 yrs Within 5 yrsWithin 10 yrs Later Never Dont knowEnterprise 481 respondents
27. Copyright 2013 Freeform Dynamics Ltd 32How much are the following standing in the way of progressingtowards the vision?0% 20% 40% 60% 80% 100%Inability of suppliers to deliver on visions and promisesLack of interest/appreciation from senior managementThe business not ready to upset the status quoIT not ready to upset the status quoCultural impediments to investment in shared infrastructureIngrained IT funding models don’t support new ways ofworkingHistorical under-investment means the mountain’s too high toclimbLack of a formally defined vision and strategyWe simply don’t have the time, resources or budget to focuson anything other than short term prioritiesBig impediment Significant challenge Minor challenge Not a problem UnsureEnterprise 481 respondents
28. Copyright 2013 Freeform Dynamics Ltd 33When it comes to driving forwards, how is it playing out, orhow do you think it’s going to play out in your organisation?Other4%Things will be left largelyas they are for theforeseeable future18%Just let adoption ofmodern architectures,tools and techniquescreep along in an ad hocmanner20%Build a modernenvironment for new stuff,and migrate oldersystems into itincrementally43%Build a modernenvironment for new stuff,and leave older systemswhere they are11%Single bigtransformational initiativeto modernise thingsacross the board4%Enterprise 481 respondents
29. Copyright 2013 Freeform Dynamics Ltd 34Summing Up Can you have you cake(OK Apple) and eat it?
30. Copyright 2013 Freeform Dynamics Ltd 36Moving forwards• Understanding is needed• Communicate – often and in language business users can understand• Ensure governance processes are ‘big data aware’Senior businessawareness raising• Proactive local involvement in planning/prioritisation• Don’t forget about getting ‘big data’ derived results back into business use• Clear policies/discipline around data usageMinimisation of adhoc adoption•Monitor usage patterns to spot trends early•Don’t get hung up on transient fads and fashions•Embrace, substitute or block more persistent activityIdentify and deal withreal business issues•More of an orchestration approach to IT leadership•Architect systems with hard core and flexible edges•Virtualise the edge to handle diversity and personal useFacilitate flexibility viacore IT
31. Copyright 2013 Freeform Dynamics Ltd 37Adoption attitudes and tacticsResist new ideas and solutionsAllow new stuff to creep in passivelyAd hoc opportunistic adoptionDraw a line, target new apps onlyEstablish beachhead, then expandBig bang migration of everythingRecommendedby most earlymoversRecipe fordisappointment
32. Copyright 2013 Freeform Dynamics Ltd 38Available for download now:Big Data and AnalyticsDazzling new solutions or irritating new hype?Available from www.freeformdynamics.comhttp://www.freeformdynamics.com/fullarticle.asp?aid=1590A Vision for the Data CentreAre you a Mover, Dreamer or Traditionalist?Available from www.freeformdynamics.comhttp://www.freeformdynamics.com/fullarticle.asp?aid=1604