Delivery 2015 pini

Sigal Russin's, Galit Fein's & Pini Cohen's work Copyright@2015
Do not remove source or attribution from any slide, graph or portion of graph
From systems of operations
to emotion-based systems
How to win the delivery game and which
technologies to use?
P

Presentation’s mission and agenda
•Provide vision about strategic directions in IT delivery (operations and infrastructure)
•Provide pragmatic advice in various IT delivery dilemmas
•Agenda:
• End user environment
• call center and collaboration
• Cloud ready architecture and public cloud
• Modular DC
• HW and servers
• Storage
• Opensource
• Devops and infrastructure as code and delivery organization
• Development trends
• Microservices
• Dbms
• APaaS
• When to leave legacy platforms
2

The delivery department perspective
3P

What is the “Ideal” delivery department?
I am cool
and I am
happy
4P

In many aspects: Internet companies
Source: http://highscalability.com/blog/2012/5/7/startups-are-creating-a-new-system-of-the-world-for-it.html STKI modifications
I am cool
and I am
happy
5P

Downtime, bugs are
un-acceptable
Downtime, bugs are bad
Many legacy technologies
Many business processes
New company-new technologies
Few business processes
Internet Companies
Eventually Consistant
Enterprise IT
Consistant (ACID)
However, internet companies are not enterprise IT
Conclusion – not
all new
technologies are
relevant to
Enterprise IT today
Still – try to get to
“there” are much
as possible
6P

How will the “new IT” look like?
Source: ids.snu.ac.kr/w/images/1/18/2011SS-03.ppt
Traditional IT Ideas from Internet companies
New IT?
7P

Don’t allow history to repeat itself – don’t say ‘No No No’
8P

Delivering service to end users
9P

Moving to browser based
applications
Social communication
& collaboration
WebRTC
Next generation ITSM
Steps for improve your End User service
Cloud networking
ibeacons
Mixed communication (VOIP,DATA,VIDEO)
ADC, Optimization
& automation
Browser-based Call
Center Software
10P

The Early Adopter “End User Environment” CIO:
IT
All PC applications
are html5 based
Chrome books or thin
clients running browser
can deliver all
applications
No software distribution
issues to end users. No
PC OS update issues
Some applications can
work off line
Input with natural
language and gestures
4K display
Some applications can
work off line
Top
Security
11P

The road to advanced end user delivery
End user delivery
VDI (and TS) is a viable option (not cheap, no promise for ROI)
Mid term era will have application delivered via web but some via VDITS
Users should develop all their new application via web technologies
Focus on central management, standardization
12P

Microsoft’s changing products, licensing and lifecycle policy
13P

Microsoft’s changing products, licensing and lifecycle policy
"Spartan" does not support legacy
technologies such as ActiveX and
Browser Helper Objects
Explorer will be offered side by side to
Spartan
Migration might be challenging
Office 365 is certainly delivering in
general with better end user
experience
Can (should) users resist strategic
moves by strategic (and sticky)
suppliers?
14P

IT Service Management
Traditional
ITSM
• Incident status
• SLA tracking
• Basic workflow
Advanced
ITSM
• Complex
workflows
• Knowledge base
• Email and voice
integration, chat
• Incident to
problem
automation
• Self service
(example
password reset)
Complete IT
Solution
• Change and asset
management
• IT portfolio
• CMDB
• IT service catalogue
• IT resource
management
• IT project
management and
SDLC
Enterprise
Service
management
(not just IT)
• Order repair at office
• Order supplies (pencils,
etc.)
• Automation employee
roll in and roll changes
15P

Which CIO type are you? Your winning hand for End User service is…
Conservative CIO
Mixed
communication
(VOIP,DATA,VIDEO)
IPV6 and ADC
Output management
systems
RFID- biometric
Advanced ITSM (KM,
Chat, ..)
VDI and TS
Modern CIO
Cloud based call
center
Unified
communication as
a service
Ibeacons
webRTC
Most apps in WEB
Early adopter CIO
Social networks with
wireless
Browser-based Call
Center Software
Social communication
& collaboration
Cloud networking
All delivery via Web
and new devices
(ChromePC)
16P

Devops
Chargeback
New sourcing models
Steps for Cloud
Hybrid cloud
Join the System, Network
DC and Storage forces
Infrastructure as Code
Public cloud for
specific projects
17P

Cloud Ready Architecture
There is not ‘private
cloud’ just
‘cloud ready architecture’
Cloud Ready example (in production):
• Automate environment construction including server, storage, network, dbms, middleware
• Self service portal for building environments.
• Moving environments from dev to test to prod automatically
Next near steps:
• Elasticity (at the environment level)
• Automatic procedure (restart environment each day at 24:00 because of memory leak)
• Devops = application deployment
• Next advanced steps: Hybrid clouds (private to public)
• APaaS is part of cloud initiative
Achieving:
• Standard environments. No human errors.
• Faster delivery cycle (from days to minutes)
• Chargebackshow back option
• Better capacity planning
18P

Cloud ready architecture solutions
• As Vmware dominates the IT enterprise vRealize automation will be a dominate cloud ready
architecture choice
• Legacy cloudautomation tools are also valid tactical option
• Openstack based solution has strategic advantage. Openstack API’s are already de facto standard.
19P

Public cloud for Modern CIO
• Building and maintaining private DC is not relevant anymore
• The co-location services market is booming : Bynet, Bezeqint, Veola, IBM, HP,
Med1, CCC, MalamTeam, Adgar and more are expanding and investing and
partnering for various cloud capabilities
• Hosted clients with burst out at their co-location provider
• Later, on premise clients will burst out to local cloud providers
Cloud
Architecture
local court jurisdiction
20P

Cloud migration principals
Cloud
migration
Elasticity is the main
factor not cost
“killing app”
technology not
available on premise
Look for brand new
projects
Encryption in public
cloud has many
options but requires
application
modification
Migrate existing project
only if it failed (not
delivering or delivering
with much too high
cost)
21P

Tools & Technologies to secure cloud services
• Encryption gateways
• Governance and compliance
• Identity gateway
SaaS
• Database monitoring and
encryption
• Dynamic and static analysis tools
PaaS
• Governance & compliance
• Encryption
• Multi cloud management
IaaS
Source:MosheFerber,CloudSecurityAllianceIsrael
34
AWS architecture options for key management:
• Server side encryption
• Key management service - KMS
• CloudHSM
• Server side encryption with Client Keys
P

Your DMZ is perfect for
public cloud
Extend your current
ESBETL for cloud
integration
DRaaS and
secondthird
copy of backup
Use service virtualization
CALisa, IBMGreenhat Public cloud
Try to utilize PaaS or MBaaS
Public cloud migration use cases and tips (order is not so important)
Testing, Development, Training
23P

Public cloud for the early adaptor CIO
Large percentage of
delivery is done via
“locked free” public
clouds
Regulation “can handle”
global clouds
Top cloud functionality
AWS Lambda – run your code in
response to events
Google Firebase – store and sync
data across multiple clients
Bare metal provisioned servers
Efficiency and portability in
Linux (cloud & on premise)
37P

Cloud service broker
•New (update) resource is needed (VM, Storage,
complete application provisioning)
•Many options around:
•Cloud service broker selects the option according to:
• Policy (ex. Production services for system of records with
sensitive data will be deployed internally)
• Price
• Availability of service and possible scale out needs
• Technical (latency, etc.)
• Relation to other services status
• Etc.
Cloud
architecture
environment 1
Cloud
architecture
environment 2
25P

Public cloud
•Wait!!
•Public cloud is not just ‘neat technology’ it’s a Sourcing
model
Source: http://www.robinhasanidea.com/2013/03/interpreting-fiction-step-1.html
26P

Which CIO type are you? Your winning hand for Cloud adoption…
Conservative CIO
Cloud Ready
architecture
Scale out to local
public providers
Migrate DMZ apps to
public cloud
(preferably to PaaS)
Modern CIO
Utilize new sourcing
models ‘cloud like’
Build new
applications in
cloud
Early adopter CIO
Hybrid cloud
Multi cloud
Utilize modern
cloud functionality
27P

Join the System, Network DC
and Storage forces
Utilize Modular DC
Software
defined storage
and networks
DC based on commodity HW
New DC
technology
SSD as general storage
The road to new Data Center technology
Converged Infrastructure
28P

Modular DC
• Many modular DC options (ISO vs, none-ISO, movable, etc.
• Time to market and better PUE main perceived advantages
• Example – 40 feet long modular DC (about 15 square meters) for 15 racks delivered in 6 months for
about 500K$
• First projects in Israel
29P

Lets talk about HW
Electronic manufacturing services (EMS) design, test, manufacture, distribute, and
provide return/repair services. EMS typically avoid ownership of IP (Intellectual Property).
Original design manufacturer (ODM) designs and manufactures a product as specified
and eventually rebranded by another firm for sale. ODM often retain some degree of IP
ownership of the products they design
Source:Wikipedia,https://www.ventureoutsource.com/contract-manufacturing/question/whats-the-difference-between-ems-and-odm/STKImodifications
Source: https://www.ventureoutsource.com/contract-manufacturing/top-10-ems-odm-reviews-ratings/
47P

•94% of global laptop computers are assembled by Taiwanese vendors, mostly
ODMs instead of OEMs.
•Majority of vendors just define the product positioning, rather than design.
•ODMs assume not only design and production but component procurement.
Source:http://www.prnewswire.com/news-releases/global-and-china-laptop-and-tablet-pc-industry-report-2011-2012-152524185.html
Laptops computers as example of HW market
31P

The big HW buyers: cloud providers
•In the big public clouds (Google, Amazon, Microsoft, etc.) HW is :
• Cheap because of scale
• Cheap because its “No name” or specific manufactured
• Cheap because SW automatically fail-over in case of HW failure. HW reliability is less
important issue.
• No need to repair HW (replace instead of fix)
• So – buying the HW with no warrantyservice
•This makes HW even cheaper!!
•As IT implements cloud ready and microservices architecture (auto. failover) and
ODM ecosystem matures (service for ‘no name HW’) White Boxes HW will become
the standard HW.
32P

Emerging option: converged infrastructure:
33P

About converged infrastructure
With ODM’s the traditional HW manufactures are forced
into converged infrastructure
The ‘traditional converged’ (sometime called ‘monolithic) vs.
‘hyper-converged debate
Converged infrastructure already provide value to customers for
specific projects (ex. VDI) especially time to market
One size does not fit all (example: if you want to grow only in storage) so clients
do not consider currently converged infrastructure as general DC environment
In long run converged infrastructure will grow until it will be
merged with SDDC (software based on standard HW)
34P

How can I strategically select between Cisco’s ACI vs. VMware's NSX?
When considering ACI and NSX always remember Openflow the open source SDN leader.
When (if) Openflow matures and spread both ACI and NSX will follow.
When your strategic supplier offers you his next generation technology, is it a strategic
decision?
No! You will do it anyway (as long as you stay with your strategic supplier and as long as the
supplier looks at this technology strategically)
This means:
If you will stay with VMWARE you will have NSX
If you will stay with CISCO you will have ACI
Recommendation:
• If you have specific need – test it. SDN
are not mature for all scenarios.
• Otherwise –wait for the market to
decide
• Defence, High-tech, Finance should
experiment Openflow
• Internal politics is key issue 35P

Storage evolution
Traditional
• Using SSD’s and
all flash array as
standard storage
• Buying SLA and
not storage
components
Conservative
• Considering
active-active
solutions and
storage
virtualization HW
• Using converged
or hyper-
converged
solutions
Modern
• Using SW storage
appliances on
dedicated HW
providing
replications, snaps,
backup etc.
• Considering all
data copies (prod,
drp, snaps, test,
dev, backup,
archive) as a whole
Early Adaptor
• No central storage! No
dedicated storage HW
• Just servers with SSD
• With SW storage part of
cloud architecture (no
dedicated HW for
storage)
• Using cloud storage for
backup and primary
• In general object storage
is preferred
36P

Pragmatic storage - The (near) future is SSD storage
•What with data reduction
(compression , dedup )?
•The future is FlashSSD
based storage
•HDD will still be used but
less
Source: http://wikibon.org/wiki/v/Evolution_of_All-Flash_Array_Architectures

Active Active Storage virtualization
Modern
IT
• Active Active storage solutions based on storage virtualization technologies gives
real value to users:
• Less downtime better RTO
• Higher flexibility between vendors easier migration
• However when software defined storage matures (VMWARE’s vvol, EMC’s
scaleioviper, HP’s storevirtual, Netapp ontap-edge, maxta, etc.) it will be the
preferred solution.
Source: http://cloudarchitectmusings.com/2013/01/03/datacenter-extension-with-vcloud-connector-2-0/

Opensource in delivery
• Open source (mainly community but also commercial) is de
facto standard in Internet companies and maturing fast in
enterprise IT as well:
• RDBMS
• NoSQL
• ESB
• WCM
• ETL
• ESM (enterprise system management)
• Microsoft has opensourced parts of .NET
39P

How to deal with HW - Infrastructure as code
It’s Code but not as we know it!
IT is another perspective to look at DevOps!!
Srouce:http://www.slideshare.net/jedi4ever/code-but-not-as-we-know-it-infrastructure-as-code
41P

Be the automator – not the automated
› Automation architect
› Business relationship mgr
› Service engineer
› Workflow/process experts
› Customer advocates
› Focus on services
Sustainers Innovators
› Systems administrator
› Network administrator
› Storage administrator
› Database administrator
› ITIL & tech zealots
› Focus on silo technologies
Source: Glenn O'donnell , Forrester 64P

SRAM
DRAM
• Flash
• Hard disk
On-chip
cache
Main
memory
Mass
storage
Speed
Costperbit
Universal memory obsoletes this hierarchy
Capacity
Emerging new radical HW technology - The Machine - HP
43P

Summary of Memristor attributes
• Memristor memory chip
• Nearly as fast as DRAM
• Lower power
• MUCH higher density
• Significantly cheaper/bit
• Nonvolatile
44P

HP the Machine
Memory = RAM DISKCPU Memory = MemistorCPU
The Machine requires new OS, new compilers etc.
45P

Another important technology – docker!
46

Invoking container machines
• LXC (Linux Containers) is an operating-system-level virtualization environment for running multiple
isolated Linux systems (containers) on a single Linux control host.
• # Download an ubuntu image
• $ sudo docker pull ubuntu
• #run a container
• $ sudo docker run -i -t ubuntu /bin/bash
• $ sudo docker ps # Lists only running containers
• $ sudo docker ps -a # Lists all containers$ sudo docker stop $JOB
• # Start the container
• $ sudo docker start $JOB
• # Restart the container
• $ sudo docker restart $JOB
• # KILL a container
• $ sudo docker kill $JOB
47

Googl’s kubernetes
• What do you do when you want Docker
containers managed across vast fleets of
servers and infrastructure? You use Docker
orchestration tools like Kubernetes
• Kubernetes is essentially a cluster
manager for Docker. With it, you can
schedule and deploy any number of
container replicas onto a node cluster and
Kubernetes will take care of making
decisions like which containers go on
which servers for you.
Source: http://www.centurylinklabs.com/what-is-kubernetes-and-how-to-use-it/

Behind the scene:
Source: http://blog.sciencelogic.com/tale-of-the-tape-aws-vs-google/12/2012

ITSM as center universe of the organizations
•ITSM (service desk) – Less incidents (systems are more reliable) and higher percent of
incidents are handled via self service.
•ITSM can provide:
• Password reset
• Identity management
• IT Asset management
• Change management in IT – all requests, CAB
• Configuration management infrastructure automation (puppet chef…)
• IT project management (related to change management)
• IT portfolio management (project priorities)
• IT vendor management. SLA management
• SDLC – software development lifecycle (related to project management)
• CMDB (related to change and asset) and IT process discovery
• Field service (2nd level)
• IT resource management (who is using what) –related to project and portfolio management
50

ESM (System management)
•Systems are more reliable in general
•Basically is part of Cloud (private, public, APaaS, SaaS)
•Since all environments are configured and built automatically – no need for CMDB
51

Which CIO type are you? Your winning hand for HW is…
Conservative CIO
Converged
infrastructure for
specific projects (ex.
VDI)
Use SDN for specific
need – vendor based
SSD for standard
storage
Active Active storage
solution if needed
Modern CIO
Converged for
mainstream DC
usage
SDN for broader
usage possibly with
Openflow
Openstack broader
usage
DevOps =
infrastructure as
code
Early adopter CIO
Just ‘no name hw’ for
all purposes =SDDC =
all automated
Mainly servers with SSD
Replace when break
Complete SW stack for
OS virtualization
network storage based
on Openstack
52P

Final word – join the System Storage and Network DC forces
Source: http://www.jta.org/2011/11/20/news-opinion/united-states/twinning-project-brings-muslims-and-jews-together
53P

Check for suitable process
Select HUB technology
Build initial IoCT based on Hub
Utilize fog computing for critical processes Magic
moments
54
Robust ESB infrastructure
The road to developing magic moment apps
Use HUB in relevant projects
P

Emotion-Oriented Systems: Magic Moments
55P

State Model
Card # 4: “The hub” basic technology – states and time based BRMS
56
New
Order
New
Order
New
Order
Condition /
TimeEvent
State 1 State 2
Source: Tibco
“The Hub”
P 56

The Hub conceptual architecture
57
Central state machine
that keeps status of
each object and
process events
Example * : If I get 4 ‘bad milkis’ calls from the same area in 4 hours retrieve tracks data. if
you get more 3 ‘bad milki calls in 3 hours send inspection right away. If get 10 in less than
5 hours more calls – report ceo
Management and
monitoring console
Events are
arriving to the
Hub from
outside
Give actions/alerts to other
systems
Rules and patterns* that
change the state,
aggregates, kill/create
events etc according to
events, objects and time.
Input from other
system regarding
rules, patterns
and objects
P

Two ways to look at the current interest in IoCT
We always had sensors connected to IT - in production floors. However:
• Effordable, capable
new sensors
• Better communications
• Big Data analytics and
cloud
Enterprise ‘got envy’
of consumer IoT
capabilities
P

Card # 5: IoT basic layers – the sensor
59Source: http://www.in-jet.dk/en/print.php?type=A&item_id=24
IoT tools and
technologies
P

Card # 5: IoT basic layers – the gateway/aggregator
IoT tools and
technologies
Sends the data from sensor (or many sensors) to the central hub
P

Card # 5: IoT basic layers – Fog computing
IoT tools and
technologies
Srouce: https://www.linkedin.com/pulse/20140822065109-246665791-fog-computing-from-the-center-to-the-edge-of-the-cloud
P

Card # 5: IoT basic layers – the center – Hub, Big Data Analytics
62
IoT tools and
technologies
Srouce http://www.techrepublic.com/article/datameer-4-0-enables-instant-visualization-in-big-data-analytics/

IoT communication – from sensors to gateway to hub
• Wifi
• Cellular
• Zigbee
• Bluethuth
LE
• More
arriving
63
Srouce:http://www.slideshare.net/aviiandevil7/security-protocols-platform-for-wsn-based-medical-applications-37863943
P

IoT standards
•Why having different standards?
•Different needs: Airplane vs. Street light
•Who is involved:
• Governments
• Industries
• Commercial vendors
64P

Which CIO type are you? Your winning hand for Magic moment technologies is…
65
Conservative CIO
Not sure if there are
appropriate business
processes
Modern CIO
Implementing hub
and IoT in initial
projects
Early adopter CIO
Many business
processes are based
on central Hub and
IoT
Systems of
engagement
Systems of
Immersion
P

Enforce DevOps metrics
Test Micro-Services
architecture. Increase
agile development
Deploy microservices
vis APaaS
continuous deployment
for web and mobile apps
Next gen.
development &
architecture
66
Build Devops
team. Tests
automation
The road for next generation development & architecture
Utilize APaaS for new projects
All Web Development
P

Card # 6: Modern Web Development – short history
67
Static web
pages
Web server is
“printing”
dynamic html
pages (cgi,
asp, jsp)
Some Ajax
but still web
server is
doing most
of the job
All is done at
the browser
level with JS
frameworks
Communicati
on to
backend via
Rest api
Web
development
Airbnb : how did we get there ?
P

Card # 6: Modern Web Development – short history
68
Static web
pages
Web server is
“printing”
dynamic html
pages (cgi, asp,
jsp)
Some Ajax but
still web server
is doing most of
the job
All is done at
the browser
level with JS
frameworks
Communication
to backend via
Rest api
Web
development
P

The thing with modern Web applications
69
Image processing in the cloud
Because it is REST
Web development is becoming the standard – even for internal PC applications
P

DBMS, Identity,
Messeging
APaaS
IDE, Source Control
Built in processes:
dev to test to prod
Public or private
clouds
Devops
Add ons
components:
BRMS, CEP, IoT
Card # 7: APaaS
APaaS
P 70

71
APAAS
Source: openshift redhat
Source: cloud foundry
P

72
Software development and deployment should be very smooth:
P

73
But in reality:Source:http://dltj.org/article/software-development-practice/
this without mentioning: build the environment, data
refresh, middleware updates, infrastructure updates,
different setup between dev/test/stage/prod etc.
P

Card # 8: Devops: continuous deployment
Enables the business:
Source: http://notafactoryanymore.com/tag/continuous-deployment/
Devops
74

Devops process example:
75
Srouce:https://programmaticponderings.wordpress.com/2013/11/13/building-a-deployment-pipeline-using-git-maven-jenkins-and-glassfish-part-2-of-2/
P

Special care: Devops and test automation
•Test automation is a must for complete DevOps
implementation and it is currently (with internal
politics) the main barrier to DevOps adoption.
•Testing automation ROI is tricky
• # of testing cycles
• Type of test (UI, none UI)
• Changes in the test cases during time. How does the testing automation
technology can support these changes without new coding
• Type and complexity of validation needed
• # of test cases, # of executions per cycle, test coverage,
• Cost of manual tester, automation programmer, tools, servers
Source: http://www.precisetestingsolution.com/wp-content/uploads/automated-300x283.gif
P 76

Card # 9: Microservices architecture: a different point of view
77
Source: http://martinfowler.com/articles/microservices.html
Micro-
services
P

Micros Services advantage and disadvantages
•Benefits:
• Small, easy to understand code base.
• Easy (and efficient) to scale
• Easy to throw away
• Easy to Deploy (goes with continuous
integrationsdeployment)
• Ability to use a different technology stack
• System resilience
•However –
•monitoring a complete process might be a
challenge,
•httpinvoking the service is a waste
•Conclusion – the debate on microservices is
on!!
78
Source: http://eugenedvorkin.com/seven-micro-services-architecture-advantages/ stki modifications
P

79
Agile development reminder
“Best fit” for first agile project:
Total staff of about 6-7 people
End users of project that can be approached once in 2-3 weeks
Not a new area (business and technology)
Bonus: incremental delivery is viable
P

Which CIO type are you? Your winning hand for modern dev&architecture is…
80
Conservative CIO
Only considering APaaS
Just build Devops team
Starting test automation
Agile development few
projects
Modern CIO
First steps in APaaS
mainly on premise
Several auto. tests
Many agile projects
Devops several
processes
Early adopter CIO
All agile development
Utilizing APaaS both on
premise and in cloud
Incorporate their own
services in APaaS
Advanced Devops
processes based on
complete auto. tests
Standardized on
microservices
architecture
P

Big data analytics use case - reference architecture
81
(a.k.a. – Groupon like application)

Source: http://www.slideshare.net/SessionsEvents/ml-conf-axp2013finalversion8am

Chose a first pilot use case
Build a data lake
Consider moving to vendors
offering
Build knowledge and check
open source tools
Plan
architecture
The road to big data analytics tools
Migrate to cloud big
data services
Big data
analytics tools

1nd card: Big data analytics tools
•Big Data – parallel, fault proof, scalable, on commodity HW,
(many) with open source offering
•Highest level of abstraction:
• Where you store the data – repository or storage
• How you deliver data to and from the repository
• How you analyze the data at the repository
•Many tools have integrated solutions
•Management and extra layers (quality, governance, policy etc.)
•Cloud offering of the above
84
Store
Deliver
Analyze
Big data
analytics
tools

Big data analytics tools: Storage
•Hadoop – the basics – traditionally by servers with hard drives
•NoSQL DBMS: – Mongo, Cassandra (can run MapReduce), Hbase (runs on HDFS)
•NoSQL DBMS on RAM: Redis, Aerospike
•Other storage with HDFS capabilities: EMC Isilon (emulating HDFS on OneFS)
•Other file system (and cloud services): Ceph, IBM’s GPFS, AWS S3
85
Store
Big data
analytics
tools

Big Data analytics tools – pushing the data to and from
•Hadoop based tools: Flume, Sqoop, BigSQL
•Traditional ETL tools that can integrate with Hadoop and other big data tools:
Informatica, IBM’s datastage, SAS, Oracle’s ODI, Talend, etc.
•Analytics tools with ETL capabilities: Pentahoo
•Real time integration (some with reasoning for streaming) : RabitMQ, Apache Kafka
86
Deliver
Big data
analytics
tools

Big Data analytics tools: Analyze
•Map Reduce
•SQL on Hadoop: Hive, Pig latin, Cloudera’s Impala, Drill, Pivotal’s Hawq, Apache
Phoenix (for Hbase)
•General analytical tools: Platfora, Panteho, RapidMiner, SAS, SAP, IBM, Oracle
87
Analyze
Big data
analytics
tools

Analytic tools: a mix
Chose a mix of 2-3 tools
Try to include at least one open source tool
•R (many statistical libraries)
•Rapidminer (UI)
•Knime (UI)
•Weka (UI)
•Python (programming)
•SAS
•SPSS-IBM
•SAP (Kxen)
•…
88
Analyze
Big data
analytics
tools

Big Data analytics – combined tools (storage and analytics)
•Search environments (based on Lucene) : ElasticSearch, SolR
•Streaming environments (some act like messaging with analytics) :
Apache Spark, IBM Streams, Samza, Apache Kafka
•Special purpose integrated tools (some are HW appliances): IBM’s
Netezza, Teradata and Teradata’s Rainstor, Vertica, Oracle’s Exadata,
89
Store Analyze
Big data
analytics
tools

Big Data analytics tools: Apache Spark
•Hadoop is the basic of big data (google ‘basics’)
•However Hadoop writes to hard drive after each operation – less
efficient for algorithms that uses the same data over and over
90
Source: cloudera
blog
Big data
analytics
tools

Elasticsearch
•Founded in 2012 by Israeli Shay Banon
•ElasticSearch is an open source RESTful search solution built on top of
Apache Lucene
•Near real time searches and analytics any type of document and diverse formats
•Schema-free
•Kibana: Exploration and visualization layer on top of Elasticsearch
91
Big data
analytics
tools

Cloud offering of the above
•Cloud offering of on premise solution (almost every tools is available on cloud as
IaaS).
•Cloud special offering – PaaS services of “on premise” tools and cloud big data
special purpose tools
92

General DBMS trends
•Oracle and MSSQL are the definite leaders
•But they are not “hot” anymore
• Open source RDBMS
• Not as easy as “next next” migration but a viable option
• NoSQL (different way of thinking by application designers!)
•PaaS for DBMS execution is an alternative
93

When do I migrate from my beloved legacy
•When critical mass if out of the legacy (example
75% of applications are out because of new
ERP CRM projects)
•When you hear the squeal:
• Availability issues (at your organization or with
neighbors )
• “Must have” functionality is missing
• The organization can not keep with current version of
technology
•In theory when legacy is too expensive (but only
in theory…)
94

95
Pini Cohen
Pini@stki.info

Delivery 2015 pini

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Delivery 2015 pini

Similar to Delivery 2015 pini (20)

More from Pini Cohen

More from Pini Cohen (17)

Recently uploaded

Recently uploaded (20)

Delivery 2015 pini