2. Introduction to enterprise data storage
• There are two types of digital information input and output data
• Users provide the input data, computers provide the output data.
• But a computer’s CPU can't compute anything or produce output data without the
user’s input.
• Users can enter the input data directly into computer.
• With data storage space, users can save data onto a device and it is saved even if the
device is powered down
• Also instead of manually entering data into a computer you can construct the
computer to pull data from a storage devices
• Computers can read input data from successes as needed and it can then create and
save the output to the same sources or other storage location
• Organizations and users require data storage to meet today’s high-level
computational needs like big data projects, Ai, ML, IOT.
• Huge data storage is required to protect against data loss due to disaster, failure or
fraud.
• To avoid data loss, you can also employ data storage as backup solutions.
3. Direct Area/Attached Storage (DAS)
• Is often in the immediate area and directly connected to
the computing machine accessing it
• Often only one
• machine connects to it
• For example the
memory card on your
phone or a hard disk
attached to your laptop.
• DAS can provide
decent local backup services, too but sharing is limited.
• DAS devices include
floppy disk optical disk compact discs(CDs) and digital video
discs(DVDs) hard disk drives(HDD) flash drives and solid-
state drives(SSD).
4. Network Based Storage
• Allows more than one computer to access storage over a network,
making it better for data sharing and collaboration
• It's off-site storage capability also makes it better suited for
backups and data protection
• The storage can reside anywhere while the machines accessing it
could be somewhere else.
• For example when you store your data on Google Drive it
is stored on the storage owned and operated by Google
• You don't have control over the storage itself you can just use the
storage quota that you're eligible for
• For accessing Google Drive storage, you need to have a network
connection available
• Two common networks based-storage types are network attached
storage(NAS) and storage area network(SAN).
5. NAS(Network Attached Storage)
• NAS – are storage devices which connect to a network.
• NAS is often a single device made-up of redundant storage containers or a
redundant array of independent disks(RAID).
NAS typically has the following characteristics
• Single storage device
• File storage system
• TCP/ IP Ethernet network
• Limited users
• Limited speed
• Limited expansion options
• Lower cost and easy setup
• NAS systems are a type of the file service device
• A NAS is connected to the LAN just like a file server
• Rather than containing a full blown OS, it typically uses a slim microkernel
specialized for handling only I/O requests such as NFS( Unix), GIFS/8MB(
windows 2000/NT) and NCP( Netware)
• Adding or removing a NAS system is like adding or removing any network code.
6. Storage Area Network(SAN)
SAN is a computer network which provides access to consolidated, block-level data storage.
• SAN storage is a network of a multiple devices of various types, including SSD and flash storage,
hybrid storage, hybrid cloud storage, backup
software add appliances and cloud storage.
SAN typically has the following characteristics
• Network of multiple devices
• Block storage system
• Fibre Channel network
• Optimized for multiple users
• Faster performance
• Highly expendable
• Higher cost and complex setup
• In SAN, data is presented from storage devices to
machine such that the storage looks like it is locally attached.
• This is achieved through various types of data visualization techniques
• SAN storage provides a high speed network storage.
• In some cases SANS can be so large that they span multiple sites as well as internal data centers and
the cloud.
7. Data storage management
• It refers to the software and processes that improve the performance of data
storage resources
• It may include network virtualization, replication, mirroring, security, compression,
deduplication, traffic analysis, process automation, storage provisioning, and
memory management.
• These processes help businesses store more data on existing hardware, speed up data
retrieval, prevent data loss, mid data retention requirements, and reduce it expenses.
• Storage management makes it possible to reassign storage capacity quickly as business
needs change.
• Storage management techniques can be applied to primary, backup or archived storage.
• Primary storage holds effectively or frequently accessed data; backup storage holds copies
of primary storage data for use in disaster recovery; an archive storage holds outdated or
seldom used data that must be written for compliance or business continuity.
• Storage provisioning is management technique asset assigns storage capacity to servers,
computers, virtual machines and other devices.
• It may use automation to allocate storage space in a network environment.
• Intelligent storage management uses software policies and algorithms to automate the
provisioning and de provisioning of storage resources, continuously monitoring data
utilization and rebalancing data placement without human intervention
8. Cloud file system
• File system is an approach to manage an operate files and data on a
storage systems.
• There are various file systems, such as NTFS, FAT32,EXT4,etc that
are commonly used today in operating systems.
• File systems typically provide mechanism for reading, writing,
modifying, deleting or organizing files in folders and directories
• Cloud file systems are specifically designed to be distributed and
operated in the cloud based environment.
• Files are typically stored in chunks on various storage
servers( devices) such a distributed nature of file systems makes it
fault tolerant and also high performance due to the possible
parallelism on file operations.
Architecture for cloud systems fall into two categories
1) Client server architecture
2) Cluster based architecture hi
9. Client server architecture
• In client server architecture, the file server host the file
system that can be mounted( attached) by the clients.
• One file server can host multiple file shares and each file
share can be mounted and operated by multiple clients.
• All file operations are then synchronized back to the file
server so that the other clients that have mounted the same
file share can get the updates as well.
• One example of such a file system is network file system or
NFS.
• Client server based file system architecture could be limited
due to dependency on the availability of the file server and
the need to synchronize the file operations periodically.
11. Cluster-based Architecture
• In a cluster based architecture, the file is broken into smaller
parts called chunks and each chunk is stored on the storage
server or devices.
• The chunks are redundantly stored on several
servers too withstand any fault and have high availability.
• This architecture does not depend upon a single server for
hosting the file system.
• The file system is distributed and provides parallelism that
significantly improves the scale and performance.
• This architecture is commuted to use today in the
cloud environment
for example Google file system, Amazon S3
13. Google file systems
• The Google file system (GFS) is a distributed file system (DFS)
for data-centric applications with robustness, scalability, and
reliability.
• GFS can be implemented in commodity servers to support large-
scale file applications with high performance and high reliability.
14. Characteristics and features of GFS
1. fault tolerant: if a few disks are corrupted, the data stored on
them can still be restored and used.
2. big data size: the file system can manage several petabytes
of data without crashing.
3 high availability: the data is highly available( copied to
several disks) and is present across various clusters of disk.
4 performance: the file system provides very high performance
for read and write from the disks.
5 resource sharing: the file system allows sharing disk
resources across users.
6 Google cloud services: there are quite few Google cloud
services, such as big table that are built on GFS2 also other
Google Apps such as Gmail and maps use GFS2 as well.
15. • Cluster-Based architecture
Redundantly stored
Direct access to chunks
Chunk Mapping
File
Chunk
1
Chunk
2
Chunk
3
Chunk
Server
1
Chunk
Server
2
Chunk
Server
3
Master Server
Application
16. Hadoop Distributed File System(HDFS)
• Hadoop is an open source framework based on java that
manages the storage and processing of large amounts of data for
applications.
• Hadoop comes with a distributed file system called HDFS.
• In HDFS data is distributed over several machines and
replicated to ensure their durability to failure and high
availability to parallel application.
• It is cost effective as it uses commodity hardware.
• It involves the concept of blocks, data nodes and node name.
Where to use HDFS?
• Very Large Files: Files should be of hundreds of megabytes,
gigabytes or more.
• Streaming Data Access: The time to read whole data set is
more important than latency in reading the first.
HDFS is built on write-once and read-many-times pattern.
• Commodity Hardware: It works on low cost hardware.
18. Features of HDFS
• Highly Scalable - HDFS is highly scalable as it can scale hundreds
of nodes in a single cluster.
• Replication - Due to some unfavorable conditions, the node
containing the data may be loss.
• So, to overcome such problems, HDFS always maintains the copy of
data on a different machine.
• Fault tolerance - The HDFS is highly fault-tolerant that if any
machine fails, the other machine containing the copy of that data
automatically become active.
• Distributed data storage - This is one of the most important
features of HDFS that makes Hadoop very powerful.
• Here, data is divided into multiple blocks and stored into nodes.
• Portable - HDFS is designed in such a way that it can easily
portable from platform to another.
19. Goals of HDFS
• Handling the hardware failure - The HDFS
contains multiple server machines.
Anyhow, if any machine fails, the HDFS goal is to
recover it quickly.
• Streaming data access - The HDFS applications
usually run on the general-purpose file system.
This application requires streaming access to their
data sets.
• Coherence Model - The application that runs on
HDFS require to follow the write-once-ready-many
approach.
So, a file once created need not to be changed.
However, it can be appended and truncate.
22. Bigtable
• Cloud Bigtable is a sparsely populated table that can scale
to billions of rows and thousands of columns, enabling you
to store terabytes or even petabytes of data.
• A single value in each row is indexed; this value is known
as the row key.
• Bigtable is a fully managed wide-column and key-
value NoSQL database service for large analytical
and operational workloads as part of the Google
Cloud portfolio.
25. High Level architecture of BigTable
• A big table implementation has three major component
1 One master server
• The master server is responsible for assigning tablets to tablet servers, detecting
the addition and expiration of a tablet servers, balancing tablet server load, and
garbage collection of files in GFS.
• in addition it handle schema changes such as table and column family creations.
2 Many tablet server
• each tablet server manages a set of tablets( typically you can have somewhere
between 10 to 1000 tablets per tablet server)
• tablet servers can be dynamically added or removed from a cluster
to accommodate changes in workloads.
• the tablet server handles read and write requests to the table that it has loaded,
and also splits tablets that have grown too large
3 Chubby
Is a highly available and persistent distributed lock service that manages
leases for resources and stores configuration information.
The service runs with five replicas, one of which is elected as a master to
serve request.
26. Features and characteristics of a big table
• Massive scale: big table is designed to store and
process massive( petabytes and more) volumes of
data.
• High performance: bigtable is designed to provide
very high performance with under less than 10
millisecond latency
• Run on commodity hardware: bigtable is
distributed in nature that allows it to run in parallel
on commodity hardware. you do not require any
specialized hardware to run big table.
• Flexibility: big table schema parameters let users
dynamically control whether to serve data out of
memory or from the disk. data is indexed using row
and column names that can be arbitrary strings.
27. HBase
HBase is an open-source non-relational distributed
database modeled after Google's Bigtable and written in
Java.
• It is developed as part of Apache Software Foundation's
Apache Hadoop project and runs on top of HDFS.
• Apache HBase is known as the Hadoop database.
• It is a column oriented, distributed and scalable big data
store.
• It is also known as a type of NoSQL database that is not
a relational database management system.
28. • HBase applications are also written in Java, built on top of
Hadoop and runs on HDFS.
• HBase is used when you need real-time read/write and random
access to big data.
• HBase is modeled based on Google's BigTable concepts.
• HBase is a column-oriented non-relational database
management system that runs on top of Hadoop Distributed
File System (HDFS).
• HBase provides a fault-tolerant way of storing sparse data
sets, which are common in many big data use cases.
29. Characteristics and features of a Hbase
1 Highly scalable: at your base is highly scalable and is designed to
handle petabytes of data. it can run on thousands of servers in parallel
2 High performance: HBase provides low latency reads and writes
to data and thus allowing for fast processing of massive datasets
3 No SQL database: this is not a traditional relational database. it is
a no SQL database that allows storing arbitrary key value pairs.
4 Fault tolerant: it reads splits data stored in tables across
multiple machines in the cluster and is built to withstand individual
machine failures in a cluster
5 API support: enterprise provides Java APIs using which you can
perform several operations on HBase data is stored in it
31. 1 HDFS: all HBase data is stored on HDFS.
2 Regions: Tables in HBase are divided horizontally by row key range in two regions. A
region contains all rows in the table between the regions start key and end key. regions are
assigned to the nodes in the cluster called region. Servers and these serve data for reads and
writes to the clients. A region server can serve around 1000 regions.
3 Master server( HMaster): The master server coordinates the cluster and performs
administrative operations such as assigning regions to the region servers and balancing the
load. it also performed other administrative operations such as creating and deleting the
tables.
4 Region Servers( HRegion): The region servers perform data processing. each region
server stores a subset of the data of each table. clients talk to region servers to access the data
in HBase.
5 Zookeeper: is the centralized service for maintaining configuration information, naming,
providing distributed synchronization and providing group services.
zookeeper maintains which region servers(HRegion) are alive and available and provides
server failure notification to the master server(HMaster) to coordinate administrative tasks
such as region assignment
• Establishing communication across the Hadoop cluster
• Maintaining configuration information
• Tracking Region Server and HMaster failure
• Maintaining Region Server information
32. DynamoDB
• DynamoDB is a fully managed NoSQL database service that
allows to create database tables that can store and retrieve any
amount of data.
• It automatically manages the data traffic of tables over
multiple servers and maintains performance.
• It also relieves the customers from the burden of operating and
scaling a distributed database.
• Hence, hardware provisioning, setup, configuration,
replication, software patching, cluster scaling, etc. is managed
by Amazon.
• With DynamoDB, you can create database tables that can store
and retrieve any amount of data and serve any level of request
traffic.
• It is one of the main components of Amazon.com, the
biggest e-commerce stores in the world.
33. Characteristics and Features of DynamoDB
• Scalable − Amazon DynamoDB is designed to scale. There is no need to
worry about predefined limits to the amount of data each table can store.
Any amount of data can be stored and retrieved. DynamoDB will spread
automatically with the amount of data stored as the table grows.
• Fast − Amazon DynamoDB provides high throughput at very low latency.
As datasets grow, latencies remain stable due to the distributed nature of
DynamoDB's data placement and request routing algorithms.
• Durable and highly available − Amazon DynamoDB replicates data over
at least 3 different data centers’ results. The system operates and serves data
even under various failure conditions.
• Flexible: Amazon DynamoDB allows creation of dynamic tables, i.e. the
table can have any number of attributes, including multi-valued attributes.
• Cost-effective: Payment is for what we use without any minimum charges.
Its pricing structure is simple and easy to calculate.
34. Architecture of Dynamo
Client Interface
Clients
Dynamo Node 1
Request Coordination
Membership and failure
detection
Local Persistence engine
Dynamo Node 2
Request Coordination
Membership and failure detection
Local Persistence engine
Dynamo Node 3
Request Coordination
Membership and failure
detection
Local Persistence engine
35. In Dynamo each storage node has three main software components that are implemented in Java
1 Request coordination
• The coordinator executes the read and write request on behalf of clients by collecting data
from one or more nodes( for reads) or storing data at one or more nodes( for writes).
• Each client requests result in the creation of a state machine on the node that received the
client request.
• The state machine contains all the logic for identifying the nodes responsible for a key,
sending the request, waiting for the responses, potentially doing retries, processing the replies
and packaging the response to the client. each state machine instance handles exactly 1
client request.
2 Membership and failure detection
• Failure detection in Dynamo is used to avoid attempts to communicate with unreachable peer
nodes.
• For the purpose of avoiding failed attempts at communication, a purely local mechanism of a
failure detection is used.
• For example node A may consider node B failed if node B does not respond to node A's
messages. Node A quickly discovers that node B is unresponsive when B fails to respond to
A’s message. Node A then uses alternate nodes to service request that map to B’s partitions.
Node A periodically retries node B to check for node B’s recovery. decentralized failure
detection protocols using simple gossip style protocol that enable each node in the system to
learn about the arrival of other nodes
36. 3 A local persistence engine
• Dynamo provides the flexibility to choose the
underlying persistent storage based on application
requirements.
• The main reason for designing a pluggable persistent
component is to choose the storage engine best suited
for an applications access patterns.
• For instance some database can handle objects typically
in the order of 10s of kilobytes whereas some can
handle objects of larger sizes.
• Applications choose Dynamos local persistence engine
based on their object size distribution
37. Google cloud data store
• Cloud storage is a cloud computing model that stores data on the internet through a
cloud computing provider who manages and operates data storage as a service.
• In this fast-moving world it become necessary to store data on the cloud storage.
• The biggest advantage of cloud storage is that we can store any type of data in
digital form on the cloud.
• Another advantage of cloud storage is that we can access data from anywhere
anytime on any device.
• There are many cloud storage providers such as, Google Drive, Dropbox,
OneDrive, iCloud, etc.
• They provide free service for limited storage but if you want to store beyond
the limit, you have to pay.
39. Cloud Storage
Cloud storage is a data deposit model in which digital information such as
documents, photos, videos and other forms of media are stored on virtual or
cloud servers hosted by third parties.
It allows you to transfer data on an offsite storage system and access them
whenever needed.
Cloud storage is a cloud computing model that allows users to save
important data or media files on remote, third-party servers.
Users can access these servers at any time over the internet. Also known as
utility storage, cloud storage is maintained and operated by a cloud-based
service provider.
40. Data Management in Cloud Storage
Cloud data management is the practice of storing a company’s
data at an offsite data center that is typically owned and overseen
by a vendor who specializes in public cloud infrastructure, such as
AWS or MicrosoftAzure.
Managing data in the cloud provides an automated backup
strategy, professional support, and ease of access from any
location.
41. Cloud Provisioning
Cloud provisioning means allocating a cloud service provider’s resources to
a customer.
It is a key feature of cloud computing.
It refers to how a client gets cloud services and resources from a provider.
The cloud services that customers can subscribe to include infrastructure-as-
a-service (IaaS), software-as-a-service (SaaS), and platform-as-a-service
(PaaS) in public or private environments.
42. Types of Cloud Provisioning
Network Provisioning: Network Provisioning in the telecom industry is a means of
referring to the provisions of telecommunications services to a client.
Server Provisioning: Datacenter’s physical infrastructure, installation, configuration of
the software, and linking it to middleware, networks, and storage.
User Provisioning: It is a method of identity management that helps us in keeping a
check on the access and privileges of authorization. Provisioning is featured by the
artifacts such as equipment, suppliers, etc.
Service Provisioning: It requires setting up a service and handling its related data.
43. Data Intensive Technology in Cloud Computing
Data Intensive Computing is a class of parallel computing which uses data parallelism
in order to process large volumes of data.
The size of this data is typically in terabytes or petabytes.
This large amount of data is generated each day and it is referred to Big Data.
Data intensive computing has some characteristics which are different from other
forms of computing. They are:
1. In order to achieve high performance in data intensive computing, it is necessary to
minimize the movement of data. This reduces system overhead and increases
performance by allowing the algorithms to execute on the node where the data
resides.
2. The data intensive computing system utilizes a machine independent approach
where the run time system controls the scheduling, execution, load balancing,
communications and the movement of programs.
44. Data Intensive Technology in Cloud Computing
Data intensive computing has some characteristics which are different from other
forms of computing. They are:
3. Data intensive computing hugely focuses on reliability and availability of data.
Traditional large scale systems may be susceptible to hardware failures,
communication errors and software bugs, and data intensive computing is designed
to overcome these challenges.
4. Data intensive computing is designed for scalability so it can accommodate any
amount of data and so it can meet the time critical requirements. Scalability of the
hardware as well as the software architecture is one of the biggest advantages of
data intensive computing.
45. Cloud Storage from LANs to WANs
Characteristics :
1. Computer power is elastic, when it can perform parallel operations. In
general, applications conceived to run on the peak of a shared-nothing
architecture are well matched for such an environment. Some cloud computing
goods, for example, Google’s App Engine, supply not only a cloud computing
infrastructure, but also an entire programs stack with a constrained API so that
software developers are compelled to compose programs that can run in a
shared-nothing natural environment and therefore help elastic scaling.
46. Cloud Storage from LANs to WANs
Characteristics :
2. Data is retained at an unknown host server. In general, letting go off data is
a threat to many security issues and thus suitable precautions should be taken.
The title ‘cloud computing’ implies that the computing and storage resources are
being operated from a celestial position.
The idea is that the data is physically stored in a specific host country and is
subject to localized laws and regulations. Since most cloud computing vendors
give their clientele little command over where data is stored, the clientele has no
alternative but to expect the least that the data is encrypted utilizing a key
unavailable with the owner, the data may be accessed by a third party without
the customer’s knowledge.
47. Cloud Storage from LANs to WANs
Characteristics :
3. Data is duplicated often over distant locations. Data accessibility and
durability is paramount for cloud storage providers, as data tampering can be
impairing for both the business and the
accessibility and durability are normally
organization’s reputation. Data
accomplished through hidden
replications. Large cloud computing providers with data hubs dispersed all
through the world have the proficiency to provide high levels of expected error
resistance by duplicating data at distant locations across continents. Amazon’s
S3 cloud storage service replicates data over ‘regions’ and ‘availability zones’ so
that data and applications can survive even when the whole location collapses.
48. Cloud Storage from LANs to WANs
Distributed Data Storage :
Distributed storage means are evolving from the existing practices of data
storage for the new generation of WWW applications through organizations like
Google, Amazon and Yahoo. There are some reasons for distributed storage
means to be favoured over traditional relational database systems encompassing
scalability, accessibility and performance. The new generation of applications
require processing of data to a tune of terabytes and even peta bytes. This is
accomplished by distributed services. Distributed services means distributed
data.
49. CouchDB
• CouchDB is a document-oriented database server.
• Couch is an acronym for ‘Cluster Of Unreliable Commodity Hardware’, emphasizing the
distributed environment of the database.
• CouchDB is designed for document-oriented applications for example, forums, bug following,
wiki, Internet note, etc. CouchDB is ad-hoc and schema-free with a flat address space.
• CouchDB aspires to persuade the Four Pillars of Data Management by these methods:
1. Save: ACID(Atomicity, Consistency, Isolation, and Durability). compliant, save efficiently
2. See: Easy retrieval, straightforward describing procedures, fulltext search
3. Secure: Strong compartmentalization, ACL, connections over SSL
4. Share: Distributed means
• A purchaser sees a snapshot of the data and works with it even if it is altered at the same time by a
distinct client.
• CouchDB actually has no apparent authentication scheme, i.e., it is in-built.
• The replication is distributed. A server can revise others once the server is made offline and data is
changed.
• If there are confrontations, CouchDB will choose a survivor and hold that as latest.
• Users can manually suspend this surviving alternative later.
• Importantly, the confrontation tenacity yields identical results comprehensively double-checking on
the offline revisions.
