1. A SEMINAR REPORT ON
BIG DATA, INTELLIGENT DEVICES USING
EMBEDDED SYSTEMS
SUBMITTED IN PARTIAL FULFILMENT FOR
AWARD OF THE DEGREE OF
BACHELOR OF TECHNOLOGY
IN
ELECTRONICS AND COMMUNICATION ENGINEERING
Session: 2014-2015
SUBMITTED BY:-
NAME: - VIPUL KAUSHIK
ROLL NO: - 1216431177
DEPARTMENT OF ELECTRONICS & COMMUNICATION
ENGINEERING
PRANVEER SINGH INSTITUTE OF TECHNOLOGY
KANPUR-208020
2. Abstract
It’s become clear in the past few years that few technologies live in a vacuum.
They’re more likely to be connected or related and sharing data, which is why
it’s always better to think of the enterprise holistically rather than in silos.
(Imagine how much more efficiently the federal government would run if it
stored one record of each citizen, rather than one at the Internal Revenue
Service, another at the Social Security Administration, another at the
Transportation Security Administration, and so on.)
Similarly, the close sibling of analytics, big data, also feeds off the Internet of
Things.
TThhee eexxpplloossiioonn ooff bbiigg ddaattaa iiss tteessttiinngg tthhee ccaappaabbiilliittiieess ooff eevveenn tthhee mmoosstt aaddvvaanncceedd
aannaallyyttiiccss ttoooollss.. IITT iiss cchhaalllleennggeedd bbyy tthhee sshheeeerr vvoolluummee,, vvaarriieettyy,, aanndd vveelloocciittyy ooff
tthhiiss fflloooodd ooff ccoommpplleexx,, ssttrruuccttuurreedd,, sseemmii ssttrruuccttuurreedd,, aanndd uunnssttrruuccttuurreedd ddaattaa——
wwhhiicchh aallssoo ooffffeerrss oorrggaanniizzaattiioonnss eexxcciittiinngg ooppppoorrttuunniittiieess ttoo ggaaiinn rriicchheerr,, ddeeeeppeerr aanndd
mmoorree aaccccuurraattee iinnssiigghhttss iinnttoo tthheeiirr bbuussiinneessss.. IIDDCC ddeessccrriibbeedd iinntteelllliiggeenntt ssyysstteemmss aass
tthhoossee eennaabblleedd wwiitthh hhiigghh--ppeerrffoorrmmaannccee mmiiccrroopprroocceessssoorrss,, ccoonnnneeccttiivviittyy,, aanndd hhiigghh
lleevveell ooppeerraattiinngg ssyysstteemmss.. EEmmbbeeddddeedd pprroocceessssoorrss nnoo lloonnggeerr ppeerrffoorrmm aass ffiixxeedd
ffuunnccttiioonnss tthhaatt ssttaanndd aalloonnee,, bbuutt ppaacckk ccoommppuutteerr ppeerrffoorrmmaannccee aanndd iinntteeggrraattiioonn iinnttoo
ddeevviicceess tthhaatt ffuueell iinntteelllliiggeenntt ssyysstteemmss.. CCoommbbiinneedd wwiitthh cclloouudd--bbaasseedd aapppplliiccaattiioonnss
aanndd aannaallyyttiiccss ccaappaabbiilliittiieess,, tthheessee iinntteelllliiggeenntt ssyysstteemmss ccaann ddeerriivvee vvaalluuee ffrroomm eeddggee
ddaattaa aanndd bbrriinngg tthhee IInntteerrnneett ooff TThhiinnggss ((IIooTT)) ttoo rreeaalliittyy..
FFoorr tthhee hhuunnddrreeddss ooff ppeettaabbyytteess ooff ddaattaa ggeenneerraatteedd bbyy iinntteelllliiggeenntt ssyysstteemmss aanndd
sseennssoorrss,, iitt’’ss ttoooo eexxppeennssiivvee aanndd iinneeffffiicciieenntt ttoo mmoovvee tthheemm ttoo aa cceennttrraall cclloouudd..
AAnn eexxppaannddiinngg wweeaalltthh ooff uubbiiqquuiittoouuss,, hheetteerrooggeenneeoouuss,, aanndd iinntteerr ccoonnnneecctteedd
eemmbbeeddddeedd ddeevviicceess iiss bbeehhiinndd mmoosstt ooff tthhee eexxppoonneennttiiaall ggrroowwtthh ooff tthhee ““BBiigg DDaattaa””
pphheennoommeennoonn.. MMeeaannwwhhiillee,, tthhee ssaammee eemmbbeeddddeedd ddeevviicceess ccoonnttiinnuuee ttoo iimmpprroovvee
iinn tteerrmmss ooff ccoommppuuttaattiioonnaall ccaappaabbiilliittiieess,, tthhuuss cclloossiinngg tthhee ggaapp wwiitthh mmoorree
ttrraaddiittiioonnaall ccoommppuutteerrss.. BBiigg ddaattaa iiss aa ggaammee cchhaannggeerr——aanndd iitt’’ss aallrreeaaddyy hheerree..
WWhhiillee mmoosstt ooff tthhee mmoommeennttuumm aarroouunndd bbiigg ddaattaa ttooddaayy iiss aarroouunndd ssoocciiaall mmeeddiiaa
ssoouurrcceess,, iitt iiss bbeelliieevveedd tthhaatt rreeaalliizziinngg tthhee pprroommiissee ooff bbiigg ddaattaa aannaallyyttiiccss mmuusstt
iinncclluuddee aa wwaayy ttoo hhaarrnneessss tthhee ppootteennttiiaall ooff bbiigg ddaattaa ffrroomm iinntteelllliiggeenntt ssyysstteemmss aanndd
sseennssoorrss..
HHeennccee tthhiiss ggiivveess rriissee oorr aa qquueesstt ttoo ddeevveelloopp aa ddeevviiccee wwiitthh vveerryy hhiigghh
ccoommppuuttaattiioonnaall ccaappaabbiilliittiieess ttoo mmaannaaggee bbiigg ddaattaa iinn aaccccoorrddaannccee wwiitthh mmiinniimmuumm hheeaatt
aanndd ootthheerr lloosssseess..
4. 1
Introduction
The Planet is growing a Central Nervous System. Humans, natural systems and
physical objects have generated vast amount of data. But until recently, even if
that data was captured, it was difficult and time consuming to use and analyze it
in an intelligent and smart way. There are now more things on the Internet
than people. The “Internet of Things” means physical objects are seamlessly
integrated into the information network, becoming active participants in
business processes while still protecting security and privacy.
This is one of the major trends in information technology that can drive
competitive advantage for enterprises. For modern enterprises, therefore, it is no
longer adequate to have individual technology solutions for various aspects of
their businesses ,back-end processing, business applications, data capture and
others — that can’t “talk” to one another. Those solutions need to be connected
in an intelligent way. The pressure for intelligent connection is also coming
from a second major trend — the “consumerization of IT,” in which customers,
collaborators, suppliers and employees are all demanding integration of their
multiple smart devices onto a uniform platform. According to International Data
Corp., the market for intelligent systems will grow from 19 percent of all
electronic system unit shipments in 2010 to more than one-third of all systems
by 2015. That kind of intelligent interconnectedness unlocks the
Power to gather “big data” like never before. Big-data analytics will be a
challenge — traditional databases and software are not equipped to handle it.
But its value in predicting customer needs, improving supply chain economics,
adapting business practices to user preferences and behavior, and helping an
enterprise position itself more effectively in the market are well worth the
investment. In short, the route to competitive advantage is an “intelligent
system” that connects devices that operate on the edge of a business to the core
infrastructure, in a continuous, two-way flow of information.
The route to an intelligent system starts with connecting “embedded systems”
that are now well established in both the business and consumer world. IDC
forecasts that the volume for embedded systems will outpace any other
mainstream system type, reaching 8.9 billion unit shipments by 2015.
The term Big Data refers not just to the explosive growth in data that almost
all organizations are experiencing, but also the emergence of data technologies
that allow that data to be leveraged. Big Data is a holistic term used to describe
the ability of any company, in any industry, to find advantage in the ever
increasingly large amount of data that now flows continuously into those
enterprises, as well as the semi-structured and unstructured data that was
previously either ignored or too costly to deal with.
5. 2
The problem is that as the world becomes more connected via technology, the
amount of data flowing into companies is growing exponentially and
identifying value in that data becomes more difficult - as the data haystack
grows larger, the needle becomes more difficult to find. So Big Data is really
about finding the needles – gathering, sorting and analyzing the flood of data to
find the valuable information on which sound business decisions are made.
The Big Data management, its analytics and flexibility along with IoT (Internet
of Things) evolving through embedded systems give rise to an ultimate solution
in form of an highly intelligent computing system known to be as
INTELLIGENT DEVICES.
6. 3
BIG DATA
Introduction to Big Data:
In 2004, Wal-Mart claimed to have the largest data warehouse with 500
terabytes storage (equivalent to 50 printed collections of the US Library of
Congress). In 2009, eBay storage amounted to eight petabytes (think of 104
years of HD-TV video). Two years later, the Yahoo warehouse totalled
170 petabytes(8.5 times of all hard disk drives created in 1995).Since the
rise of digitisation, enterprises from various verticals have amassed
burgeoning amounts of digital data,
capturing trillions of bytes of information about their customers, suppliers
and operations. Data volume is also growing exponentially due to the explosion
of machine-generated data (data records, web-log files, sensor data) and from
growing human engagement within the social networks. The growth of data will
never stop. According to the 2011 IDC Digital Universe Study, 130 exabytes
of data were created and stored in 2005. The amount grew to 1,227
exabytes in 2010 and is projected to grow at 45.2% to 7,910 exabytes in
2015.The growth of data constitutes the “Big Data” phenomenon – a
technological phenomenon brought about by the rapid rate of data growth
and parallel advancements in technology that have given rise to an
ecosystem of software and hardware products that are enabling users to
analyse this data to produce new and more granular levels of insight.
What is Big Data?
Big Data refers to datasets whose size are beyond the ability of typical
Database software tools to capture, store, manage and analyse. There is no
explicit definition of how big a dataset should be in order to be considered Big
data. New technology has to be in place to manage this Big Data phenomenon.
IDC defines Big Data technologies as a new generation of technologies and
architectures designed to extract value economically from very large volumes of
a wide variety of data by enabling high velocity capture, discovery and
analysis. “Big data is data that exceeds the processing capacity of conventional
database systems. The data is too big, moves too fast, or does not fit the
structures of existing database architectures, to gain value from these data,
there must be an alternative way to process it.”
7. 4
Characteristics of Big Data:
Volume is synonymous with the “big” in the term “Big Data”. Volume is
a relative term–some smaller-sized organisations are likely to have mere
gigabytes or terabytes of data storage as opposed to the petabytes or exabytes of
data that big global enterprises have. Data volume will continue to grow,
regardless of the organisation’s size. There is a natural tendency for companies
to store data of all sorts: financial data, medical data, environmental data and so
on. Many of these companies’ datasets are within the terabytes range today
but, soon they could reach petabytes or even exabytes.
Types of Data:
DATA
Structured Semi-Structured Unstructured
8. 5
Data can come from a variety of sources (typically both internal and external to
an organisation) and in a variety of types. With the explosion of sensors, smart
devices as well as social networking, data in an enterprise has become complex
because it includes not only structured traditional relational data, but also semi-
structured and unstructured data.
Structured data: This type describes data which is grouped into a relational
scheme (e.g. rows and columns within a standard database). The data
configuration and consistency allows it to respond to simple queries to arrive
at usable information, based on an organisation’s parameters and
operational needs.
Semi-structured data: This is a form of structured data that does not
conform to an explicit and fixed schema. The data is inherently self-
describing and contains tags or other markers to enforce hierarchies of records
and fields within the data. Examples include weblogs and social media feeds.
Unstructured data: This type of data consists of formats which cannot
easily be indexed into relational tables for analysis or querying. Examples
include images, audio and video files.
The velocity of data in terms of the frequency of its generation and delivery is
also a characteristic of big data. Conventional understanding of velocity
typically considers how quickly the data arrives and is stored, and how
quickly it can be retrieved. In the context of Big Data, velocity should
also be applied to data in motion: the speed at which the data is flowing. The
various information streams and the increase in sensor network deployment
have led to a constant flow of data at a pace that has made it impossible for
traditional systems to handle.
Why is Big Data important?
The convergence across business domains has ushered in a new economic
system that is re-defining relationships among producers, distributors, and
consumers or goods and services. In an increasingly complex world, business
verticals are intertwined and what happens in one vertical has direct
impacts on other verticals. Within an organisation, this complexity makes
it difficult for business leaders to rely solely on experience (or intuition)
to make decisions. They need to rely on data as structured, unstructured or
semi-structured - to back up their decisions. In the context of Big Data, the
amount of information for assimilation goes way beyond the human capacity.
Moreover, current data technologies have their limitations in processing huge
volumes and a variety of data within a reasonable time frame. To do so, new
technologies, more aptly known as “Big Data” technologies, must be in place.
Therefore from both the demand and supply perspectives, Big Data represents a
particularly “big” opportunity.
10. 7
Continuous growth of digital content
The increasing market adoption of mobile devices that are cheaper, more
powerful and packed with apps and functionalities is a major driver of the
continuing growth of unstructured data. It was an estimate that in 2012
smartphone shipment to reach 467.7 million units. By 2015, the expected
number of smartphones in the market will reach 1.1 billion.The market
adoption of tablets is also expected to increase significantly over the next
few years, further contributing to the growth of data. In 2012, shipment
of tablets is expected to reach 118.9 million tablets with the number
projected to rise to 369.3 million by 2015.
This market adoption of mobile devices and the prevalence of mobile
Internet will see consumers increasingly being connected, using social media
networks as the communication platform as well as the source of information.
The convergence of mobile device adoption, the mobile Internet and social
networking provides an opportunity for organisations to derive competitive
advantage through an efficient analysis of unstructured data. Businesses
that are early adopters of Big Data technologies and based their business
on data-driven decision-making were able to achieve greater productivity of up
to 5% or 6% higher than the norm.
Big Data technology early adopters such as Facebook, Linkedln, Walmart
and Amazon are good examples for companies that plan to deploy Big Data
analytics.
11. 8
Embedded Systems or Devices
Embedded Systems are Computer Systems that are designed for specific
applications. According to the definition from IEEE: “ an embedded computer
system is a computer system that is a part of a larger system and performs some
of the requirement of that system; for example a computer system used in an
aircraft or rapid transit system. Embedded systems are unlike a general purpose
computer system, they are always dedicated for a special application, so that the
developer could optimize them in order to reduce the size, power consumption
and the cost. In recent years, with the development of the embedded
technologies, embedded systems have been more and more widely used.
From the small mp3 player, microwave oven to the big plane, the devices with
embedded systems are all around us. It could be said, embedded systems have
already changed our way of life.
EMBEDDED DEVICES ACTS AND WILL ACT AS A MAJOR PART IN
DEVELOPING INTELLIGENT DEVICES
AN ESTIMATION BY AN RENOWNED IT GIANT- ORACLE
12. 9
EMBEDDED DEVICES PRESENT AND
FUTURE CAPABILITIES
It seems each major movement in digital technology is seen as its own
separate era. You may have heard people make big proclamations over the
last few years like, “This is the cloud era, ” or “This is the era of the
smart phone, ” and “We’re entering an era of big dat a and advanced
analytics. ” But it’s time to stop looking at these as disparate technologies
and see them as systems of discovery—where cloud, big data and the
internet of things collide. As this occurs, true innovation becomes possible.
According to a recent Gartner study, in 2020, more than 30 billion
connected devices will be in use. An IDC study reports that there will be
212 Billion devices or things connected to networks by 2020.Think about
the scope of that that for a moment . It makes me optimistic. I see so much
potential when it comes to what we can understand about one another and
the world around us. It also represents a significant business opportunity
f or large enterprises and start-ups alike—provided they can manage to
stay innovative. Certain reports says that sized applications of the Internet of
Things could have direct economic impact of $2. 7 trillion to $6.2 trillion per
year in 2025.
13. 10
But it’s not going to just happen on its own.
How do we get from today the relative early days of cloud, mobility,
wearable and interconnected smart devices to this hyper- connected world
of tomorrow?
This is where systems of discovery come into play.
Evolutions in mobility and cloud have spurred developments in the Int ernet
of Things. Embedded systems and sensors have been around f or some
time—as have business intelligence and analytics. But these “systems of
discovery, ” which are at the crossroads of cloud, mobile, data/ analytics
and embedded sensors, are now allowing
the industry to go to the next level with IoT.
MAJOR CAPITAL INVESTMENT
Start-ups focused on the “edge, ” which is comprised of embedded
systems and gateways (connectivity, messaging, security).
Start-ups focused on back- end cloud services (data services, analytic
services, security services etc).
These start-ups are introducing embedded systems in numerous services
that touch our daily life: televisions, refrigerators, trains, traffic lights
event he shoes on your feet can contain a sensor that tracks your
movement and provides valuable, usable data.
14. 11
THE INTERNET OF THINGS(IoT)
Humans, natural systems and physical objects have always generated vast
amounts of data. But until recently, even if that data was captured, it was
difficult and time-consuming to use and analyse it in an intelligent and useful
way.
The “Internet of Things” means physical objects are seamlessly integrated into
the information network, becoming active participants in business processes
while still protecting security and privacy.
However, many IoT examples are more focused on industrial applications,
including Fleet Management, Telematics, Smart Metering and Smart Grids,
Tele Health and so on. On the other hand we can see how the transformational
power of the IoT, and how it can leverage Big Data.
The following are areas where IoT can be beneficial:
Retail and Logistics
Retail and logistics is one key area where IoT is expected to have a
huge impact as an enabling technology. RFID (Radio Frequency
Identification) has been used successfully in logistics to track
containers, pallets and crates for some time now, primarily in closed
loop systems and mostly with high-value goods. The massive
investments in IoT technologies are promising to help reduce costs for
RFID and similar technologies, eventually making the tracking of
goods on an item-level a feasible business case. For retailers, this has
many advantages, including inventory accuracy, reduction of
administrative overhead, automated customer check-out processes and
a reliable anti-theft system.
Other emerging technologies are so-called “beacons”. These beacons
are indoor positioning systems, which can interact directly with
modern smart phones, e.g. using Bluetooth Low Energy (BLE). A
network of in-store beacons can identify the location of a customer in
a store and send them push notifications. For example, a user might
create a shopping list on their smart phone and share it with the store
app. Upon entering the store, the store app will display a map to the
customer, which highlights all the products on his shopping list. Every
time the customer gets close to a position where a group of products
from their shopping list is located, the app will notify them and make
a recommendation for a particular brand. At the check-out point, the
system could identify all the products in the shopping cart
automatically via RFID, create and confirm an invoice, and use the
smart phone to process the payment. The store’s inventory system is
automatically updated when the checkout process is complete.
15. 12
Manufacturing
“Industry 4.0”, “Smart Factory” and “Industrial Internet” – these are
some of the terms used to describe the social and technological
revolution that promises to change the current industrial landscape.
There are many examples discussed and explored in this area, from
leveraging IoT supply chain optimization to the modularization of
production lines with the help of intelligent products. One interesting
example that we explore here is related to the increasing use of hand-
held tools in manufacturing, e.g. for the assembly of automobiles,
airplanes, trains and ships. In recent years, these tools have become
more powerful (e.g. torque) and are now equipped with long lasting
batteries, enabling workers to use them without the limitations of
power cables or a fixed connection to an air compressor. This greatly
enhances flexibility, but also poses certain challenges from a
manufacturing process point of view, which can be addressed by
leveraging IoT capabilities. One of the key IoT concepts is the
development of intelligent, connected “edge” devices. One example
for such an IoT device is the Bosch Rexroth Nexo, a powerful nut
runner which is equipped with an on-board computer and wireless
connectivity. The on-board computer supports many aspects of the
tightening process, from configuration (e.g. which torque to use) to
creating a protocol of the work completed (e.g. which torque was
actually measured). In addition, the Nexo features a laser scanner for
component identification. By integrating such an intelligent edge
device into the IoT, very powerful services can be developed that can
help with supply chain optimization and modularizing the production
line. For example, these intelligent tightening tools can now be
managed by a central asset management application, which provides
different services:
•Basic services could include features like helping to actually locate
the equipment in a large production facility
•Geo-fencing concepts can be applied to help ensure only an approved
tool with the right specification and configuration can be used on a
specific product in a production cell.
MOBILITY
Rapid developments in mobility and automation are driving
significant transformations across many industries – especially in the
creation of new services and customer experiences. Telematics is a
prime example of an industry harnessing the power of mobile
connectivity and IoT. While the engine data bus has long served to
16. 13
aggregate sensor events for engine diagnostics or geo-location, each
new generation of vehicle is equipped with more sensors to extend
services into fuel efficiency, driver safety, theft prevention and more.
The availability of these sensors, coupled with the integration of data
to back-end enterprise systems via IoT application middleware is
creating entirely new business models. For example, auto-makers and
car rental companies have introduced new vehicle sharing offerings
enabling customers to locate cars using their smartphones, rent them
for a short time, and then park and return them anywhere within a
defined zone (e.g. DriveNow and Car2Go in Europe and ZipCar in
the US as well as OlaCabs in India). They may partner with local
property owners to provide secure parking for the vehicles, and in the
case of electric cars, with power companies for the location of the
charging points.
An increased interest in dynamic leasing contracts with flexible
mileage and duration terms, offering better flexibility. In a project
with a leading leasing provider, Bosch Software Innovations
implemented a connected fleet solution addressing many of these
challenges and therefore enabling the leasing provider to successfully
compete in this market. Leveraging an on-board, built-in unit and
remotely connecting this unit with a backend application allows the
fleet operator to get real-time information about fleet performance,
individual vehicle status, and so on. In the enterprise backoffice
systems, this information is consolidated and fed into the relevant
backend processes. Established approaches such as Business Process
Management (BPM) and Business Rules Management (BRM) provide
valuable tools and techniques to enable integration and automation –
for example to schedule preventative maintenance and repair. Web-
based access to vehicle information can be provided to the individual
car lessees. Other mobility providers, such as gas station operators, are
also integrated into the enterprise processes.
18. 15
This is how little data is transferred through Embedded Devices to a Cloud
Server ,as the little data is generated by multiple devices hence it will
exponentially increase and turns into Big Data and managed as an IoT Service
or application.
19. 16
THE POWER OF CLOUD ORIENTED
ARCHITECTURE
When talking about a cloud computing system, it's helpful to divide it into two
sections: the front end and the back end. They connect to each other through
a network, usually the Internet. The front end is the side the computer user, or
client, sees. The back end is the "cloud" section of the system.
The front end includes the client's computer (or computer network) and the
application required to access the cloud computing system. Not all cloud
computing systems have the same user interface. Services like Web-based e-
mail programs leverage existing Web browsers like Internet Explorer
or Firefox. Other systems have unique applications that provide network access
to clients.
On the back end of the system are the various computers, servers and data
storage systems that create the "cloud" of computing services. In theory, a cloud
computing system could include practically any computer program you can
imagine, from data processing to video games. Usually, each application will
have its own dedicated server.
20. 17
A central server administers the system, monitoring traffic and client demands
to ensure everything runs smoothly. It follows a set of rules
called protocols and uses a special kind of software called middleware.
Middleware allows networked computers to communicate with each other. Most
of the time, servers don't run at full capacity. That means there's unused
processing power going to waste. It's possible to fool a physical server into
thinking it's actually multiple servers, each running with its own independent
operating system. The technique is called server virtualization. By maximizing
the output of individual servers, server virtualization reduces the need for more
physical machines. If a cloud computing company has a lot of clients, there's
likely to be a high demand for a lot of storage space. Some companies require
hundreds of digital storage devices. Cloud computing systems need at least
twice the number of storage devices it requires to keep all its clients'
information stored. That's because these devices, like all computers,
occasionally break down. A cloud computing system must make a copy of all its
clients' information and store it on other devices. The copies enable the central
server to access backup machines to retrieve data that otherwise would be
unreachable. Making copies of data as a backup is called redundancy.
Real Picture Of Cloud Computing
There is nothing like cloud in reality only intensively joint or
HYPERCONNECTED DEVICES exist in reality.
In the above picture we can see a Hardware Analyst of Google holding a
collection of intensively connected Hard disks containing TBs and PBs of
useful data uploaded and retrieved by Google not in any cloud, these are just
storage media ranging from thousands to lakhs in number and also requires a
huge workforce to manage such big data utility hardware as well as its proper
21. 18
functioning each day. These Web Server Giants generate a lot of excessive heat
which is one of the major concerns.
GOOGLE & FACEBOOK DATA
CENTERS
22. 19
In order to manage such huge Data Center huge Data
Cooling or Server Thermal Stabilization Centers are
required in big sizes.
GOOGLE & FACEBOOK DATA
COOLING CENTERS
23. 20
CURRENT ENTERPRISE VIEW
An enterprise thinking about unlocking more hidden value by extending the
intelligence of its systems should consider the following:
1. Which systems in your business are still discrete, standalone? Which other
Systems can use the data being collected by that system?
2. What is the best way to connect the devices and systems in your company?
Are you able to cost-effectively connect your systems? What additional benefits
might that yield?
3. Can you remotely manage the devices at the edge of your system? Would you
benefit from being able to conduct remote updates to your devices or change
device configuration?
4. Are you using the data you already collect effectively? What other pieces of
intelligence can you leverage to drive your business forward?
5. Will you benefit from having more capacity and capability in your back end
available to you on demand?
6. If you were able to connect your company’s systems and extend that into “the
cloud,” what would you do?
24. 21
BIG DATA MANAGEMENT RELATED
PROBLEMS AND THEIR POTENTIAL
SOLUTIONS
To fully take advantage of visual analytics, organizations will
need to address several challenges related to visualization and
big data. Here we’ve outlined some of those key challenges –
and potential solutions.
Meeting the need for speed
In today’s hypercompetitive business environment, companies not
only have to find and analyze the relevant data they need, they must
find it quickly. Visualization helps organizations perform analyses and
make decisions much more rapidly, but the challenge is going through the
sheer volumes of data and accessing the level of detail needed, all at a
high speed. The challenge only grows as the degree of granularity
increases. One possible solution is hardware. Some vendors are using
increased memory and powerful parallel processing to crunch large
volumes of data extremely quickly. Another method is putting data in-
memory but using a grid computing approach, where many machines are
used to solve a problem. Both approaches allow organizations to explore
huge data volumes and gain business insights in near-real time.
Understanding The Data
It takes a lot of understanding to get data in the right shape so that
you can use visualization as part of data analysis. For example, if the
data comes from social media content, you need to know who the user is
in a general sense such as a customer using a particular set of products –
and understand what it is you’re trying to visualize out of the data.
Without some sort of context, visualization tools are likely to be of less
value to the user. One solution to this challenge is to have the proper
domain expertise in place. Make sure the people analyzing the data have
a deep understanding of where the data comes from, what audience will
be consuming the data and how that audience will interpret the
information.
Addressing Data Quality
Even if you can find and analyze data quickly and put it in the proper
context for the audience that will be consuming the information, the
value of data for decision-making purposes will be jeopardized if the data
is not accurate or timely. This is a challenge with any data analysis, but
when considering the volumes of information involved in big data
25. 22
projects, it becomes even more pronounced. Again, data visualization will
only prove to be a valuable tool if the data quality is assured. To address
this issue, companies need to have a data governance or information
management process in place to ensure the data is clean. It’s always best
to have a proactive method to address data quality issues so problems
won’t arise later.
Displaying Meaningful Results
Plotting points on a graph for analysis becomes difficult when
dealing with extremely large amounts of information or a variety of
categories of information. For example, imagine you have 10 billion
rows of retail SKU data that you’re trying to compare. The user trying to
view 10 billion plots on the screen will have a hard time seeing so many
data points. One way to resolve this is to cluster data into a higher-
level view where smaller groups of data become visible. By grouping
the data together, or “binning,” you can more effectively visualize the
data.
Dealing With Outliers
The graphical representations of data made possible by visualization can
communicate trends and outliers much faster than tables containing
numbers and text. Users can easily spot issues that need attention simply
by glancing at a chart. Outliers typically represent about 1 to 5 per cent of
data, but when you’re working with massive amounts of data, viewing 1
to 5 per cent of the data is rather difficult. How do you represent those
points without getting into plotting issues? Possible solutions are to
remove the outliers from the data (and therefore from the chart) or to
create a separate chart for the outliers. You can also bin the results to both
view the distribution of data and see the outliers. While outliers may not
be representative of the data, they may also reveal previously unseen
and potentially valuable insights.
Major Problem Is To Minimize Heat Generation Or
Develop Highly Thermally Stabilized Devices.
26. 23
For modern enterprises, therefore, it is no longer adequate to have individual
technology solutions for various aspects of their businesses — back-end
processing, business applications, data capture and others — that can’t “talk” to
one another. Those solutions need to be connected in an intelligent way. The
pressure for intelligent connection is also coming from a second major trend
— the “consumerization of IT,” in which customers, collaborators, suppliers
and employees are all demanding integration of their multiple smart devices
onto a uniform platform. According to International Data Corp., the market
for intelligent systems will grow from 19 per cent of all electronic system unit
shipments in 2010 to more than one-third of all systems by 2015. That kind of
intelligent interconnectedness unlocks the power to gather “big data” like never
before.
HENCE THIS GIVE RISE TO AN ADVANCED HIGH CAPABLE BOTH IN
TERMS OF HARDWARE AND SOFTWARE OR CALLED AS
“INTELLIGENT DEVICES”
27. 24
ROUTE TO AN INTELLIGENT SYSTEM
The route to an intelligent system starts with connecting “embedded systems”
that are now well established in both the business and consumer world. IDC
forecasts that the volume for embedded systems will outpace any other
mainstream system type, reaching 8.9 billion unit shipments by 2015.
Indeed, 98 per cent of computing devices are now embedded in electronic
equipment and machines, vastly out numbering those on the desktop. They
include credit and debit card readers; security and energy systems in homes;
sensors for traffic; cars that communicate their location in the event of a crash
or theft; and multiple “smart” devices that people use in their business and
personal lives.
For enterprises, just a partial list includes the mobile devices used by
employees; radio tags on products sold to consumers; the registers used to
record sales; the sensors that can do everything from tracking shipments to
making buildings operate more efficiently; and all the multiple devices that
serve as interfaces between a business process and a user, from digital signs to
ATMs, manufacturing controllers or X-ray machines. The data generated by
those systems can significantly impact business success, but only if it can be
accessed, understood, shared, and then acted upon — in a timely fashion and
from anywhere.
“Data is really the new currency for enterprises,” says Barb Edson, senior
director of marketing and business development for Microsoft’s Windows
Embedded. “And connectivity is what makes an intelligent system possible.”
KEY ATTRIBUTES OF AN INTEGLLIGENT DRIVEN
SYSTEM
• Connectivity
• Manageability
• User Experience
• Analytics
• Security
• Minimum Thermal Loss
28. 25
EFFECTIVE OR PROBABLE METHODS
TO DEVELOP AN INTELLIGENT
DEVICE
• Electronic equipment needs an efficient means of managing and
dispersing heat as systems continue to shrink in size.
• Changes in hardware fabrication & embedding.
Use of PGS
Heat is a killer for electronic systems. As applications get thinner and lighter,
this statement has never been more true, yet space and weight restrictions –
especially in portable mobile devices –mean that conventional solutions may
not be feasible. But it’s not just consumer products such as smartphones, tablets
and cameras that are at risk. Communications infrastructure equipment cram
more and more complex electronics systems into a small space; electric (Eco)
and hybrid cars require long-lasting, lightweight batteries; the advent of the
smart factory calls for greater levels of monitoring and control; solar panels
need to be able to cope with constant exposure to the sun; modern medical
devices must be able to be worn comfortably. All these examples require heat to
be transferred or dispersed effectively, using a minimum amount of space.
Pyrolytic Graphite Sheet (PGS) is a new, ultra-light graphite interface film
material, developed by Panasonic, which has a thermal conductivity up to five
times greater than copper. It is pliable enough to be cut and folded into complex
three dimensional shapes then simply stuck on to the heat source to diffuse the
heat or provide a path for heat to flow to a cold wall.
What is PGS?
Pyrolytic Highly Oriented Graphite Sheet is made of graphite with a structure
that is close to a single crystal. It is produced from polymeric film using a heat
de-composition process. The hexagonal crystal structure of graphite is arranged
uniformly in a horizontal 2D structure.
Features
PGS has a number of features which make it highly suitable as an easy-to-use,
space-saving, thermal management solution:
It is very thin – available in a range of thicknesses from 100µm down to
10µm – and has excellent thermal conductivity from 700 to 1950W/m.K
which is two to five times higher than copper and upto seven times better
than aluminium.
It is flexible and pliable so it can be easily cut and folded into a complex
shape. With a bend radius or 2mm, sheets can be bent through 180
29. 26
degrees more than 3000 times, and its thermal conductivity is unaffected
if sharp folds are avoided;
The material is very stable so it is resistant to environmental effects and
shows no deterioration with age;
PGS can provide some shielding to electromagnetic noise, providing a
simultaneous EMI and thermal solution.
• On the Software ends
Hadoop Map Reduce & Hadoop Distributed File System(HDFS)
Hadoop is a framework that provides open source libraries for distributed
computing using MapReduce software and its own distributed file
system, simply known as the Hadoop Distributed File System (HDFS). It
is designed to scale out from a few computing nodes to thousands of
machines, each offering local computation and storage. One of Hadoop's
main value propositions is that it is designed to run on commodity
hardware such as commodity servers or personal computers and has high
tolerance for hardware failure. In Hadoop, hardware failure is treated as a
rule rather than an exception. The HDFS is a fault-tolerant storage system
that can store huge amounts of information, scale up
30. 27
incrementally and survive storage failure without losing data. Hadoop
clusters are built with inexpensive computers. If one computer (or node)
fails, the cluster can continue to operate without losing data or
interrupting work by simply re-distributing the work to the remaining
machines in the cluster. HDFS manages storage on the cluster by
breaking files into small blocks and store duplicated copies of them
across the pool of nodes. The figure below illustrates how a data set is
typically stored across a cluster of five nodes. In this example, the entire
data set will still be available even if two of the servers have failed.
Compared to other redundancy techniques, including the strategies
employed by Redundant Array of Independent Disks (RAID) machines,
HDFS offers two key advantages. Firstly, HDFS requires no special
hardware as it can be built from common hardware. Secondly, it enables
an efficient technique of data processing in the form of MapReduce.
MapReduce
Most enterprise data management tools (database management systems) are
designed to make simple queries run quickly. Typically, the data is indexed so
that only small portions of the data need to be examined in order to answer a
query. This solution, however, does not work for data that cannot be indexed,
namely in semi-structured form (text files) or unstructured form (media files).
To answer a query in this case, all the data has to be examined. Hadoop uses the
MapReduce technique to carry out this exhaustive analysis quickly. MapReduce
is a data processing algorithm that uses a parallel programming implementation.
In simple terms, MapReduce is a programming paradigm that involves
distributing a task across multiple nodes running a "map" function. The map
function takes the problem, splits it into subparts and sends them to different
machines so that all the sub-parts can run concurrently. The results from the
parallel map functions are collected and distributed to a set of servers running
31. 28
"reduce" functions, which then takes the results from the sub-parts and re-
combines them to get the single answer.
Concept of Torrents & Parallel Computing
32. 29
CONCLUSION
The value of — indeed, the necessity for — intelligent systems can be summed
up in a single statistic: The first Internet experience for the next billion users
will not be primarily on a PC. They will use mobile devices to browse, social
network, entertain and conduct commerce. This means mobile apps will
continue to explode. Consumers will have them on multiple devices, and will
expect to have the same rich experience on every device they own. It takes
intelligent systems to deliver that experience, and those who do so will gain
significant competitive advantage. Capturing and holding that advantage will
take evolution and adaptation of the foundational attributes of identity, security
and connectivity. It will take an aggressive pursuit of the advanced attributes of
manageability, better user experience and analytics. The benefits of such
systems are just beginning to be realized, but they all flow from the ability to
collect, communicate and analyse data in ways that will predict customer
preferences and behaviour, giving the enterprise an ability to be proactive
instead of reactive.
As more and more businesses are discovering, data visualization is becoming
an increasingly important component of analytics in the age of big data.
The availability of new in-memory technology and high-performance analytics
that use data visualization is providing a better way to analyze data more
quickly than ever.Visual analytics enables organizations to take raw data and
present it in a meaningful way that generates the most value. Nevertheless,
when used with big data, visualization is bound to lead to some challenges.
If you’re prepared to deal with these hurdles, the opportunity for success with
a data visualization strategy is much greater.
33. 30
REFERENCES
A Whitepaper on “Device to Intelligence, IoT and Big
Data In Oracle” by Saint Kim Senior Director,
Enterprise Architect, Oracle Korea.
IoT and Big Data-A Joint Whitepaper by Bosch
Software Innovations and MongoDB October 2014.
Five Big Data Challenges by SAS
Intel IT Center- Vision Paper Distributed Data
Mining and Big Data, Intel’s Perspective on the Data
at the edge.
Intelligent Devices and Smart Journeys- Peter Whale,
Director Product Management, Qualcomm
Technologies ,Inc.
“Intelligent Systems- Connecting Data, Devices and
People” by Microsoft Embedded Community.
Big Data Cloud Database & Computing from
www.qubole.com
“Solving Thermal Management Challenges In A
Minimum Space” by Panasonic
“What is Big Data” by Edd Dumbill
Websites include
www.google.co.in
www.wikipedia.com
34. 31
ACKNOWLEDGMENT
I would like to express my sense of gratitude and my sincere thanks to the
following persons who have made the completion of this Seminar Report
possible:
MMrr.. SSaannkkhhaayyaann CChhaakkrraabbaarrttyy, Seminar Incharge, for his vital encouragement,
support, constant reminders and much needed motivation.
MMrr.. RRaagghhvveennddrraa SSiinngghh ,HOD, Department of Electronics & Communication
Engineering for his understanding and assistance.
My sincere thanks to all the Faculty Members of the Department of Electronics
& Communication Engineering for assisting in the collection of the topics for
my Seminar Report.
Most especially to my family and friends without their co-operation, this
training as well as my project work would not have been a success.
3rd
year, EC DEPTT.
35. 32
DECLARATION
I hereby declare that this submission is my own work and that to the best of my
knowledge and belief, it contains no matter previously published or written by
another neither person nor material which to a substantial extent has been
accepted for the award of my other degree or diploma of the university or other
institution of Higher learning except where due acknowledgement has been
made in the text.
Signature:
Name:
Roll No.:
Date:
36. 33
CERTIFICATE
Certified that this is a bona fide record of the seminar work entitled
BIG DATA,INTELLIGENT DEVICES USING EMBEDDED
SYSTEMS
Done by
VIPUL KAUSHIK
Of the VI semester, ELECTRONICS & COMMUNICATION ENGINEERING
in the year 2014-15 for the partial fulfillment of the requirements to the award
of degree of Bachelor of Technology in ELECTRONICS & COMMUNICATION
ENGINEERING of PRANVEER SINGH INSTITUTE OF TECHNOLOGY.
MR. SANKHAYAN CHAKRABARTY MR.RAGHVENDRA SINGH
SEMINAR –IN-CHARGE (HOD, EC DEPT.)
