The story of how data became big starts many years before the current buzz around big data.The history of Big Data as a term may be brief – but many of the foundations it is built on were laid many years ago. Now, let’s look at a detailed account of the major milestones in the history of sizing data volumes in the evolution of the idea of “big data” and observations pertaining to data or information explosion:
for getting the library resources fro the libraries entire world, the important tool is Library catalogues. every can browse all most all the world literature through WorldCat fro the INTERNET.
Many believe Big Data is a brand new phenomenon. It isn't, it is part of an evolution that reaches far back history. Here are some of the key milestones in this development.
for getting the library resources fro the libraries entire world, the important tool is Library catalogues. every can browse all most all the world literature through WorldCat fro the INTERNET.
Many believe Big Data is a brand new phenomenon. It isn't, it is part of an evolution that reaches far back history. Here are some of the key milestones in this development.
Disclaimer :
The images, company, product and service names that are used in this presentation, are for illustration purposes only. All trademarks and registered trademarks are the property of their respective owners.
Data/Image collected from various sources from Internet.
Intention was to present the big picture of Big Data & Hadoop
Data: A Timeline - How Data Came To Rule The WorldRibbonfish
Data: A Timeline - How Data Came To Rule The World
At Ribbonfish, we work with data all the time. Organisations use data to understand their customers, test new products, manage processes, and much more. This presentation looks at the timeline of how data came to such importance in this noisy world.
COMEX2017 Smart Talks by Amjid Ali , Muscat, Oman. Covering Introduction to big data, Big Data Definitions, Big Data Revolution, Big Data Timeline, Hadoop and Map Reduce covers importance of storage and DNA, Oceanstore 9000, Microsoft R, Spark,
In this paper, we discuss about the Big Data. We
analyze and reveals the benefits of Big Data. We analyze the
big data challenges and how Hadoop gives solution to it. This
research paper gives the comparison between relational
databases and Hadoop. This research paper also gives reason
of why Big Data and Hadoop.
General Terms
Data Explosion, Big Data, Big Data Analytics, Hadoop, Hadoop
Distributed File System, MapReduce
Disclaimer :
The images, company, product and service names that are used in this presentation, are for illustration purposes only. All trademarks and registered trademarks are the property of their respective owners.
Data/Image collected from various sources from Internet.
Intention was to present the big picture of Big Data & Hadoop
Data: A Timeline - How Data Came To Rule The WorldRibbonfish
Data: A Timeline - How Data Came To Rule The World
At Ribbonfish, we work with data all the time. Organisations use data to understand their customers, test new products, manage processes, and much more. This presentation looks at the timeline of how data came to such importance in this noisy world.
COMEX2017 Smart Talks by Amjid Ali , Muscat, Oman. Covering Introduction to big data, Big Data Definitions, Big Data Revolution, Big Data Timeline, Hadoop and Map Reduce covers importance of storage and DNA, Oceanstore 9000, Microsoft R, Spark,
In this paper, we discuss about the Big Data. We
analyze and reveals the benefits of Big Data. We analyze the
big data challenges and how Hadoop gives solution to it. This
research paper gives the comparison between relational
databases and Hadoop. This research paper also gives reason
of why Big Data and Hadoop.
General Terms
Data Explosion, Big Data, Big Data Analytics, Hadoop, Hadoop
Distributed File System, MapReduce
Check out this SlideShare to understand the challenges of BCBS 239 and learn ways to collect, measure, monitor and report on data to achieve better data integrity and data quality. Both G-SIBs and D-SIBS will learn how to help better govern their data.
A brief guide on how to optimize creatives on digital for better conversions and higher retention. These learnings are provided as a selection of takeaways that are readily implementable as part of your company's digital strategy.
These slides were originally presented by me at the Google Startup Bootcamp, New Delhi in October 2015. They have subsequently presented at Adwords conferences at Berlin and at several other events.
Moving Toward Big Data: Challenges, Trends and PerspectivesIJRESJOURNAL
Abstract: Big data refers to the organizational data asset that exceeds the volume, velocity, and variety of data typically stored using traditional structured database technologies. This type of data has become the important resource from which organizations can get valuable insightand make business decision by applying predictive analysis. This paper provides a comprehensive view of current status of big data development,starting from the definition and the description of Hadoop and MapReduce – the framework that standardizes the use of cluster of commodity machines to analyze big data. For the organizations that are ready to embrace big data technology, significant adjustments on infrastructure andthe roles played byIT professionals and BI practitioners must be anticipated which is discussed in the challenges of big data section. The landscape of big data development change rapidly which is directly related to the trend of big data. Clearly, a major part of the trend is the result ofthe attempt to deal with the challenges discussed earlier. Lastly the paper includes the most recent job prospective related to big data. The description of several job titles that comprise the workforce in the area of big data are also included.
Advanced live Final Year CSE Academic IEEE Major Big Data Projects in Hyderabad for Final Year Students of Engineering. Computer Science and Engineering latest major Big Data Projects."
As the time has grown the mankind has also grown in terms of technology,
living standard, the way of keeping records. In the initial years the
requirement of remembering information and keeping information intact
for future use was very less. But as the human developed himself, the
requirement of keeping information in an effective manner and in terms of
capacity becomes a crucial issue to be resolved. To solve this problem
several techniques came up in the meanwhile. And now when data capacity
has no limit and humans doesn’t want the pattern of storage and
processing to be a hurdle; BIG DATA came up as a rescue plan.The motive
of writing this paper is to understand the storage pattern of data
developed over the years.
As the time has grown the mankind has also grown in terms of technology, living standard, the way of keeping records. In the initial years the requirement of remembering information and keeping information intact for future use was very less. But as the human developed himself, the requirement of keeping information in an effective manner and in terms of capacity becomes a crucial issue to be resolved. To solve this problem several techniques came up in the meanwhile. And now when data capacity has no limit and humans doesn’t want the pattern of storage and processing to be a hurdle; BIG DATA came up as a rescue plan.The motive of writing this paper is to understand the storage pattern of data developed over the years.
SWOT of Bigdata Security Using Machine Learning Techniquesijistjournal
This paper gives complete guidelines on BigData, Different Views of BigData, etc.How the BigData is useful to us and what are the factors affecting BigData all the things are covered under this paper. The paper also contains the BigData Machine learning techniques and how the Hadoop comes into the picture. It also contains the what is importance of BigData security. The paper mostly covers all the main point that affect Big Data and Machine Learning.
Big data is the term that characterized by its increasing
volume, velocity, variety and veracity. All these characteristics
make processing on this big data a complex task. So, for
processing such data we need to do it differently like map reduce
framework. When an organization exchanges data for mining
useful information from this big data then privacy of the data
becomes an important problem. In the past, several privacy
preserving algorithms have been proposed. Of all those
anonymizing the data has been the most efficient one.
Anonymizing the dataset can be done on several operations like
generalization, suppression, anatomy, specialization, permutation
and perturbation. These algorithms are all suitable for dataset
that does not have the characteristics of the big data. To preserve
the privacy of the large dataset an algorithm was proposed
recently. It applies the top down specialization approach for
anonymizing the dataset and the scalability is increasing my
applying the map reduce frame work. In this paper we survey the
growth of big data, characteristics, map-reduce framework and
all the privacy preserving mechanisms and propose future
directions of our research.
Slides of my presentation at 9th Amirkabir Linux & Open-source Softwares Festival, about Big Data Computing Platforms and the rise of the so-called "Fast Data" phenomenon, and the architectures and state-of-the-art platforms for dealing with them.
JIMS IT Flash , a monthly newsletter-An Initiative by the students of IT Department, shares the knowledge to its readers about the latest IT Innovations, Technologies and News.Your suggestions, thoughts and comments about latest in IT are always welcome at itflash@jimsindia.org.
Visit Website : http://jimsindia.org/
Quontra solutions is your premier online IT educational destination in UK. It provides online IT courses like Selenium , Hadoop ,CCNA ,Cloud Computing ,Business Analyst and Many other IT courses. All the courses are designed by experienced instructors and designers. Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment there is an urgent need for IT professional to keep themselves in trend with Hadoop and Big Data technologies
.
Quontra Specialties :
***All the courses are designed by Experienced Instructors and Designers.
***. Trainers are not limited to the syllabus, they explain off –the-shelf content also.
*** 24X7 technical support team .
***Unlimited access to all recorded sessions ,available after every live class.
***Syllabus built based on professional standards and employer insights.
***Trainers are Certified Experts in their corresponding field and they bring years of industry experience in to the training classes
#RegReporting is a tough nut to crack! In his recent blog, Prakash Jalihal writes on why the process has become so complicated and explains how HEXANIKA can streamline Regulatory Reporting for banks using #BigData technology:
Traditionally, data integration has meant compromise. No matter how rapidly data architects and developers could complete a project before its deadline, speed would always come at the expense of quality. On the other hand, if they focused on delivering a quality project, it would generally drag on for months thus exceeding its deadline. Finally, if the teams concentrated on both quality and rapid delivery, the costs would invariably exceed the budget. Regardless of which path you chose, the end result would be less than desirable. This led some experts to revisit the scope of data integration. This write up shall focus on the same issue.
How Big Data helps banks know their customers betterHEXANIKA
Enterprises today mine customer data to ensure maximum success by targeting their products and solutions to the right audience. Let us have a look at how Big Data and Customer Analytics are helping businesses use their customer data for maximum benefits.
On April 14, 2016, the FCA (Financial Conduct Authority), one of the prime regulators in the United Kingdom, announced that it was preparing to launch a ‘Regulatory Sandbox’. It has started accepting applications from May 9, 2016 and successful applicants will be able to directly deploy their products in this sandbox for testing purposes. Quick to follow in their heels are regulators from Singapore and Australia, who are contemplating setting up a sandbox environment of their own.
So what is a sandbox and
High regulatory costs for small and mid sized banksHEXANIKA
Anecdotal evidence from bankers suggests that the cost of complying usually increases with new rules and regulations when large statutory changes are made to financial laws and rules of any country or region[1]. This burden increases significantly when such changes are made especially after a financial crisis. New regulations stemming from the financial crisis has cost the six largest U.S. banks $70.2 billion as of the end of last year[2]. Between the end of 2007 and the end of 2015, regulatory fines rose by more than 100% – or $35.5 billion- according to data from policy-analysis firm Federal Financial Analytics Inc. As per Federal Financial Analytics, the reporting costs come from a mix of requirements that are specific to these banks, e.g. particular capital surcharges that apply to banks with assets over $50 billion but impose the largest cost on the six biggest banks due to their size or risk
Automation is fast becoming a strategic business imperative for banks seeking to innovate whether through internal channels, acquisition or partnership. Implementing integrated automation solutions will enable banks to streamline the very tasks that are holding them back – removing manual intervention and ensuring that simple tasks are handled with speed and agility without error.
The financial industry has seen a sort of technological renaissance in the past couple of years. But this has also lead to a complex scenario where the problem has to be addressed from a global perspective; otherwise there arises the risk of running into an operational and technological chaos.
Some of the advantages of software automation are:
Regulatory Pain Points For Small And Medium Sized BanksHEXANIKA
Community and mid-sized banks are facing added regulatory burden post the implementation of the Dodd-Frank Act. This pressure has caused one in four local banks to close down since 2008. Let us review the key pain points that are the reason for small and medium sized banks to feel the regulatory pressure.
Understanding SAR (Suspicious Activity Reporting)HEXANIKA
To successfully identify the parties involved in any suspicious activity or money laundering/fraud processes, timely identification and reporting of the same is crucial. The Financial Crimes Enforcement Network (‘FinCEN’) has instituted various changes and updates to the requirements to enhance the process.
We will take a look at SAR requirements and challenges for financial institutions and focus on the solutions that can be enacted to stay compliant.
The process of data warehousing is undergoing rapidtransformation, giving rise to various new terminologies, especially due to theshift from the traditional ETL to the new ELT. Forsomeone new to the process, these additional terminologies and abbreviationsmight seem overwhelming, some may even ask, “Why does it matter if the L comesbefore the T?”
The answer lies in the infrastructure and the setup. Here iswhat the fuss is all about, the sequencing of the words and more importantly,why you should be shifting from ETL to ELT.
FATCA: why is it so difficult even after so many years?HEXANIKA
Under this law, all non-U.S. Foreign Financial Institutions (FFI’s) are required to search their records for U.S. persons and to report the assets and identities of such persons to the U.S. Department of the Treasury. Read the detailed report here:
The Volcker Rule: Its Implications and AftereffectsHEXANIKA
The Volcker Rule is named after Paul A. Volcker, chairman of the Federal Reserve during the 1980s and an elder statesman of the financial world. He acted as an advisor for President Obama in 2008 and was instrumental in the passing and creation of the Rule. It aims to prevent large banks from engaging in speculative trading activity with the idea that important banks support the economy by lending to consumers and businesses. We briefly explain the Volcker Rule, the challenges it brings to banks and how they can be addressed:
The Solvency II Directive, along with the Omnibus II Directive that amended it became a law on March 31, 2015. On April 1, 2015 the approval processes began, and after years of delay and negotiations, the Europe-wide capital regime for insurance companies came into effect on January 1, 2016. Insurers will have to comply with new rules and capital requirements of Solvency II across the EU.
Here is a short summary of what Solvency II is and how it’ll impact financial services institutions in the US (most of which are deemed to have fully or partly equivalent rules) along with EU.
A Review of BCBS 239: Helping banks stay compliantHEXANIKA
Although the challenge to comply with BCBS 239 is vital, the scope is immense. Now that the Jan 2016 deadline for the G-SIBs is up, the rule is expected to extend to other financial institutions and banks. The principles will also apply to all key internal risk management models including market, credit, and counterparty risk. Establishing the principle guidelines and putting core capabilities in place has its merits.
The clarity that effective risk data aggregation provides will help banks streamline their businesses, and can allow banks to make better judgments through more accurate risk analysis. Aggregated information across all channels will enable to provide comprehensive support and services to existing customers. The robust data framework also helps banks supervise and anticipate future problems, giving them a clear view for data analysis.
It can lead to gains in efficiency, reduce probability of losses and enhance strategic decision making, ultimate benefiting a bank’s profitability.
Dodd-Frank's Impact on Regulatory ReportingHEXANIKA
We previously analyzed how Dodd-Frank and how the new regulations have impacted large banks as well as midsize and small banks. This time, we will look at how the law meant to address one issue (avoid a financial meltdown similar to 2008) might have created other challenges for banks – the most important one that of regulatory reporting:
Regulatory impact on small and midsize banksHEXANIKA
Recent CCAR and Dodd-Frank Stress Tests have concluded that banks are severely under the lurch due to regulatory compliance issues, small and midsize banks being the most affected. We explain the impact in brief.
Even tho Pi network is not listed on any exchange yet.
Buying/Selling or investing in pi network coins is highly possible through the help of vendors. You can buy from vendors[ buy directly from the pi network miners and resell it]. I will leave the telegram contact of my personal vendor.
@Pi_vendor_247
how to sell pi coins in all Africa Countries.DOT TECH
Yes. You can sell your pi network for other cryptocurrencies like Bitcoin, usdt , Ethereum and other currencies And this is done easily with the help from a pi merchant.
What is a pi merchant ?
Since pi is not launched yet in any exchange. The only way you can sell right now is through merchants.
A verified Pi merchant is someone who buys pi network coins from miners and resell them to investors looking forward to hold massive quantities of pi coins before mainnet launch in 2026.
I will leave the telegram contact of my personal pi merchant to trade with.
@Pi_vendor_247
What website can I sell pi coins securely.DOT TECH
Currently there are no website or exchange that allow buying or selling of pi coins..
But you can still easily sell pi coins, by reselling it to exchanges/crypto whales interested in holding thousands of pi coins before the mainnet launch.
Who is a pi merchant?
A pi merchant is someone who buys pi coins from miners and resell to these crypto whales and holders of pi..
This is because pi network is not doing any pre-sale. The only way exchanges can get pi is by buying from miners and pi merchants stands in between the miners and the exchanges.
How can I sell my pi coins?
Selling pi coins is really easy, but first you need to migrate to mainnet wallet before you can do that. I will leave the telegram contact of my personal pi merchant to trade with.
Tele-gram.
@Pi_vendor_247
Poonawalla Fincorp and IndusInd Bank Introduce New Co-Branded Credit Cardnickysharmasucks
The unveiling of the IndusInd Bank Poonawalla Fincorp eLITE RuPay Platinum Credit Card marks a notable milestone in the Indian financial landscape, showcasing a successful partnership between two leading institutions, Poonawalla Fincorp and IndusInd Bank. This co-branded credit card not only offers users a plethora of benefits but also reflects a commitment to innovation and adaptation. With a focus on providing value-driven and customer-centric solutions, this launch represents more than just a new product—it signifies a step towards redefining the banking experience for millions. Promising convenience, rewards, and a touch of luxury in everyday financial transactions, this collaboration aims to cater to the evolving needs of customers and set new standards in the industry.
US Economic Outlook - Being Decided - M Capital Group August 2021.pdfpchutichetpong
The U.S. economy is continuing its impressive recovery from the COVID-19 pandemic and not slowing down despite re-occurring bumps. The U.S. savings rate reached its highest ever recorded level at 34% in April 2020 and Americans seem ready to spend. The sectors that had been hurt the most by the pandemic specifically reduced consumer spending, like retail, leisure, hospitality, and travel, are now experiencing massive growth in revenue and job openings.
Could this growth lead to a “Roaring Twenties”? As quickly as the U.S. economy contracted, experiencing a 9.1% drop in economic output relative to the business cycle in Q2 2020, the largest in recorded history, it has rebounded beyond expectations. This surprising growth seems to be fueled by the U.S. government’s aggressive fiscal and monetary policies, and an increase in consumer spending as mobility restrictions are lifted. Unemployment rates between June 2020 and June 2021 decreased by 5.2%, while the demand for labor is increasing, coupled with increasing wages to incentivize Americans to rejoin the labor force. Schools and businesses are expected to fully reopen soon. In parallel, vaccination rates across the country and the world continue to rise, with full vaccination rates of 50% and 14.8% respectively.
However, it is not completely smooth sailing from here. According to M Capital Group, the main risks that threaten the continued growth of the U.S. economy are inflation, unsettled trade relations, and another wave of Covid-19 mutations that could shut down the world again. Have we learned from the past year of COVID-19 and adapted our economy accordingly?
“In order for the U.S. economy to continue growing, whether there is another wave or not, the U.S. needs to focus on diversifying supply chains, supporting business investment, and maintaining consumer spending,” says Grace Feeley, a research analyst at M Capital Group.
While the economic indicators are positive, the risks are coming closer to manifesting and threatening such growth. The new variants spreading throughout the world, Delta, Lambda, and Gamma, are vaccine-resistant and muddy the predictions made about the economy and health of the country. These variants bring back the feeling of uncertainty that has wreaked havoc not only on the stock market but the mindset of people around the world. MCG provides unique insight on how to mitigate these risks to possibly ensure a bright economic future.
how to sell pi coins in South Korea profitably.DOT TECH
Yes. You can sell your pi network coins in South Korea or any other country, by finding a verified pi merchant
What is a verified pi merchant?
Since pi network is not launched yet on any exchange, the only way you can sell pi coins is by selling to a verified pi merchant, and this is because pi network is not launched yet on any exchange and no pre-sale or ico offerings Is done on pi.
Since there is no pre-sale, the only way exchanges can get pi is by buying from miners. So a pi merchant facilitates these transactions by acting as a bridge for both transactions.
How can i find a pi vendor/merchant?
Well for those who haven't traded with a pi merchant or who don't already have one. I will leave the telegram id of my personal pi merchant who i trade pi with.
Tele gram: @Pi_vendor_247
#pi #sell #nigeria #pinetwork #picoins #sellpi #Nigerian #tradepi #pinetworkcoins #sellmypi
If you are looking for a pi coin investor. Then look no further because I have the right one he is a pi vendor (he buy and resell to whales in China). I met him on a crypto conference and ever since I and my friends have sold more than 10k pi coins to him And he bought all and still want more. I will drop his telegram handle below just send him a message.
@Pi_vendor_247
BYD SWOT Analysis and In-Depth Insights 2024.pptxmikemetalprod
Indepth analysis of the BYD 2024
BYD (Build Your Dreams) is a Chinese automaker and battery manufacturer that has snowballed over the past two decades to become a significant player in electric vehicles and global clean energy technology.
This SWOT analysis examines BYD's strengths, weaknesses, opportunities, and threats as it competes in the fast-changing automotive and energy storage industries.
Founded in 1995 and headquartered in Shenzhen, BYD started as a battery company before expanding into automobiles in the early 2000s.
Initially manufacturing gasoline-powered vehicles, BYD focused on plug-in hybrid and fully electric vehicles, leveraging its expertise in battery technology.
Today, BYD is the world’s largest electric vehicle manufacturer, delivering over 1.2 million electric cars globally. The company also produces electric buses, trucks, forklifts, and rail transit.
On the energy side, BYD is a major supplier of rechargeable batteries for cell phones, laptops, electric vehicles, and energy storage systems.
USDA Loans in California: A Comprehensive Overview.pptxmarketing367770
USDA Loans in California: A Comprehensive Overview
If you're dreaming of owning a home in California's rural or suburban areas, a USDA loan might be the perfect solution. The U.S. Department of Agriculture (USDA) offers these loans to help low-to-moderate-income individuals and families achieve homeownership.
Key Features of USDA Loans:
Zero Down Payment: USDA loans require no down payment, making homeownership more accessible.
Competitive Interest Rates: These loans often come with lower interest rates compared to conventional loans.
Flexible Credit Requirements: USDA loans have more lenient credit score requirements, helping those with less-than-perfect credit.
Guaranteed Loan Program: The USDA guarantees a portion of the loan, reducing risk for lenders and expanding borrowing options.
Eligibility Criteria:
Location: The property must be located in a USDA-designated rural or suburban area. Many areas in California qualify.
Income Limits: Applicants must meet income guidelines, which vary by region and household size.
Primary Residence: The home must be used as the borrower's primary residence.
Application Process:
Find a USDA-Approved Lender: Not all lenders offer USDA loans, so it's essential to choose one approved by the USDA.
Pre-Qualification: Determine your eligibility and the amount you can borrow.
Property Search: Look for properties in eligible rural or suburban areas.
Loan Application: Submit your application, including financial and personal information.
Processing and Approval: The lender and USDA will review your application. If approved, you can proceed to closing.
USDA loans are an excellent option for those looking to buy a home in California's rural and suburban areas. With no down payment and flexible requirements, these loans make homeownership more attainable for many families. Explore your eligibility today and take the first step toward owning your dream home.
The European Unemployment Puzzle: implications from population agingGRAPE
We study the link between the evolving age structure of the working population and unemployment. We build a large new Keynesian OLG model with a realistic age structure, labor market frictions, sticky prices, and aggregate shocks. Once calibrated to the European economy, we quantify the extent to which demographic changes over the last three decades have contributed to the decline of the unemployment rate. Our findings yield important implications for the future evolution of unemployment given the anticipated further aging of the working population in Europe. We also quantify the implications for optimal monetary policy: lowering inflation volatility becomes less costly in terms of GDP and unemployment volatility, which hints that optimal monetary policy may be more hawkish in an aging society. Finally, our results also propose a partial reversal of the European-US unemployment puzzle due to the fact that the share of young workers is expected to remain robust in the US.
when will pi network coin be available on crypto exchange.DOT TECH
There is no set date for when Pi coins will enter the market.
However, the developers are working hard to get them released as soon as possible.
Once they are available, users will be able to exchange other cryptocurrencies for Pi coins on designated exchanges.
But for now the only way to sell your pi coins is through verified pi vendor.
Here is the telegram contact of my personal pi vendor
@Pi_vendor_247
how can i use my minded pi coins I need some funds.DOT TECH
If you are interested in selling your pi coins, i have a verified pi merchant, who buys pi coins and resell them to exchanges looking forward to hold till mainnet launch.
Because the core team has announced that pi network will not be doing any pre-sale. The only way exchanges like huobi, bitmart and hotbit can get pi is by buying from miners.
Now a merchant stands in between these exchanges and the miners. As a link to make transactions smooth. Because right now in the enclosed mainnet you can't sell pi coins your self. You need the help of a merchant,
i will leave the telegram contact of my personal pi merchant below. 👇 I and my friends has traded more than 3000pi coins with him successfully.
@Pi_vendor_247
Turin Startup Ecosystem 2024 - Ricerca sulle Startup e il Sistema dell'Innov...Quotidiano Piemontese
Turin Startup Ecosystem 2024
Una ricerca de il Club degli Investitori, in collaborazione con ToTeM Torino Tech Map e con il supporto della ESCP Business School e di Growth Capital
Turin Startup Ecosystem 2024 - Ricerca sulle Startup e il Sistema dell'Innov...
History of Big Data
1. A HISTORY OF BIG DATA
What is Big Data?
In essence, Big Data is a term for data sets that are so large or
complex that traditional data processing applications are
inadequate. It usually includes data sets with sizes beyond the
ability of commonly used software tools to capture, curate, manage
and process data within a tolerable elapsed time1
. The “size” of Big
Data is a constantly moving target, which doesn’t remain stable at
any given point of time. As per a recent report, its size ranges from
a few dozen terabytes to many petabytes of data.
The story of how data became big starts many years before the
current buzz around big data. About seventy years ago we
encountered the first attempts to quantify the growth rate in the
volume of data or what has popularly been known as the
Information Explosion (a term first used in 1941). The history of
Big Data as a term may be brief – but many of the foundations it is
built on were laid many years ago2
. Long before computers (as we
know today) were commonplace, the idea that we were creating an
ever-expanding body of knowledge ripe for analysis was popular
in academia.
Now, let’s look at a detailed account of the major milestones in the
history of sizing data volumes in the evolution of the idea of “big
data” and observations pertaining to data or information explosion:
1932 Skipping the important milestone of the population boom
would not do justice to the history of Big Data. Information
1
Source: Wikipedia
2
Link: https://www.linkedin.com/pulse/brief-history-big-data-everyone-should-read-bernard-marr
2. overload continued with the boom in the US population, the
issuing of social security numbers, and the general growth of
knowledge (research) which demanded more thorough and
organized record-keeping.
1941 Scholars began referring to this incredible expansion of
information as the “Information Explosion”. First referenced by
the Lawton Constitution (newspaper) in 1941, the term was
expanded upon in a New Statesman article in March 1964, which
referred to the difficulty of managing the amount of information
available.
1944 The first flag of warning on the growth of knowledge
storage and the retrieval problem came in 1944, when Fremont
Rider, a Wesleyan University Librarian estimated that American
university libraries were doubling in size every sixteen years. At
this growth rate, Rider speculated that the Yale Library in 2040
would have “approximately 200,000,000 volumes, which will
occupy over 6,000 miles of shelves… [requiring] a cataloging staff
of over six thousand persons.”
Schematic showing a general communication system3
.
3
Link: http://www.winshuttle.com/big-data-timeline/
3. 1948 Claude Shannon published “Shannon’s Information
Theory” which established a framework for determining the
minimal data requirements to transmit information over a noisy
(imperfect) channel. This was a landmark work that enabled much
of today’s infrastructure. Without this understanding, data would
be “bigger” than it is today.
1956 The concept of virtual memory was developed by German
physicist Fritz-Rudolf Guntsch as an idea that treated finite storage
as infinite. Storage, managed by integrated hardware and software
to hide the details from the user, permitted us to process data
without the hardware memory constraints that previously forced
the problem to be partitioned.
Information Overload4
4
Image source: Google images
4. 1961 Information Scientist, Derek Price, generalized Rider’s
findings to include almost the entire range of scientific knowledge.
The scientific revolution, as he called it, was responsible for the
rapid communication of new ideas as scientific information. This
rapid growth was in the form of new journals doubling every 15
years.
1963 In the early 1960’s, Price observed that the vast amount of
scientific research was too much for humans to keep abreast of.
Abstract journals, which were created in the late 1800’s as a way
to manage the increasing knowledge-base, were also growing at
the same trajectory and had already reached a “critical magnitude”.
They were no longer a storage or organization solution for
information.
1966 At around this time, the Centralized Computing Systems
entered the scene. Not only was information booming in the
science sector, it was booming in the business sector as well. Due
to the information influx in the 1960’s, most organizations began
to design, develop and implement centralized computing systems
that allowed them to automate their inventory systems.
1970 Edgar F. Codd, an Oxford-educated mathematician
working at the IBM Research Lab, published a paper showing how
information stored in large databases could be accessed without
knowing how the information was structures or where it resided on
the database. Until then, retrieving information required relatively
sophisticated computer knowledge, or even the services of
specialists —a time-consuming and expensive task. Today, most
routine data transactions—accessing bank accounts, using credit
5. cards, trading stocks, making travel reservations, buying things
online—all use structures based on relational database theory.
A relational database system5
1976 In the mid-1970’s, Materials Requirements Planning
(MRP) systems were designed as a tool to help manufacturing
firms to organize and schedule their information. Around the same
time, PC’s were gaining huge popularity gradually which marked a
shift in focus toward business processes and accounting
capabilities. Companies like Oracle and SAP were founded around
the same time.
5
Image source: IBM.com
6. 1983 As advancements in technology continued further, every
industry began to benefit from new ways to organize, store and
produce data.
Information Explosion6
1996 Digital storage became more cost-effective for storing
data than paper. Also, the boom in data brought more challenges to
ERP vendors. The need to redesign ERP products, including
breaking the barrier of proprietorship and customization, forced
vendors to embrace the collaborative business over the internet in a
seamless manner.
1997 The term “Big Data” was used for the first time in an
article by NASA researchers Michael Cox and David Ellsworth.
6
Image source: IBM.com
7. The pair claimed that the rise of data was becoming an issue for
current computer systems. This was also known as the “problem of
big data”.
The 4 V’s of Big Data7
.
1998 By the end of 90’s, many businesses began to believe that
their data mining systems were not up to the mark and still needed
improvements. Business workers were unable to get access to or
answer the data they needed from searches. Also, IT resources
were not so easily available at their disposal. So, whenever the
employees needed access, they had to call the IT department due to
lack of easily accessible information.
2001 The acronym SaaS (Software as a Service) first appeared
around this time. It basically means an “on-demand software”
7
Image source: IBM.com
8. delivery model which is licensed on a subscription basis and is
centrally hosted.
Software as a Service8
2005 SaaS companies began appearing on the scene to offer an
alternative to Oracle and SAP that was more focused on the
usability of the end user. Adding to this was the creation of a new
programming language named Hadoop. Free to download, use,
enhance and improve, Hadoop is 100% open source way pf storing
and processing data that enables distributed parallel procession of
huge amounts of data across inexpensive, industry-standard servers
that both store and process the data with extreme scalability.
2009 Business Intelligence became a top priority for Chief
Information Officers in 2009. Tim Berners, director of the World
Wide Web Consortium (W3C) was the first to use the term “linked
8
Image source: Google images
9. data” during a presentation on the subject at the TED 2009
conference. A set of best practices for using the Web to create
links between structured data is known as Linked Data.
2011 By this time, nearly all sectors in the US economy had at
least an average of 200 terabytes of stored data per company with
more than 1000 employees. The writers also estimated the
securities and investment industries led in terms of stored data per
organization. The scientists calculated that 7.4 exabytes of original
data were saved by enterprises and 6.8 exabytes by consumers in
2010 alone.
2012 After the launch of IPv6, identification and location
system for computers on the networks and traffic routes across the
internet became much faster. Technologically advanced features
such as ability to generate reports from in-memory databases
which provide faster and more predictable performance were also
on the rise. Businesses began to implement new in-memory
technology such as SAP HANA to analyze and optimize mass
quantities of data. Companies became ever more reliant on
utilizing data as a business asset to gain a competitive advantage,
with big data leading the charge as arguably the most important
new technology to understand and make use of in day-to-day
business.
How does Hexanika make use of Big Data?
Hexanika is a FinTech big data software company which has
developed an end-to-end solution for financial institutions to
address data sourcing and reporting challenges for regulatory
compliance. Hexanika’s innovative solution improves data quality,
keeps regulatory reporting in harmony with the dynamic regulatory
10. requirements and keeps pace with the new developments and latest
regulatory updates.
Hexanika’s unique Big Data deployment approach by experienced
professionals will simplify, optimize and reduce costs of
deployment. It strives to achieve this by following the process as
shown below:
Hexanika addresses Big Data using its unique product and
solutions. To know more about us,
see: http://hexanika.com/company-profile/
Feel free to get in touch with our experts to know more
at: http://hexanika.com/contact-us-big-data-company/
11. CONTACT US
USA
249 East 48 Street,
New York, NY 10017
Tel: +1 646.733.6636
INDIA
Krupa Bungalow 1187/10,
Shivaji Nagar, Pune 411005
Tel: +91 9850686861
Email: info@hexanika.com
Follow Us