challenges of bigdata

•Download as PPTX, PDF•

0 likes•39 views

This seminar presentation discusses the challenges of big data. It defines big data as extremely large and complex data sets that cannot be processed by traditional data processing software. The presentation outlines issues with big data like high data volume, velocity, and variety. It also discusses challenges in areas such as privacy and security, data access and sharing, analytics, human resources, and technical fault tolerance and scalability. Solutions proposed include using specialized software frameworks and new technologies.

Course name : Seminar(IT331)
Course co-ordinator : Prof. Zankhana Shah
Presented By : Ravi Vaniya (15IT036)
Sanat Dhobi (15IT027)

Synopsis
 Introduction
 Issues
 Challenges

Introduction
 What is big data ?
Big data is a term for data sets that are so large or complex that
traditional data processing application software is inadequate to deal
with them.
 Problem for existing DBMS…
 Solutions..
1. Use software/framework
2. Some new technology

Issues
1. Data Volume
2. Data Velocity
3. Data Variety
4. Data Value
5. Data Complexity
Example : Google map

Challenges
 Privacy and security
 Data access and sharing of information
 Analytical challenges
 Human resource and manpower
 Technical – Fault tolerance , Scalability , Quality of data

References ...
 Official website : University of Washington
 Official website : Microsoft Inc.
 https://en.wikipedia.org/wiki/Big_data
 For images : Google images
 https://www.slideshare.net/HarshMishra3/harsh-big-data-seminar-
report

Various portals have been developed to provide an easy way to discover and access public research data sets from various organizations. Data sets are made available with descriptive metadata based on common (e.g., OGC, CUAHSI, FGDC, INSPIRE, ISO, Dublin Core) or proprietary standards to facilitate better understanding and use of the data sets. Provenance descriptions may be included as part of the metadata and are specified from a data provider’s perspective. These can include, for example, different entities and activities involved in a data creation flow, such as sensing platforms, personnel, and data calculation and transformation processes. Moving beyond the provider-centric descriptions, data provenance may be complemented with forward provenance records supplied by data consumers. The records may be gathered via a user-driven feedback approach. The feedback information from data consumers gives valuable insights into application and assessment of published data sets. This might include descriptions about a scientific analysis in which the data sets were used, the corrected version of an actual data set or any discovered issues and suggestions concerning the quality of the published data sets. Data providers might then use this information to handle erroneous data and improve existing metadata, their data collection and processing methods. Contributors can use the feedback channel to share their scientific analyses. Data consumers can learn more about data sets based on other people’s experiences, and potentially save time by avoiding the need for interpreting or cleaning data sets. The goals of the study are to capture feedback from data users on published research data sets, link this to actual data sets, and finally support search and discovery of research data using feedback information. This paper reports preliminary results addressing the goals. We provide a summary of current practices on gathering feedback from end-users on research data portals, and discuss their relevance and limitations. Examples from the Earth Science domain on how commentaries from data users might be useful in practice are also included. Then, we present a data model representing key aspects of user feedback. We propose a system architecture to gather and manage feedback from end-users. We describe how the core PROV model may be used to represent the provenance of user feedback information. Technical solutions for linking feedback to existing data portals are also specified.

Understanding Metadata Needs when Migrating DAMS

Ayla Stein

This study identifies and explores metadata needs associated with migrating to a new Digital Asset Management System (DAMS). Drawing upon results from a 2014 survey, titled “Identifying Motivations for DAMS Migration: A Survey,” this paper analyzes survey questions related to metadata, interoperability, and digital preservation. Results indicate three distinct metadata needs for future system development, including support for multiple or all metadata schema, metadata reuse, and digital object identifiers. While some of these needs resemble long-standing conversations in the professional literature, others offer new areas for system development moving forward. Co-author and presenter: Santi Thompson

Open Innovation and Semantic Web

Milan Stankovic

Pedro-Combining rapid data modelling and ontology servicesAIMS (Agricultural Information Management Standards)

Studying Public Medical Images from Open Access Literature and Social Network...

Institute of Information Systems (HES-SO)

2012.10 - DDI Lifecycle - Moving ForwardDr.-Ing. Thomas Hartmann

Special Issue on: "Advances in Neural Network Models and Algorithms for Data ...

gerogepatton

Recent advances in storage, hardware, information technology, communication, and networking have resulted in increasingly large and complex heterogeneous data. This has powered the demand to extract useful and actionable insights from data in an automatic, reliable and scalable way. Neural networks are widely used learning machines with powerful learning ability and adaptability, which have achieved remarkable performance in the data analytical tasks, such as computer vision, face/speech recognition, video surveillance, document summarization, distributed and/or real-time resource allocation, etc. Recently there is a surge of research activities devoted to theoretical development of scalable and robust learning models on deep neural networks, neurodynamics, combinatorial optimization techniques.

WWW2014: Long Time No See: The Probability of Reusing Tags as a Function of F...

Dominik Kowald

Data quality information is essential to facilitate reuse of Earth science data. Recorded quality information must be sufficient for other researchers to select suitable data sets for their analysis and confirm the results and conclusions. In the research data ecosystem, several entities are responsible for data quality. Data producers (researchers and agencies) play a major role in this aspect as they often include validation checks or data cleaning as part of their work. It is possible that the quality information is not supplied with published data sets; if it is available, the descriptions might be incomplete, ambiguous or address specific quality aspects. Data repositories have built infrastructures to share data, but not all of them assess data quality. They normally provide guidelines of documenting quality information. Some suggests that scholarly and data journals should take a role in ensuring data quality by involving reviewers to assess data sets used in articles, and incorporating data quality criteria in the author guidelines. However, this mechanism primarily addresses data sets submitted to journals. We believe that data consumers will complement existing entities to assess and document the quality of published data sets. This has been adopted in crowd-source platforms such as Zooniverse, OpenStreetMap, Wikipedia, Mechanical Turk and Tomnod. This paper presents a framework designed based on open source tools to capture and share data users’ feedback on the application and assessment of research data. The framework comprises a browser plug-in, a web service and a data model such that feedback can be easily reported, retrieved and searched. The feedback records are also made available as Linked Data to promote integration with other sources on the Web. Vocabularies from Dublin Core and PROV-O are used to clarify the source and attribution of feedback. The application of the framework is illustrated with the CSIRO’s Data Access Portal.

Mootnz13 Moodle Analytics

NetSpot Pty Ltd

The robustness of distance education

Martin Weller

Agile Curation Poster

Josh Young

RDAP 15: Data Curation Issues for Electronic Theses and Dissertations at Corn...

ASIS&T

Utilitarian aggregation of open data

UK Science & Innovation Network in India

Seeing the Wood For the Trees The Challenges of mobile device logging

jontrinder

Kwon Ph.D. Dissertation 2016

Karl Kwon, Ph.D.

Ontologies for Emergency & Disaster Management

Stephane Fellah

How to improve the acceptance of AltMetrics

uherb

Ontologies for Crisis Management: A Review of State of the Art in Ontology De...

streamspotter

Open and Collaborative Software for Digital Pathology

William Baird

Dataset-driven research to improve TEL recommender systemsKatrien Verbert

Data Services at the American University of Beirut: Challenges and Opportunit...

Dalal Rahme

Adopting data8 at a two year college

Ava Meredith

Supporting Big Data, Open Data, Data Analytics and Data Science

Simon Price

An Image Retrieval System for Digital Pathology

William Baird

The influence of social status on consensus building in collaboration networks

Ilire Hasani-Mavriqi

In this paper, we analyze the influence of social status on opinion dynamics and consensus building in collaboration networks. To that end, we simulate the diffusion of opinions in empirical collaboration networks by taking into account both the network structure and the individual differences of people reflected through their social status. For our simulations, we adapt a well-known Naming Game model and extend it with the Probabilistic Meeting Rule to account for the social status of individuals participating in a meeting. This mechanism is sufficiently flexible and allows us to model various situations in collaboration networks, such as the emergence or disappearance of social classes. In this work, we concentrate on studying three well-known forms of class society: egalitarian, ranked and stratified. In particular, we are interested in the way these society forms facilitate opinion diffusion. Our experimental findings reveal that (i) opinion dynamics in collaboration networks is indeed affected by the individuals’ social status and (ii) this effect is intricate and non-obvious. In particular, although the social status favors consensus building, relying on it too strongly can slow down the opinion diffusion, indicating that there is a specific setting for each collaboration network in which social status optimally benefits the consensus building process. Paper: http://www.know-center.tugraz.at/cms/wp-content/uploads/2015/08/ASONAM_2015_Paper.pdf Reference: Hasani-Mavriqi I, Geigl F, Pujari SC, Lex E, Helic D (2015) The influence of social status on consensus building in collaboration networks. In: Proceedings of the 2015 IEEE/ACM international conference on advances in social networks analysis and mining 2015, ASONAM ’15ACM, New York, NY, USA, pp 162–169 http://dl.acm.org/citation.cfm?id=2808887&CFID=851242713&CFTOKEN=32991930

How do data analysts work with big data and distributed computing frameworks.pdf

Soumodeep Nanee Kundu

The era of big data has ushered in a new paradigm for data analysis, presenting unique challenges and opportunities. This article delves into the world of big data analytics and explores how data analysts work with distributed computing frameworks to handle large and complex datasets. We'll discuss the concept of big data, the challenges it poses, and the evolution of distributed computing frameworks. Furthermore, we'll dive into the role of data analysts, their skills and tools, and the practical applications of big data analytics. By the end of this article, readers will have a comprehensive understanding of how data analysts leverage distributed computing frameworks to extract valuable insights from vast datasets.

Thoughts on Knowledge Graphs & Deeper Provenance

Paul Groth

What's hot

Linked Data Quality Assessment – daQ and Luzzu

jerdeb

Trends in machine learning (1)

Viswanath Shenoi

Using Feedback from Data Consumers to Capture Quality Information on Environm...

Anusuriya Devaraju

Mootnz13 Moodle Analytics

NetSpot Pty Ltd

The robustness of distance education

Martin Weller

Agile Curation Poster

Josh Young

RDAP 15: Data Curation Issues for Electronic Theses and Dissertations at Corn...

ASIS&T

Utilitarian aggregation of open data

UK Science & Innovation Network in India

Seeing the Wood For the Trees The Challenges of mobile device logging

jontrinder

Kwon Ph.D. Dissertation 2016

Karl Kwon, Ph.D.

Ontologies for Emergency & Disaster Management

Stephane Fellah

How to improve the acceptance of AltMetrics

uherb

Ontologies for Crisis Management: A Review of State of the Art in Ontology De...

streamspotter

Open and Collaborative Software for Digital Pathology

William Baird

Dataset-driven research to improve TEL recommender systemsKatrien Verbert

Data Services at the American University of Beirut: Challenges and Opportunit...

Dalal Rahme

Adopting data8 at a two year college

Ava Meredith

Supporting Big Data, Open Data, Data Analytics and Data Science

Simon Price

An Image Retrieval System for Digital Pathology

William Baird

The influence of social status on consensus building in collaboration networks

Ilire Hasani-Mavriqi

What's hot (20)

Linked Data Quality Assessment – daQ and Luzzu

Trends in machine learning (1)

Using Feedback from Data Consumers to Capture Quality Information on Environm...

Mootnz13 Moodle Analytics

The robustness of distance education

Agile Curation Poster

RDAP 15: Data Curation Issues for Electronic Theses and Dissertations at Corn...

Utilitarian aggregation of open data

Seeing the Wood For the Trees The Challenges of mobile device logging

Kwon Ph.D. Dissertation 2016

Ontologies for Emergency & Disaster Management

How to improve the acceptance of AltMetrics

Ontologies for Crisis Management: A Review of State of the Art in Ontology De...

Open and Collaborative Software for Digital Pathology

Dataset-driven research to improve TEL recommender systems

Data Services at the American University of Beirut: Challenges and Opportunit...

Adopting data8 at a two year college

Supporting Big Data, Open Data, Data Analytics and Data Science

An Image Retrieval System for Digital Pathology

The influence of social status on consensus building in collaboration networks

Similar to challenges of bigdata

How do data analysts work with big data and distributed computing frameworks.pdf

Soumodeep Nanee Kundu

Thoughts on Knowledge Graphs & Deeper Provenance

Paul Groth

Machine Learning On Big Data: Opportunities And Challenges- Future Research D...

PhD Assistance

Machine Learning (ML) is rapidly used in a variety of applications. It has risen to prominence in recent years, owing in part to the emergence of big data. When it comes to big data, ML algorithms have never been more promising. Big data allows machine learning algorithms to discover finer-grained patterns and make more timely and precise predictions than ever before; however, it also poses significant challenges to machine learning, such as model scalability and distributed computing. Learn More: https://bit.ly/2RB1buD Contact Us: Website: https://www.phdassistance.com/ UK NO: +44–1143520021 India No: +91–4448137070 WhatsApp No: +91 91769 66446 Email: info@phdassistance.com

Agent-Based Problem Solving Methods In Big Data Environment

Laurie Smith

Big-data analytics: challenges and opportunities

台灣資料科學年會

One View of Data Science

Philip Bourne

Top (10) challenging problems in data mining

Ahmedasbasb

ccna course 2S Sridhar

Identifying and analyzing the transient and permanent barriers for big data

sarfraznawaz

Auspiciously, big data analytics had made it possible to generate value from immense amounts of raw data. Organizations are able to seek incredible insights which assist them in effective decision making and providing quality of service by establishing innovative strategies to recognize, examine and address the customers’ preferences. However, organizations are reluctant to adopt big data solutions due to several barriers such as data storage and transfer, scalability, data quality, data complexity, timeliness, security, privacy, trust, data ownership, and transparency. Despite the discussion on big data opportunities, in this paper, we present the findings of our in-depth review process that was focused on identifying as well as analyzing the transient and permanent barriers for adopting big data. Although, the transient barriers for big data can be eliminated in the near future with the advent of innovative technical contributions, however, it is challenging to eliminate the permanent barriers enduringly, though their impact could be recurrently reduced with the efficient and effective use of technology, standards, policies, and procedures.

Big data privacy issues in public social media

Supriya Radhakrishna

Research paper on big data and hadoop

Shree M.L.Kakadiya MCA mahila college, Amreli

Twdatasci cjlin-big data analytics - challenges and opportunities

Aravindharamanan S

Seminar presentation

Klawal13

The concept of big data has been endemic within computer science since the earliest days of computing. “Big Data” originally meant the volume of data that could not be processed (efficiently) by traditional database methods and tools. In a broad term Big data can be describe as a data sets which is so large or complex that can not be handle by traditional data processing applications. More especially unstructured or semi-structured data.

CyberistructureLab Southwest

Big data processing using - Hadoop Technology

Shital Kat

Meeting the NSF DMP Requirement June 13, 2012

IUPUI

Big data deep learning: applications and challenges

fazail amin

Big Data: Issues and ChallengesHarsh Kishore Mishra

Top 10 Ways to Make Your Digital Content AccessibleD2L Barry

Experience from 10 months of University Linked Data

Mathieu d'Aquin

Similar to challenges of bigdata (20)

How do data analysts work with big data and distributed computing frameworks.pdf

Thoughts on Knowledge Graphs & Deeper Provenance

Machine Learning On Big Data: Opportunities And Challenges- Future Research D...

Agent-Based Problem Solving Methods In Big Data Environment

Big-data analytics: challenges and opportunities

One View of Data Science

Top (10) challenging problems in data mining

ccna course 2

Identifying and analyzing the transient and permanent barriers for big data

Big data privacy issues in public social media

Research paper on big data and hadoop

Twdatasci cjlin-big data analytics - challenges and opportunities

Seminar presentation

Cyberistructure

Big data processing using - Hadoop Technology

Meeting the NSF DMP Requirement June 13, 2012

Big data deep learning: applications and challenges

Big Data: Issues and Challenges

Top 10 Ways to Make Your Digital Content Accessible

Experience from 10 months of University Linked Data

Recently uploaded

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Generating a custom Ruby SDK for your web service or Rails API using Smithy

g2nightmarescribd

Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Recently uploaded (20)