Apache Zeppelin and Spark for Enterprise Data Science
•Download as PPTX, PDF•
2 likes•514 views
Apache Zeppelin and Spark are turning out to be useful tools in the toolkit of the modern data scientist when working on large scale datasets for machine learning. Zeppelin makes Big Data accessible with minimal effort using web browser based notebooks to interact with data in Hadoop. It enables data scientists to interactively explore and visualize their data and collaborate with others to develop models. Zeppelin has great integration with Apache Spark that delivers many machine learning algorithms out of the box to Zeppelin users as well as providing a fast engine to run custom machine learning on Big Data. The talk will describe the latest in Zeppelin and focus on how it has been made ready for the enterprise. With support for secure Hadoop clusters, LDAP/AD integration, user impersonation and session separation, Zeppelin can now be confidently used in secure and multi-tenant enterprise domains.
Report
Share
Report
Share
Recommended
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
Running Zeppelin in Enterprise
Apache Zeppelin has become a popular way to unlock the value of data lake due to its user interface and appeal to business users. These business users ask their IT department for access to Zeppelin. Enterprise IT department want to help their business users but they have several enterprise concerns such as enterprise security, integration with their corporate LDAP/AD, scalability and multi-user environment, integration with Ranger and Kerberos. This session will walk through enterprise concerns and how these concerns can be handled with Zeppelin.
Speaker
Simon Elliston Ball, Director Product Management, Cyber Security, Hortonworks
Getting involved with Open Source at the ASF
The document discusses getting involved with open source projects at the Apache Software Foundation. It provides an overview of the ASF, how it works, and how to contribute to Apache projects. The key points are:
- The ASF is a non-profit organization that oversees hundreds of open source projects and thousands of volunteers. Popular projects include Hadoop, Hive, and Pig.
- To get involved, individuals can start by joining mailing lists, reviewing documentation, reporting issues, and submitting code patches. More responsibilities come with becoming a committer or PMC member.
- Projects follow an open development process based on consensus. Voting on decisions helps include contributors from different time zones.
- Contributing is rewarding
Building a Smarter Home with Apache NiFi and Spark
This document discusses using Apache NiFi and Spark to build a smarter home. It describes using NiFi on a Raspberry Pi and EC2 to collect sensor data from smart home devices and transmit it to an HDP cluster for storage and analysis with Pig and Spark. It outlines the architecture as a hub-and-spoke model and shows the evolution of the NiFi flows from sequential blocking writes to attribute-based routing. Key discoveries include privacy issues, using MAC addresses to predict arrivals, and motion sensors being less useful alone. Challenges involved Oracle vs OpenJDK, backpressure, and site-to-site configuration.
Scalable OCR with NiFi and Tesseract
This document describes a scalable optical character recognition (OCR) pipeline using Apache NiFi and Tesseract to extract text from PDF documents at large scale. The pipeline includes steps for converting PDFs to images, preprocessing the images using ImageMagick to enhance text, and performing OCR with Tesseract to extract text. NiFi is used to operationalize the pipeline and handle large-scale processing. Example use cases discussed include analyzing medical records and extracting text from large datasets to enable better journalism.
Dataflow with Apache NiFi
This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
What’s new in Apache Spark 2.3 and Spark 2.4
Apache Spark 2.0 set the architectural foundations of structure in Spark, unified high-level APIs, structured streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. Likewise, Apache Spark 2.4 will have many JIRA issues resolved over 1100. In this talk, I want to skim and go through those notable features and changes.
The Elephant in the Clouds
The document discusses key considerations for running Hadoop in the cloud. It notes that running Hadoop in the cloud provides unlimited elastic scale, ephemeral and long-running workloads, no upfront hardware costs, and IT and business agility. It outlines some of the major cloud Hadoop solutions according to a Forrester Wave report and discusses architectural considerations like shared data and storage, on-demand ephemeral workloads, elastic resource management, and shared metadata, security, and governance.
Recommended
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
Running Zeppelin in Enterprise
Apache Zeppelin has become a popular way to unlock the value of data lake due to its user interface and appeal to business users. These business users ask their IT department for access to Zeppelin. Enterprise IT department want to help their business users but they have several enterprise concerns such as enterprise security, integration with their corporate LDAP/AD, scalability and multi-user environment, integration with Ranger and Kerberos. This session will walk through enterprise concerns and how these concerns can be handled with Zeppelin.
Speaker
Simon Elliston Ball, Director Product Management, Cyber Security, Hortonworks
Getting involved with Open Source at the ASF
The document discusses getting involved with open source projects at the Apache Software Foundation. It provides an overview of the ASF, how it works, and how to contribute to Apache projects. The key points are:
- The ASF is a non-profit organization that oversees hundreds of open source projects and thousands of volunteers. Popular projects include Hadoop, Hive, and Pig.
- To get involved, individuals can start by joining mailing lists, reviewing documentation, reporting issues, and submitting code patches. More responsibilities come with becoming a committer or PMC member.
- Projects follow an open development process based on consensus. Voting on decisions helps include contributors from different time zones.
- Contributing is rewarding
Building a Smarter Home with Apache NiFi and Spark
This document discusses using Apache NiFi and Spark to build a smarter home. It describes using NiFi on a Raspberry Pi and EC2 to collect sensor data from smart home devices and transmit it to an HDP cluster for storage and analysis with Pig and Spark. It outlines the architecture as a hub-and-spoke model and shows the evolution of the NiFi flows from sequential blocking writes to attribute-based routing. Key discoveries include privacy issues, using MAC addresses to predict arrivals, and motion sensors being less useful alone. Challenges involved Oracle vs OpenJDK, backpressure, and site-to-site configuration.
Scalable OCR with NiFi and Tesseract
This document describes a scalable optical character recognition (OCR) pipeline using Apache NiFi and Tesseract to extract text from PDF documents at large scale. The pipeline includes steps for converting PDFs to images, preprocessing the images using ImageMagick to enhance text, and performing OCR with Tesseract to extract text. NiFi is used to operationalize the pipeline and handle large-scale processing. Example use cases discussed include analyzing medical records and extracting text from large datasets to enable better journalism.
Dataflow with Apache NiFi
This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
What’s new in Apache Spark 2.3 and Spark 2.4
Apache Spark 2.0 set the architectural foundations of structure in Spark, unified high-level APIs, structured streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. Likewise, Apache Spark 2.4 will have many JIRA issues resolved over 1100. In this talk, I want to skim and go through those notable features and changes.
The Elephant in the Clouds
The document discusses key considerations for running Hadoop in the cloud. It notes that running Hadoop in the cloud provides unlimited elastic scale, ephemeral and long-running workloads, no upfront hardware costs, and IT and business agility. It outlines some of the major cloud Hadoop solutions according to a Forrester Wave report and discusses architectural considerations like shared data and storage, on-demand ephemeral workloads, elastic resource management, and shared metadata, security, and governance.
Apache NiFi Meetup - Princeton NJ 2016
The document discusses an overview presentation on Apache NiFi given by Timothy Spann. The presentation covered what NiFi is, how to install it, its terminology, user interface, extensibility, and ecosystem. It also included a demonstration of how to add a processor for data intake within 1 minute. The presentation was part of a larger meetup event on the future of data.
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
In this talk I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets and facebook posts.
In the same flow I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, Email and Slack.
I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy and Python TextBlob.
This talk will be a deep dive into how to manage complex dataflow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning and business rules.
Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in stream and
storing data into Apache Phoenix on HBase and Apache Hive as ORC tables.
In this talk, I will walk through each step in the process from ingest of each source, applying filters, performing transformations, converting types, picking and converting fields and finally storing data to Apache Phoenix on HBase.
A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x.
This is based on a few talks I have given at the Future of Data - Princeton meetup on various ingestion and processing patterns with Apache NiFi.
Speaker:
TImothy Spann, Solutions Engineer, Hortonworks
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Imagine if you could build and deploy an end to end complex streaming analytics app on a streaming engine like Storm or Flink that did the following:
1. Joining Streams
2. Aggregations over Windows (Time or Count based)
3. Complex Event Processing
4. Pattern Matching
5. Model scoring.
Now imagine implementing and deploying this without writing a single line of code in under 10 mins.
Imagine no more; it is indeed here. In this talk, we will discuss an exciting open source project led by Hortonworks on building and deploying streaming applications using a drag and drop paradigm.
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
An introduction to Apache NiFi presented at the Apache NiFi Meetup as part of the 2016 Hadoop Summit held in San Jose.
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.
Expected prior knowledge / intended audience: developers and data flow managers should have passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.
Speaker
Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Running Apache NiFi with Apache Spark : Integration Options
A walk-through of various options in integration Apache Spark and Apache NiFi in one smooth dataflow. There are now several options in interfacing between Apache NiFi and Apache Spark with Apache Kafka and Apache Livy.
Webinar Series Part 5 New Features of HDF 5
Overview of the newest features of Hortonworks DataFlow highlighting the new processors, new user interface, edge intelligence powered by Apache MiNiFi and new support for multi-tenancy and new zero master clustering architecture
What the #$* is a Business Catalog and why you need it
The document discusses Apache Atlas, which is a data governance solution for Hadoop. It provides a centralized business catalog to organize data assets along business terms. This helps improve data governance, compliance, and faster discovery of data. The business catalog provides a common taxonomy and supports features like tagging, lineage tracking, and dynamic access control integrated with Ranger. It aims to reduce the time analysts spend searching for data from 50-80% to less than 25%.
Dataflow with Apache NiFi - Crash Course - HS16SJ
This document provides an overview of Apache NiFi, a dataflow management software. It begins with an introduction to dataflow and challenges in moving data effectively. It then discusses key features of Apache NiFi like guaranteed delivery, data buffering, and data provenance. The document outlines NiFi's architecture including repositories and extension points. It also advertises an upcoming Birds of a Feather session on streaming, dataflow and cybersecurity. Finally, it encourages learning more about NiFi and getting involved in the community.
Dataflow Management From Edge to Core with Apache NiFi
What is “dataflow?” — the process and tooling around gathering necessary information and getting it into a useful form to make insights available. Dataflow needs change rapidly — what was noise yesterday may be crucial data today, an API endpoint changes, or a service switches from producing CSV to JSON or Avro. In addition, developers may need to design a flow in a sandbox and deploy to QA or production — and those database passwords aren’t the same (hopefully). Learn about Apache NiFi — a robust and secure framework for dataflow development and monitoring.
Abstract: Identifying, collecting, securing, filtering, prioritizing, transforming, and transporting abstract data is a challenge faced by every organization. Apache NiFi and MiNiFi allow developers to create and refine dataflows with ease and ensure that their critical content is routed, transformed, validated, and delivered across global networks. Learn how the framework enables rapid development of flows, live monitoring and auditing, data protection and sharing. From IoT and machine interaction to log collection, NiFi can scale to meet the needs of your organization. Able to handle both small event messages and “big data” on the scale of terabytes per day, NiFi will provide a platform which lets both engineers and non-technical domain experts collaborate to solve the ingest and storage problems that have plagued enterprises.
Expected prior knowledge / intended audience: developers and data flow managers should be interested in learning about and improving their dataflow problems. The intended audience does not need experience in designing and modifying data flows.
Takeaways: Attendees will gain an understanding of dataflow concepts, data management processes, and flow management (including versioning, rollbacks, promotion between deployment environments, and various backing implementations).
Current uses: I am a committer and PMC member for the Apache NiFi, MiNiFi, and NiFi Registry projects and help numerous users deploy these tools to collect data from an incredibly diverse array of endpoints, aggregate, prioritize, filter, transform, and secure this data, and generate actionable insight from it. Current users of these platforms include many Fortune 100 companies, governments, startups, and individual users across fields like telecommunications, finance, healthcare, automotive, aerospace, and oil & gas, with use cases like fraud detection, logistics management, supply chain management, machine learning, IoT gateway, connected vehicles, smart grids, etc.
Intro to Spark with Zeppelin
This document provides an introduction to Apache Spark and Zeppelin. It describes Spark as an open source cluster computing framework, and its APIs for Scala, Java, Python and R. Key Spark components are outlined like Spark Core, Spark SQL, MLlib and GraphX. RDDs are defined as Spark's primary abstraction, and DataFrames/Datasets are presented as higher-level APIs built on RDDs. The benefits of Spark SQL for structured data are highlighted. Examples demonstrate basic Spark and SQL usage. Finally, Apache Zeppelin and the Hortonworks sandbox are introduced as tools for interactive data analytics on Spark and Hadoop clusters.
Protecting Enterprise Data in Apache Hadoop
The document discusses security threats and protections for enterprise data stored in Apache Hadoop. It outlines various attack vectors including accidental damage, remote access, eavesdropping, and users accessing private data. It then describes Hortonworks' security architecture and encryption for HDFS and Hive, including encryption schemes and changes to Hive architecture that add security. Finally, it notes threats of users reading hidden values or from shadow security before concluding and thanking the reader.
#HSTokyo16 Apache Spark Crash Course
This document provides an overview and crash course on Apache Spark and related big data technologies. It discusses the history and components of Spark including Spark Core, SQL, Streaming, and MLlib. It also discusses data sources, challenges of big data, and how Spark addresses them through its in-memory computation model. Finally, it introduces Apache Zeppelin for interactive notebooks and the Hortonworks Data Platform sandbox for experimenting with these technologies.
Double Your Hadoop Hardware Performance with SmartSense
Hortonworks SmartSense provides proactive recommendations that improve cluster performance, security and operations. And since 30% of issues are configuration related, Hortonworks SmartSense makes an immediate impact on Hadoop system performance and availability, in some cases boosting hardware performance by two times. Learn how SmartSense can help you increase the efficiency of your Hadoop hardware, through customized cluster recommendations.
View the on-demand webinar: https://hortonworks.com/webinar/boosts-hadoop-hardware-performance-2x-smartsense/
Securing Hadoop in an Enterprise Context
Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Apache Hadoop Crash Course
The document provides an overview of Apache Hadoop and how it addresses challenges with traditional data architectures. It discusses how Hadoop provides a distributed storage and processing framework to allow businesses to store all of their data in its native format and access it via different engines. The key components of Hadoop include the Hadoop Distributed File System (HDFS) for storage and Yet Another Resource Negotiator (YARN) for distributed computing. Hadoop addresses issues around cost, speed, and the ability to leverage both new and cold data with modern data applications.
Apache NiFi Crash Course - San Jose Hadoop Summit
This document provides an overview of Apache NiFi and dataflow. It begins with defining what dataflow is and the challenges of moving data effectively. It then introduces Apache NiFi, describing its key features like guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document discusses NiFi's architecture including its use of FlowFiles to move data agnostically through processors. It also covers NiFi's extension points and integration with other systems. Finally, it describes a live demo use case of using NiFi to integrate real-time traffic data for urban planning.
Multi User Data science with Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It summarizes Zeppelin's features such as support for multiple programming languages and deep integration with Apache Spark. The document also outlines Zeppelin's roadmap, including goals to improve support for authentication, authorization, and collaboration features.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.Running Zeppelin in Enterprise
Zeppelin has become a popular way to unlock the value of data lake due to its user interface and appeal to business users. These business users ask their IT department for access to Zeppelin. Enterprise IT department want to help their business users but they have several enterprise concerns such as enterprise security, integration with their corporate LDAP/AD, scalability and multi-user environment, integration with Ranger and Kerberos. This session will walk through enterprise concerns and how these concerns can be handled with Zeppelin.
Spark-Zeppelin-ML on HWX
Kirk Haslbeck gave a presentation on data science at scale using Apache Spark. He discussed how Spark can handle large, distributed datasets and supports multiple programming languages. Spark addresses limitations of single-machine analysis and allows horizontal scaling. Haslbeck demonstrated how to build machine learning models for credit card fraud detection using Spark and showed visualizations created with R and Matplotlib in Apache Zeppelin.
Apache Zeppelin 소개
Zeppelin is an open-source notebook application for interactive data analytics. It allows for data exploration, visualization, and sharing. The presenter is Lee moon soo, who helped start Zeppelin at NFLabs. Zeppelin began as a commercial product but is now open-source with Apache Spark integration. It aims to simplify data analysis with interpreters for languages like Spark, PySpark, and SQL. The roadmap includes adding more framework integrations and output streaming capabilities.
More Related Content
What's hot
Apache NiFi Meetup - Princeton NJ 2016
The document discusses an overview presentation on Apache NiFi given by Timothy Spann. The presentation covered what NiFi is, how to install it, its terminology, user interface, extensibility, and ecosystem. It also included a demonstration of how to add a processor for data intake within 1 minute. The presentation was part of a larger meetup event on the future of data.
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
In this talk I will show data engineers and architects how to run real-time TensorFlow Inception Image Recognition on images captured by remote sensors and images in tweets and facebook posts.
In the same flow I will also demonstrate how to apply real-time sentiment analysis and intelligent routing of data to Phoenix, Email and Slack.
I will elaborate on a number of different sentiment analysis frameworks available for use within Apache NiFi including Python NLTK, Stanford CoreNLP, Python SpaCy and Python TextBlob.
This talk will be a deep dive into how to manage complex dataflow pipelines ingesting from multiple streaming sources including social, public open data feeds, logs, drones, RDBMS and IoT with transformations, deep learning, machine learning and business rules.
Data engineers will be shown the power of Apache NiFi for loading diverse sources of data, applying transformations in-stream, routing based on attributes, adding sentiment data to workflows, running deep learning algorithms in stream and
storing data into Apache Phoenix on HBase and Apache Hive as ORC tables.
In this talk, I will walk through each step in the process from ingest of each source, applying filters, performing transformations, converting types, picking and converting fields and finally storing data to Apache Phoenix on HBase.
A quick data analysis to show streaming updates to data will be done in Apache Zeppelin running on HDP 2.x.
This is based on a few talks I have given at the Future of Data - Princeton meetup on various ingestion and processing patterns with Apache NiFi.
Speaker:
TImothy Spann, Solutions Engineer, Hortonworks
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Imagine if you could build and deploy an end to end complex streaming analytics app on a streaming engine like Storm or Flink that did the following:
1. Joining Streams
2. Aggregations over Windows (Time or Count based)
3. Complex Event Processing
4. Pattern Matching
5. Model scoring.
Now imagine implementing and deploying this without writing a single line of code in under 10 mins.
Imagine no more; it is indeed here. In this talk, we will discuss an exciting open source project led by Hortonworks on building and deploying streaming applications using a drag and drop paradigm.
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
An introduction to Apache NiFi presented at the Apache NiFi Meetup as part of the 2016 Hadoop Summit held in San Jose.
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.
Expected prior knowledge / intended audience: developers and data flow managers should have passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.
Speaker
Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Running Apache NiFi with Apache Spark : Integration Options
A walk-through of various options in integration Apache Spark and Apache NiFi in one smooth dataflow. There are now several options in interfacing between Apache NiFi and Apache Spark with Apache Kafka and Apache Livy.
Webinar Series Part 5 New Features of HDF 5
Overview of the newest features of Hortonworks DataFlow highlighting the new processors, new user interface, edge intelligence powered by Apache MiNiFi and new support for multi-tenancy and new zero master clustering architecture
What the #$* is a Business Catalog and why you need it
The document discusses Apache Atlas, which is a data governance solution for Hadoop. It provides a centralized business catalog to organize data assets along business terms. This helps improve data governance, compliance, and faster discovery of data. The business catalog provides a common taxonomy and supports features like tagging, lineage tracking, and dynamic access control integrated with Ranger. It aims to reduce the time analysts spend searching for data from 50-80% to less than 25%.
Dataflow with Apache NiFi - Crash Course - HS16SJ
This document provides an overview of Apache NiFi, a dataflow management software. It begins with an introduction to dataflow and challenges in moving data effectively. It then discusses key features of Apache NiFi like guaranteed delivery, data buffering, and data provenance. The document outlines NiFi's architecture including repositories and extension points. It also advertises an upcoming Birds of a Feather session on streaming, dataflow and cybersecurity. Finally, it encourages learning more about NiFi and getting involved in the community.
Dataflow Management From Edge to Core with Apache NiFi
What is “dataflow?” — the process and tooling around gathering necessary information and getting it into a useful form to make insights available. Dataflow needs change rapidly — what was noise yesterday may be crucial data today, an API endpoint changes, or a service switches from producing CSV to JSON or Avro. In addition, developers may need to design a flow in a sandbox and deploy to QA or production — and those database passwords aren’t the same (hopefully). Learn about Apache NiFi — a robust and secure framework for dataflow development and monitoring.
Abstract: Identifying, collecting, securing, filtering, prioritizing, transforming, and transporting abstract data is a challenge faced by every organization. Apache NiFi and MiNiFi allow developers to create and refine dataflows with ease and ensure that their critical content is routed, transformed, validated, and delivered across global networks. Learn how the framework enables rapid development of flows, live monitoring and auditing, data protection and sharing. From IoT and machine interaction to log collection, NiFi can scale to meet the needs of your organization. Able to handle both small event messages and “big data” on the scale of terabytes per day, NiFi will provide a platform which lets both engineers and non-technical domain experts collaborate to solve the ingest and storage problems that have plagued enterprises.
Expected prior knowledge / intended audience: developers and data flow managers should be interested in learning about and improving their dataflow problems. The intended audience does not need experience in designing and modifying data flows.
Takeaways: Attendees will gain an understanding of dataflow concepts, data management processes, and flow management (including versioning, rollbacks, promotion between deployment environments, and various backing implementations).
Current uses: I am a committer and PMC member for the Apache NiFi, MiNiFi, and NiFi Registry projects and help numerous users deploy these tools to collect data from an incredibly diverse array of endpoints, aggregate, prioritize, filter, transform, and secure this data, and generate actionable insight from it. Current users of these platforms include many Fortune 100 companies, governments, startups, and individual users across fields like telecommunications, finance, healthcare, automotive, aerospace, and oil & gas, with use cases like fraud detection, logistics management, supply chain management, machine learning, IoT gateway, connected vehicles, smart grids, etc.
Intro to Spark with Zeppelin
This document provides an introduction to Apache Spark and Zeppelin. It describes Spark as an open source cluster computing framework, and its APIs for Scala, Java, Python and R. Key Spark components are outlined like Spark Core, Spark SQL, MLlib and GraphX. RDDs are defined as Spark's primary abstraction, and DataFrames/Datasets are presented as higher-level APIs built on RDDs. The benefits of Spark SQL for structured data are highlighted. Examples demonstrate basic Spark and SQL usage. Finally, Apache Zeppelin and the Hortonworks sandbox are introduced as tools for interactive data analytics on Spark and Hadoop clusters.
Protecting Enterprise Data in Apache Hadoop
The document discusses security threats and protections for enterprise data stored in Apache Hadoop. It outlines various attack vectors including accidental damage, remote access, eavesdropping, and users accessing private data. It then describes Hortonworks' security architecture and encryption for HDFS and Hive, including encryption schemes and changes to Hive architecture that add security. Finally, it notes threats of users reading hidden values or from shadow security before concluding and thanking the reader.
#HSTokyo16 Apache Spark Crash Course
This document provides an overview and crash course on Apache Spark and related big data technologies. It discusses the history and components of Spark including Spark Core, SQL, Streaming, and MLlib. It also discusses data sources, challenges of big data, and how Spark addresses them through its in-memory computation model. Finally, it introduces Apache Zeppelin for interactive notebooks and the Hortonworks Data Platform sandbox for experimenting with these technologies.
Double Your Hadoop Hardware Performance with SmartSense
Hortonworks SmartSense provides proactive recommendations that improve cluster performance, security and operations. And since 30% of issues are configuration related, Hortonworks SmartSense makes an immediate impact on Hadoop system performance and availability, in some cases boosting hardware performance by two times. Learn how SmartSense can help you increase the efficiency of your Hadoop hardware, through customized cluster recommendations.
View the on-demand webinar: https://hortonworks.com/webinar/boosts-hadoop-hardware-performance-2x-smartsense/
Securing Hadoop in an Enterprise Context
Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Apache Hadoop Crash Course
The document provides an overview of Apache Hadoop and how it addresses challenges with traditional data architectures. It discusses how Hadoop provides a distributed storage and processing framework to allow businesses to store all of their data in its native format and access it via different engines. The key components of Hadoop include the Hadoop Distributed File System (HDFS) for storage and Yet Another Resource Negotiator (YARN) for distributed computing. Hadoop addresses issues around cost, speed, and the ability to leverage both new and cold data with modern data applications.
Apache NiFi Crash Course - San Jose Hadoop Summit
This document provides an overview of Apache NiFi and dataflow. It begins with defining what dataflow is and the challenges of moving data effectively. It then introduces Apache NiFi, describing its key features like guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document discusses NiFi's architecture including its use of FlowFiles to move data agnostically through processors. It also covers NiFi's extension points and integration with other systems. Finally, it describes a live demo use case of using NiFi to integrate real-time traffic data for urban planning.
Multi User Data science with Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It summarizes Zeppelin's features such as support for multiple programming languages and deep integration with Apache Spark. The document also outlines Zeppelin's roadmap, including goals to improve support for authentication, authorization, and collaboration features.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.Running Zeppelin in Enterprise
Zeppelin has become a popular way to unlock the value of data lake due to its user interface and appeal to business users. These business users ask their IT department for access to Zeppelin. Enterprise IT department want to help their business users but they have several enterprise concerns such as enterprise security, integration with their corporate LDAP/AD, scalability and multi-user environment, integration with Ranger and Kerberos. This session will walk through enterprise concerns and how these concerns can be handled with Zeppelin.
What's hot (20)
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Its Finally Here! Building Complex Streaming Analytics Apps in under 10 min w...
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Running Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration Options
What the #$* is a Business Catalog and why you need it
What the #$* is a Business Catalog and why you need it
Dataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
Double Your Hadoop Hardware Performance with SmartSense
Double Your Hadoop Hardware Performance with SmartSense
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Viewers also liked
Spark-Zeppelin-ML on HWX
Kirk Haslbeck gave a presentation on data science at scale using Apache Spark. He discussed how Spark can handle large, distributed datasets and supports multiple programming languages. Spark addresses limitations of single-machine analysis and allows horizontal scaling. Haslbeck demonstrated how to build machine learning models for credit card fraud detection using Spark and showed visualizations created with R and Matplotlib in Apache Zeppelin.
Apache Zeppelin 소개
Zeppelin is an open-source notebook application for interactive data analytics. It allows for data exploration, visualization, and sharing. The presenter is Lee moon soo, who helped start Zeppelin at NFLabs. Zeppelin began as a commercial product but is now open-source with Apache Spark integration. It aims to simplify data analysis with interpreters for languages like Spark, PySpark, and SQL. The roadmap includes adding more framework integrations and output streaming capabilities.
Data Science with Spark & Zeppelin
This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It can be used for data exploration, visualization, collaboration and publishing. Zeppelin has deep integration with Apache Spark and supports multiple languages including Scala, Python, and SQL. It provides a Spark interpreter that allows users to analyze data using Spark without having to configure Spark themselves. The document demonstrates Zeppelin's functionality through examples and encourages readers to try it out and get involved in the community.
Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark
In this session, we will discuss:
* reactive architecture tenets
* distributed “fast data” streams
* application and analytics focused Data Lake
Enterprise level concerns and the importance of holistic governance, operational management, and a Metadata Lake will be conceptually investigated. The next level of detail will be to explore what a prospective architecture looks like at scale with Terabytes of ingestion per day, how scale puts pressure on an architecture, and how to be successful without losing data in a mission critical system via resilient, self-healing, scalable technologies. DevOps and application architecture concerns will be first-class themes throughout.
Reactive principles and technology will be the second act of this talk. Kafka. Akka. Spark. Various streaming technologies (Kafka Streams, Akka Streams, Spark Streaming) will be reviewed to identify what they are best suited for. The fast data pipeline discussion will center around Kafka, Akka, and Apache Flink (Lightbend Fast Data platform). We’ll also walk through an exciting addition to the Akka family, Alpakka, which is a Camel equivalent for Enterprise Integration Patterns.
The final act will be to dive into the Data Lake, from both an analytics and application development perspective. Technologies used to explain concepts will include Amazon and Hadoop. A Data Lake may service multiple analytics consumers with various “views” (and access levels) of data. It may also be a participant of various applications, perhaps by acting as a centralized source for reference data or common middleware (in turn feeding the analytics aspect). The concept of the Metadata Lake to apply structure, meaning and purpose will be an over-arching success factor for a Data Lake. The difference between the Data Lake and Metadata Lake is conceptually similar to a Halocline… Various technologies (Iglu/Snowplow and more) will be discussed from a feature standpoint to flesh out the technology capabilities needed for Data Lake governance.
A Deep Dive into Structured Streaming in Apache Spark
This document provides an overview of Structured Streaming in Apache Spark. It begins with a brief history of streaming in Spark and outlines some of the limitations of the previous DStream API. It then introduces the new Structured Streaming API, which allows for continuous queries to be expressed as standard Spark SQL queries against continuously arriving data. It describes the new processing model and how queries are executed incrementally. It also covers features like event-time processing, windows, joins, and fault-tolerance guarantees through checkpointing and write-ahead logging. Overall, the document presents Structured Streaming as providing a simpler way to perform streaming analytics by allowing streaming queries to be expressed using the same APIs as batch queries.
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
This document discusses Apache Zeppelin, an open-source notebook for interactive data analytics. It provides an overview of Zeppelin's features, including interactive notebooks, multiple backends, interpreters, and a display system. The document also covers Zeppelin's adoption timeline, from its origins as a commercial product in 2012 to becoming an Apache Incubator project in 2014. Future projects involving Zeppelin like Helium and Z-Manager are also briefly described.
67 Weeks of TensorFlow
This document discusses various topics including the open sourcing of Tensor Flow in November 2015, 67 weeks passing since then in February 2017, capturing video and audio of passing trains to validate Caltrain API data, DeepMind Lab, Alzheimer's disease, DICOM for digital imaging, and contact information for Gorka Bengochea. It jumps between unrelated topics without much context or connection between the sections.
Saving Environment with IoT: Smart Watering with Predix
Learn how IoT and Predix are fighting droughts in California using the Smart Watering System developed at the San Francisco University.
Viewers also liked (8)
Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark
Building Reactive Fast Data & the Data Lake with Akka, Kafka, Spark
A Deep Dive into Structured Streaming in Apache Spark
A Deep Dive into Structured Streaming in Apache Spark
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Saving Environment with IoT: Smart Watering with Predix
Saving Environment with IoT: Smart Watering with Predix
Similar to Apache Zeppelin and Spark for Enterprise Data Science
introduction-to-apache-kafka
This document provides an introduction to Apache Kafka. It begins with an overview of Kafka as a distributed messaging system that is real-time, scalable, low latency, and fault tolerant. It then covers key concepts such as topics, partitions, producers, consumers, and replication. The document explains how Kafka achieves fast reads and writes through its design and use of disk flushing and replication for durability. It also discusses how Kafka can be used to build real-time systems and provides examples like connected cars. Finally, it introduces Apache Metron as an example of a cyber security solution built on Kafka.
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
The document discusses bringing multi-tenancy to Apache Zeppelin through the use of Apache Livy. Livy is an open-source REST interface that allows interacting with Spark from anywhere and enables features like multi-user sessions and security. It improves on previous versions of interactive analysis in Zeppelin by allowing custom user sessions through Livy and improving security and isolation between users through mechanisms like SPNEGO and impersonation. The integration of Livy provides multi-tenancy, security, and isolation for interactive analysis in Zeppelin.YARN Ready: Apache Spark
http://hortonworks.com/hadoop/spark/
Recording:
https://hortonworks.webex.com/hortonworks/lsr.php?RCID=03debab5ba04b34a033dc5c2f03c7967
As the ratio of memory to processing power rapidly evolves, many within the Hadoop community are gravitating towards Apache Spark for fast, in-memory data processing. And with YARN, they use Spark for machine learning and data science use cases along side other workloads simultaneously. This is a continuation of our YARN Ready Series, aimed at helping developers learn the different ways to integrate to YARN and Hadoop. Tools and applications that are YARN Ready have been verified to work within YARN.
Spark Security
This document discusses security requirements and solutions for Apache Spark production deployments. It covers authenticating users with Kerberos/AD, authorizing access to Spark jobs and data with Ranger, auditing access, and encrypting data at rest and in motion. It provides examples of configuring Kerberos authentication for Spark, using Ranger to control authorization to HDFS and SparkSQL, and demonstrates dynamic row filtering and masking of sensitive data in SparkSQL queries based on user policies.
Storm Demo Talk - Colorado Springs May 2015
The document discusses real-time processing capabilities in Hadoop and Hortonworks Data Platform (HDP). It begins with an introduction to Hortonworks and an overview of real-time streaming architectures on HDP. It then demonstrates streaming capabilities with and without predictive analytics additions. The document highlights how HDP provides a centralized architecture and open data platform to enable real-time and batch processing of any type of data for analytics applications.
Storm Demo Talk - Denver Apr 2015
The document discusses real-time processing in Hadoop and provides an overview of streaming architectures using the Hortonworks Data Platform (HDP). It includes two demos, the first showing a basic streaming scenario and the second integrating predictive analytics. The document aims to introduce HDP's capabilities for real-time streaming and predictive analytics and demonstrate them through examples relevant to logistics companies.
Row/Column- Level Security in SQL for Apache Spark
Security is one of fundamental features for enterprise adoption. Specifically, for SQL users, row/column-level access control is important. However, when a cluster is used as a data warehouse accessed by various user groups via different ways, it is difficult to guarantee data governance in a consistent way. In this talk, we focus on SQL users and talk about how to provide row/column-level access controls with common access control rules throughout the whole cluster with various SQL engines, e.g., Apache Spark 2.1, Apache Spark 1.6 and Apache Hive 2.1. If some of rules are changed, all engines are controlled consistently in near real-time. Technically, we enables Spark Thrift Server to work with an identify given by JDBC connection and take advantage of Hive LLAP daemon as a shared and secured processing engine. We demonstrate row-level filtering, column-level filtering and various column maskings in Apache Spark with Apache Ranger. We use Apache Ranger as a single point of security control center.
Hadoop in adtech
This document discusses the evolution of Hadoop and its use cases in the adtech industry. It describes how Hadoop was initially used primarily for batch processing via Hive and MapReduce. Over time, improvements like Tez, Presto, and Impala enabled faster interactive SQL queries on big data. The document also outlines how the Hadoop ecosystem is now used for real-time log collection, reporting, model generation, and more across the entire adtech stack. Key recent developments discussed include improvements in Hive like LLAP that enable sub-second SQL and ACID transactions, as well as tools like Cloudbreak for deploying Hadoop clusters in the cloud.
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
The document discusses real-time processing in Hadoop using the Hortonworks Data Platform (HDP). It provides an overview of using HDP for real-time streaming analytics in a logistics scenario. Example applications and architectures are presented, including using Kafka for ingesting sensor data, Storm for stream processing, and HBase for real-time querying. Demos will also illustrate integrating predictive analytics into streaming scenarios.
Internet of things Crash Course Workshop
This document provides an overview of real-time processing capabilities on Hortonworks Data Platform (HDP). It discusses how a trucking company uses HDP to analyze sensor data from trucks in real-time to monitor for violations and integrate predictive analytics. The company collects data using Kafka and analyzes it using Storm, HBase and Hive on Tez. This provides real-time dashboards as well as querying of historical data to identify issues with routes, trucks or drivers. The document explains components like Kafka, Storm and HBase and how they enable a unified YARN-based architecture for multiple workloads on a single HDP cluster.
Internet of Things Crash Course Workshop at Hadoop Summit
This document provides an overview of how a trucking company can use Hortonworks Data Platform (HDP) to gain insights from real-time streaming data generated by sensors in its trucks. The company wants to monitor trucks for locations, violations, and other events. HDP allows the company to ingest streaming data from trucks using Kafka and analyze it in real-time with Storm for alerts or serve it to applications with HBase. The company can also run interactive queries on historical data with Hive and Tez. All of this is run on a single HDP cluster for consistent governance, security, and operations across batch and real-time workloads.
Intro to Big Data Analytics using Apache Spark and Apache Zeppelin
This workshop will provide an introduction to Big Data Analytics using Apache Spark and Apache Zeppelin.
https://github.com/zeltovhorton/intro_spark_zeppelin_meetup
There will be a short lecture that includes an introduction to Spark, the Spark components.
Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes.
The lecture will be followed by demo . There will be a short lecture on Hadoop and how Spark and Hadoop interact and compliment each other. You will learn how to move data into HDFS using Spark APIs, create Hive table, explore the data with Spark and SQL, transform the data and then issue some SQL queries. We will be using Scala and/or PySpark for labs.
Spark and Hadoop Perfect Togeher by Arun Murthy
Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Spark Summit EMEA - Arun Murthy's Keynote
Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Don't Let the Spark Burn Your House: Perspectives on Securing Spark
Apache Spark is emerging as a key enabler for various enterprise use cases including customer intelligence applications, data warehousing, real-time or streaming, recommendation engines, and log processing. Even the most common use case for Spark around business intelligence (BI) or customer intelligence applications via data science encompasses the complete data worker lifecycle from file processing, workflows, cleansing, enrichment, model building and deployments to dash boarding and reporting. However, many aspects of security and governance with Spark are still emerging and pose challenges to enterprise adoption including areas of authorization, authentication, and comprehensive auditing as well as metadata harvesting and governance. We will demonstrate some examples of the current the state of the art in terms of different open source approaches to Spark security and governance. For example, we will show how Spark technologies can be integrated with enterprise identity providers, and how we can enable fine-grained access control for processes, and how to harvest process metadata while providing detailed audits. We will also provide best practices and common usage patterns to secure your Spark clusters and how best to support enterprise compliance and governance needs when using Spark.
Accumulo Summit 2016: Apache Accumulo on Docker with YARN Native Services
YARN native services is an exciting new feature that can be used to configure and run Accumulo easily and flexibly on YARN, with or without the use of Docker containers.
Previously, Apache Accumulo has been run on Apache Hadoop YARN through Apache Slider (incubating). Slider is designed to make it easier to run existing long-running distributed applications on YARN. Recently in YARN-4692, there began an effort to enhance support for long-running services natively in YARN. A central part of this effort is introducing a framework layer based on Slider into YARN itself in YARN-5079. Other useful features include a REST API for running services in YARN-4793 and a capability to discover services via DNS in YARN-4757.
This talk will show how to run Accumulo on YARN using the new mechanisms of YARN native services and will highlight differences in the capabilities provided in YARN native services vs. Slider. This will include how to configure, launch, and manage Accumulo on YARN. Additionally, the talk will introduce the concept of application assemblies, complex services that have entire applications as subcomponents that are run together in a single YARN service. To provide an example of how assemblies work, the talk will demonstrate the use of Accumulo on YARN as a building block for complex services on YARN.
Note the YARN work is in progress in the yarn-native-services branch of Apache Hadoop.
– Speaker –
Billie Rinaldi
Principal Software Engineer, Hortonworks
Billie Rinaldi is a Principal Software Engineer I at Hortonworks, Inc., currently prototyping new features related to application monitoring and deployment in the Apache Hadoop ecosystem. Prior to August 2012, Billie engaged in big data science and research at the National Security Agency. Since 2008, she has been providing technical leadership regarding the software that is now Apache Accumulo. Billie is the VP of Apache Accumulo, the Accumulo Project Management Committee Chair, and a member of the Apache Software Foundation. She holds a Ph.D. in applied mathematics from Rensselaer Polytechnic Institute.
— More Information —
For more information see http://www.accumulosummit.com/
Using Apache® NiFi to Empower Self-Organising Teams
The document discusses how Apache NiFi can empower self-organizing teams by allowing them to control their own data movement pipelines. It describes NiFi as a data-movement production line that enables teams to build and change pipelines quickly. This helps reduce time-to-production and feedback loops by moving responsibility for data integration to individual teams rather than having changes controlled in a centralized manner.
Mission to NARs with Apache NiFi
This document discusses extending the functionality of Apache NiFi through custom processors and controller services. It provides an overview of the NiFi architecture and repositories, describes how to create extensions with minimal dependencies using Maven archetypes, and notes that most extensions can be developed within hours. Quick prototyping of data flows is possible using existing binaries, applications, and scripting languages. Resources for the NiFi developer guide and example Maven projects are also listed.
Apache Phoenix Query Server PhoenixCon2016
This document discusses Apache Phoenix Query Server, which provides a client-server abstraction for Apache Phoenix using Apache Calcite's Avatica sub-project. It allows Phoenix to have thin clients by offloading computational resources to query servers running on Hadoop clusters. This enables non-Java clients through a standardized HTTP API. The query server implementation uses HTTP, Protocol Buffers for serialization, and common libraries like Jetty and Dropwizard Metrics. It aims to simplify Phoenix client development and improve performance and scalability.
Running Apache Zeppelin production
Discussion around concerns related to deploying Apache Zeppelin in production, including deployment choices, security, performance and integration.
Similar to Apache Zeppelin and Spark for Enterprise Data Science (20)
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + LIvy: Bringing Multi Tenancy to Interactive Data Analysis
Row/Column- Level Security in SQL for Apache Spark
Row/Column- Level Security in SQL for Apache Spark
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Real-Time Processing in Hadoop for IoT Use Cases - Phoenix HUG
Internet of Things Crash Course Workshop at Hadoop Summit
Internet of Things Crash Course Workshop at Hadoop Summit
Intro to Big Data Analytics using Apache Spark and Apache Zeppelin
Intro to Big Data Analytics using Apache Spark and Apache Zeppelin
Don't Let the Spark Burn Your House: Perspectives on Securing Spark
Don't Let the Spark Burn Your House: Perspectives on Securing Spark
Accumulo Summit 2016: Apache Accumulo on Docker with YARN Native Services
Accumulo Summit 2016: Apache Accumulo on Docker with YARN Native Services
Using Apache® NiFi to Empower Self-Organising Teams
Using Apache® NiFi to Empower Self-Organising Teams
Recently uploaded
The Building Blocks of QuestDB, a Time Series Database
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Population Growth in Bataan: The effects of population growth around rural pl...
A population analysis specific to Bataan.
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
毕业原版【微信:41543339】【(Coventry毕业证书)考文垂大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
原版定制【微信:41543339】【(Bradford毕业证书)布拉德福德大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
End-to-end pipeline agility - Berlin Buzzwords 2024
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
学校原件一模一样【微信：741003700 】《(Deakin毕业证书)迪肯大学毕业证学位证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
原版定制【微信:41543339】【(UniSA毕业证书)南澳大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
原版定制【微信:41543339】【(UofS毕业证书)萨省大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
毕业原版【微信:41543339】【(牛布毕业证书)牛津布鲁克斯大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
毕业原版【微信:41543339】【(Chester毕业证书)切斯特大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
毕业原版【微信:176555708】【(UMN毕业证书)明尼苏达大学毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
毕业原版【微信:176555708】【(UCSF毕业证书)旧金山分校毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
Analysis insight about a Flyball dog competition team's performance
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Natural Language Processing (NLP), RAG and its applications .pptx
1. In the realm of Natural Language Processing (NLP), knowledge-intensive tasks such as question answering, fact verification, and open-domain dialogue generation require the integration of vast and up-to-date information. Traditional neural models, though powerful, struggle with encoding all necessary knowledge within their parameters, leading to limitations in generalization and scalability. The paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" introduces RAG (Retrieval-Augmented Generation), a novel framework that synergizes retrieval mechanisms with generative models, enhancing performance by dynamically incorporating external knowledge during inference.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
一比一原版(UO毕业证)渥太华大学毕业证如何办理
UO毕业证录取书【微信95270640】购买（渥太华大学毕业证成绩单硕士学历）Q微信95270640代办UO学历认证留信网伪造渥太华大学学位证书精仿渥太华大学本科/硕士文凭证书补办渥太华大学 diplomaoffer,Transcript购买渥太华大学毕业证成绩单购买UO假毕业证学位证书购买伪造渥太华大学文凭证书学位证书,专业办理雅思、托福成绩单，学生ID卡，在读证明，海外各大学offer录取通知书，毕业证书，成绩单，文凭等材料:1:1完美还原毕业证、offer录取通知书、学生卡等各种在读或毕业材料的防伪工艺（包括 烫金、烫银、钢印、底纹、凹凸版、水印、防伪光标、热敏防伪、文字图案浮雕，激光镭射，紫外荧光，温感光标）学校原版上有的工艺我们一样不会少，不论是老版本还是最新版本，都能保证最高程度还原，力争完美以求让所有同学都能享受到完美的品质服务。
文凭办理流程：
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：微信95270640我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。
7完成交易删除客户资料
高精端提供以下服务：
一：渥太华大学渥太华大学毕业证文凭证书全套材料从防伪到印刷水印底纹到钢印烫金
二：真实使馆认证（留学人员回国证明）使馆存档
三：真实教育部认证教育部存档教育部留服网站可查
四：留信认证留学生信息网站可查
五：与学校颁发的相关证件1:1纸质尺寸制定（定期向各大院校毕业生购买最新版本毕,业证成绩单保证您拿到的是鲁昂大学内部最新版本毕业证成绩单微信95270640）
A.为什么留学生需要操作留信认证?
留信认证全称全国留学生信息服务网认证,隶属于北京中科院。①留信认证门槛条件更低,费用更美丽,并且包过,完单周期短,效率高②留信认证虽然不能去国企,但是一般的公司都没有问题,因为国内很多公司连基本的留学生学历认证都不了解。这对于留学生来说,这就比自己光拿一个证书更有说服力,因为留学学历可以在留信网站上进行查询!
B.为什么我们提供的毕业证成绩单具有使用价值？
查询留服认证是国内鉴别留学生海外学历的唯一途径但认证只是个体行为不是所有留学生都操作所以没有办理认证的留学生的学历在国内也是查询不到的他们也仅仅只有一张文凭。所以这时候我们提供的和学校颁发的一模一样的毕业证成绩单就有了使用价值。只硕大的蛇皮袋手里拎着长铁钩正站在门口朝黑色的屋内张望不好坏人小偷山娃一怔却也灵机一动立马仰起头双手拢在嘴边朝楼上大喊：“爸爸爸——有人找——那人一听朝山娃尴尬地笑笑悻悻地走了山娃立马“嘭的一声将铁门锁死心却咚咚地乱跳当山娃跟父亲说起这事时父亲很吃惊抚摸着山娃的头说还好醒得及时要不家早被人掏空了到时连电视也没得看啰不过父亲还是夸山娃能临危不乱随机应变有胆有谋山娃笑笑说那都是书上学的看童话和小说时多
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
原版定制【微信:41543339】【(Adelaide毕业证书)阿德莱德大学毕业证】【微信:41543339】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信41543339】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信41543339】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
Recently uploaded (20)
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
Natural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptx
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Apache Zeppelin and Spark for Enterprise Data Science
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enabling Apache Zeppelin* and Spark* for Data Science in the Enterprise Bikas Saha @bikassaha *Apache Hadoop, Falcon, Atlas, Tez, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie, Zeppelin and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Making Big Data Science easy to approach What are the current issues for the enterprise Making Apache Zeppelin enterprise ready Future Roadmap
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Zeppelin
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin makes Big Data Science Easy to Approach Zero install – Just connect via a web browser and ready to run Support for multiple execution platforms (Apache Spark, JDBC, Hive…) Support for multiple languages (Scala, SQL, Python…) Support for built-in visualizations Support for reporting Support for sharing and collaborative work Does NOT have machine learning built-in – that’s where Apache Spark comes in (or your favorite SQL engine Apache Flink/Drill/Hive… and 30+ others)
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin for Sharing
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Making Big Data Science easy to approach What are the current issues for the enterprise Making Apache Zeppelin enterprise ready Future Roadmap
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Current Apache Zeppelin and Spark integration Zeppelin Server Spark Driver U s e r Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architectural Issue with Secure Data Access Zeppelin Server Spark Driver U s e r 1 Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Zeppelin Server User H D F S
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architectural Issues with Multi-Tenancy – Fault Tolerance Zeppelin Server Spark Driver U s e r 1 Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor U s e r 2 User 1 failure affects User 2 Heavy-weight Spark drivers
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architectural Issues with Multi-Tenancy – Privacy Zeppelin Server Spark Driver U s e r 1 Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor Spark Executor U s e r 2 User 1 can access User 2 Data
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Making Big Data Science easy to approach What are the current issues for the enterprise Enterprise Ready Big Data Science Future Roadmap
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Livy Server as a Session Management Service Livy Server Remote Spark Driver Session Remote Context Interactive REST API Batch REST API Standard Spark Batch Job Spark Executor Spark Executor Spark Executor Spark Executor
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Secure Data Access - Solved Zeppelin Server Livy Interpreter U s e r Spark Executor Spark Executor Livy Server Remote Spark Driver Session Remote Context User HDFS
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Multi Tenancy - Solved Zeppelin Server Livy Interpreter Livy Server Session 1 U s e r 1 U s e r 2 Livy Interpreter Session 2 Remote Spark Driver Remote Context Spark Executor Remote Spark Driver Remote Context Spark Executor
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Making Big Data Science easy to approach What are the current issues for the enterprise Making Apache Zeppelin enterprise ready Future Roadmap
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Near Term Improvements Session Management Debuggability Unified session for all languages Better visualizations for Machine Learning Support for Spark 2.0
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Long Term Improvements Controlled sharing of sessions for collaboration Data exploration and browsing with metadata Taking the model from training to production
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You