The DAP - Where YARN, HBase, Kafka and Spark go to Production
•Download as PPTX, PDF•
2 likes•1,131 views
The document summarizes the Cask Data Application Platform (CDAP), which provides an integrated framework for building and running data applications on Hadoop and Spark. It consolidates the big data application lifecycle by providing dataset abstractions, self-service data, metrics and log collection, lineage, audit, and access control. CDAP has an application container architecture with reusable programming abstractions and global user and machine metadata. It aims to simplify deploying and operating big data applications in enterprises by integrating technologies like YARN, HBase, Kafka and Spark.
Report
Share
Report
Share
Recommended
Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop
Successes, Challenges, and Pitfalls Migrating a SAAS business to HadoopDataWorks Summit/Hadoop Summit
The document discusses Marketo's migration of their SAAS business analytics platform to Hadoop. It describes their requirements of near real-time processing of 1 billion activities per customer per day at scale. They conducted a technology selection process between various Hadoop components and chose HBase, Kafka and Spark Streaming. The implementation involved building expertise, designing and building their first cluster, implementing security including Kerberos, validation through passive testing, deploying the new system through a migration, and ongoing monitoring, patching and upgrading of the new platform. Challenges included managing expertise retention, Zookeeper performance on VMs, Kerberos integration, and capacity planning for the shared Hadoop cluster.Integrating Apache Spark and NiFi for Data Lakes
This document discusses using Apache Spark and Apache NiFi together for data lakes. It outlines the goals of a data lake including having a central data repository, reducing costs, enabling easier discovery and prototyping. It also discusses what is needed for a Hadoop data lake, including automation of pipelines, governance, and interactive data discovery. The document then provides an example ingestion project and describes using Apache Spark for functions like cleansing, validating, and profiling data. It outlines using Apache NiFi for the pipeline design with drag and drop functionality. Finally, it demonstrates ingesting and preparing data, data self-service and transformation, data discovery, and operational monitoring capabilities.
Filling the Data Lake
This document discusses strategies for filling a data lake by improving the process of data onboarding. It advocates using a template-based approach to streamline data ingestion from various sources and reduce dependence on hardcoded procedures. The key aspects are managing ELT templates and metadata through automated metadata extraction. This allows generating integration jobs dynamically based on metadata passed at runtime, providing flexibility to handle different source data with one template. It emphasizes reducing the risks associated with large data onboarding projects by maintaining a standardized and organized data lake.
Insights into Real World Data Management Challenges
Data is your most valuable business asset and it's also your biggest challenge. This challenge and opportunity means we continually face significant road blocks toward becoming a data driven organisation. From the management of data, to the bubbling open source frameworks, the limited industry skills to surmounting time and cost pressures, our challenge in data is big.
We all want and need a “fit for purpose” approach to management of data, especially Big Data, and overcoming the ongoing challenges around the ‘3Vs’ means we get to focus on the most important V - ‘Value’.Come along and join the discussion on how Oracle Big Data Cloud provides Value in the management of data and supports your move toward becoming a data driven organisation.
Speaker
Noble Raveendran, Principal Consultant, Oracle
Discover HDP 2.1: Apache Solr for Hadoop Search
This document appears to be a presentation about Apache Solr for Hadoop search using the Hortonworks Data Platform (HDP). The agenda includes an overview of Apache Solr and Hadoop search, a demo of Hadoop search, and a question and answer section. The presentation discusses how Solr provides scalable indexing of data stored in HDFS and powerful search capabilities. It also includes a reference architecture showing how Solr integrates with Hadoop for search and indexing.
Beyond TCO
This document discusses architecting Hadoop for adoption and data applications. It begins by explaining how traditional systems struggle as data volumes increase and how Hadoop can help address this issue. Potential Hadoop use cases are presented such as file archiving, data analytics, and ETL offloading. Total cost of ownership (TCO) is discussed for each use case. The document then covers important considerations for deploying Hadoop such as hardware selection, team structure, and impact across the organization. Lastly, it discusses lessons learned and the need for self-service tools going forward.
Accelerating Big Data Insights
This document discusses Dell's solutions for big data and analytics workloads. It describes Dell's portfolio for unstructured analytics including storage, servers, and reference architectures. It also outlines Dell's vision for a unified streaming and batch analytics platform called Project Nautilus that would integrate Isilon storage with real-time stream processing.
Data-In-Motion Unleashed
The document discusses Hortonworks DataFlow (HDF), which is a platform for data in motion. HDF allows users to collect data at the edge, route and process streaming data with Apache NiFi and Kafka, and analyze, visualize, predict and prescribe outcomes from the data using HDF platform services. The HDF platform provides scalable stream processing, security, data provenance, and management capabilities for data in motion applications across the enterprise.
Recommended
Successes, Challenges, and Pitfalls Migrating a SAAS business to Hadoop
Successes, Challenges, and Pitfalls Migrating a SAAS business to HadoopDataWorks Summit/Hadoop Summit
The document discusses Marketo's migration of their SAAS business analytics platform to Hadoop. It describes their requirements of near real-time processing of 1 billion activities per customer per day at scale. They conducted a technology selection process between various Hadoop components and chose HBase, Kafka and Spark Streaming. The implementation involved building expertise, designing and building their first cluster, implementing security including Kerberos, validation through passive testing, deploying the new system through a migration, and ongoing monitoring, patching and upgrading of the new platform. Challenges included managing expertise retention, Zookeeper performance on VMs, Kerberos integration, and capacity planning for the shared Hadoop cluster.Integrating Apache Spark and NiFi for Data Lakes
This document discusses using Apache Spark and Apache NiFi together for data lakes. It outlines the goals of a data lake including having a central data repository, reducing costs, enabling easier discovery and prototyping. It also discusses what is needed for a Hadoop data lake, including automation of pipelines, governance, and interactive data discovery. The document then provides an example ingestion project and describes using Apache Spark for functions like cleansing, validating, and profiling data. It outlines using Apache NiFi for the pipeline design with drag and drop functionality. Finally, it demonstrates ingesting and preparing data, data self-service and transformation, data discovery, and operational monitoring capabilities.
Filling the Data Lake
This document discusses strategies for filling a data lake by improving the process of data onboarding. It advocates using a template-based approach to streamline data ingestion from various sources and reduce dependence on hardcoded procedures. The key aspects are managing ELT templates and metadata through automated metadata extraction. This allows generating integration jobs dynamically based on metadata passed at runtime, providing flexibility to handle different source data with one template. It emphasizes reducing the risks associated with large data onboarding projects by maintaining a standardized and organized data lake.
Insights into Real World Data Management Challenges
Data is your most valuable business asset and it's also your biggest challenge. This challenge and opportunity means we continually face significant road blocks toward becoming a data driven organisation. From the management of data, to the bubbling open source frameworks, the limited industry skills to surmounting time and cost pressures, our challenge in data is big.
We all want and need a “fit for purpose” approach to management of data, especially Big Data, and overcoming the ongoing challenges around the ‘3Vs’ means we get to focus on the most important V - ‘Value’.Come along and join the discussion on how Oracle Big Data Cloud provides Value in the management of data and supports your move toward becoming a data driven organisation.
Speaker
Noble Raveendran, Principal Consultant, Oracle
Discover HDP 2.1: Apache Solr for Hadoop Search
This document appears to be a presentation about Apache Solr for Hadoop search using the Hortonworks Data Platform (HDP). The agenda includes an overview of Apache Solr and Hadoop search, a demo of Hadoop search, and a question and answer section. The presentation discusses how Solr provides scalable indexing of data stored in HDFS and powerful search capabilities. It also includes a reference architecture showing how Solr integrates with Hadoop for search and indexing.
Beyond TCO
This document discusses architecting Hadoop for adoption and data applications. It begins by explaining how traditional systems struggle as data volumes increase and how Hadoop can help address this issue. Potential Hadoop use cases are presented such as file archiving, data analytics, and ETL offloading. Total cost of ownership (TCO) is discussed for each use case. The document then covers important considerations for deploying Hadoop such as hardware selection, team structure, and impact across the organization. Lastly, it discusses lessons learned and the need for self-service tools going forward.
Accelerating Big Data Insights
This document discusses Dell's solutions for big data and analytics workloads. It describes Dell's portfolio for unstructured analytics including storage, servers, and reference architectures. It also outlines Dell's vision for a unified streaming and batch analytics platform called Project Nautilus that would integrate Isilon storage with real-time stream processing.
Data-In-Motion Unleashed
The document discusses Hortonworks DataFlow (HDF), which is a platform for data in motion. HDF allows users to collect data at the edge, route and process streaming data with Apache NiFi and Kafka, and analyze, visualize, predict and prescribe outcomes from the data using HDF platform services. The HDF platform provides scalable stream processing, security, data provenance, and management capabilities for data in motion applications across the enterprise.
Big Data Simplified - Is all about Ab'strakSHeN
This document discusses designing a new big data platform to replace an existing complex and outdated one. It analyzes challenges with the current platform, including inability to keep up with business needs. The proposed new platform called Dredge would use abstraction layers to integrate big data tools in a loosely coupled and scalable way. This would simplify development and maintenance while supporting business goals. Key aspects of Dredge include declarative configuration, logical workflows, and plug-and-play integration of tools like HDFS, Hive, HBase, Kafka and Spark in a reusable and event-driven manner. The new platform aims to improve scalability, reduce costs and better support analytics needs over time.
Is your Enterprise Data lake Metadata Driven AND Secure?
This document discusses how Apache Atlas and Apache Ranger can be used together to provide a metadata-driven and secure data lake. Apache Atlas provides metadata services and tagging capabilities. Apache Ranger uses the tags in Atlas to dynamically define and enforce access policies. The integration allows Ranger policies to automatically apply and change as Atlas metadata such as tags are updated. The document demonstrates how tags in Atlas for columns and tables can be used to create time-based and PII data access policies in Ranger.
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
In this interactive webinar, we'll walk through use cases on how you can use advanced analytics like SAS Visual Statistics and In-Memory Statistic with Hortonworks’ data platform (HDP) to reveal insights in your big data and redefine how your organization solves complex problems.
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
This document discusses using Hadoop and the Hortonworks Data Platform (HDP) for big data applications. It outlines how HDP can help organizations optimize their existing data warehouse, lower storage costs, unlock new applications from new data sources, and achieve an enterprise data lake architecture. The document also discusses how Talend's data integration platform can be used with HDP to easily develop batch, real-time, and interactive data integration jobs on Hadoop. Case studies show how companies have used Talend and HDP together to modernize their data architecture and product inventory and pricing forecasting.
Big Data at your Desk with KNIME
This document provides an agenda and overview of a presentation by Data Transformed on big data analytics using the KNIME platform. The presentation includes an introduction of Data Transformed and KNIME, a live demonstration of forecasting energy usage from customer data using KNIME and Hadoop tools, and a question and answer session. It promotes Data Transformed's services around data management, analytics, and consulting and highlights KNIME's capabilities for comprehensive data processing.
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...DataWorks Summit/Hadoop Summit
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditional EDW and Hadoop with Apache AtlasCloudy with a chance of Hadoop - real world considerations
Over the last eighteen months, we have seen significant adoption of Hadoop eco-system centric big data processing in Microsoft Azure and Amazon AWS. In this talk we present some of the lessons learned and architectural considerations for cloud-based deployments including security, fault tolerance and auto-scaling.
We look at how Hortonworks Data Cloud and Cloudbreak can automate that scaling of Hadoop clusters, showing how it can react dynamically to workloads, and what that can deliver in cost-effective Hadoop-in-cloud deployments.
Modernise your EDW - Data Lake
The document discusses modernizing enterprise data warehouses by using a Hadoop data lake solution with EMC Isilon storage. This provides benefits like offloading expensive ETL processing to reduce costs, archiving cold data for cheaper storage, and enabling analytics on new data sources like semi-structured data. The solution leverages Hortonworks Data Platform for open, interoperable analytics and provides enterprise-grade data management capabilities on Hadoop at lower costs than traditional EDWs.
Splunk-hortonworks-risk-management-oct-2014
Slide deck from Splunk and Hortonworks joint webinar on October 1, 2014. Title Building a Modern Data Architecture for Risk Management.
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Back in 2014, our team set out to change the way the world exchanges and collaborates with data. Our vision was to build a single tenant environment for multiple organisations to securely share and consume data. And we did just that, leveraging multiple Hadoop technologies to help our infrastructure scale quickly and securely.
Today Data Republic’s technology delivers a trusted platform for hundreds of enterprise level companies to securely exchange, commercialise and collaborate with large datasets.
Join Head of Engineering, Juan Delard de Rigoulières and Senior Solutions Architect, Amin Abbaspour as they share key lessons from their team’s journey with Hadoop:
* How a startup leveraged a clever combination of Hadoop technologies to build a secure data exchange platform
* How Hadoop technologies helped us deliver key solutions around governance, security and controls of data and metadata
* An evaluation on the maturity and usefulness of some Hadoop technologies in our environment: Hive, HDFS, Spark, Ranger, Atlas, Knox, Kylin: we've use them all extensively.
* Our bold approach to expose APIs directly to end users; as well as the challenges, learning and code we created in the process
* Learnings from the front-line: How our team coped with code changes, performance tuning, issues and solutions while building our data exchange
Whether you’re an enterprise level business or a start-up looking to scale - this case study discussion offers behind-the-scenes lessons and key tips when using Hadoop technologies to manage data governance and collaboration in the cloud.
Speakers:
Juan Delard De Rigoulieres, Head of Engineering, Data Republic Pty Ltd
Amin Abbaspour, Senior Solutions Architect, Data Republic
Discover.hdp2.2.h base.final[2]
Hortonworks Data Platform 2.2 includes Apache HBase for fast NoSQL data access. In this 30-minute webinar, we discussed HBase innovations that are included in HDP 2.2, including: support for Apache Slider; Apache HBase high availability (HA); block ache compression; and wire-level encryption.
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
This document discusses how organizations can leverage data and analytics to power their business models. It provides examples of Fortune 100 companies that are using Attunity products to build data lakes and ingest data from SAP and other sources into Hadoop, Apache Kafka, and the cloud in order to perform real-time analytics. The document outlines the benefits of Attunity's data replication tools for extracting, transforming, and loading SAP and other enterprise data into data lakes and data warehouses.
Scaling Data Science on Big Data
Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production ?
In this session learn the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of DSX with HDP with the focus on integration, security and model deployment and management.
Speakers:
Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM
Vikram Murali, Program Director, Data Science and Machine Learning, IBM
Swimming Across the Data Lake, Lessons learned and keys to success
This document discusses strategies for successfully utilizing a data lake. It notes that creating a data lake is just the beginning and that challenges include data governance, metadata management, access, and effective use of the data. The document advocates for data democratization through discovery, accessibility, and usability. It also discusses best practices like self-service BI and automated workload migration from data warehouses to reduce costs and risks. The key is to address the "data lake dilemma" of these challenges to avoid a "data swamp" and slow adoption.
Format Wars: from VHS and Beta to Avro and Parquet
The document discusses different data storage formats such as text, Avro, Parquet, and their suitability for writing and reading data. It provides examples of how to choose a format based on factors like query needs, data types, and whether schemas need to evolve. The document also demonstrates how Avro can handle schema evolution by adding or changing fields while still reading existing data.
HDFS: Optimization, Stabilization and Supportability
This document summarizes improvements made to HDFS to optimize performance, stabilize operations, and improve supportability. Key areas discussed include logging enhancements, metrics and tools for troubleshooting, load management through RPC improvements, and changes to reduce garbage collection overhead and improve liveness detection. Specific optimizations covered range from code changes to reduce logging verbosity to adding batch processing of block reports.
End-to-End Security and Auditing in a Big Data as a Service Deployment
End-to-End Security and Auditing in a Big Data as a Service DeploymentDataWorks Summit/Hadoop Summit
1. The document discusses security considerations for deploying big data as a service (BDaaS) across multiple tenants and applications. It focuses on maintaining a single user identity to prevent data duplication and enforce access policies consistently.
2. It describes using Apache Ranger to centrally define and enforce policies across Hadoop services like HDFS, HBase, Hive. Ranger integrates with LDAP/AD for authentication.
3. The key challenge is propagating user identities from the application layer to the data layer. This can be done by connecting HDFS directly via Kerberos or using a "super-user" that impersonates other users when accessing HDFS.Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the CloudDataWorks Summit/Hadoop Summit
The world’s largest enterprises run their infrastructure on Oracle, DB2 and SQL and their critical business operations on SAP applications. Organisations need this data to be available in real-time to conduct necessary analytics. However, delivering this heterogeneous data at the speed it’s required can be a huge challenge because of the complex underlying data models and structures and legacy manual processes which are prone to errors and delays.
Unlock these silos of data and enable the new advanced analytics platforms by attending this session.
Find out how to:
• To overcome common challenges faced by enterprises trying to access their SAP data
• You can integrate SAP data in real-time with change data capture (CDC) technology
• Organisations are using Attunity Replicate for SAP to stream SAP data in to KafkaHigh Performance Spatial-Temporal Trajectory Analysis with Spark
This document discusses high performance spatial-temporal trajectory analysis using Spark. It covers the background of analyzing mobile signaling data to enable smarter urban planning. The solution architecture includes data sources, distributed file system, computation engine, and visualization. Technical designs address the big data platform, data governance, algorithm models, and Spark spatial computing. Example scenarios are presented for population heatmaps, commute routes, and office-residence imbalance analysis.
Hybrid Data Platform
The document discusses the need for a hybrid data platform that combines on-premise and cloud-based infrastructure. A hybrid data platform can address issues like mixed workloads, long-tail data usage, and data movement costs. It also allows tapping into the scale of cloud while keeping some data and workloads on-premise. The proposed architecture shows data integration between on-premise Hadoop and cloud services like S3, Redshift, and EMR through private tunnels and security measures. Case studies demonstrate how a large commodities producer built a hybrid analytics environment for productivity, supply chain, and operations.
Cassandra summit 2015 - Simplifying Streaming Analytics
Using Apache Cassandra, Apache Spark and Apache Kafka is a powerful combination to realize streaming analytics, from online recommendations to metrics applications. We will have a look at this setup in the context of popular architectures such as Lambda and Kappa and discuss requirements such as latency and processing guarantees in a practical setup. This talk includes a demo of a streaming analytics application using the Mesosphere DCOS as a deployment environment.
Manage your compactions before they manage you!
Manage your compactions before they manage you! This document discusses compaction in Cassandra databases. It covers why compaction is needed, different compaction strategies like size-tiered, levelled, and time-series, how to tune strategies, monitoring compactions, and disk tuning tips. The key messages are to select the right strategy for your usage, monitor compactions closely, and tune disk configuration, as compactions involve heavy I/O that must be well managed.
More Related Content
What's hot
Big Data Simplified - Is all about Ab'strakSHeN
This document discusses designing a new big data platform to replace an existing complex and outdated one. It analyzes challenges with the current platform, including inability to keep up with business needs. The proposed new platform called Dredge would use abstraction layers to integrate big data tools in a loosely coupled and scalable way. This would simplify development and maintenance while supporting business goals. Key aspects of Dredge include declarative configuration, logical workflows, and plug-and-play integration of tools like HDFS, Hive, HBase, Kafka and Spark in a reusable and event-driven manner. The new platform aims to improve scalability, reduce costs and better support analytics needs over time.
Is your Enterprise Data lake Metadata Driven AND Secure?
This document discusses how Apache Atlas and Apache Ranger can be used together to provide a metadata-driven and secure data lake. Apache Atlas provides metadata services and tagging capabilities. Apache Ranger uses the tags in Atlas to dynamically define and enforce access policies. The integration allows Ranger policies to automatically apply and change as Atlas metadata such as tags are updated. The document demonstrates how tags in Atlas for columns and tables can be used to create time-based and PII data access policies in Ranger.
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
In this interactive webinar, we'll walk through use cases on how you can use advanced analytics like SAS Visual Statistics and In-Memory Statistic with Hortonworks’ data platform (HDP) to reveal insights in your big data and redefine how your organization solves complex problems.
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
This document discusses using Hadoop and the Hortonworks Data Platform (HDP) for big data applications. It outlines how HDP can help organizations optimize their existing data warehouse, lower storage costs, unlock new applications from new data sources, and achieve an enterprise data lake architecture. The document also discusses how Talend's data integration platform can be used with HDP to easily develop batch, real-time, and interactive data integration jobs on Hadoop. Case studies show how companies have used Talend and HDP together to modernize their data architecture and product inventory and pricing forecasting.
Big Data at your Desk with KNIME
This document provides an agenda and overview of a presentation by Data Transformed on big data analytics using the KNIME platform. The presentation includes an introduction of Data Transformed and KNIME, a live demonstration of forecasting energy usage from customer data using KNIME and Hadoop tools, and a question and answer session. It promotes Data Transformed's services around data management, analytics, and consulting and highlights KNIME's capabilities for comprehensive data processing.
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...DataWorks Summit/Hadoop Summit
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditional EDW and Hadoop with Apache AtlasCloudy with a chance of Hadoop - real world considerations
Over the last eighteen months, we have seen significant adoption of Hadoop eco-system centric big data processing in Microsoft Azure and Amazon AWS. In this talk we present some of the lessons learned and architectural considerations for cloud-based deployments including security, fault tolerance and auto-scaling.
We look at how Hortonworks Data Cloud and Cloudbreak can automate that scaling of Hadoop clusters, showing how it can react dynamically to workloads, and what that can deliver in cost-effective Hadoop-in-cloud deployments.
Modernise your EDW - Data Lake
The document discusses modernizing enterprise data warehouses by using a Hadoop data lake solution with EMC Isilon storage. This provides benefits like offloading expensive ETL processing to reduce costs, archiving cold data for cheaper storage, and enabling analytics on new data sources like semi-structured data. The solution leverages Hortonworks Data Platform for open, interoperable analytics and provides enterprise-grade data management capabilities on Hadoop at lower costs than traditional EDWs.
Splunk-hortonworks-risk-management-oct-2014
Slide deck from Splunk and Hortonworks joint webinar on October 1, 2014. Title Building a Modern Data Architecture for Risk Management.
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Back in 2014, our team set out to change the way the world exchanges and collaborates with data. Our vision was to build a single tenant environment for multiple organisations to securely share and consume data. And we did just that, leveraging multiple Hadoop technologies to help our infrastructure scale quickly and securely.
Today Data Republic’s technology delivers a trusted platform for hundreds of enterprise level companies to securely exchange, commercialise and collaborate with large datasets.
Join Head of Engineering, Juan Delard de Rigoulières and Senior Solutions Architect, Amin Abbaspour as they share key lessons from their team’s journey with Hadoop:
* How a startup leveraged a clever combination of Hadoop technologies to build a secure data exchange platform
* How Hadoop technologies helped us deliver key solutions around governance, security and controls of data and metadata
* An evaluation on the maturity and usefulness of some Hadoop technologies in our environment: Hive, HDFS, Spark, Ranger, Atlas, Knox, Kylin: we've use them all extensively.
* Our bold approach to expose APIs directly to end users; as well as the challenges, learning and code we created in the process
* Learnings from the front-line: How our team coped with code changes, performance tuning, issues and solutions while building our data exchange
Whether you’re an enterprise level business or a start-up looking to scale - this case study discussion offers behind-the-scenes lessons and key tips when using Hadoop technologies to manage data governance and collaboration in the cloud.
Speakers:
Juan Delard De Rigoulieres, Head of Engineering, Data Republic Pty Ltd
Amin Abbaspour, Senior Solutions Architect, Data Republic
Discover.hdp2.2.h base.final[2]
Hortonworks Data Platform 2.2 includes Apache HBase for fast NoSQL data access. In this 30-minute webinar, we discussed HBase innovations that are included in HDP 2.2, including: support for Apache Slider; Apache HBase high availability (HA); block ache compression; and wire-level encryption.
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
This document discusses how organizations can leverage data and analytics to power their business models. It provides examples of Fortune 100 companies that are using Attunity products to build data lakes and ingest data from SAP and other sources into Hadoop, Apache Kafka, and the cloud in order to perform real-time analytics. The document outlines the benefits of Attunity's data replication tools for extracting, transforming, and loading SAP and other enterprise data into data lakes and data warehouses.
Scaling Data Science on Big Data
Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production ?
In this session learn the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of DSX with HDP with the focus on integration, security and model deployment and management.
Speakers:
Sriram Srinivasan, Senior Technical Staff Member, Analytics Platform Architect, IBM
Vikram Murali, Program Director, Data Science and Machine Learning, IBM
Swimming Across the Data Lake, Lessons learned and keys to success
This document discusses strategies for successfully utilizing a data lake. It notes that creating a data lake is just the beginning and that challenges include data governance, metadata management, access, and effective use of the data. The document advocates for data democratization through discovery, accessibility, and usability. It also discusses best practices like self-service BI and automated workload migration from data warehouses to reduce costs and risks. The key is to address the "data lake dilemma" of these challenges to avoid a "data swamp" and slow adoption.
Format Wars: from VHS and Beta to Avro and Parquet
The document discusses different data storage formats such as text, Avro, Parquet, and their suitability for writing and reading data. It provides examples of how to choose a format based on factors like query needs, data types, and whether schemas need to evolve. The document also demonstrates how Avro can handle schema evolution by adding or changing fields while still reading existing data.
HDFS: Optimization, Stabilization and Supportability
This document summarizes improvements made to HDFS to optimize performance, stabilize operations, and improve supportability. Key areas discussed include logging enhancements, metrics and tools for troubleshooting, load management through RPC improvements, and changes to reduce garbage collection overhead and improve liveness detection. Specific optimizations covered range from code changes to reduce logging verbosity to adding batch processing of block reports.
End-to-End Security and Auditing in a Big Data as a Service Deployment
End-to-End Security and Auditing in a Big Data as a Service DeploymentDataWorks Summit/Hadoop Summit
1. The document discusses security considerations for deploying big data as a service (BDaaS) across multiple tenants and applications. It focuses on maintaining a single user identity to prevent data duplication and enforce access policies consistently.
2. It describes using Apache Ranger to centrally define and enforce policies across Hadoop services like HDFS, HBase, Hive. Ranger integrates with LDAP/AD for authentication.
3. The key challenge is propagating user identities from the application layer to the data layer. This can be done by connecting HDFS directly via Kerberos or using a "super-user" that impersonates other users when accessing HDFS.Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the CloudDataWorks Summit/Hadoop Summit
The world’s largest enterprises run their infrastructure on Oracle, DB2 and SQL and their critical business operations on SAP applications. Organisations need this data to be available in real-time to conduct necessary analytics. However, delivering this heterogeneous data at the speed it’s required can be a huge challenge because of the complex underlying data models and structures and legacy manual processes which are prone to errors and delays.
Unlock these silos of data and enable the new advanced analytics platforms by attending this session.
Find out how to:
• To overcome common challenges faced by enterprises trying to access their SAP data
• You can integrate SAP data in real-time with change data capture (CDC) technology
• Organisations are using Attunity Replicate for SAP to stream SAP data in to KafkaHigh Performance Spatial-Temporal Trajectory Analysis with Spark
This document discusses high performance spatial-temporal trajectory analysis using Spark. It covers the background of analyzing mobile signaling data to enable smarter urban planning. The solution architecture includes data sources, distributed file system, computation engine, and visualization. Technical designs address the big data platform, data governance, algorithm models, and Spark spatial computing. Example scenarios are presented for population heatmaps, commute routes, and office-residence imbalance analysis.
Hybrid Data Platform
The document discusses the need for a hybrid data platform that combines on-premise and cloud-based infrastructure. A hybrid data platform can address issues like mixed workloads, long-tail data usage, and data movement costs. It also allows tapping into the scale of cloud while keeping some data and workloads on-premise. The proposed architecture shows data integration between on-premise Hadoop and cloud services like S3, Redshift, and EMR through private tunnels and security measures. Case studies demonstrate how a large commodities producer built a hybrid analytics environment for productivity, supply chain, and operations.
What's hot (20)
Is your Enterprise Data lake Metadata Driven AND Secure?
Is your Enterprise Data lake Metadata Driven AND Secure?
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) ...
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...
Implementing the Business Catalog in the Modern Enterprise: Bridging Traditio...
Cloudy with a chance of Hadoop - real world considerations
Cloudy with a chance of Hadoop - real world considerations
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Bring Your SAP and Enterprise Data to Hadoop, Kafka, and the Cloud
Swimming Across the Data Lake, Lessons learned and keys to success
Swimming Across the Data Lake, Lessons learned and keys to success
Format Wars: from VHS and Beta to Avro and Parquet
Format Wars: from VHS and Beta to Avro and Parquet
HDFS: Optimization, Stabilization and Supportability
HDFS: Optimization, Stabilization and Supportability
End-to-End Security and Auditing in a Big Data as a Service Deployment
End-to-End Security and Auditing in a Big Data as a Service Deployment
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
High Performance Spatial-Temporal Trajectory Analysis with Spark
High Performance Spatial-Temporal Trajectory Analysis with Spark
Viewers also liked
Cassandra summit 2015 - Simplifying Streaming Analytics
Using Apache Cassandra, Apache Spark and Apache Kafka is a powerful combination to realize streaming analytics, from online recommendations to metrics applications. We will have a look at this setup in the context of popular architectures such as Lambda and Kappa and discuss requirements such as latency and processing guarantees in a practical setup. This talk includes a demo of a streaming analytics application using the Mesosphere DCOS as a deployment environment.
Manage your compactions before they manage you!
Manage your compactions before they manage you! This document discusses compaction in Cassandra databases. It covers why compaction is needed, different compaction strategies like size-tiered, levelled, and time-series, how to tune strategies, monitoring compactions, and disk tuning tips. The key messages are to select the right strategy for your usage, monitor compactions closely, and tune disk configuration, as compactions involve heavy I/O that must be well managed.
A Generative Method for Infrastructure Emergence
Every society has structures which influence collective behavior. With all of the possible configurations of people in a population, the question emerges for designers of how to implement a method to use the collective information and create a successful design solution. We show how the fractal shape can emerge from a generative process that takes information on the scale of individuals or groups, and uses it to design a permanent infrastructure on the scale of a city. In this sense, we grow cities consisting of individuals and roads, starting from just individuals.
Streamline Hadoop DevOps with Apache Ambari
Ambari talk at Hadoop Summit, Tokyo 2016
Abstract
Apache Ambari has become an indispensable tool for operating Hadoop clusters from as small as 10s of nodes to 1000s of nodes. Ambari’s deep knowledge of the Hadoop stack allows it to deploy a cluster within minutes and manage the entire lifecycle: scaling, security, upgrades, and more. This talk will cover the central features important to cluster operators and the latest innovations from the community. We will discuss automatically deploying clusters with Blueprints, adding custom services, scaling the number of hosts as the data needs grow, adding High Availability for critical services, securing with MIT kerberos, and upgrading the Hadoop stack with features like Rolling & Express Upgrade. More advanced users will also be interested in using Ambari’s powerful REST API to automate workflows. For users and data scientists, Ambari provides LDAP sync, Role-Based Access Control to handle user permissions, and a framework to host Ambari Views such as as the newly added Views for Hive, Oozie, Capacity Scheduler, Tez, Storm, and Zeppelin. Lastly, we will cover how to monitor the health of the cluster via Alerts and troubleshoot problems by using new features like LogSearch and Ambari Metrics Systems integrated with Grafana UI.
SparkSQL et Cassandra - Tool In Action Devoxx 2015
Cassandra ne permet ni jointure, ni agrégats et limite drastiquement vos capacités à requêter vos données pour permettre une scalabilité linéaire dans une architecture masterless. L'outil de choix pour effectuer des traitements analytiques sur vos tables Cassandra est Spark mais ce dernier complexifie des opérations pourtant simples en SQL. SparkSQL permet de retrouver une syntaxe SQL dans Spark et nous allons voir comment l'utiliser en Scala, Java et en Python pour travailler sur des tables Cassandra, et retrouver jointures et agrégats (entre autres).
A complete hadoop stack
A brief presentation where I talk about an E2E Hadoop and open source data warehouse and BI stack - bringing in the power of hadoop and online dashboards.
The SparkSQL things you maybe confuse
SparkSQL, SchemaRDD, DataFrame, and Dataset are Apache Spark APIs for structured data processing. SparkSQL is a high-level module introduced in Spark 1.0. SchemaRDD was introduced in Spark 1.0 from the Shark project and was later renamed to DataFrame in Spark 1.3. Dataset, introduced experimentally in Spark 1.6, allows SparkSQL optimizations while working with RDDs. DataFrame and Dataset were unified under a single API in Spark 2.0.
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
Speaker: Russ Savage, from Cask
Big Data Applications Meetup, 09/14/2016
Palo Alto, CA
More info here: http://www.meetup.com/BigDataApps/
Link to talk: https://youtu.be/4j78g3WvC4Y
About the talk:
As data lake sizes grow, and more users begin exploring and including that data in their everyday analysis, keeping track of the sources for data becomes critical. Understanding how a dataset was generated and who is using it allows users and companies to ensure their analysis is leveraging the most accurate and up to date information. In this talk, we will explore the different techniques available to keep track of your data in your data lake and demonstrate how we at Cask approached and attempted to mitigate this issue.
Getting started with SparkSQL - Desert Code Camp 2016
The document discusses Spark SQL, an Apache Spark module for structured data processing. It provides an agenda that covers Spark concepts, Spark SQL, the Catalyst optimizer, Project Tungsten, and a demo. Spark SQL allows users to perform SQL queries and use the DataFrame and Dataset APIs to interact with structured data in a Spark cluster.
Introducing Athena: 08/19 Big Data Application Meetup, Talk #3
Athena is Uber's platform for stream processing built on Apache Samza. It uses Kafka as the messaging system and allows for near real-time computation of data streams. Some current uses at Uber include pricing calculations, event processing, and driver status updates. Athena provides tooling like a job manager, configuration files, testing frameworks, and integration with services like Graphite. Ongoing work includes improving upgrades, adding more metrics, and enhancing configuration validation.
ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...
Apache Tephra provides ACID transaction support for Apache HBase. It uses a transaction manager and transaction-aware clients to coordinate transaction lifecycles and ensure consistency across HBase regions. Transactions use multi-version concurrency control and optimistic concurrency control, allowing concurrent transactions while detecting conflicts at commit time to ensure isolation. Tephra integrates with Apache Phoenix to enable transactional operations on Phoenix tables.
Webinar: What's new in CDAP 3.5?
Cask Webinar
Date: 08/10/2016
Link to video recording: https://www.youtube.com/watch?v=XUkANr9iag0
In this webinar, Nitin Motgi, CTO of Cask, walks through the new capabilities of CDAP 3.5 and explains how your organization can benefit.
Some of the highlights include:
- Enterprise-grade security - Authentication, authorization, secure keystore for storing configurations. Plus integration with Apache Sentry and Apache Ranger.
- Preview mode - Ability to preview and debug data pipelines before deploying them.
- Joins in Cask Hydrator - Capabilities to join multiple data sources in data pipelines
- Real-time pipelines with Spark Streaming - Drag & drop real-time pipelines using Spark Streaming.
- Data usage analytics - Ability to report application usage of data sets.
- And much more!
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0
The document discusses scaling R using Hadoop and Spark. It provides an overview of IBM's approach to big data, which leverages open source technologies like Hadoop, Spark, and R. It then summarizes IBM's investments in Spark and the Open Data Platform initiative. The rest of the document focuses on describing Big R, IBM's tool for scaling R to big data using Hadoop. Big R allows users to run R scripts on large datasets in Hadoop and provides functions for machine learning algorithms and accessing Hadoop data from within R.
HBaseConEast2016: HBase and Spark, State of the Art
Jean-Marc Spaggiari of Cloudera at HBaseConEast2016: http://www.meetup.com/HBase-NYC/events/233024937/
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...
Many people refer to Apache Hadoop as their system of choice for big data management but few actually use just Apache Hadoop. Hadoop has become a proxy for a much larger system which has HDFS storage at its core. The Apache Hadoop based "big data stack" has changed dramatically over the past 24 months and will chance even more over the next 24 months. This talk talks about trends in the evolution of the Hadoop stack, change in architecture and changes in the kinds of use cases that are supported. It will also talk about the role of interoperability and cohesion in the Apache Hadoop stack and the role of Apache Bigtop in this regard.
Transactions Over Apache HBase
Transactions for Apache HBase™: Apache Tephra provides globally consistent transactions on top of Apache HBase. While HBase provides strong consistency with row- or region-level ACID operations, it sacrifices cross-region and cross-table consistency in favor of scalability. This trade-off requires application developers to handle the complexity of ensuring consistency when their modifications span region boundaries. By providing support for global transactions that span regions, tables, or multiple RPCs, Tephra simplifies application development on top of HBase, without a significant impact on performance or scalability for many workloads.
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
Apache Spark is the next big data processing tool for Data Scientist. As seen on the recent StackOverflow analysis, it's the hottest big data technology on their site! In this talk, I'll use the PySpark interface to leverage the speed and performance of Apache Spark. I'll focus on the end to end workflow for getting data into a distributed platform, and leverage Spark to process the data for advanced analytics. I'll discuss the popular Spark APIs used for data preparation, SQL analysis, and ML algorithms. I'll explain the performance differences between Scala and Python, and how Spark has bridged the gap in performance. I'll focus on PySpark as the interface to the platform, and walk through a demo to showcase the APIs.
Talk Overview:
Spark's Architecture. What's out now and what's in Spark 2.0Spark APIs: Most common APIs used by Spark Common misconceptions and proper techniques for using Spark.
Demo:
Walk through ETL of the Reddit dataset. SparkSQL Analytics + Visualizations of the Dataset using MatplotLibSentiment Analysis on Reddit Comments
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Given at Data Day Seattle 2015.
Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.
Spark meetup v2.0.5
In this talk, we’ll discuss technical designs of support of HBase as a “native” data source to Spark SQL to achieve both query and load performance and scalability: near-precise execution locality of query and loading, fine-tuned partition pruning, predicate pushdown, plan execution through coprocessor, and optimized and fully parallelized bulk loader. Point and range queries on dimensional attributes will benefit particularly well from the techniques. Preliminary test results vs. established SQL-on-HBase technologies will be provided. The speaker will also share the future plan and real-world use cases, particularly in the telecom industry.
Parquet Hadoop Summit 2013
Parquet is a columnar storage format for Hadoop data. It was developed by Twitter and Cloudera to optimize storage and querying of large datasets. Parquet provides more efficient compression and I/O compared to traditional row-based formats by storing data by column. Early results show a 28% reduction in storage size and up to a 114% improvement in query performance versus the original Thrift format. Parquet supports complex nested schemas and can be used with Hadoop tools like Hive, Pig, and Impala.
Viewers also liked (20)
Cassandra summit 2015 - Simplifying Streaming Analytics
Cassandra summit 2015 - Simplifying Streaming Analytics
SparkSQL et Cassandra - Tool In Action Devoxx 2015
SparkSQL et Cassandra - Tool In Action Devoxx 2015
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
"Who Moved my Data? - Why tracking changes and sources of data is critical to...
Getting started with SparkSQL - Desert Code Camp 2016
Getting started with SparkSQL - Desert Code Camp 2016
Introducing Athena: 08/19 Big Data Application Meetup, Talk #3
Introducing Athena: 08/19 Big Data Application Meetup, Talk #3
ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...
ACID Transactions in Apache Phoenix with Apache Tephra™ (incubating), by Poor...
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0
HBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the Art
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Similar to The DAP - Where YARN, HBase, Kafka and Spark go to Production
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Hadoop or Spark: is it an either-or proposition? An exodus away from Hadoop to Spark is picking up steam in the news headlines and talks! Away from marketing fluff and politics, this talk analyzes such news and claims from a technical perspective.
In practical ways, while referring to components and tools from both Hadoop and Spark ecosystems, this talk will show that the relationship between Hadoop and Spark is not of an either-or type but can take different forms such as: evolution, transition, integration, alternation and complementarity.
Transitioning Compute Models: Hadoop MapReduce to Spark
This presentation is an analysis of the observed trends in the transition from the Hadoop ecosystem to the Spark ecosystem. The related talk took place at the Chicago Hadoop User Group (CHUG) meetup held on February 12, 2015.
spark_v1_2
Spark is an open-source cluster computing framework that can run analytics applications much faster than Hadoop by keeping data in memory rather than on disk. While Spark can access Hadoop's HDFS storage system and is often used as a replacement for Hadoop's MapReduce, Hadoop remains useful for batch processing and Spark is not expected to fully replace it. Spark provides speed, ease of use, and integration of SQL, streaming, and machine learning through its APIs in multiple languages.
Accelerating R analytics with Spark and Microsoft R Server for Hadoop
This document discusses how Microsoft R Server accelerates predictive analytics by enabling R users to leverage the performance of Apache Spark on Hadoop clusters without having to learn Spark or Hadoop. It summarizes that Microsoft R Server for Hadoop allows R users to conduct data exploration, transformation, modeling and other analytics on large datasets in Hadoop using R, while Microsoft R Server handles the parallelization of R scripts across Spark to provide significantly faster performance compared to traditional Hadoop approaches.
The Big Picture on Hadoop
Apache Hadoop is an open source software framework created in 2005. Engineered for Big Data and large-scale procession applications.
Spark vs Hadoop: Which Big Data Framework to Choose?
This is a comparison presentation between two popular big data frameworks, Hadoop and Spark. Here you will get detailed information about their pros and cons, alongside getting familiar with different factors to consider during ‘Spark vs Hadoop’ battle.
Hadoop Vs Spark — Choosing the Right Big Data Framework
The data is increasing, and to digest all this data, there are many distributed systems available. Hadoop and Spark are the most famous ones. Choosing one out of two depends entirely upon the requirement of your project. Read more to know which of these two frameworks is right for you.
View on big data technologies
Big Data is still a challenge for many companies to collect, process, and analyze large amounts of structured and unstructured data. Hadoop provides an open source framework for distributed storage and processing of large datasets across commodity servers to help companies gain insights from big data. While Hadoop is commonly used, Spark is becoming a more popular tool that can run 100 times faster for iterative jobs and integrates with SQL, machine learning, and streaming technologies. Both Hadoop and Spark often rely on the Hadoop Distributed File System for storage and are commonly implemented together in big data projects and platforms from major vendors.
Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...
Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...Impetus Technologies
In spite of investments in big data lakes, there is wide use of expensive proprietary products for data ingestion, integration, and transformation (ETL) while bringing and processing data on the lake.
Enterprises have successfully tested Apache Spark for its versatility and strengths as a distributed computing framework that can completely handle all needs for data processing, analytics, and machine learning workloads.
Since the Hadoop distributions and the public cloud already include Apache Spark, there is nothing new to be procured. However, the skills required to put Spark to good use are typically unavailable today.
In this webinar, we will discuss how Apache Spark can be an inexpensive enterprise backbone for all types of data processing workloads. We will also demo how a visual framework on top of Apache Spark makes it much more viable.
The following scenarios will be covered:
On-Prem
Data quality and ETL with Apache Spark using pre-built operators
Advanced monitoring of Spark pipelines
On Cloud
Visual interactive development of Apache Spark Structured Streaming pipelines
IoT use-case with event-time, late-arrival and watermarks
Python based predictive analytics running on SparkSunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala
Mopuru Babu has over 9 years of experience in software development using Java technologies and 3 years experience in Hadoop development. He has extensive experience designing, developing, and deploying multi-tier and enterprise-level distributed applications. He has expertise in technologies like Hadoop, Hive, Pig, Spark, and frameworks like Spring and Struts. He has worked on both small and large projects for clients in various industries.
Athens BigData Meetup - Sept 17
Landoop presentation in the Athens Big Data meetup, about streaming technologies on Apache Kafka. Introduction to the Lenses SQL engine and the Lenses platform and our open-source projects.
SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments
SnapLogic announced the Fall 2014 release of its integration platform with new capabilities for data wrangling and enhanced support for Hadoop 2.0 deployments. Key updates included the Hadooplex option to run SnapLogic pipelines natively on Hadoop as YARN applications and expanded support for Hadoop file formats. The release also improved usability, deployability, security, and performance. SnapLogic's enhancements help solidify its position as a provider of integration solutions for big data environments.
Custom Development - SAP HANA
This document summarizes a presentation on SAP HANA custom development. It introduces the presenter and outlines their experience working with SAP technologies. The presentation agenda is then outlined, covering the history and evolution of SAP HANA runtimes, programming models, and services. It also provides a sneak peek at a live demo and discusses differences between ABAP, HANA and HANA custom development.
Sandish3Certs
The document provides a summary of a senior big data consultant with over 4 years of experience working with technologies such as Apache Spark, Hadoop, Hive, Pig, Kafka and databases including HBase, Cassandra. The consultant has strong skills in building real-time streaming solutions, data pipelines, and implementing Hadoop-based data warehouses. Areas of expertise include Spark, Scala, Java, machine learning, and cloud platforms like AWS.
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Introduction to Big Data Analytics using Apache Spark on HDInsights on Azure (SaaS) and/or HDP on Azure(PaaS)
This workshop will provide an introduction to Big Data Analytics using Apache Spark using the HDInsights on Azure (SaaS) and/or HDP deployment on Azure(PaaS) . There will be a short lecture that includes an introduction to Spark, the Spark components.
Spark is a unified framework for big data analytics. Spark provides one integrated API for use by developers, data scientists, and analysts to perform diverse tasks that would have previously required separate processing engines such as batch analytics, stream processing and statistical modeling. Spark supports a wide range of popular languages including Python, R, Scala, SQL, and Java. Spark can read from diverse data sources and scale to thousands of nodes.
The lecture will be followed by demo . There will be a short lecture on Hadoop and how Spark and Hadoop interact and compliment each other. You will learn how to move data into HDFS using Spark APIs, create Hive table, explore the data with Spark and SQL, transform the data and then issue some SQL queries. We will be using Scala and/or PySpark for labs.
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Analytics and machine learning continue to be the top use cases for deploying big data platforms such as Hadoop. SAS recognised the potential and power of Hadoop platform early on and has been integrating analytical solutions with Hadoop to leverage the power and flexibility of Hadoop for analytical workloads. The combination of SAS and Hadoop offers developers and organisations an approach that can accelerate the development and deployment of big data analytics applications that are mature, proven and scalable. Furthermore, by giving developers and analysts analytical applications that are rich, proven and collaborative, SAS allows more users across different skill levels to unleash the value of data stored in big data platform more easily and quickly.
In this session, we will cover common big data analytics use cases, the depth and breadth of SAS analytical capabilities on Hadoop, and how SAS solutions are integrated into the Hadoop ecosystem via technologies such as Hive, YARN and Spark.
Speaker
Felix Liao, SAS Institute Australia & New Zealand
Emerging trends in data analytics
- Deep learning (Tensorflow) and microservices (Kubernetes, Docker, Kafka) are emerging trends.
- While batch processing is still dominant, stream processing is gaining traction with technologies like Kafka, Flink, and Beam.
- Python has surpassed Java as the most popular language for data analytics, according to Stack Overflow trends.
Storm Demo Talk - Colorado Springs May 2015
The document discusses real-time processing capabilities in Hadoop and Hortonworks Data Platform (HDP). It begins with an introduction to Hortonworks and an overview of real-time streaming architectures on HDP. It then demonstrates streaming capabilities with and without predictive analytics additions. The document highlights how HDP provides a centralized architecture and open data platform to enable real-time and batch processing of any type of data for analytics applications.
Learn Apache Spark: A Comprehensive Guide
Want to learn Apache Spark and become big data expert in 2018? This guide will help you learn everything you need to know about Apache Spark!
Leveraging SAP HANA with Apache Hadoop and SAP Analytics
The rise of big data and the Apache Hadoop platform allows for the capture and processing of data at an unprecedented scale and velocity. Watch this slide deck to get a comprehensive overview of the Apache Hadoop platform architecture and learn how to leverage the strengths of both the Apache Hadoop and SAP HANA platforms.
Similar to The DAP - Where YARN, HBase, Kafka and Spark go to Production (20)
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Transitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to Spark
Accelerating R analytics with Spark and Microsoft R Server for Hadoop
Accelerating R analytics with Spark and Microsoft R Server for Hadoop
Spark vs Hadoop: Which Big Data Framework to Choose?
Spark vs Hadoop: Which Big Data Framework to Choose?
Hadoop Vs Spark — Choosing the Right Big Data Framework
Hadoop Vs Spark — Choosing the Right Big Data Framework
Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...
Apache Spark – The New Enterprise Backbone for ETL, Batch Processing and Real...
Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala
Sunshine consulting mopuru babu cv_java_j2ee_spring_bigdata_scala
SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments
SnapLogic Enhancements Support iPaaS for Hadoop 2.0 Environments
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Leveraging SAP HANA with Apache Hadoop and SAP Analytics
Leveraging SAP HANA with Apache Hadoop and SAP Analytics
More from DataWorks Summit/Hadoop Summit
Running Apache Spark & Apache Zeppelin in Production
This document discusses running Apache Spark and Apache Zeppelin in production. It begins by introducing the author and their background. It then covers security best practices for Spark deployments, including authentication using Kerberos, authorization using Ranger/Sentry, encryption, and audit logging. Different Spark deployment modes like Spark on YARN are explained. The document also discusses optimizing Spark performance by tuning executor size and multi-tenancy. Finally, it covers security features for Apache Zeppelin like authentication, authorization, and credential management.
State of Security: Apache Spark & Apache Zeppelin
This document discusses Spark security and provides an overview of authentication, authorization, encryption, and auditing in Spark. It describes how Spark leverages Kerberos for authentication and uses services like Ranger and Sentry for authorization. It also outlines how communication channels in Spark are encrypted and some common issues to watch out for related to Spark security.
Unleashing the Power of Apache Atlas with Apache Ranger
The document discusses the Virtual Data Connector project which aims to leverage Apache Atlas and Apache Ranger to provide unified metadata and access governance across data sources. Key points include:
- The project aims to address challenges of understanding, governing, and controlling access to distributed data through a centralized metadata catalog and policies.
- Apache Atlas provides a scalable metadata repository while Apache Ranger enables centralized access governance. The project will integrate these using a virtualization layer.
- Enhancements to Atlas and Ranger are proposed to better support the project's goals around a unified open metadata platform and metadata-driven governance.
- An initial minimum viable product will be built this year with the goal of an open, collaborative ecosystem around shared
Enabling Digital Diagnostics with a Data Science Platform
This document discusses using a data science platform to enable digital diagnostics in healthcare. It provides an overview of healthcare data sources and Yale/YNHH's data science platform. It then describes the data science journey process using a clinical laboratory use case as an example. The goal is to use big data and machine learning to improve diagnostic reproducibility, throughput, turnaround time, and accuracy for laboratory testing by developing a machine learning algorithm and real-time data processing pipeline.
Revolutionize Text Mining with Spark and Zeppelin
This document discusses using Apache Spark and MLlib for text mining on big data. It outlines common text mining applications, describes how Spark and MLlib enable scalable machine learning on large datasets, and provides examples of text mining workflows and pipelines that can be built with Spark MLlib algorithms and components like tokenization, feature extraction, and modeling. It also discusses customizing ML pipelines and the Zeppelin notebook platform for collaborative data science work.
Double Your Hadoop Performance with Hortonworks SmartSense
This document compares the performance of Hive and Spark when running the BigBench benchmark. It outlines the structure and use cases of the BigBench benchmark, which aims to cover common Big Data analytical properties. It then describes sequential performance tests of Hive+Tez and Spark on queries from the benchmark using a HDInsight PaaS cluster, finding variations in performance between the systems. Concurrency tests are also run by executing multiple query streams in parallel to analyze throughput.
Hadoop Crash Course
The document discusses modern data applications and architectures. It introduces Apache Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. Hadoop provides massive scalability and easy data access for applications. The document outlines the key components of Hadoop, including its distributed storage, processing framework, and ecosystem of tools for data access, management, analytics and more. It argues that Hadoop enables organizations to innovate with all types and sources of data at lower costs.
Data Science Crash Course
This document provides an overview of data science and machine learning. It discusses what data science and machine learning are, including extracting insights from data and computers learning without being explicitly programmed. It also covers Apache Spark, which is an open source framework for large-scale data processing. Finally, it discusses common machine learning algorithms like regression, classification, clustering, and dimensionality reduction.
Apache Spark Crash Course
This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
Dataflow with Apache NiFi
This document provides an overview of Apache NiFi and dataflow. It begins with an introduction to the challenges of moving data effectively within and between systems. It then discusses Apache NiFi's key features for addressing these challenges, including guaranteed delivery, data buffering, prioritized queuing, and data provenance. The document outlines NiFi's architecture and components like repositories and extension points. It also previews a live demo and invites attendees to further discuss Apache NiFi at a Birds of a Feather session.
Schema Registry - Set you Data Free
Many Organizations are currently processing various types of data and in different formats. Most often this data will be in free form, As the consumers of this data growing it’s imperative that this free-flowing data needs to adhere to a schema. It will help data consumers to have an expectation of about the type of data they are getting and also they will be able to avoid immediate impact if the upstream source changes its format. Having a uniform schema representation also gives the Data Pipeline a really easy way to integrate and support various systems that use different data formats.
SchemaRegistry is a central repository for storing, evolving schemas. It provides an API & tooling to help developers and users to register a schema and consume that schema without having any impact if the schema changed. Users can tag different schemas and versions, register for notifications of schema changes with versions etc.
In this talk, we will go through the need for a schema registry and schema evolution and showcase the integration with Apache NiFi, Apache Kafka, Apache Storm.
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...DataWorks Summit/Hadoop Summit
There is increasing need for large-scale recommendation systems. Typical solutions rely on periodically retrained batch algorithms, but for massive amounts of data, training a new model could take hours. This is a problem when the model needs to be more up-to-date. For example, when recommending TV programs while they are being transmitted the model should take into consideration users who watch a program at that time.
The promise of online recommendation systems is fast adaptation to changes, but methods of online machine learning from streams is commonly believed to be more restricted and hence less accurate than batch trained models. Combining batch and online learning could lead to a quickly adapting recommendation system with increased accuracy. However, designing a scalable data system for uniting batch and online recommendation algorithms is a challenging task. In this talk we present our experiences in creating such a recommendation engine with Apache Flink and Apache Spark.Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...DataWorks Summit/Hadoop Summit
DeepLearning is not just a hype - it outperforms state-of-the-art ML algorithms. One by one. In this talk we will show how DeepLearning can be used for detecting anomalies on IoT sensor data streams at high speed using DeepLearning4J on top of different BigData engines like ApacheSpark and ApacheFlink. Key in this talk is the absence of any large training corpus since we are using unsupervised machine learning - a domain current DL research threats step-motherly. As we can see in this demo LSTM networks can learn very complex system behavior - in this case data coming from a physical model simulating bearing vibration data. Once draw back of DeepLearning is that normally a very large labaled training data set is required. This is particularly interesting since we can show how unsupervised machine learning can be used in conjunction with DeepLearning - no labeled data set is necessary. We are able to detect anomalies and predict braking bearings with 10 fold confidence. All examples and all code will be made publicly available and open sources. Only open source components are used.Mool - Automated Log Analysis using Data Science and ML
QE automation for large systems is a great step forward in increasing system reliability. In the big-data world, multiple components have to come together to provide end-users with business outcomes. This means, that QE Automations scenarios need to be detailed around actual use cases, cross-cutting components. The system tests potentially generate large amounts of data on a recurring basis, verifying which is a tedious job. Given the multiple levels of indirection, the false positives of actual defects are higher, and are generally wasteful.
At Hortonworks, we’ve designed and implemented Automated Log Analysis System - Mool, using Statistical Data Science and ML. Currently the work in progress has a batch data pipeline with a following ensemble ML pipeline which feeds into the recommendation engine. The system identifies the root cause of test failures, by correlating the failing test cases, with current and historical error records, to identify root cause of errors across multiple components. The system works in unsupervised mode with no perfect model/stable builds/source-code version to refer to. In addition the system provides limited recommendations to file/open past tickets and compares run-profiles with past runs.
How Hadoop Makes the Natixis Pack More Efficient
Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together.
This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear:
• How and why the business and IT requirements originated
• How we leverage the platform to fulfill security and production requirements
• How we organize a community to:
o Guard all the players, no one gets left on the ground!
o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead)
• What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community
We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match!
DETAILS
This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.
HBase in Practice
HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)DataWorks Summit/Hadoop Summit
There has been an explosion of data digitising our physical world – from cameras, environmental sensors and embedded devices, right down to the phones in our pockets. Which means that, now, companies have new ways to transform their businesses – both operationally, and through their products and services – by leveraging this data and applying fresh analytical techniques to make sense of it. But are they ready? The answer is “no” in most cases.
In this session, we’ll be discussing the challenges facing companies trying to embrace the Analytics of Things, and how Teradata has helped customers work through and turn those challenges to their advantage.Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...DataWorks Summit/Hadoop Summit
In high-risk manufacturing industries, regulatory bodies stipulate continuous monitoring and documentation of critical product attributes and process parameters. On the other hand, sensor data coming from production processes can be used to gain deeper insights into optimization potentials. By establishing a central production data lake based on Hadoop and using Talend Data Fabric as a basis for a unified architecture, the German pharmaceutical company HERMES Arzneimittel was able to cater to compliance requirements as well as unlock new business opportunities, enabling use cases like predictive maintenance, predictive quality assurance or open world analytics. Learn how the Talend Data Fabric enabled HERMES Arzneimittel to become data-driven and transform Big Data projects from challenging, hard to maintain hand-coding jobs to repeatable, future-proof integration designs.
Talend Data Fabric combines Talend products into a common set of powerful, easy-to-use tools for any integration style: real-time or batch, big data or master data management, on-premises or in the cloud.Backup and Disaster Recovery in Hadoop
While you could be tempted assuming data is already safe in a single Hadoop cluster, in practice you have to plan for more. Questions like: "What happens if the entire datacenter fails?, or "How do I recover into a consistent state of data, so that applications can continue to run?" are not a all trivial to answer for Hadoop. Did you know that HDFS snapshots are handling open files not as immutable? Or that HBase snapshots are executed asynchronously across servers and therefore cannot guarantee atomicity for cross region updates (which includes tables)? There is no unified and coherent data backup strategy, nor is there tooling available for many of the included components to build such a strategy. The Hadoop distributions largely avoid this topic as most customers are still in the "single use-case" or PoC phase, where data governance as far as backup and disaster recovery (BDR) is concerned are not (yet) important. This talk first is introducing you to the overarching issue and difficulties of backup and data safety, looking at each of the many components in Hadoop, including HDFS, HBase, YARN, Oozie, the management components and so on, to finally show you a viable approach using built-in tools. You will also learn not to take this topic lightheartedly and what is needed to implement and guarantee a continuous operation of Hadoop cluster based solutions.
More from DataWorks Summit/Hadoop Summit (20)
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
Recently uploaded
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Pitangent Analytics & Technology Solutions Pvt. Ltd
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfarePapadakis K.-Cyber-Information Warfare Analyst & Cyber Defense/Security Consultant-Hellenic MoD
AI & Electronic WarfareConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Recently uploaded (20)
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
The DAP - Where YARN, HBase, Kafka and Spark go to Production
- 1. The DAP – Where YARN, HBase, Kafka and Spark Go to Production Hadoop Summit - June 30th, 2016 cask.co Cask, CDAP, Cask Hydrator and Cask Tracker are trademarks or registered trademarks of Cask Data. Apache Spark, Spark, the Spark logo, Apache Hadoop, Hadoop and the Hadoop logo are trademarks or registered trademarks of the Apache Software Foundation. All other trademarks and registered trademarks are the property of their respective owners.
- 3. cask.co The Many Faces of Hadoop 3 Developer Data Scientist IT Pro / Ops LOB Manager Advanced Programming Focuses on App Logic Basic Programming Focuses on Data Configuration & Monitoring Focuses on Operations Analysis & Decision Making Focuses on Insights
- 5. cask.co Building a Big Data App 5
- 6. cask.co Deploying and Operating a Big Data App 6
- 7. cask.co Today’s Integration Solutions are Silo’ed 7 Data Integration App Integration Cloud Integration Governance
- 9. cask.co9 Enter Cask Key Customers and Partners Named a Gartner Cool Vendor 2016 Founded in 2011 by early Hadoop engineers from Facebook and Yahoo!
- 10. cask.co Introducing the Cask Data App Platform 10
- 11. cask.co CDAP Overview 11 Open Source, Integrated Framework for Building and Running Data Applications on Hadoop and Spark
- 12. cask.co12 ● Provides a platform with framework level correctness ● Dataset abstractions & self-service data ● One framework: Prototype to Production ● Unified approach across all paradigms ○ Metrics & Log collection ○ Lineage, Audit, Access Control CDAP Consolidates Big Data App Lifecycle
- 14. cask.co CDAP Architecture 14 ● Application Container Architecture ● Reusable Programming Abstractions ● Global User and Machine Metadata
- 17. cask.co Hadoop in the Enterprise – Simplified with CDAP 17