Adam bosc-071114
•
1 like•903 views
ADAM is an open source, scalable genome analysis platform developed by researchers at UC Berkeley and other institutions. It includes tools for processing, analyzing and accessing large genomic datasets using Apache Spark. ADAM provides efficient data formats, rich APIs, and scalable algorithms to allow genome analysis to be performed on clusters and clouds. The goal is to enable fast, distributed analysis of genomic data across platforms while enhancing data access and flexibility.
Report
Share
Report
Share
Download to read offline
Recommended
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Surge 2013 presentation which covers how Netflix maximizes engineering velocity while keeping risks to scalability, reliability, and performance in check.
Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bour...
This document summarizes a presentation about linking reactive applications to Spark Streaming using Reactive Streams. It discusses back pressure in Spark Streaming, how Spark 1.5 introduced dynamic rate limiting to support back pressure, and how the rate is estimated using a PID controller. It also describes reactive applications as being responsive, resilient, elastic, and message-driven. Reactive Streams is presented as a specification that allows connecting systems using a back pressure interface in the JVM. Finally, it demonstrates how end-to-end back pressure can be achieved between a reactive application, Spark Streaming, and a Reactive Streams receiver.
Performance improvements in etcd 3.5 release
Etcd is a distributed key-value store that provides strong consistency. The presenter discussed recent performance improvements made in Etcd 3.5, including optimizing concurrent read transactions to reduce latency by 90% while improving throughput. Inefficient warning log calls were optimized to reduce memory usage by up to 50%. A new benchmark command was added to test mixed read/write workloads. Testing showed Etcd 3.5 has lower CPU and memory costs than previous versions under the same workloads. Further work is ongoing to improve various Etcd components and support multiple database backends.
Prashant Vichare Resume
Prashant Vichare is seeking a software development position utilizing his skills in cloud computing, virtualization, and networking technologies. He has a Master's degree in Computer Networking from North Carolina State University and a Bachelor's degree in Electronics Engineering. His experience includes internships at Cisco Systems and Citi Research Technology where he developed automation testing tools and delivered application functionality. Prashant has strong skills in programming languages, networking, Linux, AWS, Docker, Hadoop, and data analysis tools. At NCSU he completed projects involving DevOps pipelines, IoT device modeling, key-value filesystems, and PaaS for big data analytics.
Scaling Graphite At Yelp
This document discusses metrics and the Graphite monitoring system. It describes the main components of Graphite including Carbon, which persists metrics to disk and supports replication and sharding. It also describes Whisper for data storage and aggregation, and the web interface for rendering graphs. The document provides an overview of how these components work together and tips for optimizing performance such as aggregating metrics before ingestion and controlling the metrics that can be sent. It also briefly mentions alternative time-series databases like InfluxDB and Cassandra that could be used in the future.
Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013
This document discusses the care and feeding of large scale Graphite installations. It begins with introductions and then discusses Graphite components like carbon-cache, carbon-aggregator, carbon-relay and StatsD. It covers Graphite storage, installation, documentation, middleware, backups, monitoring and the web UI. It provides tips on tuning, debugging and visualizing metrics in Graphite.
Vineetha.ppt
The document proposes improving the performance of the Needleman-Wunsch algorithm for global sequence alignment to efficiently handle next-generation sequencing (NGS) data. It suggests implementing the algorithm using approximation algorithms, parallel processing on Apache Hadoop, and running on low-cost commodity hardware. The results show that a Hadoop version of the algorithm performs better than a sequential version as input size increases. The improved algorithm allows for parallel execution, fault tolerance, and efficient handling of massive NGS data loads.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Recommended
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Surge 2013 presentation which covers how Netflix maximizes engineering velocity while keeping risks to scalability, reliability, and performance in check.
Reactive Streams, linking Reactive Application to Spark Streaming by Luc Bour...
This document summarizes a presentation about linking reactive applications to Spark Streaming using Reactive Streams. It discusses back pressure in Spark Streaming, how Spark 1.5 introduced dynamic rate limiting to support back pressure, and how the rate is estimated using a PID controller. It also describes reactive applications as being responsive, resilient, elastic, and message-driven. Reactive Streams is presented as a specification that allows connecting systems using a back pressure interface in the JVM. Finally, it demonstrates how end-to-end back pressure can be achieved between a reactive application, Spark Streaming, and a Reactive Streams receiver.
Performance improvements in etcd 3.5 release
Etcd is a distributed key-value store that provides strong consistency. The presenter discussed recent performance improvements made in Etcd 3.5, including optimizing concurrent read transactions to reduce latency by 90% while improving throughput. Inefficient warning log calls were optimized to reduce memory usage by up to 50%. A new benchmark command was added to test mixed read/write workloads. Testing showed Etcd 3.5 has lower CPU and memory costs than previous versions under the same workloads. Further work is ongoing to improve various Etcd components and support multiple database backends.
Prashant Vichare Resume
Prashant Vichare is seeking a software development position utilizing his skills in cloud computing, virtualization, and networking technologies. He has a Master's degree in Computer Networking from North Carolina State University and a Bachelor's degree in Electronics Engineering. His experience includes internships at Cisco Systems and Citi Research Technology where he developed automation testing tools and delivered application functionality. Prashant has strong skills in programming languages, networking, Linux, AWS, Docker, Hadoop, and data analysis tools. At NCSU he completed projects involving DevOps pipelines, IoT device modeling, key-value filesystems, and PaaS for big data analytics.
Scaling Graphite At Yelp
This document discusses metrics and the Graphite monitoring system. It describes the main components of Graphite including Carbon, which persists metrics to disk and supports replication and sharding. It also describes Whisper for data storage and aggregation, and the web interface for rendering graphs. The document provides an overview of how these components work together and tips for optimizing performance such as aggregating metrics before ingestion and controlling the metrics that can be sent. It also briefly mentions alternative time-series databases like InfluxDB and Cassandra that could be used in the future.
Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013
This document discusses the care and feeding of large scale Graphite installations. It begins with introductions and then discusses Graphite components like carbon-cache, carbon-aggregator, carbon-relay and StatsD. It covers Graphite storage, installation, documentation, middleware, backups, monitoring and the web UI. It provides tips on tuning, debugging and visualizing metrics in Graphite.
Vineetha.ppt
The document proposes improving the performance of the Needleman-Wunsch algorithm for global sequence alignment to efficiently handle next-generation sequencing (NGS) data. It suggests implementing the algorithm using approximation algorithms, parallel processing on Apache Hadoop, and running on low-cost commodity hardware. The results show that a Hadoop version of the algorithm performs better than a sequential version as input size increases. The improved algorithm allows for parallel execution, fault tolerance, and efficient handling of massive NGS data loads.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Fast and Reliable Apache Spark SQL Engine
Building the next generation Spark SQL engine at speed poses new challenges to both automation and testing. At Databricks, we are implementing a new testing framework for assessing the quality and performance of new developments as they produced. Having more than 1,200 worldwide contributors, Apache Spark follows a rapid pace of development. At this scale, new testing tooling such as random query and data generation, fault injection, longevity stress, and scalability tests are essential to guarantee a reliable and performance Spark later in production. By applying such techniques, we will demonstrate the effectiveness of our testing infrastructure by drilling-down into cases where correctness and performance regressions have been found early. In addition, showing how they have been root-caused and fixed to prevent regressions in production and boosting the continuous delivery of new features.
Dataflow in 104corp - AWS UserGroup TW 2018
This document discusses migrating data processing workflows from on-premises to cloud-based serverless architectures. It outlines reasons for upgrading systems like HDFS and Pig to cloud services including AWS EMR, Kinesis, and S3 for improved resource utilization, high availability, and performance. The document then details considerations for how to migrate components like streaming, storage, computing, exploration and serving to various AWS services, and concludes with recommendations to leverage AWS services where possible for ease of use while balancing controllability and cost versus performance.
IGUANA: A Generic Framework for Benchmarking the Read-Write Performance of Tr...
Iguana is a framework for benchmarking the read-write performance of triple stores. It provides a realistic scenario by simulating multiple concurrent users querying and updating a triple store. Iguana executes benchmarks on different datasets and triple stores, measuring key performance indicators like queries per second. Results are stored in files and triple stores for analysis. The framework is extensible and can benchmark any dataset, SPARQL/update queries, and triple store configuration.
Realizing the promise of portability with Apache Beam
The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms.
In this talk, I will:
Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem.
Discuss the benefits Beam provides regarding portability and ease-of-use.
Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise).
Give a glimpse at some of the challenges Beam aims to address in the future.
Monitoring Microservices
By Tom Wilkie, delivered at London Microservices User Group on 2/12/15
The rise of microservice-based applications has had many knock-on effects, not least on the complexity of monitoring your application. Order-of-magnitude increase in the number of moving parts and rate of change of the application require us to reassess traditional monitoring techniques.
In this talk we will discuss some different approaches to monitoring, visualising and tracing containerised, microservices-based applications. We’ll present different techniques to some of the emergent problems, and try not to rant too much.
Flink Forward Berlin 2017: Francesco Versaci - Integrating Flink and Kafka in...
High-throughput DNA sequencing is a key data acquisition technology which enables dozens of important applications, from oncology to personalized diagnostics. We extended work presented last year to port additional portions of the standard genomics data processing pipeline to Flink. Our Flink-based processor consists of two distinct specialized modules (reader and writer) that are loosely linked via Kafka streams, thus allowing for easy composability and integration into already existing Hadoop workflows. To extend our work we had to manage the dynamical creation and detection of the data streams: the set of output files is not known in advance by the writer, which learns it at running time. Particular care had to be taken to handle the finite nature of the genomic streams: since we use some already existing Hadoop output formats, we had to properly handle the flow of end-of-streams markers through Flink and Kafka, in order to have the final output files correctly finalized.
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Flink provides fault tolerance guarantees through checkpointing and recovery mechanisms. Checkpoints take consistent snapshots of distributed state and data, while barriers mark checkpoints in the data flow. This allows Flink to recover jobs from failures and resume processing from the last completed checkpoint. Flink also implements high availability by persisting metadata like the execution graph and checkpoints to Apache Zookeeper, enabling a standby JobManager to take over if the active one fails.
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Mac...
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai"
Kai Wähner works as Technology Evangelist at Confluent. Kai’s main area of expertise lies within the fields of Big Data Analytics, Machine Learning, Integration, Microservices, Internet of Things, Stream Processing and Blockchain. He is regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture or ApacheCon, writes articles for professional journals, and shares his experiences with new technologies on his blog (www.kai-waehner.de/blog). Contact and references: kontakt@kai-waehner.de / @KaiWaehner / www.kai-waehner.de
Abstract:
Big Data and Machine Learning are key for innovation in many industries today. The first part of this session explains how to build analytic models with R, Python or Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses the deployment of these built analytic models to your own applications or microservices leveraging the Apache Kafka cluster and Kafka Streams. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
Circonus: Design failures - A Case Study
SRECon18 Keynote https://www.usenix.org/conference/srecon18europe/presentation/schlossnagle
Joint talk Theo Schlossnagle (p1-26), Heinrich Hartmann (p26-44ff).
Spark Summit EU talk by Sital Kedia
This document summarizes a case study of using Apache Spark at scale to process over 60 TB of data at Facebook. It describes the previous implementation using Hive, which involved many small jobs that were unmanageable and slow. The new Spark implementation processes the entire dataset in a single job with two stages, shuffling over 90 TB of intermediate data. It provides performance comparisons showing significant reductions in CPU time, latency, and resource usage. It also details reliability and performance improvements made to Spark, such as fixing memory leaks, enabling seamless cluster restarts, and reducing shuffle write latency. Configuration tuning tips are provided to optimize memory usage and shuffle processing.
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
Attend this session to learn how to easily share state in-memory across multiple Spark jobs, either within the same application or between different Spark applications using an implementation of the Spark RDD abstraction provided in Apache Ignite. During the talk, attendees will learn in detail how IgniteRDD – an implementation of native Spark RDD and DataFrame APIs – shares the state of the RDD across other Spark jobs, applications and workers. Examples will show how IgniteRDD, with its advanced in-memory indexing capabilities, allows execution of SQL queries many times faster than native Spark RDDs or Data Frames.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
Autoscaling with Kubernetes
An interactive hands-on introduction of autoscaling in Kubernetes featuring Cluster Autoscaling, Horizontal Pod Autoscaling (HPA) with resource metrics, custom (object) metrics and external metrics and Vertical Pod Autoscaling (VPA).
Featuring a demo application showcasing the different scaling approaches for stateful and stateless applications.
Lifting the Blinds: Monitoring Windows Server 2012
Operating systems monitor resources continuously in order to effectively schedule processes.
In this webinar, Evan Mouzakitis (Datadog) discusses how to get operational data from Windows Server 2012 using a variety of native tools.
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
This document provides an overview of single-pass graph stream analytics using Apache Flink. It discusses why graph streaming is useful, provides examples of single-pass graph algorithms like connected components and bipartite detection, and introduces the GellyStream API in Apache Flink for working with streaming graphs. GellyStream represents streaming graphs as GraphStreams and enables neighborhood aggregations through windows and graph aggregations like connected components that operate on the streaming graph in a single pass.
Querying Dynamic Datasources with Continuously Mapped Sensor Data
Triple Pattern Fragments and continuous ETL processes are used to publish raw sensor data as RDF on the web and allow clients to query current temperature and humidity readings through a lightweight query interface. Sensor measurements are extracted, transformed by adding metadata on measurement time, and loaded as RDF which is then queried using SPARQL to retrieve live temperature and humidity values from a sensor.
Cloud Computing: Safe Haven from the Data Deluge? AGBT 2011
This document discusses moving a genomic data analysis pipeline to the cloud. It provides context on cloud computing and describes an experiment to port an Illumina sequencing analysis pipeline to Amazon Web Services. The document outlines challenges in moving to the cloud like data transfer speeds, efficient resource utilization, and security concerns for controlled-access data. It also compares performance and costs between running analysis locally versus in the cloud. While the cloud poses challenges, the document concludes it is a viable option especially for small centers and large collaborations, though costs can be difficult to predict.
Autoscaling on Kubernetes
You've built your app on Kubernetes and now you're ready to scale. Where do you begin and how do you scale using custom application and external metrics? In this talk, you learn the basics of autoscaling your deployments in Kubernetes and then see how to scale your application using Kubernetes custom metric adapters, including using metrics from any Azure service available. We will look into the details of how the metric adapters are built and see it in action. You will leave with a strong understanding of how to autoscale deployments in Kubernetes and the tools to accomplish automated scale so you can go home and have dinner.
Portable Streaming Pipelines with Apache Beam
1) Apache Beam is an open source unified model for defining both batch and streaming data processing pipelines. It allows writing pipelines once that can run on multiple distributed processing backends.
2) The Beam model separates the data processing logic from runtime requirements. It defines concepts like processing time vs event time to allow portability across batch and streaming runners.
3) Beam supports extensible IO connectors and aims to allow pipelines written in one language to run on different runtimes through language-specific SDKs. Currently, Java and Python SDKs can run on backends like Apache Spark, Flink, and Google Cloud Dataflow.
Gobblin on-aws
Gobblin on AWS allows running Gobblin data integration jobs on AWS infrastructure. It uses auto scaling groups to dynamically scale worker nodes and runs the Gobblin master in another auto scaling group. Network storage is provided by EFS. Gobblin as a Service will build on this to provide a fully managed service, adding monitoring, a UI, and job configuration storage in S3. It will also support additional cloud platforms and continuous execution of jobs.
AWS re:Invent 2016: Large-Scale, Cloud-Based Analysis of Cancer Genomes: Less...
The PanCancer Analysis of Whole Genomes (PCAWG) project is a large-scale, highly distributed research collaboration designed to identify common patterns of mutations across 2,800 cancer genomes. The use of public and private clouds were instrumental in analyzing this dataset using current best practice containerized pipelines. This session describes the technical infrastructure built for the project, how we leveraged cloud environments to perform the “core” analysis, and the lessons learned along the way.
The Open Chemistry Project
The document summarizes the Open Chemistry Project, which aims to develop open-source software for computational chemistry. It describes applications like Avogadro for molecular editing, MoleQueue for job management, and MongoChem for data storage and analysis. The project uses open frameworks and a collaborative development process to advance chemistry research and education through tightly integrated, user-friendly tools.
More Related Content
What's hot
Fast and Reliable Apache Spark SQL Engine
Building the next generation Spark SQL engine at speed poses new challenges to both automation and testing. At Databricks, we are implementing a new testing framework for assessing the quality and performance of new developments as they produced. Having more than 1,200 worldwide contributors, Apache Spark follows a rapid pace of development. At this scale, new testing tooling such as random query and data generation, fault injection, longevity stress, and scalability tests are essential to guarantee a reliable and performance Spark later in production. By applying such techniques, we will demonstrate the effectiveness of our testing infrastructure by drilling-down into cases where correctness and performance regressions have been found early. In addition, showing how they have been root-caused and fixed to prevent regressions in production and boosting the continuous delivery of new features.
Dataflow in 104corp - AWS UserGroup TW 2018
This document discusses migrating data processing workflows from on-premises to cloud-based serverless architectures. It outlines reasons for upgrading systems like HDFS and Pig to cloud services including AWS EMR, Kinesis, and S3 for improved resource utilization, high availability, and performance. The document then details considerations for how to migrate components like streaming, storage, computing, exploration and serving to various AWS services, and concludes with recommendations to leverage AWS services where possible for ease of use while balancing controllability and cost versus performance.
IGUANA: A Generic Framework for Benchmarking the Read-Write Performance of Tr...
Iguana is a framework for benchmarking the read-write performance of triple stores. It provides a realistic scenario by simulating multiple concurrent users querying and updating a triple store. Iguana executes benchmarks on different datasets and triple stores, measuring key performance indicators like queries per second. Results are stored in files and triple stores for analysis. The framework is extensible and can benchmark any dataset, SPARQL/update queries, and triple store configuration.
Realizing the promise of portability with Apache Beam
The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms.
In this talk, I will:
Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem.
Discuss the benefits Beam provides regarding portability and ease-of-use.
Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise).
Give a glimpse at some of the challenges Beam aims to address in the future.
Monitoring Microservices
By Tom Wilkie, delivered at London Microservices User Group on 2/12/15
The rise of microservice-based applications has had many knock-on effects, not least on the complexity of monitoring your application. Order-of-magnitude increase in the number of moving parts and rate of change of the application require us to reassess traditional monitoring techniques.
In this talk we will discuss some different approaches to monitoring, visualising and tracing containerised, microservices-based applications. We’ll present different techniques to some of the emergent problems, and try not to rant too much.
Flink Forward Berlin 2017: Francesco Versaci - Integrating Flink and Kafka in...
High-throughput DNA sequencing is a key data acquisition technology which enables dozens of important applications, from oncology to personalized diagnostics. We extended work presented last year to port additional portions of the standard genomics data processing pipeline to Flink. Our Flink-based processor consists of two distinct specialized modules (reader and writer) that are loosely linked via Kafka streams, thus allowing for easy composability and integration into already existing Hadoop workflows. To extend our work we had to manage the dynamical creation and detection of the data streams: the set of output files is not known in advance by the writer, which learns it at running time. Particular care had to be taken to handle the finite nature of the genomic streams: since we use some already existing Hadoop output formats, we had to properly handle the flow of end-of-streams markers through Flink and Kafka, in order to have the final output files correctly finalized.
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Flink provides fault tolerance guarantees through checkpointing and recovery mechanisms. Checkpoints take consistent snapshots of distributed state and data, while barriers mark checkpoints in the data flow. This allows Flink to recover jobs from failures and resume processing from the last completed checkpoint. Flink also implements high availability by persisting metadata like the execution graph and checkpoints to Apache Zookeeper, enabling a standby JobManager to take over if the active one fails.
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Mac...
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Machine Learning Microservices with Apache Kafka Streams and H2O.ai"
Kai Wähner works as Technology Evangelist at Confluent. Kai’s main area of expertise lies within the fields of Big Data Analytics, Machine Learning, Integration, Microservices, Internet of Things, Stream Processing and Blockchain. He is regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture or ApacheCon, writes articles for professional journals, and shares his experiences with new technologies on his blog (www.kai-waehner.de/blog). Contact and references: kontakt@kai-waehner.de / @KaiWaehner / www.kai-waehner.de
Abstract:
Big Data and Machine Learning are key for innovation in many industries today. The first part of this session explains how to build analytic models with R, Python or Scala leveraging open source machine learning / deep learning frameworks like Apache Spark, TensorFlow or H2O.ai. The second part discusses the deployment of these built analytic models to your own applications or microservices leveraging the Apache Kafka cluster and Kafka Streams. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.
Circonus: Design failures - A Case Study
SRECon18 Keynote https://www.usenix.org/conference/srecon18europe/presentation/schlossnagle
Joint talk Theo Schlossnagle (p1-26), Heinrich Hartmann (p26-44ff).
Spark Summit EU talk by Sital Kedia
This document summarizes a case study of using Apache Spark at scale to process over 60 TB of data at Facebook. It describes the previous implementation using Hive, which involved many small jobs that were unmanageable and slow. The new Spark implementation processes the entire dataset in a single job with two stages, shuffling over 90 TB of intermediate data. It provides performance comparisons showing significant reductions in CPU time, latency, and resource usage. It also details reliability and performance improvements made to Spark, such as fixing memory leaks, enabling seamless cluster restarts, and reducing shuffle write latency. Configuration tuning tips are provided to optimize memory usage and shuffle processing.
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
Attend this session to learn how to easily share state in-memory across multiple Spark jobs, either within the same application or between different Spark applications using an implementation of the Spark RDD abstraction provided in Apache Ignite. During the talk, attendees will learn in detail how IgniteRDD – an implementation of native Spark RDD and DataFrame APIs – shares the state of the RDD across other Spark jobs, applications and workers. Examples will show how IgniteRDD, with its advanced in-memory indexing capabilities, allows execution of SQL queries many times faster than native Spark RDDs or Data Frames.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
Autoscaling with Kubernetes
An interactive hands-on introduction of autoscaling in Kubernetes featuring Cluster Autoscaling, Horizontal Pod Autoscaling (HPA) with resource metrics, custom (object) metrics and external metrics and Vertical Pod Autoscaling (VPA).
Featuring a demo application showcasing the different scaling approaches for stateful and stateless applications.
Lifting the Blinds: Monitoring Windows Server 2012
Operating systems monitor resources continuously in order to effectively schedule processes.
In this webinar, Evan Mouzakitis (Datadog) discusses how to get operational data from Windows Server 2012 using a variety of native tools.
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
This document provides an overview of single-pass graph stream analytics using Apache Flink. It discusses why graph streaming is useful, provides examples of single-pass graph algorithms like connected components and bipartite detection, and introduces the GellyStream API in Apache Flink for working with streaming graphs. GellyStream represents streaming graphs as GraphStreams and enables neighborhood aggregations through windows and graph aggregations like connected components that operate on the streaming graph in a single pass.
Querying Dynamic Datasources with Continuously Mapped Sensor Data
Triple Pattern Fragments and continuous ETL processes are used to publish raw sensor data as RDF on the web and allow clients to query current temperature and humidity readings through a lightweight query interface. Sensor measurements are extracted, transformed by adding metadata on measurement time, and loaded as RDF which is then queried using SPARQL to retrieve live temperature and humidity values from a sensor.
Cloud Computing: Safe Haven from the Data Deluge? AGBT 2011
This document discusses moving a genomic data analysis pipeline to the cloud. It provides context on cloud computing and describes an experiment to port an Illumina sequencing analysis pipeline to Amazon Web Services. The document outlines challenges in moving to the cloud like data transfer speeds, efficient resource utilization, and security concerns for controlled-access data. It also compares performance and costs between running analysis locally versus in the cloud. While the cloud poses challenges, the document concludes it is a viable option especially for small centers and large collaborations, though costs can be difficult to predict.
Autoscaling on Kubernetes
You've built your app on Kubernetes and now you're ready to scale. Where do you begin and how do you scale using custom application and external metrics? In this talk, you learn the basics of autoscaling your deployments in Kubernetes and then see how to scale your application using Kubernetes custom metric adapters, including using metrics from any Azure service available. We will look into the details of how the metric adapters are built and see it in action. You will leave with a strong understanding of how to autoscale deployments in Kubernetes and the tools to accomplish automated scale so you can go home and have dinner.
Portable Streaming Pipelines with Apache Beam
1) Apache Beam is an open source unified model for defining both batch and streaming data processing pipelines. It allows writing pipelines once that can run on multiple distributed processing backends.
2) The Beam model separates the data processing logic from runtime requirements. It defines concepts like processing time vs event time to allow portability across batch and streaming runners.
3) Beam supports extensible IO connectors and aims to allow pipelines written in one language to run on different runtimes through language-specific SDKs. Currently, Java and Python SDKs can run on backends like Apache Spark, Flink, and Google Cloud Dataflow.
Gobblin on-aws
Gobblin on AWS allows running Gobblin data integration jobs on AWS infrastructure. It uses auto scaling groups to dynamically scale worker nodes and runs the Gobblin master in another auto scaling group. Network storage is provided by EFS. Gobblin as a Service will build on this to provide a fully managed service, adding monitoring, a UI, and job configuration storage in S3. It will also support additional cloud platforms and continuous execution of jobs.
What's hot (20)
IGUANA: A Generic Framework for Benchmarking the Read-Write Performance of Tr...
IGUANA: A Generic Framework for Benchmarking the Read-Write Performance of Tr...
Realizing the promise of portability with Apache Beam
Realizing the promise of portability with Apache Beam
Flink Forward Berlin 2017: Francesco Versaci - Integrating Flink and Kafka in...
Flink Forward Berlin 2017: Francesco Versaci - Integrating Flink and Kafka in...
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Mac...
Kai Wähner, Technology Evangelist at Confluent: "Development of Scalable Mac...
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Lifting the Blinds: Monitoring Windows Server 2012
Lifting the Blinds: Monitoring Windows Server 2012
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
Querying Dynamic Datasources with Continuously Mapped Sensor Data
Querying Dynamic Datasources with Continuously Mapped Sensor Data
Cloud Computing: Safe Haven from the Data Deluge? AGBT 2011
Cloud Computing: Safe Haven from the Data Deluge? AGBT 2011
Similar to Adam bosc-071114
AWS re:Invent 2016: Large-Scale, Cloud-Based Analysis of Cancer Genomes: Less...
The PanCancer Analysis of Whole Genomes (PCAWG) project is a large-scale, highly distributed research collaboration designed to identify common patterns of mutations across 2,800 cancer genomes. The use of public and private clouds were instrumental in analyzing this dataset using current best practice containerized pipelines. This session describes the technical infrastructure built for the project, how we leveraged cloud environments to perform the “core” analysis, and the lessons learned along the way.
The Open Chemistry Project
The document summarizes the Open Chemistry Project, which aims to develop open-source software for computational chemistry. It describes applications like Avogadro for molecular editing, MoleQueue for job management, and MongoChem for data storage and analysis. The project uses open frameworks and a collaborative development process to advance chemistry research and education through tightly integrated, user-friendly tools.
Avogadro, Open Chemistry and Semantics
Avogadro is being rewritten and architected to put semantic chemical meaning at the center of its internal data structures in order to fully support data-centric workflows. Computational and experimental chemistry both suffer when semantic meaning is lost; through the use of expressive formats such as CML, along with lightweight data-exchange formats such as JSON, workflows that previously demanded manual intervention to retain semantic meaning can be used. Integration with projects like JUMBO and Open Babel when conversion is required, coupled with codes such as NWChem where direct support for CML is being added, allow for much richer storage, analysis, and indexing of data. As web-based data sources add more semantic structure to their data, Avogadro will take advantage of those resources.
Open Chemistry: Input Preparation, Data Visualization & Analysis
The document outlines an open-source software development project called Open Chemistry that aims to integrate desktop chemistry applications, high-performance computing resources, and database/informatics resources. It describes several software applications being developed as part of Open Chemistry, including Avogadro 2 for structure editing and visualization, MoleQueue for running computational jobs on local and remote systems, and MongoChem for storing and searching chemistry data. The goal of Open Chemistry is to advance computational chemistry tools through open-source development and tight integration of related applications.
HPC and cloud distributed computing, as a journey
Introducing an internal cloud brings new paradigms, tools and infrastructure management. When placed alongside traditional HPC the new opportunities are significant But getting to the new world with micro-services, autoscaling and autodialing is a journey that cannot be achieved in a single step.
GlobusWorld 2020 Keynote
This presentation was given at the GlobusWorld 2020 Virtual Conference, by Ian Foster, Rachana Ananthakrishnan, and Vas Vasiliadis from the University of Chicago.
Big Data Streams Architectures. Why? What? How?
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Processing 70Tb Of Genomics Data With ADAM And Toil
This document discusses analyzing large genomic datasets with ADAM and Toil. It summarizes the sequencing and analysis process, and how ADAM implemented on Spark can provide horizontal scalability and speedups of 30-50x over traditional tools. Toil is introduced as a pipeline system for massive genomic workflows that can run on thousands of nodes and is resilient to failures. Results show ADAM produces equivalent variants to GATK while being 3.5x faster and 4x cheaper.
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS present at Los Angeles UUAC and Los Angeles Mesos users group meetup on Jan 11, 2018
Ceph used in Cancer Research at OICR
This document discusses a Ceph storage cluster used by the Cancer Genome Collaboratory at the Ontario Institute for Cancer Research to store and analyze large genomic datasets. It provides details on the hardware architecture including compute nodes, Ceph storage nodes and OpenStack controllers. It also describes the software stack, including tools for monitoring Ceph performance and usage. The storage is used to support cancer research by hundreds of users analyzing petabytes of genomic data.
Scientific
Apache Airavata is an open source science gateway software framework that allows users to compose, manage, execute, and monitor distributed computational workflows. It provides tools and services to register applications, schedule jobs on various resources, and manage workflows and generated data. Airavata is used across several domains to support scientific workflows and is largely derived from academic research funded by the NSF.
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
1) The document discusses improving the efficiency of machine learning algorithms using the HPCC Systems platform through parallelization.
2) It describes the HPCC Systems architecture and its advantages for distributed machine learning.
3) A parallel DBSCAN algorithm is implemented on the HPCC platform which shows improved performance over the serial algorithm, with execution times decreasing as more nodes are used.
Application Profiling at the HPCAC High Performance Center
Pak Lui from the HPC Advisory Council presented this deck at the 2017 Stanford HPC Conference.
"To achieve good scalability performance on the HPC scientific applications typically involves good understanding of the workload though performing profile analysis, and comparing behaviors of using different hardware which pinpoint bottlenecks in different areas of the HPC cluster. In this session, a selection of HPC applications will be shown to demonstrate various methods of profiling and analysis to determine the bottleneck, and the effectiveness of the tuning to improve on the application performance from tests conducted at the HPC Advisory Council High Performance Center."
Watch the video presentation: http://wp.me/p3RLHQ-gpY
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Scaling Hadoop at LinkedIn
LinkedIn leverages the Apache Hadoop ecosystem for its big data analytics. Steady growth of the member base at LinkedIn along with their social activities results in exponential growth of the analytics infrastructure. Innovations in analytics tooling lead to heavier workloads on the clusters, which generate more data, which in turn encourage innovations in tooling and more workloads. Thus, the infrastructure remains under constant growth pressure. Heterogeneous environments embodied via a variety of hardware and diverse workloads make the task even more challenging.
This talk will tell the story of how we doubled our Hadoop infrastructure twice in the past two years.
• We will outline our main use cases and historical rates of cluster growth in multiple dimensions.
• We will focus on optimizations, configuration improvements, performance monitoring and architectural decisions we undertook to allow the infrastructure to keep pace with business needs.
• The topics include improvements in HDFS NameNode performance, and fine tuning of block report processing, the block balancer, and the namespace checkpointer.
• We will reveal a study on the optimal storage device for HDFS persistent journals (SATA vs. SAS vs. SSD vs. RAID).
• We will also describe Satellite Cluster project which allowed us to double the objects stored on one logical cluster by splitting an HDFS cluster into two partitions without the use of federation and practically no code changes.
• Finally, we will take a peek at our future goals, requirements, and growth perspectives.
SPEAKERS
Konstantin Shvachko, Sr Staff Software Engineer, LinkedIn
Erik Krogen, Senior Software Engineer, LinkedIn
QuTrack: Model Life Cycle Management for AI and ML models using a Blockchain ...
As the complexity in AI and Machine Learning processes increases, robust data pipelines need to be
developed for industrial scale model development and deployment. . In regulated industries such as
Finance, Healthcare etc. where automated decision making is increasingly becoming used, tracking
design of experiments and from inception to deployment is critical to ensure a robust process is
adopted. Model Life-cycle management solutions are proposed to track experiments, design robust
experiments for hyper parameter tuning, optimization and selection of models and for monitoring.
The number of choices and the parameters that need to be tracked makes is significantly
challenging to trace experiments and to address reproducibility concerns.
In this talk, we discuss QuTrack, a Blockchain-based approach to track experiment and model
changes primarily for AI and ML models. In addition, we discuss how change analytics can be used
for process improvement and to enhance the model development and deployment processes.
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Talk Abstract
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Speaker
Joe Stein
Principal Consultant, Big Data Open Source Security, LLC
Joe Stein is an Apache Kafka committer and PMC member. Joe is the Founder and Principal Architect of Big Data Open Source Security LLC a professional services and product solutions company. Joe has been a developer, architect and technologist professionally for 15 years now having built back end systems that supported over one hundred million unique devices a day processing trillions of events. He blogs and hosts a podcast about Hadoop and related systems at All Things Hadoop and tweets @allthingshadoop
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Realizing the Promise of Portable Data Processing with Apache Beam
The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere".
This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future.
Speaker
Davor Bonaci, Senior Software Engineer, Google
Getting started with postgresql
This document provides an overview of Postgresql, including its history, capabilities, advantages over other databases, best practices, and references for further learning. Postgresql is an open source relational database management system that has been in development for over 30 years. It offers rich SQL support, high performance, ACID transactions, and extensive extensibility through features like JSON, XML, and programming languages.
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...
This document outlines a proposal to conceptualize and prototype a scalable genomic data analysis pipeline using Project Glow and Apache Spark on Amazon Web Services' (AWS) Databricks platform. It discusses the growing amounts of human genomic data, limitations of existing bioinformatics tools, and how distributed computing frameworks like Apache Spark and cloud platforms can enable analysis of huge genomic datasets.
Similar to Adam bosc-071114 (20)
AWS re:Invent 2016: Large-Scale, Cloud-Based Analysis of Cancer Genomes: Less...
AWS re:Invent 2016: Large-Scale, Cloud-Based Analysis of Cancer Genomes: Less...
Open Chemistry: Input Preparation, Data Visualization & Analysis
Open Chemistry: Input Preparation, Data Visualization & Analysis
Processing 70Tb Of Genomics Data With ADAM And Toil
Processing 70Tb Of Genomics Data With ADAM And Toil
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Improving Efficiency of Machine Learning Algorithms using HPCC Systems
Application Profiling at the HPCAC High Performance Center
Application Profiling at the HPCAC High Performance Center
QuTrack: Model Life Cycle Management for AI and ML models using a Blockchain ...
QuTrack: Model Life Cycle Management for AI and ML models using a Blockchain ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Accumulo Summit 2015: Real-Time Distributed and Reactive Systems with Apache ...
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo
Realizing the Promise of Portable Data Processing with Apache Beam
Realizing the Promise of Portable Data Processing with Apache Beam
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...
Conceptualizing And Prototyping A Scalable Genomic Data Analysis Pipeline: Us...
More from fnothaft
Scalable Genome Analysis with ADAM
This document discusses scalable genome analysis using ADAM (Apache Spark-based framework). It begins by describing genomes and the goal of analyzing genetic variations. The document then discusses challenges like the large size of genomes and complexity of linking variations to traits. It proposes using ADAM's schema, optimized storage and algorithms to accelerate common access patterns like overlap joins. The document also emphasizes applying biological knowledge like protein grammars to make sense of non-coding variations. Finally, it acknowledges contributions from various institutions that have helped develop ADAM and its ability to enable genome analysis at scale.
Rethinking Data-Intensive Science Using Scalable Analytics Systems
Presentation from SIGMOD 2015. With Matt Massie, Timothy Danford, Zhao Zhang, Uri Laserson, Carl Yeksigian, Jey Kottalam, Arun Ahuja, Jeff Hammerbacher, Michael Linderman, Michael J. Franklin, Anthony D. Joseph, David A. Patterson. Paper at http://dl.acm.org/citation.cfm?id=2742787.
Scalable Genome Analysis With ADAM
This document provides a summary of the Scalable Genome Analysis with ADAM project. ADAM is an open-source, high-performance, distributed platform for genomic analysis that defines a data schema, data layout on disk, and programming interface for distributed processing of genomic data using Spark and Scala. The goal of ADAM is to integrate across terabyte and petabyte-scale datasets to enable the discovery of low frequency genetic variants linked to traits and diseases.
Fast Variant Calling with ADAM and avocado
Slides from the BioBankCloud Hadoop/NGS Workshop at KTH Kista on 2/19/2015. Discusses early work on variant calling using ADAM and avocado.
Scaling Genomic Analyses
ADAM is an open source, high performance, distributed platform for genomic analysis that defines a data schema and layout on disk using Parquet and Avro, integrates with Spark's Scala and Java APIs, and provides a command line interface. ADAM achieves linear scalability out to 128 nodes for most tasks and provides a 2-4x performance improvement over other tools like GATK and samtools. The platform includes various tools like avocado for efficient local variant calling via de Bruijn graph reassembly of sequencing reads.
Scaling up genomic analysis with ADAM
Slides from DNAnexus about engineering/system design principles from ADAM, as well as emerging work in long-read OLC assembly.
Scaling up genomic analysis with ADAM
The document discusses ADAM, a new framework for scalable genomic data analysis. It aims to make genomic pipelines horizontally scalable by using a columnar data format and in-memory computing. This avoids disk I/O bottlenecks. The framework represents genomic data as schemas and stores data in Parquet for efficient column-based access. It has been shown to reduce genome analysis pipeline times from 100 hours to 1 hour by enabling analysis on large datasets in parallel across many nodes.
Reproducible Emulation of Analog Behavioral Models
1) Analog behavioral models are abstracted using SystemVerilog real numbers to allow simulation in digital emulation environments with higher throughput.
2) Key challenges to emulating analog models include converting floating-point implementations to fixed-point and handling high sampling rates in filters.
3) The document describes techniques used by Broadcom to synthesize analog behavioral models for emulation, including pragmas for sensitivity analysis and parallelizing filters.
Scalable up genomic analysis with ADAM
ADAM is an open source platform for scalable genomic analysis that defines a data schema, Scala API, and command line interface. It uses Apache Spark for efficient parallel and distributed processing of large genomic datasets stored in Parquet format. Key features of ADAM include its ability to perform iterative analysis on whole genome datasets while minimizing data movement through Spark. The document also describes using ADAM and PacMin for long read assembly through techniques like minhashing for fast read overlapping and building consensus sequences on read graphs.
CS176: Genome Assembly
The document discusses genome assembly from sequencing reads. It describes how reads can be aligned to a reference genome if available, but for a new genome the reads must be assembled without a reference. Two main assembly approaches are described: overlap-layout-consensus which builds an overlap graph, and de Brujin graph assembly which constructs a de Brujin graph from k-mers. Both approaches aim to find contiguous sequences (contigs) from the reads but face challenges from computational complexity and sequencing errors in the reads.
Execution Environments
Talk from GA4GH meeting at ASHG/San Diego meeting about support for efficiently executing code on top of the GA4GH APIs.
PacMin @ AMPLab All-Hands
Describes early work on PacMin, a ploidy aware overlap/string based approach for assembling genomes from long reads.
Design for Scalability in ADAM
ADAM is an open source, high performance, distributed platform for genomic analysis built on Apache Spark. It defines a Scala API and data schema using Avro and Parquet to store data in a columnar format, addressing the I/O bottleneck in genomics pipelines. ADAM implements common genomics algorithms as data or graph parallel computations and minimizes data movement by sending code to the data using Spark. It is designed to scale to processing whole human genomes across distributed file systems and cloud infrastructure.
ADAM—Spark Summit, 2014
ADAM is a scalable genome analysis platform that uses a column-oriented file format called Parquet to efficiently store and access large genomic datasets across distributed systems. It provides APIs and tools for transforming, analyzing, and querying genomic data in a scalable way using Apache Spark. Some key goals of ADAM include enabling efficient processing of genomes using clusters/clouds, providing a data format for parallel data access, and enhancing data semantics to allow more flexible access patterns.
More from fnothaft (14)
Rethinking Data-Intensive Science Using Scalable Analytics Systems
Rethinking Data-Intensive Science Using Scalable Analytics Systems
Reproducible Emulation of Analog Behavioral Models
Reproducible Emulation of Analog Behavioral Models
Recently uploaded
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Recently uploaded (20)
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Adam bosc-071114
- 1. ADAM: Fast, Scalable Genome Analysis Frank Austin Nothaft AMPLab, University of California, Berkeley, @fnothaft with: Matt Massie,André Schumacher,Timothy Danford, CarlYeksigian, Chris Hartl, Jey Kottalam,Arun Aruha, Neal Sidhwaney, Michael Linderman, Jeff Hammerbacher,Anthony Joseph, and Dave Patterson https://github.com/bigdatagenomics http://www.bdgenomics.org
- 2. What is in ADAM/BDG? ADAM: Core API + CLIs bdg-formats: Data schemas RNAdam: RNA analysis on ADAM avocado: Distributed local assembler Guacamole: Distributed somatic caller xASSEMBLEx: GraphX-based de novo assembler bdg-services: ADAM clusters
- 3. Design Goals • Develop processing pipeline that enables efficient, scalable use of cluster/cloud • Provide data format that has efficient parallel/distributed access across platforms • Enhance semantics of data and allow more flexible data access patterns
- 4. Implementation Overview • 27K lines of Scala code • 100% Apache-licensed open-source • 21 contributors from 8 institutions • Working towards a production quality release late 2014
- 5. ADAM Stack Physical File/Block Record/Split ‣Commodity Hardware ‣Cloud Systems - Amazon, GCE, Azure ‣Hadoop Distributed Filesystem ‣Local Filesystem ‣Schema-driven records w/ Apache Avro ‣Store and retrieve records using Parquet ‣Read BAM Files using Hadoop-BAM In-Memory RDD ‣Transform records using Apache Spark ‣Query with SQL using Shark ‣Graph processing with GraphX ‣Machine learning using MLBase
- 6. • Abstract as much as possible: schema oriented design makes format easy to evolve • Provide rich and scalable APIs for manipulating and transforming genomic data and regions • Don’t lock data in: play nicely with other tools Design Principles
- 7. • OSS Created by Twitter and Cloudera, based on Google Dremel, just entered Apache Incubator • Columnar File Format: • Limits I/O to only data that is needed • Compresses very well - ADAM files are 5-25% smaller than BAM files without loss of data • Fast scans - load only columns you need, e.g. scan a read flag on a whole genome, high- coverage file in less than a minute Parquet
- 8. Scaling Genomics: BQSR • Broadcast 3 GB table of variants, used for masking • Break reads down to bases and map bases to covariates • Calculate empirical values per covariate • Broadcast observation, apply across reads
- 9. Performance/Acc’y ADAM 0 10 20 30 40 50 GATK 0 10 20 30 40 50 • Fully concordant with Picard for MarkDup, >99% concordant with GATK for BQSR Hours 0 4 8 12 16 20 24 Sort Mark Duplicates BQSR Picard ADAM 100 EC2 Nodes
- 10. Future Work • Pushing hard towards production release • Are building out a complete analysis pipeline • Plan to release Python bindings • Work on interoperability with Global Alliance for Genomic Health API (http:// genomicsandhealth.org/)
- 11. Call for contributions • As an open source project, we welcome contributions • We maintain a list of open enhancements at our Github issue trackers • Github: https://www.github.com/bdgenomics • UC Berkeley is looking to hire two full time engineers to support this work
- 12. Acknowledgements • UC Berkeley: Matt Massie,André Schumacher, Jey Kottalam, Christos Kozanitis • Mt. Sinai: Arun Ahuja, Neal Sidhwaney, Michael Linderman, Jeff Hammerbacher • GenomeBridge: Timothy Danford, CarlYeksigian • The Broad Institute: Chris Hartl • Cloudera: Uri Laserson • Microsoft Research: Jeremy Elson, Ravi Pandya • Michael Heuer • And other open source contributors!
- 13. Acknowledgements This research is supported in part by NSF CISE Expeditions Award CCF-1139158, LBNL Award 7076018, and DARPA XData Award FA8750-12-2-0331, and gifts from Amazon Web Services, Google, SAP, The Thomas and Stacey Siebel Foundation,Apple, Inc., C3Energy, Cisco, Cloudera, EMC, Ericsson, Facebook, GameOnTalis, Guavus, HP, Huawei, Intel, Microsoft, NetApp, Pivotal, Splunk,Virdata,VMware,WANdisco and Yahoo!.