SMACK Stack @ Nitro
•
3 likes•2,011 views
SMACK Stack @ Nitro Roland Tritsch is a 50% software engineer, 50% manager/builder of organizations, and 50% seller/implementer of ideas. He previously worked at IONA and Cyclist and is a father and husband. Nitro is changing the way the world works with documents by making it easy to create, edit, share, sign and collaborate both online and offline. They are moving to a SAAS model hosted on bare metal and cloud servers, powered by machine learning, data analytics and natural language processing. Nitro's technology is based on the SMACK stack of Spark, Mesos, Akka, Cassandra and Kafka. The SMACKED stack involves
Report
Share
Report
Share
Download to read offline
Recommended
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
A talk covering the best-of-breed platform consisting of Spark, Mesos, Akka, Cassandra and Kafka. SMACK is more of a toolbox of technologies to allow the building of resilient ingestion pipelines, offering a high degree of freedom in the selection of analysis and query possibilities and baked in support for flow-control. More and more customers are using this stack, which is rapidly becoming the new industry standard for Big Data solutions. Session can be seen here - in German - https://speakerdeck.com/stefan79/fast-data-smack-down
Building a Big Data platform with the Hadoop ecosystem
This presentation provides a brief insight into a Big Data platform using the Hadoop ecosystem.
To this end the presentation will touch on:
-views of the Big Data ecosystem and it’s components
-an example of a Hadoop cluster
-considerations when selecting a Hadoop distribution
-some of the Hadoop distributions available
-a recommended Hadoop distribution
Kick-Start with SMACK Stack
SMACK is a combination of Spark, Mesos, Akka, Cassandra and Kafka. It is used for pipelined data architecture which is required for the real time data analysis and to integrate all the technology at the right place to efficient data pipeline.
Intro to Apache Spark
This presentation includes a comprehensive introduction to Apache Spark. From an explanation of its rapid ascent to performance and developer advantages over MapReduce. We also explore its built-in functionality for application types involving streaming, machine learning, and Extract, Transform and Load (ETL).
Alpine academy apache spark series #1 introduction to cluster computing wit...
Alpine academy apache spark series #1 introduction to cluster computing with python & a wee bit of scala. This is the first in the series and is aimed at the intro level, the next one will cover MLLib & ML.
Sa introduction to big data pipelining with cassandra & spark west mins...
This document provides an overview and outline of a 1-hour introduction to building a big data pipeline using Docker, Cassandra, Spark, Spark-Notebook and Akka. The introduction is presented as a half-day workshop at Devoxx November 2015. It uses a data pipeline environment from Data Fellas and demonstrates how to use scalable distributed technologies like Docker, Spark, Spark-Notebook and Cassandra to build a reactive, repeatable big data pipeline. The key takeaway is understanding how to construct such a pipeline.
Rethinking Streaming Analytics For Scale
This talk will address new architectures emerging for large scale streaming analytics. Some based on Spark, Mesos, Akka, Cassandra and Kafka (SMACK) and other newer streaming analytics platforms and frameworks using Apache Flink or GearPump. Popular architecture like Lambda separate layers of computation and delivery and require many technologies which have overlapping functionality. Some of this results in duplicated code, untyped processes, or high operational overhead, let alone the cost (e.g. ETL).
I will discuss the problem domain and what is needed in terms of strategies, architecture and application design and code to begin leveraging simpler data flows. We will cover how the particular set of technologies addresses common requirements and how collaboratively they work together to enrich and reinforce each other.
Recommended
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
A talk covering the best-of-breed platform consisting of Spark, Mesos, Akka, Cassandra and Kafka. SMACK is more of a toolbox of technologies to allow the building of resilient ingestion pipelines, offering a high degree of freedom in the selection of analysis and query possibilities and baked in support for flow-control. More and more customers are using this stack, which is rapidly becoming the new industry standard for Big Data solutions. Session can be seen here - in German - https://speakerdeck.com/stefan79/fast-data-smack-down
Building a Big Data platform with the Hadoop ecosystem
This presentation provides a brief insight into a Big Data platform using the Hadoop ecosystem.
To this end the presentation will touch on:
-views of the Big Data ecosystem and it’s components
-an example of a Hadoop cluster
-considerations when selecting a Hadoop distribution
-some of the Hadoop distributions available
-a recommended Hadoop distribution
Kick-Start with SMACK Stack
SMACK is a combination of Spark, Mesos, Akka, Cassandra and Kafka. It is used for pipelined data architecture which is required for the real time data analysis and to integrate all the technology at the right place to efficient data pipeline.
Intro to Apache Spark
This presentation includes a comprehensive introduction to Apache Spark. From an explanation of its rapid ascent to performance and developer advantages over MapReduce. We also explore its built-in functionality for application types involving streaming, machine learning, and Extract, Transform and Load (ETL).
Alpine academy apache spark series #1 introduction to cluster computing wit...
Alpine academy apache spark series #1 introduction to cluster computing with python & a wee bit of scala. This is the first in the series and is aimed at the intro level, the next one will cover MLLib & ML.
Sa introduction to big data pipelining with cassandra & spark west mins...
This document provides an overview and outline of a 1-hour introduction to building a big data pipeline using Docker, Cassandra, Spark, Spark-Notebook and Akka. The introduction is presented as a half-day workshop at Devoxx November 2015. It uses a data pipeline environment from Data Fellas and demonstrates how to use scalable distributed technologies like Docker, Spark, Spark-Notebook and Cassandra to build a reactive, repeatable big data pipeline. The key takeaway is understanding how to construct such a pipeline.
Rethinking Streaming Analytics For Scale
This talk will address new architectures emerging for large scale streaming analytics. Some based on Spark, Mesos, Akka, Cassandra and Kafka (SMACK) and other newer streaming analytics platforms and frameworks using Apache Flink or GearPump. Popular architecture like Lambda separate layers of computation and delivery and require many technologies which have overlapping functionality. Some of this results in duplicated code, untyped processes, or high operational overhead, let alone the cost (e.g. ETL).
I will discuss the problem domain and what is needed in terms of strategies, architecture and application design and code to begin leveraging simpler data flows. We will cover how the particular set of technologies addresses common requirements and how collaboratively they work together to enrich and reinforce each other.
AWS re:Invent 2016: Building Big Data Applications with the AWS Big Data Plat...
Building big data applications often requires integrating a broad set of technologies to store, process, and analyze the increasing variety, velocity, and volume of data being collected by many organizations. In this session, we show how you can build entire big data applications using a core set of managed services including Amazon S3, Amazon Kinesis, Amazon EMR, Amazon Elasticsearch Service, Amazon Redshift, and Amazon QuickSight.
We walk you through the steps of building and securing a big data application using the AWS Big Data Platform. We also share best practices and common use cases for AWS big data services, including tips to help you choose the best services for your specific application.
How to deploy Apache Spark
to Mesos/DCOS
Since 2014, Typesafe has been actively contributing to the Apache Spark project, and has become a certified development support partner of Databricks, the company started by the creators of Spark. Typesafe and Mesosphere have forged a partnership in which Typesafe is the official commercial support provider of Spark on Apache Mesos, along with Mesosphere’s Datacenter Operating Systems (DCOS).
In this webinar with Iulian Dragos, Spark team lead at Typesafe Inc., we reveal how Typesafe supports running Spark in various deployment modes, along with the improvements we made to Spark to help integrate backpressure signals into the underlying technologies, making it a better fit for Reactive Streams. He also show you the functionalities at work, and how to make it simple to deploy to Spark on Mesos with Typesafe.
We will introduce:
Various deployment modes for Spark: Standalone, Spark on Mesos, and Spark with Mesosphere DCOS
Overview of Mesos and how it relates to Mesosphere DCOS
Deeper look at how Spark runs on Mesos
How to manage coarse-grained and fine-grained scheduling modes on Mesos
What to know about a client vs. cluster deployment
A demo running Spark on Mesos
Akka in Production - ScalaDays 2015
Everyone in the Scala world is using or looking into using Akka for low-latency, scalable, distributed or concurrent systems. I'd like to share my story of developing and productionizing multiple Akka apps, including low-latency ingestion and real-time processing systems, and Spark-based applications.
When does one use actors vs futures?
Can we use Akka with, or in place of, Storm?
How did we set up instrumentation and monitoring in production?
How does one use VisualVM to debug Akka apps in production?
What happens if the mailbox gets full?
What is our Akka stack like?
I will share best practices for building Akka and Scala apps, pitfalls and things we'd like to avoid, and a vision of where we would like to go for ideal Akka monitoring, instrumentation, and debugging facilities. Plus backpressure and at-least-once processing.
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
NOTE: This was converted to Powerpoint from Keynote. Slideshare does not play the embedded videos. You can download the powerpoint from slideshare and import it into keynote. The videos should work in the keynote.
Abstract:
In this presentation, we will describe the "Spark Kernel" which enables applications, such as end-user facing and interactive applications, to interface with Spark clusters. It provides a gateway to define and run Spark tasks and to collect results from a cluster without the friction associated with shipping jars and reading results from peripheral systems. Using the Spark Kernel as a proxy, applications can be hosted remotely from Spark.
Reactive app using actor model & apache spark
Developing Application with Big Data is really challenging work, scaling, fault tolerance and responsiveness some are the biggest challenge. Realtime bigdata application that have self healing feature is a dream these days. Apache Spark is a fast in-memory data processing system that gives a good backend for realtime application.In this talk I will show how to use reactive platform, Actor model and Apache Spark stack to develop a system that have responsiveness, resiliency, fault tolerance and message driven feature.
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
We present a solution for streaming anomaly detection, named “Coral”, based on Spark, Akka and Cassandra. In the system presented, we run Spark to run the data analytics pipeline for anomaly detection. By running Spark on the latest events and data, we make sure that the model is always up-to-date and that the amount of false positives is kept low, even under changing trends and conditions. Our machine learning pipeline uses Spark decision tree ensembles and k-means clustering. Once the model is trained by Spark, the model’s parameters are pushed to the Streaming Event Processing Layer, implemented in Akka. The Akka layer will then score 1000s of event per seconds according to the last model provided by Spark. Spark and Akka communicate which each other using Cassandra as a low-latency data store. By doing so, we make sure that every element of this solution is resilient and distributed. Spark performs micro-batches to keep the model up-to-date while Akka detects the new anomalies by using the latest Spark-generated data model. The project is currently hosted on Github. Have a look at : http://coral-streaming.github.io
Reactive dashboard’s using apache spark
Apache Spark's Tutorial talk, In this talk i explained how to start working with Apache spark, feature of apache spark and how to compose data platform with spark. This talk also explains about reactive platform, tools and framework like Play, akka.
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and DatabricksLegacy Typesafe (now Lightbend)
Spark Streaming makes it easy to build scalable fault-tolerant streaming applications. In this webinar, developers will learn:
*How Spark Streaming works - a quick review.
*Features in Spark Streaming that help prevent potential data loss.
*Complementary tools in a streaming pipeline - Kafka and Akka.
*Design and tuning tips for Reactive Spark Streaming applications.Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
This talk is about architecture designs for data processing platforms based on SMACK stack which stands for Spark, Mesos, Akka, Cassandra and Kafka. The main topics of the talk are:
- SMACK stack overview
- storage layer layout
- fixing NoSQL limitations (joins and group by)
- cluster resource management and dynamic allocation
- reliable scheduling and execution at scale
- different options for getting the data into your system
- preparing for failures with proper backup and patching strategies
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
This document discusses Apache Zeppelin, an open-source notebook for interactive data analytics. It provides an overview of Zeppelin's features, including interactive notebooks, multiple backends, interpreters, and a display system. The document also covers Zeppelin's adoption timeline, from its origins as a commercial product in 2012 to becoming an Apache Incubator project in 2014. Future projects involving Zeppelin like Helium and Z-Manager are also briefly described.
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
This document provides an overview of streaming big data with Spark, Kafka, Cassandra, Akka, and Scala. It discusses delivering meaning in near-real time at high velocity and an overview of Spark Streaming, Kafka and Akka. It also covers Cassandra and the Spark Cassandra Connector as well as integration in big data applications. The presentation is given by Helena Edelson, a Spark Cassandra Connector committer and Akka contributor who is a Scala and big data conference speaker working as a senior software engineer at DataStax.
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Regardless of the meaning we are searching for over our vast amounts of data, whether we are in science, finance, technology, energy, health care…, we all share the same problems that must be solved: How do we achieve that? What technologies best support the requirements? This talk is about how to leverage fast access to historical data with real time streaming data for predictive modeling for lambda architecture with Spark Streaming, Kafka, Cassandra, Akka and Scala. Efficient Stream Computation, Composable Data Pipelines, Data Locality, Cassandra data model and low latency, Kafka producers and HTTP endpoints as akka actors...
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Scala Days, Amsterdam, 2015: Lambda Architecture - Batch and Streaming with Spark, Cassandra, Kafka, Akka and Scala; Fault Tolerance, Data Pipelines, Data Flows, Data Locality, Akka Actors, Spark, Spark Cassandra Connector, Big Data, Asynchronous data flows. Time series data, KillrWeather, Scalable Infrastructure, Partition For Scale, Replicate For Resiliency, Parallelism
Isolation, Data Locality, Location Transparency
RESTful API 제대로 만들기
This document discusses best practices for developing RESTful APIs. It begins by explaining that APIs should follow REST principles even if others are not, in order to be usable by others. It then shows an example of a non-RESTful API call. The document goes on to explain concepts like RESTful design, HTTP methods, resources, and HATEOAS. It provides examples of good and bad API patterns. It also introduces tools like appkr/fractal that can help build RESTful APIs in Laravel. Overall, the document provides guidance on how to properly structure APIs according to REST architectural principles.
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
The gambling industry has arguably been one of the most comprehensively affected by the internet revolution, and if an organization such as William Hill hadn't adapted successfully it would have disappeared. We call this, “Going Reactive.”
The company's latest innovations are very cutting edge platforms for personalization, recommendation, and big data, which are based on Akka, Scala, Play Framework, Kafka, Cassandra, Spark, and Mesos.
Streaming Analytics with Spark, Kafka, Cassandra and Akka
This document discusses a new approach to building scalable data processing systems using streaming analytics with Spark, Kafka, Cassandra, and Akka. It proposes moving away from architectures like Lambda and ETL that require duplicating data and logic. The new approach leverages Spark Streaming for a unified batch and stream processing runtime, Apache Kafka for scalable messaging, Apache Cassandra for distributed storage, and Akka for building fault tolerant distributed applications. This allows building real-time streaming applications that can join streaming and historical data with simplified architectures that remove the need for duplicating data extraction and loading.
Istanbul Spark Meetup Nov 28 2015
This document appears to be a slide deck presentation about optimizing Apache Spark performance. Some key points discussed include techniques for improving CPU cache locality through data structure design, avoiding unnecessary object creation, and using lock-free thread synchronization. Examples demonstrated include optimizing matrix multiplication and incrementing shared counters across threads. Performance improvements of 30-65% were shown from these optimizations.
Budapest Big Data Meetup Nov 26 2015
This document appears to be a slide deck presentation about Apache Spark. It discusses Spark's core concepts like tuning and mechanical sympathy. It provides examples of how to optimize Spark performance by improving CPU cache line affinity, such as through a cache-friendly matrix multiplication algorithm. The presentation also covers Spark SQL and query optimization using the Catalyst optimizer. It promotes the speaker's Advanced Apache Spark Meetup group and upcoming speaking events.
Copenhagen Spark Meetup Nov 25, 2015
The document is a slide deck presentation given by Chris Fregly on Spark and related big data technologies. It discusses techniques for improving performance through mechanical sympathy with hardware, including optimizing data layout for CPU cache locality and using lock-free thread synchronization. It also covers Spark SQL query optimization and the Spark core, with a live demo of an example "After Dark" application that uses various big data tools.
Toronto Spark Meetup Dec 14 2015
This document summarizes a presentation about optimizing Apache Spark for performance. Some key points discussed include:
- Techniques for optimizing Spark sorting and matrix multiplication to be more CPU cache friendly by using sequential access patterns and minimizing random memory access
- Different approaches for synchronizing concurrent updates to shared counters from multiple threads, including lock-based, immutable, and lock-free atomic implementations
- The IBM Project Tungsten efforts to customize Spark's data structures and algorithms to operate directly on compressed byte arrays to reduce garbage collection and maximize CPU cache usage
More Related Content
Viewers also liked
AWS re:Invent 2016: Building Big Data Applications with the AWS Big Data Plat...
Building big data applications often requires integrating a broad set of technologies to store, process, and analyze the increasing variety, velocity, and volume of data being collected by many organizations. In this session, we show how you can build entire big data applications using a core set of managed services including Amazon S3, Amazon Kinesis, Amazon EMR, Amazon Elasticsearch Service, Amazon Redshift, and Amazon QuickSight.
We walk you through the steps of building and securing a big data application using the AWS Big Data Platform. We also share best practices and common use cases for AWS big data services, including tips to help you choose the best services for your specific application.
How to deploy Apache Spark
to Mesos/DCOS
Since 2014, Typesafe has been actively contributing to the Apache Spark project, and has become a certified development support partner of Databricks, the company started by the creators of Spark. Typesafe and Mesosphere have forged a partnership in which Typesafe is the official commercial support provider of Spark on Apache Mesos, along with Mesosphere’s Datacenter Operating Systems (DCOS).
In this webinar with Iulian Dragos, Spark team lead at Typesafe Inc., we reveal how Typesafe supports running Spark in various deployment modes, along with the improvements we made to Spark to help integrate backpressure signals into the underlying technologies, making it a better fit for Reactive Streams. He also show you the functionalities at work, and how to make it simple to deploy to Spark on Mesos with Typesafe.
We will introduce:
Various deployment modes for Spark: Standalone, Spark on Mesos, and Spark with Mesosphere DCOS
Overview of Mesos and how it relates to Mesosphere DCOS
Deeper look at how Spark runs on Mesos
How to manage coarse-grained and fine-grained scheduling modes on Mesos
What to know about a client vs. cluster deployment
A demo running Spark on Mesos
Akka in Production - ScalaDays 2015
Everyone in the Scala world is using or looking into using Akka for low-latency, scalable, distributed or concurrent systems. I'd like to share my story of developing and productionizing multiple Akka apps, including low-latency ingestion and real-time processing systems, and Spark-based applications.
When does one use actors vs futures?
Can we use Akka with, or in place of, Storm?
How did we set up instrumentation and monitoring in production?
How does one use VisualVM to debug Akka apps in production?
What happens if the mailbox gets full?
What is our Akka stack like?
I will share best practices for building Akka and Scala apps, pitfalls and things we'd like to avoid, and a vision of where we would like to go for ideal Akka monitoring, instrumentation, and debugging facilities. Plus backpressure and at-least-once processing.
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
NOTE: This was converted to Powerpoint from Keynote. Slideshare does not play the embedded videos. You can download the powerpoint from slideshare and import it into keynote. The videos should work in the keynote.
Abstract:
In this presentation, we will describe the "Spark Kernel" which enables applications, such as end-user facing and interactive applications, to interface with Spark clusters. It provides a gateway to define and run Spark tasks and to collect results from a cluster without the friction associated with shipping jars and reading results from peripheral systems. Using the Spark Kernel as a proxy, applications can be hosted remotely from Spark.
Reactive app using actor model & apache spark
Developing Application with Big Data is really challenging work, scaling, fault tolerance and responsiveness some are the biggest challenge. Realtime bigdata application that have self healing feature is a dream these days. Apache Spark is a fast in-memory data processing system that gives a good backend for realtime application.In this talk I will show how to use reactive platform, Actor model and Apache Spark stack to develop a system that have responsiveness, resiliency, fault tolerance and message driven feature.
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
We present a solution for streaming anomaly detection, named “Coral”, based on Spark, Akka and Cassandra. In the system presented, we run Spark to run the data analytics pipeline for anomaly detection. By running Spark on the latest events and data, we make sure that the model is always up-to-date and that the amount of false positives is kept low, even under changing trends and conditions. Our machine learning pipeline uses Spark decision tree ensembles and k-means clustering. Once the model is trained by Spark, the model’s parameters are pushed to the Streaming Event Processing Layer, implemented in Akka. The Akka layer will then score 1000s of event per seconds according to the last model provided by Spark. Spark and Akka communicate which each other using Cassandra as a low-latency data store. By doing so, we make sure that every element of this solution is resilient and distributed. Spark performs micro-batches to keep the model up-to-date while Akka detects the new anomalies by using the latest Spark-generated data model. The project is currently hosted on Github. Have a look at : http://coral-streaming.github.io
Reactive dashboard’s using apache spark
Apache Spark's Tutorial talk, In this talk i explained how to start working with Apache spark, feature of apache spark and how to compose data platform with spark. This talk also explains about reactive platform, tools and framework like Play, akka.
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and DatabricksLegacy Typesafe (now Lightbend)
Spark Streaming makes it easy to build scalable fault-tolerant streaming applications. In this webinar, developers will learn:
*How Spark Streaming works - a quick review.
*Features in Spark Streaming that help prevent potential data loss.
*Complementary tools in a streaming pipeline - Kafka and Akka.
*Design and tuning tips for Reactive Spark Streaming applications.Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
This talk is about architecture designs for data processing platforms based on SMACK stack which stands for Spark, Mesos, Akka, Cassandra and Kafka. The main topics of the talk are:
- SMACK stack overview
- storage layer layout
- fixing NoSQL limitations (joins and group by)
- cluster resource management and dynamic allocation
- reliable scheduling and execution at scale
- different options for getting the data into your system
- preparing for failures with proper backup and patching strategies
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
This document discusses Apache Zeppelin, an open-source notebook for interactive data analytics. It provides an overview of Zeppelin's features, including interactive notebooks, multiple backends, interpreters, and a display system. The document also covers Zeppelin's adoption timeline, from its origins as a commercial product in 2012 to becoming an Apache Incubator project in 2014. Future projects involving Zeppelin like Helium and Z-Manager are also briefly described.
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
This document provides an overview of streaming big data with Spark, Kafka, Cassandra, Akka, and Scala. It discusses delivering meaning in near-real time at high velocity and an overview of Spark Streaming, Kafka and Akka. It also covers Cassandra and the Spark Cassandra Connector as well as integration in big data applications. The presentation is given by Helena Edelson, a Spark Cassandra Connector committer and Akka contributor who is a Scala and big data conference speaker working as a senior software engineer at DataStax.
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Regardless of the meaning we are searching for over our vast amounts of data, whether we are in science, finance, technology, energy, health care…, we all share the same problems that must be solved: How do we achieve that? What technologies best support the requirements? This talk is about how to leverage fast access to historical data with real time streaming data for predictive modeling for lambda architecture with Spark Streaming, Kafka, Cassandra, Akka and Scala. Efficient Stream Computation, Composable Data Pipelines, Data Locality, Cassandra data model and low latency, Kafka producers and HTTP endpoints as akka actors...
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Scala Days, Amsterdam, 2015: Lambda Architecture - Batch and Streaming with Spark, Cassandra, Kafka, Akka and Scala; Fault Tolerance, Data Pipelines, Data Flows, Data Locality, Akka Actors, Spark, Spark Cassandra Connector, Big Data, Asynchronous data flows. Time series data, KillrWeather, Scalable Infrastructure, Partition For Scale, Replicate For Resiliency, Parallelism
Isolation, Data Locality, Location Transparency
RESTful API 제대로 만들기
This document discusses best practices for developing RESTful APIs. It begins by explaining that APIs should follow REST principles even if others are not, in order to be usable by others. It then shows an example of a non-RESTful API call. The document goes on to explain concepts like RESTful design, HTTP methods, resources, and HATEOAS. It provides examples of good and bad API patterns. It also introduces tools like appkr/fractal that can help build RESTful APIs in Laravel. Overall, the document provides guidance on how to properly structure APIs according to REST architectural principles.
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
The gambling industry has arguably been one of the most comprehensively affected by the internet revolution, and if an organization such as William Hill hadn't adapted successfully it would have disappeared. We call this, “Going Reactive.”
The company's latest innovations are very cutting edge platforms for personalization, recommendation, and big data, which are based on Akka, Scala, Play Framework, Kafka, Cassandra, Spark, and Mesos.
Streaming Analytics with Spark, Kafka, Cassandra and Akka
This document discusses a new approach to building scalable data processing systems using streaming analytics with Spark, Kafka, Cassandra, and Akka. It proposes moving away from architectures like Lambda and ETL that require duplicating data and logic. The new approach leverages Spark Streaming for a unified batch and stream processing runtime, Apache Kafka for scalable messaging, Apache Cassandra for distributed storage, and Akka for building fault tolerant distributed applications. This allows building real-time streaming applications that can join streaming and historical data with simplified architectures that remove the need for duplicating data extraction and loading.
Viewers also liked (18)
AWS re:Invent 2016: Building Big Data Applications with the AWS Big Data Plat...
AWS re:Invent 2016: Building Big Data Applications with the AWS Big Data Plat...
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Similar to SMACK Stack @ Nitro
Istanbul Spark Meetup Nov 28 2015
This document appears to be a slide deck presentation about optimizing Apache Spark performance. Some key points discussed include techniques for improving CPU cache locality through data structure design, avoiding unnecessary object creation, and using lock-free thread synchronization. Examples demonstrated include optimizing matrix multiplication and incrementing shared counters across threads. Performance improvements of 30-65% were shown from these optimizations.
Budapest Big Data Meetup Nov 26 2015
This document appears to be a slide deck presentation about Apache Spark. It discusses Spark's core concepts like tuning and mechanical sympathy. It provides examples of how to optimize Spark performance by improving CPU cache line affinity, such as through a cache-friendly matrix multiplication algorithm. The presentation also covers Spark SQL and query optimization using the Catalyst optimizer. It promotes the speaker's Advanced Apache Spark Meetup group and upcoming speaking events.
Copenhagen Spark Meetup Nov 25, 2015
The document is a slide deck presentation given by Chris Fregly on Spark and related big data technologies. It discusses techniques for improving performance through mechanical sympathy with hardware, including optimizing data layout for CPU cache locality and using lock-free thread synchronization. It also covers Spark SQL query optimization and the Spark core, with a live demo of an example "After Dark" application that uses various big data tools.
Toronto Spark Meetup Dec 14 2015
This document summarizes a presentation about optimizing Apache Spark for performance. Some key points discussed include:
- Techniques for optimizing Spark sorting and matrix multiplication to be more CPU cache friendly by using sequential access patterns and minimizing random memory access
- Different approaches for synchronizing concurrent updates to shared counters from multiple threads, including lock-based, immutable, and lock-free atomic implementations
- The IBM Project Tungsten efforts to customize Spark's data structures and algorithms to operate directly on compressed byte arrays to reduce garbage collection and maximize CPU cache usage
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
High Performance, Reali-time, Stream Processing, Machine Learning, Natural Language Processing, Text Analytics, and Recommendations
Sydney Spark Meetup Dec 08, 2015
This document summarizes a presentation about optimizing Apache Spark for performance. It discusses techniques like leveraging CPU caches, reducing random memory access, avoiding thread context switches, and using immutable data structures to minimize locking. It also promotes the concept of "mechanical sympathy" - designing software and hardware to work together efficiently. The presentation contains demos showing the impact of these optimizations on sorting and matrix multiplication performance.
Melbourne Spark Meetup Dec 09 2015
1. The document discusses techniques for improving Apache Spark performance through mechanical sympathy, which means optimizing for hardware performance by considering factors like CPU cache usage and minimizing random memory access.
2. It provides examples of how to improve sorting, matrix multiplication, and thread synchronization by making them more cache-friendly and reducing cache misses and context switches.
3. The speaker demonstrates performance improvements from these techniques using Linux perf and flame graph profiling tools. Optimizations like Project Tungsten that customize Spark for the hardware are also discussed.
DC Spark Users Group March 15 2016 - Spark and Netflix Recommendations
This document discusses various techniques for providing recommendations, including non-personalized and personalized recommendations. Non-personalized recommendations include using highest rated actors, top K aggregations, social graphs, or PageRank on likes/dislikes to address the cold start problem for new users. Personalized recommendations techniques discussed include user-to-user clustering based on similarity of viewing patterns or ratings. The document also covers feature engineering techniques such as normalization, dimensionality reduction, and one-hot encoding for modeling recommendations.
Data Science with Spark
The document outlines an agenda for a conference on Apache Spark and data science, including sessions on Spark's capabilities and direction, using DataFrames in PySpark, linear regression, text analysis, classification, clustering, and recommendation engines using Spark MLlib. Breakout sessions are scheduled between many of the technical sessions to allow for hands-on work and discussion.
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
* Title *
Spark After Dark 1.5: Deep Dive Into Latest Perf and Scale Improvements in Spark Ecosystem
* Abstract *
Combining the most popular and technically-deep material from his wildly popular Advanced Apache Spark Meetup, Chris Fregly will provide code-level deep dives into the latest performance and scalability advancements within the Apache Spark Ecosystem by exploring the following:
1) Building a Scalable and Performant Spark SQL/DataFrames Data Source Connector such as Spark-CSV, Spark-Cassandra, Spark-ElasticSearch, and Spark-Redshift
2) Speeding Up Spark SQL Queries using Partition Pruning and Predicate Pushdowns with CSV, JSON, Parquet, Avro, and ORC
3) Tuning Spark Streaming Performance and Fault Tolerance with KafkaRDD and KinesisRDD
4) Maintaining Stability during High Scale Streaming Ingestion using Approximations and Probabilistic Data Structures from Spark, Redis, and Twitter's Algebird
5) Building Effective Machine Learning Models using Feature Engineering, Dimension Reduction, and Natural Language Processing with MLlib/GraphX, ML Pipelines, DIMSUM, Locality Sensitive Hashing, and Stanford's CoreNLP
6) Tuning Core Spark Performance by Acknowledging Mechanical Sympathy for the Physical Limitations of OS and Hardware Resources such as CPU, Memory, Network, and Disk with Project Tungsten, Asynchronous Netty, and Linux epoll
* Demos *
This talk features many interesting and audience-interactive demos - as well as code-level deep dives into many of the projects listed above.
All demo code is available on Github at the following link: https://github.com/fluxcapacitor/pipeline/wiki
In addition, the entire demo environment has been Dockerized and made available for download on Docker Hub at the following link: https://hub.docker.com/r/fluxcapacitor/pipeline/
* Speaker Bio *
Chris Fregly is a Principal Data Solutions Engineer for the newly-formed IBM Spark Technology Center, an Apache Spark Contributor, a Netflix Open Source Committer, as well as the Organizer of the global Advanced Apache Spark Meetup and Author of the Upcoming Book, Advanced Spark.
Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix.
When Chris isn’t contributing to Spark and other open source projects, he’s creating book chapters, slides, and demos to share knowledge with his peers at meetups and conferences throughout the world.
Spark, Similarity, Approximations, NLP, Recommendations - Boulder Denver Spar...
This document summarizes a presentation given by Chris Fregly on recommendations and similarity algorithms in Apache Spark. The presentation covered scaling techniques like parallelism and composability in Spark. It also discussed different similarity measures like Euclidean, cosine, and Jaccard similarity. Recommendation algorithms mentioned include clustering users based on behavior and item metadata to find similar users and items. The document provides an overview of the key topics presented.
Advanced Analytics and Recommendations with Apache Spark - Spark Maryland/DC ...
Title
Real-time, Advanced Analytics and Recommendations using Machine Learning, Graph Processing, Natural Language Processing, and Approximations with Apache Spark, Stanford CoreNLP, and Twitter Algebird
BONUS: Netflix Recommendations: Then and Now
Agenda
Intro
Live, Interactive Recommendations Demo
Spark ML, GraphX, Streaming, Kafka, Cassandra, Docker
Types of Similarity
Euclidean vs. Non-Euclidean Similarity
User-to-User Similarity
Content-based, Item-to-Item Similarity (Amazon)
Collaborative-based, User-to-Item Similarity (Netflix)
Graph-based, Item-to-Item Similarity Pathway (Spotify)
Similarity Approximations at Scale
Twitter Algebird
MinHash and Bucketing
Locality Sensitive Hashing (LSH)
BONUS: Netflix Recommendations: From Ratings to Real-Time
DVD-Ratings-based $1M Netflix Prize (2009)
Streaming-based "Trending Now" (2016)
Wrap Up
Q & A
Bio
Chris Fregly is a Principal Data Solutions Engineer for the newly-formed IBM Spark Technology Center, an Apache Spark Contributor, and a Netflix Open Source Committer.
Chris is also the founder of the global Advanced Apache Spark Meetup and author of the upcoming book, Advanced Spark @ advancedspark.com.
Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix.
Related Links
https://github.com/fluxcapacitor/pipeline/wiki
http://cdn.oreillystatic.com/en/assets/1/event/105/Algebra%20for%20Scalable%20Analytics%20Presentation.pdf
http://static.echonest.com/BoilTheFrog/
http://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf
http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-summary/
http://www.cc.gatech.edu/~zha/CSE8801/CF/kdd-fp074-koren.pdf
5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...
Title: Real-Time Training and Deploying Spark ML Recommendations With Kafka and NetflixOSS
Speaker: Chris Fregly (https://linkedin.com/in/cfregly/)
Date: Monday, October 17, 2016
Event: https://meetup.com/Athens-Big-Data/events/234546355/
Si lence
This document outlines a business plan for a startup that aims to combine noise cancellation technologies with intelligent music playlist algorithms to allow urban dwellers to tune out noise and find personalized music selections. The company sees opportunities to exploit existing noise cancellation and music recommendation technologies by integrating them into headphones. It believes its key advantages are investing in quality manufacturing, seeking partnerships with reputable audio companies, having talented programmers, and having an open-source component to engage the community. The document provides financial projections, lists the founding team members, and outlines some key milestones.
STC Design
Engage - Create - Consume
STC is an interface between IBM and the Spark community.
Our mission is to make data available to everyone.
Can Santa Cloud survive the Generative AI revolution?
This document discusses how Santa Cloud, Santa Claus' cloud computing business, could be impacted by generative AI technology. It notes that Santa's data collection practices may violate GDPR regulations. It then proposes adopting a new identity called Leonardo.Ai and using generative AI tools like Copilot to create content such as songs, videos, and voices. The document suggests both opportunities and risks of generative AI, and calls for human and AI collaboration to create a better future. It concludes by wishing readers a merry Christmas and happy new year.
Similar to SMACK Stack @ Nitro (16)
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
Stockholm Spark Meetup Nov 23 2015 Spark After Dark 1.5
DC Spark Users Group March 15 2016 - Spark and Netflix Recommendations
DC Spark Users Group March 15 2016 - Spark and Netflix Recommendations
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Paris Spark Meetup Oct 26, 2015 - Spark After Dark v1.5 - Best of Advanced Ap...
Spark, Similarity, Approximations, NLP, Recommendations - Boulder Denver Spar...
Spark, Similarity, Approximations, NLP, Recommendations - Boulder Denver Spar...
Advanced Analytics and Recommendations with Apache Spark - Spark Maryland/DC ...
Advanced Analytics and Recommendations with Apache Spark - Spark Maryland/DC ...
5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...
5th Athens Big Data Meetup - PipelineIO Workshop - Real-Time Training and Dep...
Can Santa Cloud survive the Generative AI revolution?
Can Santa Cloud survive the Generative AI revolution?
More from Roland Tritsch
Shared-Nothing Microservices Architectures ... and how to get there @ 404.ie ...
This document discusses shared-nothing microservice architectures and how to transition to them. It notes that shared state is problematic and microservices should be stateless when possible. It outlines benefits of microservices like independent scaling and high availability. The document also describes gradually transitioning from a monolith to microservices by eliminating shared state, introducing messaging, and breaking up databases. It shares experiences from Gilt and Nitro in making these transitions over several years.
Reactive Programming Meetup - NodeJs on K8s
This document summarizes a presentation on reactive programming, immutable deployments using Kubernetes and Google Container Engine. The presentation covers reactive programming principles, immutable deployments inspired by functional programming, and concepts of Kubernetes including clusters, nodes, pods and services. It also demonstrates upgrading a running Node.js application on a Kubernetes cluster and discusses considerations for building reactive systems including using hosted container platforms and continuous integration tools.
Scala, Android & BLE - Scala Meetup Dublin - Hands on code walkthrough
This document discusses using Scala for Android development. It introduces various build tools like Gradle and SBT that can be used to create Scala Android projects. It also presents two UI frameworks called Macroid and Scaloid that allow building Android UIs with Scala. Finally, it discusses using Scala for Bluetooth Low Energy (BLE) development and provides links to example projects on GitHub.
Immutable Deployment Platforms and Continuos Delivery - Friends or Foes?
Talk on the value of immutability for deployment platforms and how this is going to help Gilt to deliver on it's vision of "All-Ops & Zero-Waiting" and how continuos delivery and immutable deployments and continuos analysis will come together to great a feedback loop that will allow engineers to iterate and innovate even faster.
Delivered at http://futurestack.io in October 2013.
Mobile Monday - WebServices on the iPhone - 05/2008
The document discusses approaches to developing mobile applications for the iPhone. It summarizes that native development using the iPhone SDK provides high-quality results but at a high cost, while browser-based approaches using AJAX are lower cost but provide more limited capabilities. It then focuses on using the Apache CXF framework to generate code for calling web services from iPhone applications in order to reduce development time compared to manually coding web service requests in JavaScript. The presentation provides resources for learning more about developing for the iPhone and mobile applications more generally.
Sbt Concepts - Tips, Tricks, Sandbox, ... 02/2013
SBT provides several commands for debugging and exploring builds:
- settings -V and tasks -V to view settings and tasks matching a regex
- inspect and inspect tree to view types and relationships of settings/tasks
- show to display values of settings and tasks
An SBT build defines a transformation of an immutable map through Setting objects. These can define SettingKeys, TaskKeys, and InputKeys to modify the map. Settings are scoped to projects, configurations, and tasks. The sbt-assembly plugin introduces new tasks to package assemblies for publishing.
A SOA for the car - 01/2009
This document discusses using a service-oriented architecture (SOA) approach for software in cars. It proposes using Web Services Definition Language (WSDL) to define services between electronic control units (ECUs) in a car. WSDL could describe the services in a way that is transport and protocol independent. Extensions to WSDL would be needed to support fixed format messages and the CAN bus transport typically used in cars. An additional extensor could provide a way to access services uniformly. Adopting concepts from application SOA could help address challenges of increasing software complexity as more ECUs are added to modern vehicles.
Puppet Camp Dublin - 06/2012
Gilt uses Puppet for infrastructure configuration management. They have around 1000 Puppet modules and typically run Puppetmaster with Apache and Passenger on CentOS. Changes go through code review and are deployed incrementally using a "canary" flag. An internal tool called Mothership acts as an external node classifier for Puppet and also handles provisioning, assets management, users/groups and DNS. Lessons learned include keeping modules small and simple, planning for change like OS upgrades, and needing at least two views (logical and physical) for node management.
iPhone meets SOA - 06/2008
This document discusses how to use FUSE to accelerate iPhone web development by generating JavaScript code to access SOAP/HTTP web services from the iPhone. It provides an agenda that discusses why this is important, demonstrates code examples before and after using FUSE, shows the service interface, and summarizes how to generate the code. The demo shows invoking operations from a sample greeter service with simplified JavaScript code instead of manual XML handling. Resources and next steps are also provided.
Innovation, Gilt and Ireland - 11/2011
Fidelma Healy introduced Gilt.com as an online retailer that offers discounted designer clothing, accessories, home goods and travel experiences. She outlined their various business divisions including Gilt, Gilt Man, Gilt Home and international sites. She noted they have over 5 million members and sell over 10,000 packages per day. Roland Tritsch then discussed Gilt's focus on innovation and how Ireland's diverse population and culture can support becoming a hub for innovation in Europe.
Teaching Functional Programming @ TCD - 11/2012
This document provides an overview of a Scala module being taught at Trinity College Dublin in November 2012. It will include a 90 minute lecture introducing Scala, and two labs: the first rewriting a Java program in Scala, and the second integrating Scala with Java using Apache Commons. The goals are to kickstart learning Scala so students can continue on their own, focusing on the language and development environment rather than functional programming concepts. The module will use Git, SBT, Scala, ScalaTest, and an IDE like IntelliJ or Eclipse. Resources listed for further learning include the Scala website, Coursera course, and Scala puzzles and problems.
"Managed Customization” in the Garment Industry - 10/2010
“Managed Customization” in the Garment Industry
4th International Conference on Mass Customization and Personalization in Central Europe (MCP - CE 2010)
RESTful Services and Distributed OSGi - 04/2009
This document discusses RESTful services and distributed OSGi. It provides an introduction to OSGi and REST, explaining how OSGi bundles work and the key concepts of REST interfaces. It describes how services can be exposed from an OSGi container to REST, and how REST services can be consumed from OSGi. Distributed OSGi allows remote invocations between OSGi containers. The document discusses challenges in making distributed OSGi and REST work together seamlessly and potential next steps.
AJAX for Mobile Devices - 04/2009
Roland Tritsch discusses using AJAX and Apache projects to build mobile applications. He notes the rise of mobile devices and importance of developing for mobile. While native apps have high costs, AJAX has low costs but may provide good enough results. Apache ServiceMix, ActiveMQ and CXF can form a server-side SOA platform, while CXF's code generation capabilities simply client development. With the right requirements, AJAX is viable for mobile if services are properly wrapped.
More from Roland Tritsch (14)
Shared-Nothing Microservices Architectures ... and how to get there @ 404.ie ...
Shared-Nothing Microservices Architectures ... and how to get there @ 404.ie ...
Scala, Android & BLE - Scala Meetup Dublin - Hands on code walkthrough
Scala, Android & BLE - Scala Meetup Dublin - Hands on code walkthrough
Immutable Deployment Platforms and Continuos Delivery - Friends or Foes?
Immutable Deployment Platforms and Continuos Delivery - Friends or Foes?
Mobile Monday - WebServices on the iPhone - 05/2008
Mobile Monday - WebServices on the iPhone - 05/2008
"Managed Customization” in the Garment Industry - 10/2010
"Managed Customization” in the Garment Industry - 10/2010
Recently uploaded
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
"What does it really mean for your system to be available, or how to define w...
We will talk about system monitoring from a few different angles. We will start by covering the basics, then discuss SLOs, how to define them, and why understanding the business well is crucial for success in this exercise.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
What began over 115 years ago as a supplier of precision gauges to the automotive industry has evolved into being an industry leader in the manufacture of product branding, automotive cockpit trim and decorative appliance trim. Value-added services include in-house Design, Engineering, Program Management, Test Lab and Tool Shops.
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
Recently uploaded (20)
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
SMACK Stack @ Nitro
- 1. SMACK Stack @ Nitro From SMACK to SMACKED :) Created by /roland.tritsch@gonitro.com @innolocity
- 2. About ... the Roland 50% Software Engineer 50% Manager/Builder of organisations 50% Seller/Implementer of ideas , Ex-IONA,Ex-Cyclist Ex-Gilt Papa and Hubby :)
- 3. About ... the Roland
- 4. About ... the Roland
- 5. About ... Nitro At Nitro, we're changing the way the world works with documents. From the desktop to the cloud, we make it easy to create, edit, share, sign and collaborate—online or offline. Moving to a SAAS model. Hosted on bare metal and cloud. Powered by Machine Learning, Data Analytics and NLP. Based on SMACK - Spark, Mesos, Akka, Casandra, Kafka.
- 9. About ... Nitro DUBLIN, Ireland, Sept. 21, 2015 /PRNewswire/ -- By bringing Guinness- style nitrogenation to an IPA, the Irish brewer has created a hop- forward India Pale Ale with the qualities of a classic Guinness pint — namely balanced flavor and aroma, creamy mouthfeel and a perfectly formed and long-lasting head of dense bubbles.
- 10. About ... SMACK
- 11. About ... SMACK
- 12. About ... SMACK
- 13. About ... SMACK
- 14. About ... SMACK
- 15. About ... SMACK
- 16. About ... SMACK
- 17. About ... SMACKED SMACK + ??? + D(ocker) = SMACKED
- 18. About ... SMACKED SMACK + E(macs :)) + D(ocker) = SMACKED