Vizualize 300 tb in less than 5_seconds
•Download as PPTX, PDF•
0 likes•312 views
Finding the effectiveness of a Marketing campaign is complex. Working with Fortune 500 companies in their conversion of customers on their million dollar marketing campaigns is extremely complex. Data comes from a multitude of sources and algorithms must be built to understand their data and answer tough questions. Data must be accurate, algorithms must be novel and dashboards must provide correct insights. Dashboards give marketers the ability to make order, see what's important, make decisions and act. And they needs answers FAST. How fast? We visualize 300TB of data in less than 5 seconds. Why, because if we can do this marketers win and if they can win we win. This presentation was give in a talk which explained the techniques and lessons learned of how to create interactive deep analytics dashboards of very large data sets.
Report
Share
Report
Share
Recommended
Realtime Data Analytics
The document discusses challenges and approaches for real-time data analytics. It addresses issues like data volume, latency requirements, and frequent changes. It recommends using schemas, SQL, and memory to improve efficiency and speed. Specific technologies mentioned include Spark SQL, Druid, Pinot, and ELK for real-time ingestion, processing, and querying large and changing datasets at low latency. The key aspects are high-throughput writes, concurrent reads and writes, fast scans and aggregations over historical and recent data stored columnarly with inverted indexes.
Cloud Dataflow - A Unified Model for Batch and Streaming Data Processing
Dataflow is a unified programming model and a managed service for developing and executing a wide range of data processing patterns including ETL, batch computation, and continuous computation. Cloud Dataflow frees you from operational tasks like resource management and performance optimization.
Dataflow - A Unified Model for Batch and Streaming Data Processing
Batch and Streaming Data Processing and Vizualize 300Tb in 5 Seconds meetup on April 18th, 2016 (http://www.meetup.com/Big-things-are-happening-here/events/229532500)
Introduction to Apache Beam
Apache Beam (formerly Google Cloud Dataflow SDK) is an unified model and set of language-specific SDKs for defining and executing data processing workflows. You design pipelines, simplifying the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service).
This presentation introduces the Beam programming model, and how you can use it to design your pipelines, transporting PCollection and applying some PTransforms. You will see how the same code will be "translated" to a target runtimes thanks to a specific runner. You will also have an overview of the current roadmap, with the new interesting features.
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
1. Google Cloud Dataflow is a fully managed service that allows users to define data processing pipelines that can run batch or streaming computations.
2. The Dataflow programming model defines pipelines as directed graphs of transformations on collections of data elements. This provides flexibility in how computations are defined across batch and streaming workloads.
3. The Dataflow service handles graph optimization, scaling of workers, and monitoring of jobs to efficiently execute user-defined pipelines on Google Cloud Platform.
Kenneth Knowles - Apache Beam - A Unified Model for Batch and Streaming Data...
http://flink-forward.org/kb_sessions/apache-beam-a-unified-model-for-batch-and-streaming-data-processing/
Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam (incubating) defines a new data processing programming model that evolved from more than a decade of experience within Google, including MapReduce, FlumeJava, MillWheel, and Cloud Dataflow. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, et al.) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, touch on its evolution, describe main concepts in the programming model, and compare with similar systems. We’ll go from a simple scenario to a relatively complex data processing pipeline, and finally demonstrate execution of that pipeline on multiple runtimes.
Streaming Auto-scaling in Google Cloud Dataflow
This document discusses auto-scaling in Google Cloud Dataflow. It describes how Dataflow pipelines can automatically adjust parallelism levels based on throughput, backlog growth, and CPU utilization signals. The scaling policy aims to keep pipelines keeping up with input rates and reducing backlogs quickly. The mechanism for changing parallelism involves splitting computation ranges across additional machines or migrating ranges between machines. Future work may include finer-grained range splitting and approximate throughput modeling.
Sourabh Bajaj - Big data processing with Apache Beam
Apache Beam is a unified programming model for expressing efficient and portable data processing pipelines. It provides a programming model and set of language-specific SDKs for building pipelines that can run on multiple distributed processing backends. Beam pipelines consist of transformations and collections that allow expressing data processing workflows in a way that is portable across different execution environments like Apache Spark, Apache Flink, and Google Cloud Dataflow. Examples of pipelines covered include word counting from text, trending hashtags on Twitter, and billing estimates.
Recommended
Realtime Data Analytics
The document discusses challenges and approaches for real-time data analytics. It addresses issues like data volume, latency requirements, and frequent changes. It recommends using schemas, SQL, and memory to improve efficiency and speed. Specific technologies mentioned include Spark SQL, Druid, Pinot, and ELK for real-time ingestion, processing, and querying large and changing datasets at low latency. The key aspects are high-throughput writes, concurrent reads and writes, fast scans and aggregations over historical and recent data stored columnarly with inverted indexes.
Cloud Dataflow - A Unified Model for Batch and Streaming Data Processing
Dataflow is a unified programming model and a managed service for developing and executing a wide range of data processing patterns including ETL, batch computation, and continuous computation. Cloud Dataflow frees you from operational tasks like resource management and performance optimization.
Dataflow - A Unified Model for Batch and Streaming Data Processing
Batch and Streaming Data Processing and Vizualize 300Tb in 5 Seconds meetup on April 18th, 2016 (http://www.meetup.com/Big-things-are-happening-here/events/229532500)
Introduction to Apache Beam
Apache Beam (formerly Google Cloud Dataflow SDK) is an unified model and set of language-specific SDKs for defining and executing data processing workflows. You design pipelines, simplifying the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service).
This presentation introduces the Beam programming model, and how you can use it to design your pipelines, transporting PCollection and applying some PTransforms. You will see how the same code will be "translated" to a target runtimes thanks to a specific runner. You will also have an overview of the current roadmap, with the new interesting features.
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
1. Google Cloud Dataflow is a fully managed service that allows users to define data processing pipelines that can run batch or streaming computations.
2. The Dataflow programming model defines pipelines as directed graphs of transformations on collections of data elements. This provides flexibility in how computations are defined across batch and streaming workloads.
3. The Dataflow service handles graph optimization, scaling of workers, and monitoring of jobs to efficiently execute user-defined pipelines on Google Cloud Platform.
Kenneth Knowles - Apache Beam - A Unified Model for Batch and Streaming Data...
http://flink-forward.org/kb_sessions/apache-beam-a-unified-model-for-batch-and-streaming-data-processing/
Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam (incubating) defines a new data processing programming model that evolved from more than a decade of experience within Google, including MapReduce, FlumeJava, MillWheel, and Cloud Dataflow. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, et al.) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, touch on its evolution, describe main concepts in the programming model, and compare with similar systems. We’ll go from a simple scenario to a relatively complex data processing pipeline, and finally demonstrate execution of that pipeline on multiple runtimes.
Streaming Auto-scaling in Google Cloud Dataflow
This document discusses auto-scaling in Google Cloud Dataflow. It describes how Dataflow pipelines can automatically adjust parallelism levels based on throughput, backlog growth, and CPU utilization signals. The scaling policy aims to keep pipelines keeping up with input rates and reducing backlogs quickly. The mechanism for changing parallelism involves splitting computation ranges across additional machines or migrating ranges between machines. Future work may include finer-grained range splitting and approximate throughput modeling.
Sourabh Bajaj - Big data processing with Apache Beam
Apache Beam is a unified programming model for expressing efficient and portable data processing pipelines. It provides a programming model and set of language-specific SDKs for building pipelines that can run on multiple distributed processing backends. Beam pipelines consist of transformations and collections that allow expressing data processing workflows in a way that is portable across different execution environments like Apache Spark, Apache Flink, and Google Cloud Dataflow. Examples of pipelines covered include word counting from text, trending hashtags on Twitter, and billing estimates.
Virtual Flink Forward 2020: Cogynt: Flink without code - Samantha Chan, Aslam...
In our data-driven world, the need for speed has never been greater. The advent of Flink has no doubt paved the way for faster and more efficient data delivery solutions; however, it is not without its costs. The amount of time, talent and resources required to effectively manipulate streams and conduct analysis at scale is far from trivial, and it can be especially daunting to the uninitiated or technically challenged. In an effort to make scalable stream processing more readily accessible to the world, Cogility Software created Cogynt: a zero-coding analytics platform for the masses. Cogynt enables engineers and non-engineers alike to manipulate and analyze streams on an abstracted level, while leveraging the power of Flink and Kafka under the hood to declaratively build complex Flink jobs. Shielding the analyst from low-level system configuration and programming API’s lends itself to creating an environment where analysts can focus on what’s most important to their businesses – the data. This session will demonstrate, from a data science perspective, how Cogynt can easily do almost anything Flink users can do with code, and more!
Google Cloud Dataflow Two Worlds Become a Much Better One
Google Cloud Dataflow is a fully managed service that allows users to build batch or streaming parallel data processing pipelines. It provides a unified programming model and SDKs in Java and Python to process data across Google Cloud Platform services like Pub/Sub, BigQuery, and Cloud Storage. The Cloud Dataflow service automatically optimizes and runs data pipelines at scale in a reliable, cost-effective manner without requiring operational management by the user.
Databricks clusters in autopilot mode
Why building a big data platform is hard? What are the key aspects involved in providing a "Serverless" experience for data folks. And how Databricks solves infrastructure problems and provides the "Serverless" experience.
Google cloud Dataflow & Apache Flink
This presentation shows two different engines to crunch data in a batch and streaming approach:
- Google Cloud Dataflow
- Apache Flink
So you think you can stream.pptx
- Spark Streaming allows processing of live data streams using Spark's batch processing engine by dividing streams into micro-batches.
- A Spark Streaming application consists of input streams, transformations on those streams such as maps and filters, and output operations. The application runs continuously processing each micro-batch.
- Key aspects of operationalizing Spark Streaming jobs include checkpointing to ensure fault tolerance, optimizing throughput by increasing parallelism, and debugging using Spark UI.
Self-managed and automatically reconfigurable stream processing
With its superior state management and savepoint mechanism, Apache Flink is unique among modern stream processors in supporting minimal-effort job reconfiguration. Savepoints are being extensively used to enable dynamic scaling, bug fixing, upgrades, and numerous other reconfiguration use-cases, all while preserving exactly-once semantics. However, when it comes to dynamic scaling, the burden of reconfiguration decisions -when and how much to scale- is currently placed on the user.
In this talk, I share our recent work at ETH Zurich on providing support for self-managed and automatically reconfigurable stream processing. I present SnailTrail (NSDI’18), an online critical path analysis module that detects bottlenecks and provides insights on streaming application performance, and DS2 (OSDI’18), an automatic scaling controller which identifies optimal backpressure-free configurations and operates reactively online. Both SnailTrail and DS2 are integrated with Apache Flink and publicly available. I conclude with evaluation results, ongoing work, and and future challenges in this area.
Beyond the DSL-Unlocking the Power of Kafka Streams with the Processor API (A...
Kafka Streams is a flexible and powerful framework. The Domain Specific Language (DSL) is an obvious place from which to start, but not all requirements fit the DSL model. Many people are unaware of the Processor API (PAPI) – or are intimidated by it because of sinks, sources, edges and stores – oh my! But most of the power of the PAPI can be leveraged, simply through the DSL ”#process” method, which lets you attach the general building block ”Processor” interface to your -easy to use- DSL topology, to combine the best of both worlds.
In this talk you’ll get a look at the flexibility of the DSL’s process method and the possibilities it opens up. We’ll use real world use-cases borne from extensive experience in the field with multiple customers to explore power of direct write access to the state stores and how to perform range sub-selects. We’ll also see the options that punctuators bring to the table, as well as opportunities for major latency optimisations.
Key takeaways:
* Understanding of how to combine DSL and Processors
* Capabilities and benefits of Processors
* Real-world uses of Processors
Flink Forward Berlin 2017: Dr. Radu Tudoran - Huawei Cloud Stream Service in ...
Huawei Cloud Stream Service uses Flink internally. Cloud Stream firstly uses Flink as an internal job executing engine. Kinesis use Storm, Alibaba Stream Compute service use Storm. At the end of this year we will support Flink run on the kubernetes and Mesos at cloud, also CEP on SQL and other features. The presentation will show how to create a serverless cloud service from zero, how to provide streaming features with Flink, how to operate the service with quantization and visualization (by collecting yarn/flink/os metrics in real-time). This service was developed from zero and only cost around three months.
Mantis: Netflix's Event Stream Processing System
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1yyaHb8.
The authors discuss Netflix's new stream processing system that supports a reactive programming model, allows auto scaling, and is capable of processing millions of messages per second. Filmed at qconsf.com.
Danny Yuan is an architect and software developer in Netflix’s Platform Engineering team. Justin Becker is Senior Software Engineer at Netflix.
Always On: Building Highly Available Applications on Cassandra
Cassandra was built from the ground up to enable linearly scalable, always-on applications. But the path to high availability has many land mines that can mean failure for the inexperienced user. In this talk, I will offer practical advice on how to achieve 100% uptime on millions of transactions per second. I'll address all aspects of the topic, including deployment, configuration, application design, and operations.
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Streaming is the latest hot topic in the big data world. We want to process data immediately and continuously. Modern stream processors have matured significantly and offer exceptional features, including sub-second latencies, high throughput, fault-tolerance, and seamless integration with various data sources and sinks.
Many sources of streaming data consist of related or connected events: user interactions in a social network, web page clicks, movie ratings, product purchases. These connected events can be naturally represented as edges in an evolving graph.
In this talk I will explain how we can leverage a powerful stream processor, such as Apache Flink, and academic research of the past two decades, to build graph streaming applications. I will describe how we can model graphs as streams and how we can compute graph properties without storing and managing the graph state. I will introduce useful graph summary data structures and show how they allow us to build graph algorithms in the streaming model, such as connected components, bipartiteness detection, and distance estimation.
Macy's: Changing Engines in Mid-Flight
This presentation recounts the story of Macys.com and Bloomingdales.com's migration from legacy RDBMS to NoSQL Cassandra in partnership with DataStax.
One thing that differentiates this talk from others on Cassandra is Macy's philosophy of "doing more with less." You will see why we emphasize the performance tuning aspects of iterative development when you see how much processing we can support on relatively small configurations.
This session will cover:
1) The process that led to our decision to use Cassandra
2) The approach we used for migrating from DB2 & Coherence to Cassandra without disrupting the production environment
3) The various schema options that we tried and how we settled on the current one. We'll show you a selection of some of our extensive performance tuning benchmarks, as well as how these performance results figured into our final schema designs.
4) Our lessons learned and next steps
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Apache Beam is Flink’s sibling in the Apache family of streaming processing frameworks. The Beam and Flink teams work closely together on advancing what is possible in streaming processing, including Streaming SQL extensions and code interoperability on both platforms.
Beam was originally developed at Google as the amalgamation of its internal batch and streaming frameworks to power the exabyte-scale data processing for Gmail, YouTube and Ads. It now powers a fully-managed, serverless service Google Cloud Dataflow, as well as is available to run in other Public Clouds and on-premises when deployed in portability mode on Apache Flink, Spark, Samza and other runners. Users regularly run distributed data processing jobs on Beam spanning tens of thousands of CPU cores and processing millions of events per second.
In this session, Sergei Sokolenko, Cloud Dataflow product manager, and Reuven Lax, the founding member of the Dataflow and Beam team, will share Google’s learnings from building and operating a global streaming processing infrastructure shared by thousands of customers, including:
safe deployment to dozens of geographic locations,
resource autoscaling to minimize processing costs,
separating compute and state storage for better scaling behavior,
dynamic work rebalancing of work items away from overutilized worker nodes,
offering a throughput-optimized batch processing capability with the same API as streaming,
grouping and joining of 100s of Terabytes in a hybrid in-memory/on-desk file system,
integrating with the Google Cloud security ecosystem, and other lessons.
Customers benefit from these advances through faster execution of jobs, resource savings, and a fully managed data processing environment that runs in the Cloud and removes the need to manage infrastructure.
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
In this talk, we will introduce some of the new available APIs around stateful aggregation in Structured Streaming, namely flatMapGroupsWithState. We will show how this API can be used to power many complex real-time workflows, including stream-to-stream joins, through live demos using Databricks and Apache Kafka.
Predictive Maintenance with Deep Learning and Apache Flink
Flink can be used to build a predictive maintenance system using deep learning models on time-series sensor data. A Flink data stream processing pipeline is designed to handle joining streams, applying convolutional LSTM models through an ensemble, and monitoring outputs. Docker and Prometheus are used to package and monitor the solution.
Lambda at Weather Scale - Cassandra Summit 2015
This document provides an overview of Weather.com's analytics architecture using Apache Cassandra and Spark. It summarizes Weather.com's initial attempts using Cassandra, lessons learned, and its improved architecture. The improved architecture uses Cassandra for streaming event data with time-window compaction, stores all other data in Amazon S3 for batch processing in Spark, and replaces Kafka with Amazon SQS for event ingestion. It discusses best practices for data modeling in Cassandra including partitioning, secondary indexes, and avoiding wide rows and nulls. The document also highlights how Weather.com uses Apache Zeppelin notebooks for data exploration and visualization.
Distributed Stream Processing - Spark Summit East 2017
The document discusses distributed stream processing frameworks. It provides an overview of frameworks like Storm, Spark Streaming, Samza, Flink, and Kafka Streams. It compares aspects of different frameworks like programming models, delivery guarantees, fault tolerance, and state management. General guidelines are given for choosing a framework based on needs like latency requirements and state needs. Storm and Trident are recommended for low latency tasks while Spark Streaming and Flink are more full-featured but have higher latency. The document provides code examples for word count in different frameworks.
Distributed Real-Time Stream Processing: Why and How 2.0
The demand for stream processing is increasing a lot these day. Immense amounts of data has to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications include trading, social networks, Internet of things, system monitoring, and many other examples.
In this talk we are going to discuss various state of the art open-source distributed streaming frameworks, their similarities and differences, implementation trade-offs and their intended use-cases. Apart of that, I’m going to speak about Fast Data, theory of streaming, framework evaluation and so on. My goal is to provide comprehensive overview about modern streaming frameworks and to help fellow developers with picking the best possible for their particular use-case.
Our Story With ClickHouse at seo.do
Clickhouse Ankara & Istanbul meetup slide. We talked about how did we use ClickHouse to manage billions of rows
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Afterwork big data et data viz - du lac à votre écran
This document discusses a data visualization workshop hosted by OCTOSuisse on exploring and visualizing big data from a data lake. It provides an overview of OCTO's big data capabilities and projects. It then uses a case study of Swiss public transportation data to demonstrate data exploration, analysis, and visualization techniques using tools like Tableau. The goal is to understand data, identify insights, and effectively communicate findings to others.
Digital transformation with microsoft data and ai
This document discusses how data, AI, and digital transformation can drive business value. It promotes Microsoft's data and AI platform for enabling organizations to optimize their data platforms, transform with analytics and AI, and differentiate with new applications. The platform provides security, flexibility to use any data from anywhere, and capabilities for the cloud, hybrid scenarios, and on-premises. Case studies show how various companies achieved benefits like improved performance, reduced costs, accelerated innovation, and new revenue through use of Microsoft's data and AI tools.
More Related Content
What's hot
Virtual Flink Forward 2020: Cogynt: Flink without code - Samantha Chan, Aslam...
In our data-driven world, the need for speed has never been greater. The advent of Flink has no doubt paved the way for faster and more efficient data delivery solutions; however, it is not without its costs. The amount of time, talent and resources required to effectively manipulate streams and conduct analysis at scale is far from trivial, and it can be especially daunting to the uninitiated or technically challenged. In an effort to make scalable stream processing more readily accessible to the world, Cogility Software created Cogynt: a zero-coding analytics platform for the masses. Cogynt enables engineers and non-engineers alike to manipulate and analyze streams on an abstracted level, while leveraging the power of Flink and Kafka under the hood to declaratively build complex Flink jobs. Shielding the analyst from low-level system configuration and programming API’s lends itself to creating an environment where analysts can focus on what’s most important to their businesses – the data. This session will demonstrate, from a data science perspective, how Cogynt can easily do almost anything Flink users can do with code, and more!
Google Cloud Dataflow Two Worlds Become a Much Better One
Google Cloud Dataflow is a fully managed service that allows users to build batch or streaming parallel data processing pipelines. It provides a unified programming model and SDKs in Java and Python to process data across Google Cloud Platform services like Pub/Sub, BigQuery, and Cloud Storage. The Cloud Dataflow service automatically optimizes and runs data pipelines at scale in a reliable, cost-effective manner without requiring operational management by the user.
Databricks clusters in autopilot mode
Why building a big data platform is hard? What are the key aspects involved in providing a "Serverless" experience for data folks. And how Databricks solves infrastructure problems and provides the "Serverless" experience.
Google cloud Dataflow & Apache Flink
This presentation shows two different engines to crunch data in a batch and streaming approach:
- Google Cloud Dataflow
- Apache Flink
So you think you can stream.pptx
- Spark Streaming allows processing of live data streams using Spark's batch processing engine by dividing streams into micro-batches.
- A Spark Streaming application consists of input streams, transformations on those streams such as maps and filters, and output operations. The application runs continuously processing each micro-batch.
- Key aspects of operationalizing Spark Streaming jobs include checkpointing to ensure fault tolerance, optimizing throughput by increasing parallelism, and debugging using Spark UI.
Self-managed and automatically reconfigurable stream processing
With its superior state management and savepoint mechanism, Apache Flink is unique among modern stream processors in supporting minimal-effort job reconfiguration. Savepoints are being extensively used to enable dynamic scaling, bug fixing, upgrades, and numerous other reconfiguration use-cases, all while preserving exactly-once semantics. However, when it comes to dynamic scaling, the burden of reconfiguration decisions -when and how much to scale- is currently placed on the user.
In this talk, I share our recent work at ETH Zurich on providing support for self-managed and automatically reconfigurable stream processing. I present SnailTrail (NSDI’18), an online critical path analysis module that detects bottlenecks and provides insights on streaming application performance, and DS2 (OSDI’18), an automatic scaling controller which identifies optimal backpressure-free configurations and operates reactively online. Both SnailTrail and DS2 are integrated with Apache Flink and publicly available. I conclude with evaluation results, ongoing work, and and future challenges in this area.
Beyond the DSL-Unlocking the Power of Kafka Streams with the Processor API (A...
Kafka Streams is a flexible and powerful framework. The Domain Specific Language (DSL) is an obvious place from which to start, but not all requirements fit the DSL model. Many people are unaware of the Processor API (PAPI) – or are intimidated by it because of sinks, sources, edges and stores – oh my! But most of the power of the PAPI can be leveraged, simply through the DSL ”#process” method, which lets you attach the general building block ”Processor” interface to your -easy to use- DSL topology, to combine the best of both worlds.
In this talk you’ll get a look at the flexibility of the DSL’s process method and the possibilities it opens up. We’ll use real world use-cases borne from extensive experience in the field with multiple customers to explore power of direct write access to the state stores and how to perform range sub-selects. We’ll also see the options that punctuators bring to the table, as well as opportunities for major latency optimisations.
Key takeaways:
* Understanding of how to combine DSL and Processors
* Capabilities and benefits of Processors
* Real-world uses of Processors
Flink Forward Berlin 2017: Dr. Radu Tudoran - Huawei Cloud Stream Service in ...
Huawei Cloud Stream Service uses Flink internally. Cloud Stream firstly uses Flink as an internal job executing engine. Kinesis use Storm, Alibaba Stream Compute service use Storm. At the end of this year we will support Flink run on the kubernetes and Mesos at cloud, also CEP on SQL and other features. The presentation will show how to create a serverless cloud service from zero, how to provide streaming features with Flink, how to operate the service with quantization and visualization (by collecting yarn/flink/os metrics in real-time). This service was developed from zero and only cost around three months.
Mantis: Netflix's Event Stream Processing System
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1yyaHb8.
The authors discuss Netflix's new stream processing system that supports a reactive programming model, allows auto scaling, and is capable of processing millions of messages per second. Filmed at qconsf.com.
Danny Yuan is an architect and software developer in Netflix’s Platform Engineering team. Justin Becker is Senior Software Engineer at Netflix.
Always On: Building Highly Available Applications on Cassandra
Cassandra was built from the ground up to enable linearly scalable, always-on applications. But the path to high availability has many land mines that can mean failure for the inexperienced user. In this talk, I will offer practical advice on how to achieve 100% uptime on millions of transactions per second. I'll address all aspects of the topic, including deployment, configuration, application design, and operations.
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Streaming is the latest hot topic in the big data world. We want to process data immediately and continuously. Modern stream processors have matured significantly and offer exceptional features, including sub-second latencies, high throughput, fault-tolerance, and seamless integration with various data sources and sinks.
Many sources of streaming data consist of related or connected events: user interactions in a social network, web page clicks, movie ratings, product purchases. These connected events can be naturally represented as edges in an evolving graph.
In this talk I will explain how we can leverage a powerful stream processor, such as Apache Flink, and academic research of the past two decades, to build graph streaming applications. I will describe how we can model graphs as streams and how we can compute graph properties without storing and managing the graph state. I will introduce useful graph summary data structures and show how they allow us to build graph algorithms in the streaming model, such as connected components, bipartiteness detection, and distance estimation.
Macy's: Changing Engines in Mid-Flight
This presentation recounts the story of Macys.com and Bloomingdales.com's migration from legacy RDBMS to NoSQL Cassandra in partnership with DataStax.
One thing that differentiates this talk from others on Cassandra is Macy's philosophy of "doing more with less." You will see why we emphasize the performance tuning aspects of iterative development when you see how much processing we can support on relatively small configurations.
This session will cover:
1) The process that led to our decision to use Cassandra
2) The approach we used for migrating from DB2 & Coherence to Cassandra without disrupting the production environment
3) The various schema options that we tried and how we settled on the current one. We'll show you a selection of some of our extensive performance tuning benchmarks, as well as how these performance results figured into our final schema designs.
4) Our lessons learned and next steps
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Apache Beam is Flink’s sibling in the Apache family of streaming processing frameworks. The Beam and Flink teams work closely together on advancing what is possible in streaming processing, including Streaming SQL extensions and code interoperability on both platforms.
Beam was originally developed at Google as the amalgamation of its internal batch and streaming frameworks to power the exabyte-scale data processing for Gmail, YouTube and Ads. It now powers a fully-managed, serverless service Google Cloud Dataflow, as well as is available to run in other Public Clouds and on-premises when deployed in portability mode on Apache Flink, Spark, Samza and other runners. Users regularly run distributed data processing jobs on Beam spanning tens of thousands of CPU cores and processing millions of events per second.
In this session, Sergei Sokolenko, Cloud Dataflow product manager, and Reuven Lax, the founding member of the Dataflow and Beam team, will share Google’s learnings from building and operating a global streaming processing infrastructure shared by thousands of customers, including:
safe deployment to dozens of geographic locations,
resource autoscaling to minimize processing costs,
separating compute and state storage for better scaling behavior,
dynamic work rebalancing of work items away from overutilized worker nodes,
offering a throughput-optimized batch processing capability with the same API as streaming,
grouping and joining of 100s of Terabytes in a hybrid in-memory/on-desk file system,
integrating with the Google Cloud security ecosystem, and other lessons.
Customers benefit from these advances through faster execution of jobs, resource savings, and a fully managed data processing environment that runs in the Cloud and removes the need to manage infrastructure.
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
In this talk, we will introduce some of the new available APIs around stateful aggregation in Structured Streaming, namely flatMapGroupsWithState. We will show how this API can be used to power many complex real-time workflows, including stream-to-stream joins, through live demos using Databricks and Apache Kafka.
Predictive Maintenance with Deep Learning and Apache Flink
Flink can be used to build a predictive maintenance system using deep learning models on time-series sensor data. A Flink data stream processing pipeline is designed to handle joining streams, applying convolutional LSTM models through an ensemble, and monitoring outputs. Docker and Prometheus are used to package and monitor the solution.
Lambda at Weather Scale - Cassandra Summit 2015
This document provides an overview of Weather.com's analytics architecture using Apache Cassandra and Spark. It summarizes Weather.com's initial attempts using Cassandra, lessons learned, and its improved architecture. The improved architecture uses Cassandra for streaming event data with time-window compaction, stores all other data in Amazon S3 for batch processing in Spark, and replaces Kafka with Amazon SQS for event ingestion. It discusses best practices for data modeling in Cassandra including partitioning, secondary indexes, and avoiding wide rows and nulls. The document also highlights how Weather.com uses Apache Zeppelin notebooks for data exploration and visualization.
Distributed Stream Processing - Spark Summit East 2017
The document discusses distributed stream processing frameworks. It provides an overview of frameworks like Storm, Spark Streaming, Samza, Flink, and Kafka Streams. It compares aspects of different frameworks like programming models, delivery guarantees, fault tolerance, and state management. General guidelines are given for choosing a framework based on needs like latency requirements and state needs. Storm and Trident are recommended for low latency tasks while Spark Streaming and Flink are more full-featured but have higher latency. The document provides code examples for word count in different frameworks.
Distributed Real-Time Stream Processing: Why and How 2.0
The demand for stream processing is increasing a lot these day. Immense amounts of data has to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications include trading, social networks, Internet of things, system monitoring, and many other examples.
In this talk we are going to discuss various state of the art open-source distributed streaming frameworks, their similarities and differences, implementation trade-offs and their intended use-cases. Apart of that, I’m going to speak about Fast Data, theory of streaming, framework evaluation and so on. My goal is to provide comprehensive overview about modern streaming frameworks and to help fellow developers with picking the best possible for their particular use-case.
Our Story With ClickHouse at seo.do
Clickhouse Ankara & Istanbul meetup slide. We talked about how did we use ClickHouse to manage billions of rows
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
What's hot (20)
Virtual Flink Forward 2020: Cogynt: Flink without code - Samantha Chan, Aslam...
Virtual Flink Forward 2020: Cogynt: Flink without code - Samantha Chan, Aslam...
Google Cloud Dataflow Two Worlds Become a Much Better One
Google Cloud Dataflow Two Worlds Become a Much Better One
Self-managed and automatically reconfigurable stream processing
Self-managed and automatically reconfigurable stream processing
Beyond the DSL-Unlocking the Power of Kafka Streams with the Processor API (A...
Beyond the DSL-Unlocking the Power of Kafka Streams with the Processor API (A...
Flink Forward Berlin 2017: Dr. Radu Tudoran - Huawei Cloud Stream Service in ...
Flink Forward Berlin 2017: Dr. Radu Tudoran - Huawei Cloud Stream Service in ...
Always On: Building Highly Available Applications on Cassandra
Always On: Building Highly Available Applications on Cassandra
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Keynote: Building and Operating A Serverless Streaming Runtime for Apache Bea...
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
Predictive Maintenance with Deep Learning and Apache Flink
Predictive Maintenance with Deep Learning and Apache Flink
Distributed Stream Processing - Spark Summit East 2017
Distributed Stream Processing - Spark Summit East 2017
Distributed Real-Time Stream Processing: Why and How 2.0
Distributed Real-Time Stream Processing: Why and How 2.0
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Similar to Vizualize 300 tb in less than 5_seconds
Afterwork big data et data viz - du lac à votre écran
This document discusses a data visualization workshop hosted by OCTOSuisse on exploring and visualizing big data from a data lake. It provides an overview of OCTO's big data capabilities and projects. It then uses a case study of Swiss public transportation data to demonstrate data exploration, analysis, and visualization techniques using tools like Tableau. The goal is to understand data, identify insights, and effectively communicate findings to others.
Digital transformation with microsoft data and ai
This document discusses how data, AI, and digital transformation can drive business value. It promotes Microsoft's data and AI platform for enabling organizations to optimize their data platforms, transform with analytics and AI, and differentiate with new applications. The platform provides security, flexibility to use any data from anywhere, and capabilities for the cloud, hybrid scenarios, and on-premises. Case studies show how various companies achieved benefits like improved performance, reduced costs, accelerated innovation, and new revenue through use of Microsoft's data and AI tools.
big data et data viz - du lac à votre écran - afterwork
This document discusses a presentation on big data and data visualization from lake to screen. It covers exploring data in a data lake using tools like Tableau and Jupyter notebooks. Models can be built to predict things like train delays. Visualizations are then created using technologies like D3.js to communicate insights from the data and models. The goal is to extract value from large, raw data sources through the entire data science process from exploration to communication.
Logitech - LOGITECH ACCELERATES CLOUD ANALYTICS USING DATA VIRTUALIZATION
Logitech is using data virtualization and cloud analytics to accelerate insights from their growing data. They implemented Denodo on AWS to create a decentralized self-service analytics environment. This allows business users to perform descriptive, diagnostic, predictive, and prescriptive analytics. Logitech aims to provide real-time, natural language insights on desktops and phones to support business decisions. Data virtualization has helped Logitech reduce costs while improving data access, governance, and analytics capabilities.
Making the Most of In-Memory: More than Speed
The document discusses how in-memory platforms are more than just speed - they are designed to efficiently exploit RAM and are optimized for analytics workloads that involve complex "crunching" of data. It explains that analytics workloads are CPU-intensive and benefit from techniques like parallelization across CPU cores. Additionally, the document notes that declining RAM prices and interest in advanced analytics are driving more adoption of in-memory platforms for both large and small data use cases.
IoT Slam Keynote: Harnessing the Flood of Data with Heterogeneous Computing a...
This presentation was delivered as the closing keynote for the 2015 IoT Slam virtual conference. During the presentation, Ryft VP of Engineering, Pat McGarry, took a close look at how the IoT revolution is changing data analytics and driving the move of data analysis to the network’s edge where the data is being created. - See more at: http://www.ryft.com/blog/2015-iot-slam-keynote-harnessing-flood-of-iot-data-with-heterogenenous-computing-at-the-edge#sthash.x1Anoapb.dpuf
Denodo DataFest 2016: Big Data Virtualization in the Cloud
Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/kahTgf
Many firms are adopting “cloud first” strategy and are migrating their on-premises technologies to the cloud. Logitech is one of them. They have adopted the AWS platform and big data on the cloud for all of their analytical needs, including Amazon Redshift and S3.
In this presentation, the Principal of Big Data and Analytics team at Logitech, Avinash Deshpande will present:
• The business rationale for migrating to the cloud
• How data virtualization enables the migration
• Running data virtualization itself in the cloud
This session also includes a panel discussion with:
• Avinash Deshpande, Principal – Big Data and Analytics at Logitech
• Kurt Jackson, Platform Lead at Autodesk
• Dan Young, Chief Data Architect at Indiana University
• Paul Moxon, Head of Product Management at Denodo (as moderator)
This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6
How to Transform Into a Data-Driven Organization
A webinar on enabling organizations to start their journey to becoming a data-driven organization wherein data are turned into actionable insights.
Powered by EXIST Software Labs, Inc: https://www.exist.com/
Data Platform Modernization Solutions
Power your entire data estate with SQL Server 2017 and Azure Data Services
Data Sources
Azure Data Services has a variety of data management and analytics tools for all your data needs. It can store structured and unstructured data, whether it’s born in the cloud like sensor and social, or perhaps different data altogether like media.
<click>
Data Management
Azure SQL Database is the cloud answer to managing your operational data, with the ability to achieve infinite scale. It’s built on SQL Server, so your existing applications and skills transfer. And as a managed database service, it can save you time and money in both set-up and administration.
Azure Cosmos DB is a database designed for modern mobile and web applications. It is also a database-as-a-service, fully managed by Microsoft Azure. Azure Cosmos DB delivers consistently fast reads and writes, schema flexibility, and the ability to easily scale a database up and down on demand
Azure SQL Data Warehouse is the industry’s first enterprise-class cloud data warehouse that can grow, shrink, and pause in seconds. Based on the proven SQL Server relational database engine, it gives petabyte scalability with massive parallel-processing architecture that enables distributed processing to handle the rigors of modern data realities. It independently scales compute and storage in seconds.
SQL Data Warehouse works seamlessly with Power BI, Azure Machine Learning, HDInsight, and Azure Data Factory.
Azure Data Lake - Offers data management and analytics at scale to customers, as Azure Data Lake. It is composed of three parts: Azure Data Lake Store, Azure Data Lake Analytics, and HD Insight. Together with these products, you can store big data of any size and type, and analyze it with familiar open-source tools – all while getting a leading, enterprise-class SLA.
Data analytics at a petabyte scale final
Or Koren, Head of Data @ ironSource presents how re-designed it's big data analytics using Presto to analyze at petabyte scale,.
A Network Engineer's Approach to Automation
This document introduces Junos PyEZ, an open-source micro-framework for network automation using Python. It is designed for network engineers who want to automate tasks but may not have extensive programming experience. Junos PyEZ provides Python-based access to Juniper device configuration and operational data without requiring knowledge of XML or CLI screen-scraping. It presents a layered approach where users can start interactively in a Python shell and progressively build more complex scripts and applications using the Junos PyEZ abstraction layer or other open-source projects like ncclient. The project documentation is hosted on Juniper's TechWiki.
Hybrid Cloud: The Engine for Digital Transformation
Presentation video: https://youtu.be/H-guf07meWY
Company: Lenovo
Presenter: Karl Hansen
Location: Future Cloud Summit Tallinn 2018
website: https://www.futurecloudsummit2018.pilw.io
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...Lucidworks (Archived)
This document discusses building a lightweight discovery interface for Chinese patents. It describes using parsers and the cloud to ingest various patent file formats and metadata in order to build a search interface. It emphasizes spending adequate time on user experience design and sharing data with users and other applications.SQL Anywhere and the Internet of Things
This lecture will discuss and demonstrate how SQL Anywhere can be used in Internet of Things scenarios. Learn how to leverage the full power of SQL Anywhere in applications that run on single board computers like the Raspberry Pi. Topics discussed will include an overview of IoT, deployment and setup of SQL Anywhere on single board computers, and an example application to demonstrate the powerful and flexible applications that can be run anywhere and synchronize data back to an enterprise data source such as Hana Cloud Platform.
Live Coding a KSQL Application
Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.
Data Science in the Cloud @StitchFix
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu.
Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com..
Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.
Should I stay or should I go?
The document discusses factors to consider when deciding whether to migrate away from an Oracle database platform. It outlines some common reasons companies consider migrating, such as high costs of Oracle licenses and support. However, it also notes that migrating involves costs for migration, training, and operating on a new platform. The document advises evaluating database products and scenarios carefully before deciding to migrate or stay with the existing Oracle platform. It concludes that not every application needs Oracle and alternatives exist, but simplicity should be prioritized over unnecessary changes.
Database Shootout: What's best for BI?
This TDWI EU 2012 presentation looks at the various options for implementing a data store for analytical purposes and shows that there's no 'one size fits all' solution available
Delivering New Visibility and Analytics for IT Operations
The document discusses how Splunk provides visibility and analytics for IT operations. It outlines Splunk's ability to ingest data from various sources like applications, databases, networks and more. This gives organizations a universal platform to gain operational visibility, enable proactive monitoring, and obtain business insights from their machine data in real-time. Splunk differentiators include analyzing all data, scaling for large environments, and reducing MTTR, costs and improving user experiences.
SplunkLive Wellington 2015 - Operational Intelligence
The document discusses how Splunk provides visibility and analytics for IT operations. It describes how Splunk can ingest data from various sources like applications, databases, networks, virtualization and more. This gives organizations operational visibility across their infrastructure and enables proactive monitoring, search and investigation capabilities for troubleshooting and problem solving. Splunk offers a universal platform for machine data that can scale to handle large, complex environments.
Similar to Vizualize 300 tb in less than 5_seconds (20)
Afterwork big data et data viz - du lac à votre écran
Afterwork big data et data viz - du lac à votre écran
big data et data viz - du lac à votre écran - afterwork
big data et data viz - du lac à votre écran - afterwork
Logitech - LOGITECH ACCELERATES CLOUD ANALYTICS USING DATA VIRTUALIZATION
Logitech - LOGITECH ACCELERATES CLOUD ANALYTICS USING DATA VIRTUALIZATION
IoT Slam Keynote: Harnessing the Flood of Data with Heterogeneous Computing a...
IoT Slam Keynote: Harnessing the Flood of Data with Heterogeneous Computing a...
Denodo DataFest 2016: Big Data Virtualization in the Cloud
Denodo DataFest 2016: Big Data Virtualization in the Cloud
Hybrid Cloud: The Engine for Digital Transformation
Hybrid Cloud: The Engine for Digital Transformation
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...
Building a Lightweight Discovery Interface for Chinese Patents, Presented by ...
Delivering New Visibility and Analytics for IT Operations
Delivering New Visibility and Analytics for IT Operations
SplunkLive Wellington 2015 - Operational Intelligence
SplunkLive Wellington 2015 - Operational Intelligence
Recently uploaded
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Webinar: Designing a schema for a Data Warehouse
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Recently uploaded (20)
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Vizualize 300 tb in less than 5_seconds
- 1. VIZUALIZE 300TB IN 5 SECONDS
- 2. 2
- 3. Techniques, Tips & Tricks UX/IDataTechnology
- 4. I am Yaniv Shalev CTO @Convertro | Site Lead AOL Israel Hello!
- 5. TV DISPLAY SOCIAL SEARCH MOBILE VIDEO WE OPTIMIZE MARKTERS’ BUDGET INVESTMENT OUT OF HOME Weather
- 6. Technology
- 7. O(N) made easy 8-15 X
- 11. Write Once Read Many Reports Analytics Reports Analytics N:N 1:N
- 12. Sometimes more is more offline Onlinelogs online Reports Analytics N:N 1:N Slow Speed
- 13. Something is missing ConsumersData Sources Transformations
- 14. Smart Data Service offlineOnlinelogs online Reports Analytics N:N 1:N HYDRO • Extract • Transform • Render • Optimize • Stream manipulations • Rendering • Logic reuse • Caching • Modular • Pluggable SLOW SPEED API
- 15. Data
- 17. Minimize IO, Minimize CPU
- 18. Let the structures work for you Partitioning Sorting Clustering / Locality
- 20. UX/I
- 22. Value vs. Cost Dimensionality Granularity Freshness
- 23. Make time 1. Pre compute 2. 10 seconds to cache 3. Data reuse 4. In memory join 2 data sources 5. Data reuse + in memory join 6. Action via API
- 25. We did it the wrong way
- 26. First thing first UX/I Data Technology
- 27. Any questions ? You can find me at ◉ @yanivshalev ◉ yaniv.shalev@teamaol.com ◉ http://www.meetup.com/Tech-Talk-Teach/ Thanks!
Editor's Notes
- 1987 מנהל רשת הילטון צפון אמריקה מגיע פעם ראשונה לביקור בסיאול – דרום אמריקה – באחד ממולונות הילטון המפוארים פקידת הקבלה מברכת אותו ב Welcome to our hotel sir והציעה לו ביקור במלון לאחר מספר חודשים חזר עם אשתו, פקידת קבלה שונה ברכה אותו ב Welcome back sir ואל אשתו פנתה welcome to our hotel זה לא היה אקראי !!! נפעם מהחוויה התקשר לצוות יועציו על מנת לפצח את החוויה, צוות יועציו יכל לחשוב הוא על זיהוי פנים שהיה מאוד יקר בזמנו לפני ששקל להשקיע את הכסף התקשר למנהל המלון ושאל אותו איך הם מייצרים חוויה מדהימה כל כך וא ענה לו: אה, יש לנו דיל עם נהגי המוניות. כשהם מסיעים את האורחים משדה התעופה הם מבררים אם הם כבר ביקרו במלון שלנו. כדי שהעברת המידע לקבלה לא תהיה ברורה מדי, הם מניחים את המזוודה מימין לאורח, כשמדובר באורח חוזר, ומשמאל- כשמדובר באורח חדש. הסידור הזה עולה פחות מחצי דולר Lesson learned: לפעמים הפיתרון הוא לא רק טכנולוגי, ודווקא ההבנה אי Techniques to get the most from you technology