Taming the Tiger: Tips and Tricks for Using TelegrafInfluxData
Taming the Tiger: Tips and Tricks for Using Telegraf
As part of InfluxDays North America 2020 Virtual Experience, the Technical Services team will be offering a free live InfluxDB training to the first 100 registered attendees.This will be hosted over Zoom and Slack with two main trainers and there will be assistants to help participants with the course work. The training will be recorded and made available on the InfluxDays website and the InfluxData YouTube channel.
The course provides an introduction to using Telegraf within a hands-on lab setting. Attendees will be presented a series of lab exercises and get the chance to work through them with the assistance of our remote proctors. After taking this class, attendants will be able to:
Articulate the purposes and value of Telegraf
Understand the basics of configuring and running Telegraf
Understand how to manipulate incoming data to optimize InfluxDB schema
Visualize the insertion results using InfluxDB Cloud UI
Obtaining the Perfect Smoke By Monitoring Your BBQ with InfluxDB and TelegrafInfluxData
Did you know you can use InfluxDB to monitor your BBQ and to ensure the tastiest results? Join this meetup to learn two different approaches to using a time series database to monitor a BBQ or a smoker. Learn how Will Cooke uses Python, MQTT, Telegraf and InfluxDB 2.0 to monitor his smoker and to gain insight into temperature changes, the stall, and other important stats about his brisket. Scott Anderson will demonstrate how he uses a FireBoard wireless thermometer, Telegraf and InfluxDB 2.0 to continuously work towards the perfect smoke.
Taming the Tiger: Tips and Tricks for Using TelegrafInfluxData
Taming the Tiger: Tips and Tricks for Using Telegraf
As part of InfluxDays North America 2020 Virtual Experience, the Technical Services team will be offering a free live InfluxDB training to the first 100 registered attendees.This will be hosted over Zoom and Slack with two main trainers and there will be assistants to help participants with the course work. The training will be recorded and made available on the InfluxDays website and the InfluxData YouTube channel.
The course provides an introduction to using Telegraf within a hands-on lab setting. Attendees will be presented a series of lab exercises and get the chance to work through them with the assistance of our remote proctors. After taking this class, attendants will be able to:
Articulate the purposes and value of Telegraf
Understand the basics of configuring and running Telegraf
Understand how to manipulate incoming data to optimize InfluxDB schema
Visualize the insertion results using InfluxDB Cloud UI
Obtaining the Perfect Smoke By Monitoring Your BBQ with InfluxDB and TelegrafInfluxData
Did you know you can use InfluxDB to monitor your BBQ and to ensure the tastiest results? Join this meetup to learn two different approaches to using a time series database to monitor a BBQ or a smoker. Learn how Will Cooke uses Python, MQTT, Telegraf and InfluxDB 2.0 to monitor his smoker and to gain insight into temperature changes, the stall, and other important stats about his brisket. Scott Anderson will demonstrate how he uses a FireBoard wireless thermometer, Telegraf and InfluxDB 2.0 to continuously work towards the perfect smoke.
Monitoring Your ISP Using InfluxDB Cloud and Raspberry PiInfluxData
When a large group of people change their habits, it can be tricky for infrastructures! Working from home and spending time indoor today means attending video calls and streaming movies and tv shows. This leads to increased internet traffic that can create congestion on the network infrastructure. So how do you get real-time visibility into your ISP connection? In this meetup, Mirko presents his setup based on a time series database and Raspberry Pi to better understand his ISP connection quality and speed — including upload and download speeds. Join us to discover how he does it using Telegraf, InfluxDB Cloud, Astro Pi, Telegram and Grafana! Finally, proof that your ISP connection is (or is not) as fast as it promises.
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...Databricks
Meltdown and Spectre are two security vulnerabilities disclosed in early 2018 that expose systems to cross-VM and cross-process attacks. They were the first of their kind and opened up a new class of exploits that allow one program to scan another program’s memory. The kernel and VM patches released to address these vulnerabilities have shown to degrade the performance of Apache Spark workloads in the cloud by 2-5%.
This talk will dive deep into the exploits and their patches in order to help explain the origin of this decline in performance.
In this InfluxDays NYC 2019 talk, InfluxData Founder & CTO Paul Dix will outline his vision around the platform and its new data scripting and query language Flux, and he will give the latest updates on InfluxDB time series database. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.
Extending Spark SQL API with Easier to Use Array Types Operations with Marek ...Databricks
Big companies typically integrate their data from various heterogeneous systems when building a data lake as single point for accessing data. To achieve this goal technical teams often deal with data defined by complex schemas and various data formats. Spark SQL Datasets are currently compatible with data formats such as XML, Avro and Parquet by providing primitive and complex data types such as structs and arrays.
Although Dataset API offers rich set of functions, general manipulation of array and deeply nested data structures is lacking. We will demonstrate this fact by providing examples of data which is currently very hard to process in Spark efficiently. We designed and developed an extension of Dataset API to allow developers to work with array and complex type elements in a more straightforward and consistent way. The extension should help users dealing with complex and structured big data to use Apache Spark as a truly generic processing framework.
InfluxQL is a powerful query language for InfluxDB, and TICKScript is a domain specific language used by Kapacitor to define tasks involving the extraction, transformation and loading of data and also involving the tracking of arbitrary changes and detection of events within data. The combination of these two can make your monitoring apps powerful. During this session, InfluxData Engineer Michael DeSa will share best practices for using these powerful tools. Prerequisite: Intro To Kapacitor.
We'll discuss our experiences with tooling aimed at finding and fixing performance problems in a production Rust application, as experienced through the eyes of somebody who's more familiar with the Go ecosystem but grew to love Rust. We'll cover CPU and Heap profiling, and also briefly touch causal profiling.
Weather of the Century: Design and PerformanceMongoDB
This talk walks you through how you can use MongoDB to store and analyze worldwide weather data from the entire 20th century in a graphical application.
Paul will outline his vision around the platform and give the latest updates on Flux (a new query language), the decoupling of query and storage, the impact of hybrid cloud environments on architecture, cardinality, and discuss the technical directions of the platform. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.
Meet the Experts: Visualize Your Time-Stamped Data Using the React-Based Gira...InfluxData
Giraffe is the open source React-based visualization library that powers data visualizations in the InfluxDB 2.0 UI. Giraffe can be used to display your data within your own app and is Fluxlang-supported! It uses algorithms to handle visualizing high volumes of time series data that InfluxDB can ingest and query.
Kristina Robinson, the engineering manager for the Giraffe team at InfluxData, will dive into:
The basics of using the Giraffe library including how to query your data with Flux
Specific Giraffe visualization types for dashboards (e.g. single number, table and graph)
How to incorporate visualizations in your own custom apps
Using Grafana with InfluxDB 2.0 and Flux Lang by Jacob LisiInfluxData
Flux, the new InfluxData data scripting and query language (formerly IFQL), super-charges queries both for analytics and data science. Jacob Lisi from Grafana Labs will give a quick overview of the language features as well as the moving parts for a working deployment. Grafana is an open source dashboard solution that shares Flux’s passion for analytics and data science. For that reason, they are very excited to showcase the new Flux support within Grafana, and a couple of common analytics use cases to get the most out of your data.
In this InfluxDays NYC 2019 talk, Jacob Lisi will share the latest updates they have made with their Flux builder in Grafana.
k-means Clustering in Python
scikit-learn--Machine Learning in Python
from sklearn.cluster import KMeans
k-means clustering is a method of vector quantization, originally from signal processing, that is popular for cluster analysis in data mining. k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells.
The problem is computationally difficult (NP-hard); however, there are efficient heuristic algorithms that are commonly employed and converge quickly to a local optimum. These are usually similar to the expectation-maximization algorithm for mixtures of Gaussian distributions via an iterative refinement approach employed by both algorithms. Additionally, they both use cluster centers to model the data; however, k-means clustering tends to find clusters of comparable spatial extent, while the expectation-maximization mechanism allows clusters to have different shapes.[wikipedia]
ref: http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_iris.html
Big Data Day LA 2015 - Large Scale Distinct Count -- The HyperLogLog algorith...Data Con LA
"At OpenX we not only use the tools in big data ecosystems to solve our business problems, but also explore the cutting edge algorithms for practical uses. HyperLogLog is one of the algorithm that we use intensively in our internal system. It has really low computation cost and can easily plug into map-reduce framework (hadoop or spark). Some of the applications that worth to highlight are:
* high cardinality test
* distinct count of unique users over time
* Visualize hyperloglog for fraud detection"
Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
A three-part presentation on the Swift programming language:
• An introduction to Swift for Objective-C developers
• Changes in Swift 2
• What's coming in Swift 2.2 & 3.0
Evgeniy Muralev, Mark Vince, Working with the compiler, not against itSergey Platonov
The talk will look at limitations of compilers when creating fast code and how to make more effective use of both the underlying micro-architecture of modern CPU's and how algorithmic optimizations may have surprising effects on the generated code. We shall discuss several specific CPU architecture features and their pros and cons in relation to creating fast C++ code. We then expand with several algorithmic techniques, not usually well-documented, for making faster, compiler friendly, C++.
Note that we shall not discuss caching and related issues here as they are well documented elsewhere.
Monitoring Your ISP Using InfluxDB Cloud and Raspberry PiInfluxData
When a large group of people change their habits, it can be tricky for infrastructures! Working from home and spending time indoor today means attending video calls and streaming movies and tv shows. This leads to increased internet traffic that can create congestion on the network infrastructure. So how do you get real-time visibility into your ISP connection? In this meetup, Mirko presents his setup based on a time series database and Raspberry Pi to better understand his ISP connection quality and speed — including upload and download speeds. Join us to discover how he does it using Telegraf, InfluxDB Cloud, Astro Pi, Telegram and Grafana! Finally, proof that your ISP connection is (or is not) as fast as it promises.
Analyzing the Performance Effects of Meltdown + Spectre on Apache Spark Workl...Databricks
Meltdown and Spectre are two security vulnerabilities disclosed in early 2018 that expose systems to cross-VM and cross-process attacks. They were the first of their kind and opened up a new class of exploits that allow one program to scan another program’s memory. The kernel and VM patches released to address these vulnerabilities have shown to degrade the performance of Apache Spark workloads in the cloud by 2-5%.
This talk will dive deep into the exploits and their patches in order to help explain the origin of this decline in performance.
In this InfluxDays NYC 2019 talk, InfluxData Founder & CTO Paul Dix will outline his vision around the platform and its new data scripting and query language Flux, and he will give the latest updates on InfluxDB time series database. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.
Extending Spark SQL API with Easier to Use Array Types Operations with Marek ...Databricks
Big companies typically integrate their data from various heterogeneous systems when building a data lake as single point for accessing data. To achieve this goal technical teams often deal with data defined by complex schemas and various data formats. Spark SQL Datasets are currently compatible with data formats such as XML, Avro and Parquet by providing primitive and complex data types such as structs and arrays.
Although Dataset API offers rich set of functions, general manipulation of array and deeply nested data structures is lacking. We will demonstrate this fact by providing examples of data which is currently very hard to process in Spark efficiently. We designed and developed an extension of Dataset API to allow developers to work with array and complex type elements in a more straightforward and consistent way. The extension should help users dealing with complex and structured big data to use Apache Spark as a truly generic processing framework.
InfluxQL is a powerful query language for InfluxDB, and TICKScript is a domain specific language used by Kapacitor to define tasks involving the extraction, transformation and loading of data and also involving the tracking of arbitrary changes and detection of events within data. The combination of these two can make your monitoring apps powerful. During this session, InfluxData Engineer Michael DeSa will share best practices for using these powerful tools. Prerequisite: Intro To Kapacitor.
We'll discuss our experiences with tooling aimed at finding and fixing performance problems in a production Rust application, as experienced through the eyes of somebody who's more familiar with the Go ecosystem but grew to love Rust. We'll cover CPU and Heap profiling, and also briefly touch causal profiling.
Weather of the Century: Design and PerformanceMongoDB
This talk walks you through how you can use MongoDB to store and analyze worldwide weather data from the entire 20th century in a graphical application.
Paul will outline his vision around the platform and give the latest updates on Flux (a new query language), the decoupling of query and storage, the impact of hybrid cloud environments on architecture, cardinality, and discuss the technical directions of the platform. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.
Meet the Experts: Visualize Your Time-Stamped Data Using the React-Based Gira...InfluxData
Giraffe is the open source React-based visualization library that powers data visualizations in the InfluxDB 2.0 UI. Giraffe can be used to display your data within your own app and is Fluxlang-supported! It uses algorithms to handle visualizing high volumes of time series data that InfluxDB can ingest and query.
Kristina Robinson, the engineering manager for the Giraffe team at InfluxData, will dive into:
The basics of using the Giraffe library including how to query your data with Flux
Specific Giraffe visualization types for dashboards (e.g. single number, table and graph)
How to incorporate visualizations in your own custom apps
Using Grafana with InfluxDB 2.0 and Flux Lang by Jacob LisiInfluxData
Flux, the new InfluxData data scripting and query language (formerly IFQL), super-charges queries both for analytics and data science. Jacob Lisi from Grafana Labs will give a quick overview of the language features as well as the moving parts for a working deployment. Grafana is an open source dashboard solution that shares Flux’s passion for analytics and data science. For that reason, they are very excited to showcase the new Flux support within Grafana, and a couple of common analytics use cases to get the most out of your data.
In this InfluxDays NYC 2019 talk, Jacob Lisi will share the latest updates they have made with their Flux builder in Grafana.
k-means Clustering in Python
scikit-learn--Machine Learning in Python
from sklearn.cluster import KMeans
k-means clustering is a method of vector quantization, originally from signal processing, that is popular for cluster analysis in data mining. k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells.
The problem is computationally difficult (NP-hard); however, there are efficient heuristic algorithms that are commonly employed and converge quickly to a local optimum. These are usually similar to the expectation-maximization algorithm for mixtures of Gaussian distributions via an iterative refinement approach employed by both algorithms. Additionally, they both use cluster centers to model the data; however, k-means clustering tends to find clusters of comparable spatial extent, while the expectation-maximization mechanism allows clusters to have different shapes.[wikipedia]
ref: http://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_iris.html
Big Data Day LA 2015 - Large Scale Distinct Count -- The HyperLogLog algorith...Data Con LA
"At OpenX we not only use the tools in big data ecosystems to solve our business problems, but also explore the cutting edge algorithms for practical uses. HyperLogLog is one of the algorithm that we use intensively in our internal system. It has really low computation cost and can easily plug into map-reduce framework (hadoop or spark). Some of the applications that worth to highlight are:
* high cardinality test
* distinct count of unique users over time
* Visualize hyperloglog for fraud detection"
Apply Hammer Directly to Thumb; Avoiding Apache Spark and Cassandra AntiPatt...Databricks
Learn from someone who has made just about every basic Apache Spark mistake possible so you don’t have to! We’ll go over some of the most common things that users do that end up doing that cause unnecessary pain and actually explain how to avoid them.
Confused about serialization? Not sure what is meant by use a singleton to share connections? Together we will walk through concrete examples of how to handle these situation. Learn how to: do all your work remotely, not break your catalyst optimizations, use all your resources, and much more! Together lets learn how to make our Spark Applications better!
A three-part presentation on the Swift programming language:
• An introduction to Swift for Objective-C developers
• Changes in Swift 2
• What's coming in Swift 2.2 & 3.0
Evgeniy Muralev, Mark Vince, Working with the compiler, not against itSergey Platonov
The talk will look at limitations of compilers when creating fast code and how to make more effective use of both the underlying micro-architecture of modern CPU's and how algorithmic optimizations may have surprising effects on the generated code. We shall discuss several specific CPU architecture features and their pros and cons in relation to creating fast C++ code. We then expand with several algorithmic techniques, not usually well-documented, for making faster, compiler friendly, C++.
Note that we shall not discuss caching and related issues here as they are well documented elsewhere.
C++ open positions and popularity remain high as media has recently, and there is a reason for that: from the many languages and platforms that developers have available today, C++ features uncontested capabilities in power and performance, allowing innovation outside the box (just think on action games, natural user interfaces or augmented reality, to mention some). In this talk you’ll see the new features and technologies that are coming with Visual C++ vNext, helping you build compelling applications with a renewed developer experience. Don’t miss it!!
(Bill Bejeck, Confluent) Kafka Summit SF 2018
Apache Kafka added a powerful stream processing library in mid-2016, Kafka Streams, which runs on top of Apache Kafka. The community has embraced Kafka Streams with many early adopters, and the adoption rate continues to grow. Large to mid-size organizations have come to rely on Kafka Streams in their production environments. Kafka Streams has many advanced features to make applications more robust.
The point of this presentation is to show users of Kafka Streams some of the latest and greatest features, as well as some that may be advanced, that can make streams applications more resilient. The target audience for this talk are those users already comfortable writing Kafka Streams applications and want to go from writing their first proof-of-concept applications to writing robust applications that can withstand the rigor that running in a production environment demands.
The talk will be a technical deep dive covering topics like:
-Best practices on configuring a Kafka Streams application
-How to meet production SLAs by minimizing failover and recovery times: configuring standby tasks and the pros and cons of having standby replicas for local state
-How to improve resiliency and 24×7 operability: the use of different configurable error handlers, callbacks and how they can be used to see what’s going on inside the application
-How to achieve efficient scalability: a thorough review of the relationship between the number of instances, threads and state stores and how they relate to each other
While this is a technical deep dive, the talk will also present sample code so that attendees can view the concepts discussed in practice. Attendees of this talk will walk away with a deeper understanding of how Kafka Streams works, and how to make their Kafka Streams applications more robust and efficient. There will be a mix of discussion.
Java/Scala Lab: Анатолий Кметюк - Scala SubScript: Алгебра для реактивного пр...GeeksLab Odessa
SubScript - это расширение языка Scala, добавляющее поддержку конструкций и синтаксиса аглебры общающихся процессов (Algebra of Communicating Processes, ACP). SubScript является перспективным расширением, применимым как для разработки высоконагруженных параллельных систем, так и для простых персональных приложений.
O'Reilly Velocity New York 2016 presentation on modern Linux tracing tools and technology. Highlights the available tracing data sources on Linux (ftrace, perf_events, BPF) and demonstrates some tools that can be used to obtain traces, including DebugFS, the perf front-end, and most importantly, the BCC/BPF tool collection.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
28. dplyr
•
Package by Hadley Whickham
•
Plyr specialised for data frames: faster & with
remote datastores
•
Great performance thanks to C++
29. arrange
ex: Arrange by year within each player
arrange(Batting,
playerID, yearID)
Unit: milliseconds
expr
min
lq
df 186.64016 188.48495
dt 349.25496 352.12806
cpp 12.20485 13.85538
base 181.68259 182.58014
dt_raw 166.94213 170.15704
median
190.8989
357.4358
14.0081
184.6904
170.6418
uq
192.42140
403.45465
16.72979
186.33794
220.89911
max neval
195.36592
10
405.30055
10
23.95173
10
189.70377
10
223.42155
10
30. filter
Find the year for which each player played the most games
filter(Batting, G == max(G))
Unit: milliseconds
expr
min
lq
median
uq
max neval
df 371.96066 375.98652 380.92300 389.78870 430.2898
10
dt 47.37897 49.39681 51.23722 52.79181 95.8757
10
cpp 34.63382 35.27462 36.48151 38.30672 106.2422
10
base 141.81983 144.87670 147.36940 148.67299 173.8763
10
31. summarise
Compute the average number of at bats for each player
summarise(x, ab = mean(AB))
Unit: microseconds
expr
min
lq
median
uq
max neval
df 470726.569 475168.481 495500.076 498223.152 502601.494
10
dt 23002.422 23923.691 25888.191 28517.318 28683.864
10
cpp
756.265
820.921
838.529
864.624
950.079
10
base 253189.624 259167.496 263124.650 273097.845 326663.243
10
dt_raw 22462.560 23469.528 24438.422 25718.549 28385.158
10
32. Vector Visitor
Traversing an R vector of any type with the same interface
class VectorVisitor {
public:
virtual ~VectorVisitor(){}
/** hash the element of the visited vector at index i */
virtual size_t hash(int i) const = 0 ;
/** are the elements at indices i and j equal */
virtual bool equal(int i, int j) const = 0 ;
!
/** creates a new vector, of the same type as the visited vector, by
* copying elements at the given indices
*/
virtual SEXP subset( const Rcpp::IntegerVector& index ) const = 0 ;
!
}
33. Vector Visitor
inline VectorVisitor* visitor( SEXP vec ){
switch( TYPEOF(vec) ){
case INTSXP:
if( Rf_inherits(vec, "factor" ))
return new FactorVisitor( vec ) ;
return new VectorVisitorImpl<INTSXP>( vec ) ;
case REALSXP:
if( Rf_inherits( vec, "Date" ) )
return new DateVisitor( vec ) ;
if( Rf_inherits( vec, "POSIXct" ) )
return new POSIXctVisitor( vec ) ;
return new VectorVisitorImpl<REALSXP>( vec ) ;
case LGLSXP: return new VectorVisitorImpl<LGLSXP>( vec ) ;
case STRSXP: return new VectorVisitorImpl<STRSXP>( vec ) ;
default: break ;
}
// should not happen
return 0 ;
}
34. Chunked evaluation
ir <- group_by( iris, Species)
summarise(ir,
Sepal.Length = mean(Sepal.Length)
)
•
R expression to evaluate: mean(Sepal.Length)
•
Sepal.Length
•
dplyr knows mean.
•
fast and memory efficient algorithm
∊
iris
35. Hybrid evaluation
myfun <- function(x) x+x
ir <- group_by( iris, Species)
summarise(ir,
xxx = mean(Sepal.Length) + min(Sepal.Width) - myfun(Sepal.Length)
)
#1: fast evaluation of mean(Sepal.Length).
5.006 + min(Sepal.Width) - myfun(Sepal.Length)
#2: fast evaluation of min(Sepal.Width).
5.006 + 3.428 - myfun(Sepal.Length)
#3: fast evaluation of 5.006 + 3.428.
8.434 - myfun(Sepal.Length)
#4: R evaluation of 8.434 - myfun(Sepal.Length).
36. Hybrid Evaluation
!
•
mean, min, max, sum, sd, var, n, +, -, /, *, <, >,
<=, >=, &&, ||
•
packages can register their own hybrid
evaluation handler.
•
See hybrid-evaluation vignette
39. C++11 : lambda
Lambda: function defined where used. Similar to apply
functions in R.
// [[Rcpp::export]]
NumericVector foo( NumericVector v){
NumericVector res = sapply( v,
[](double x){ return x*x; }
) ;
return res ;
}
40. C++11 : for each loop
C++98, C++03
std::vector<double> v ;
for( int i=0; i<v.size(); v++){
double d = v[i] ;
// do something with d
}
C++11
for( double d: v){
// do stuff with d
}
41. C++11 : init list
C++98, C++03
NumericVector x = NumericVector::create( 1, 2 ) ;
C++11
NumericVector x = {1, 2} ;
42. Other changes
•
Move semantics : used under the hood in
Rcpp11. Using less memory.
•
Less code bloat. Variadoic templates
43. Rcpp11 article
•
I’m writing an article about C++11
•
Explain the merits of C++11
•
What’s next: C++14, C++17
•
Goal is to make C++11 welcome on CRAN
•
https://github.com/romainfrancois/cpp11_article