This document discusses the care and feeding of large scale Graphite installations. It begins with introductions and then discusses Graphite components like carbon-cache, carbon-aggregator, carbon-relay and StatsD. It covers Graphite storage, installation, documentation, middleware, backups, monitoring and the web UI. It provides tips on tuning, debugging and visualizing metrics in Graphite.
Initially presented at OpenWest 2014 conference.
Graphite and StatsD gather line series data and offer a robust set of APIs to access that data. While the tools are robust, the dashboards are straight from 1992 and alerting off the data is nonexistent. Nark, an opensource project, solves both of these problems. It provides easy to use dashboards and readily available alerts and notifications to users. It has been used in production at Lucid Software for almost a year. Related to Nark are the tools required to make Graphite highly available.
Graphite is often regarded as very slow and not easily scalable. As a data driven company, we couldn't give up the statistical functions of Graphite. In this talk we show how SimilarWeb scaled its graphite stack to meet the demand.
Federated Graphite in Docker - Denver Docker MeetupPhil Zimmerman
Slides from talk I gave at Denver Docker Meetup about a Federated Graphite Cluster in Docker using Consul, Consul-template, Registrator, Mesos and Marathon.
Initially presented at OpenWest 2014 conference.
Graphite and StatsD gather line series data and offer a robust set of APIs to access that data. While the tools are robust, the dashboards are straight from 1992 and alerting off the data is nonexistent. Nark, an opensource project, solves both of these problems. It provides easy to use dashboards and readily available alerts and notifications to users. It has been used in production at Lucid Software for almost a year. Related to Nark are the tools required to make Graphite highly available.
Graphite is often regarded as very slow and not easily scalable. As a data driven company, we couldn't give up the statistical functions of Graphite. In this talk we show how SimilarWeb scaled its graphite stack to meet the demand.
Federated Graphite in Docker - Denver Docker MeetupPhil Zimmerman
Slides from talk I gave at Denver Docker Meetup about a Federated Graphite Cluster in Docker using Consul, Consul-template, Registrator, Mesos and Marathon.
A hands-on workshop about a typical data architecture for an IoT device - how to gather data from the device, display it on a dashboard and trigger alerts based on thresholds that you set.
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData
InfluxDB 2.0 has some new dashboarding and querying capabilities that will make using a time series database even easier. This InfluxDays NYC 2019 presentation presented by David G. Simmons (Senior Developer Evangelist at InfluxData), walks you through how to set up your first dashboard.
Presto: Optimizing Performance of SQL-on-Anything EngineDataWorks Summit
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
(BDT403) Netflix's Next Generation Big Data Platform | AWS re:Invent 2014Amazon Web Services
As Netflix expands their services to more countries, devices, and content, they continue to evolve their big data analytics platform to accommodate the increasing needs of product and consumer insights. This year, Netflix re-innovated their big data platform: they upgraded to Hadoop 2, transitioned to the Parquet file format, experimented with Pig on Tez for the ETL workload, and adopted Presto as their interactive querying engine. In this session, Netflix discusses their latest architecture, how they built it on the Amazon EMR infrastructure, the contributions put into the open source community, as well as some performance numbers for running a big data warehouse with Amazon S3.
ARCHITECTING INFLUXENTERPRISE FOR SUCCESSInfluxData
In this session, everyone will learn how to architect their own InfluxEnterprise clusters to be performant and resilient whether in a single data center or spread across multiple datacenters.
Presto talk @ Global AI conference 2018 Bostonkbajda
Presented at Global AI Conference in Boston 2018:
http://www.globalbigdataconference.com/boston/global-artificial-intelligence-conference-106/speaker-details/kamil-bajda-pawlikowski-62952.html
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, LinkedIn, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years. Presto is really a SQL-on-Anything engine in a single query can access data from Hadoop, S3-compatible object stores, RDBMS, NoSQL and custom data stores. This talk will cover some of the best use cases for Presto, recent advancements in the project such as Cost-Based Optimizer and Geospatial functions as well as discuss the roadmap going forward.
tado° Makes Your Home Environment Smart with InfluxDBInfluxData
Michal Knizek, Head of Research and Development at tado° GmbH, will share how they use InfluxData to gather data collected from their Smart Thermostat to help turn any home thermostat into a smart device. This device uses a variety of information collected (geo-location, temperature, user settings, current device functional state) to serve information to automatically control the environment temperature as well as letting users know when the device may need maintenance.
Streaming Sensor Data with Grafana and InfluxDB | Ryan Mckinley | GrafanaInfluxData
In this session, Ryan will preview the new streaming and shared query support in Grafana. He will show how you can visualize high-resolution real-time sensor streams using InfluxDB and Grafana.
HBaseCon 2015: Apache Kylin - Extreme OLAP Engine for HadoopHBaseCon
Kylin is an open source distributed analytics engine contributed by eBay that provides a SQL interface and OLAP on Hadoop supporting extremely large datasets. Kylin's pre-built MOLAP cubes (stored in HBase), distributed architecture, and high concurrency helps users analyze multidimensional queries via SQL and other BI tools. During this session, you'll learn how Kylin uses HBase's key-value store to serve SQL queries with relational schema.
JAX London 2021: Jumpstart Your Cloud Native Development: An Overview of Prac...Daniel Bryant
At a previous JAX event I talked about effective cloud native Java developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I’ll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
Join this talk to:
Learn why the core tenets of continuous delivery — speed and safety — must be considered in all parts of the cloud native SDLC
Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
Explore the importance of cultivating an effective cloud platform and associated team of experts
Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
At GOTO Amsterdam in 2019 I presented how to create an effective cloud native developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I'll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
In this talk, you will:
- Learn why the core tenets of continuous delivery -- speed and safety -- must be considered in all parts of the cloud native SDLC
- Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
- Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
- Explore the importance of cultivating an effective cloud platform and associated team of experts
- Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
A hands-on workshop about a typical data architecture for an IoT device - how to gather data from the device, display it on a dashboard and trigger alerts based on thresholds that you set.
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData
InfluxDB 2.0 has some new dashboarding and querying capabilities that will make using a time series database even easier. This InfluxDays NYC 2019 presentation presented by David G. Simmons (Senior Developer Evangelist at InfluxData), walks you through how to set up your first dashboard.
Presto: Optimizing Performance of SQL-on-Anything EngineDataWorks Summit
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.
(BDT403) Netflix's Next Generation Big Data Platform | AWS re:Invent 2014Amazon Web Services
As Netflix expands their services to more countries, devices, and content, they continue to evolve their big data analytics platform to accommodate the increasing needs of product and consumer insights. This year, Netflix re-innovated their big data platform: they upgraded to Hadoop 2, transitioned to the Parquet file format, experimented with Pig on Tez for the ETL workload, and adopted Presto as their interactive querying engine. In this session, Netflix discusses their latest architecture, how they built it on the Amazon EMR infrastructure, the contributions put into the open source community, as well as some performance numbers for running a big data warehouse with Amazon S3.
ARCHITECTING INFLUXENTERPRISE FOR SUCCESSInfluxData
In this session, everyone will learn how to architect their own InfluxEnterprise clusters to be performant and resilient whether in a single data center or spread across multiple datacenters.
Presto talk @ Global AI conference 2018 Bostonkbajda
Presented at Global AI Conference in Boston 2018:
http://www.globalbigdataconference.com/boston/global-artificial-intelligence-conference-106/speaker-details/kamil-bajda-pawlikowski-62952.html
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, LinkedIn, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years. Presto is really a SQL-on-Anything engine in a single query can access data from Hadoop, S3-compatible object stores, RDBMS, NoSQL and custom data stores. This talk will cover some of the best use cases for Presto, recent advancements in the project such as Cost-Based Optimizer and Geospatial functions as well as discuss the roadmap going forward.
tado° Makes Your Home Environment Smart with InfluxDBInfluxData
Michal Knizek, Head of Research and Development at tado° GmbH, will share how they use InfluxData to gather data collected from their Smart Thermostat to help turn any home thermostat into a smart device. This device uses a variety of information collected (geo-location, temperature, user settings, current device functional state) to serve information to automatically control the environment temperature as well as letting users know when the device may need maintenance.
Streaming Sensor Data with Grafana and InfluxDB | Ryan Mckinley | GrafanaInfluxData
In this session, Ryan will preview the new streaming and shared query support in Grafana. He will show how you can visualize high-resolution real-time sensor streams using InfluxDB and Grafana.
HBaseCon 2015: Apache Kylin - Extreme OLAP Engine for HadoopHBaseCon
Kylin is an open source distributed analytics engine contributed by eBay that provides a SQL interface and OLAP on Hadoop supporting extremely large datasets. Kylin's pre-built MOLAP cubes (stored in HBase), distributed architecture, and high concurrency helps users analyze multidimensional queries via SQL and other BI tools. During this session, you'll learn how Kylin uses HBase's key-value store to serve SQL queries with relational schema.
JAX London 2021: Jumpstart Your Cloud Native Development: An Overview of Prac...Daniel Bryant
At a previous JAX event I talked about effective cloud native Java developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I’ll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
Join this talk to:
Learn why the core tenets of continuous delivery — speed and safety — must be considered in all parts of the cloud native SDLC
Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
Explore the importance of cultivating an effective cloud platform and associated team of experts
Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
At GOTO Amsterdam in 2019 I presented how to create an effective cloud native developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I'll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
In this talk, you will:
- Learn why the core tenets of continuous delivery -- speed and safety -- must be considered in all parts of the cloud native SDLC
- Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
- Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
- Explore the importance of cultivating an effective cloud platform and associated team of experts
- Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
GOTOpia 2/2021 "Cloud Native Development Without the Toil: An Overview of Pra...Daniel Bryant
At GOTO Amsterdam in 2019 I presented how to create an effective cloud native developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I'll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
Join this talk to:
Learn why the core tenets of continuous delivery -- speed and safety -- must be considered in all parts of the cloud native SDLC
Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
Explore the importance of cultivating an effective cloud platform and associated team of experts
Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu.
Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com..
Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.
This is a summary of the sessions I attended at PASS Summit 2017. Out of the week-long conference, I put together these slides to summarize the conference and present at my company. The slides are about my favorite sessions that I found had the most value. The slides included screenshotted demos I personally developed and tested alike the speakers at the conference.
Every customer-facing project aims to achieve high uptime. We, as developers, write thorough unit and integration tests to catch as many bugs as possible. We even set up staging environments to simulate production as well as we can. Yet, downtime still happens and not only once we had to revert a change.
Blue-Green deployment is a technique that allows us to ship code with high confidence, minimizing the risk of errors. We will go through the why and what of Blue-Green,as well as show how you can start doing it using Spinnaker.
BDEVOPS @ Adobe 11.04.2017
Introducing Gridiron Security and Compliance Management Platform and Enclave ...Aptible
View more (including full video and transcript) at: https://www.aptible.com/resources/update-webinar-jan-2017/
The Aptible Update Webinar Series is a quarterly presentation that covers recent features and changes to the Enclave deployment platform and Gridiron security management products.
The Aptible Update Webinar Series is a quarterly presentation that covers recent features and changes to the Enclave deployment platform and Gridiron security management products. These webinars feature technical sessions led by Aptible engineers, live demonstrations, customer examples, and Q&A with the Aptible team.
This Aptible Update Webinar covers:
- A preview of Gridiron, our security management and compliance platform
- Improvements to the Enclave deployment process for reliability, predictability and speed
- Enclave database logging
- The Enclave CLI for Windows
- Continual efforts to strengthen Enclave security
My 6th. revision of my Stackato presentation given at the German Perl Workshop 2013 in Berlin, Germany,
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Most mid-sized Django websites thrive by relying on memcached. Though what happens when basic memcached is not enough? And how can one identify when the caching architecture is becoming a bottleneck? We'll cover the problems we've encountered and solutions we've put in place.
Continuous Integration, the minimum viable productJulian Simpson
What does it mean to 'do' Continuous Integration? It used to be enough to execute your unit tests in CI. But the bar is steadily raising for engineering practices. In the last decade we've seen tremendous improvements inacceptance testing. JavaScript is now a platform in it's own right. Cloudcomputing is now vital. There's growing interest in deployment to prod.So Continuous Integration is under more pressure than ever. As the bar slowly raises for engineering practices, we ll present 2011's minimum viable feature set for Continuous Integration
Azure + DataStax Enterprise (DSE) Powers Office365 Per User StoreDataStax Academy
We will present our Office 365 use case scenarios, why we chose Cassandra + Spark, and walk through the architecture we chose for running DSE on Azure.
The presentation will feature demos on how you too can build similar applications.
Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCPDaniel Zivkovic
Enterprises traditionally think of App Platforms as PCF (Pivotal Cloud Foundry) or Red Hat OpenShift. In reality, public Clouds have evolved into Application Platforms - especially when using Managed Services & Serverless.
• If you are an IT Executive under increased pressure to cut costs, see how better Technology Stack choices – not layoffs or pay cuts, can reduce IT costs + increase business agility (while avoiding vendor lock-in):
• If you are a Developer lost in the sea of the Cloud Computing choices, watch Ray Tsang (Java Champion from GCP) live-code, and you will walk away Cloud-Native :)
See how to stop cannibalization of IT by deploying your good ol' Java Spring Boot Apps directly to Google Cloud Platform - no Servers/PCF/OpenShift/Kubernetes to manage, nor to limit your creativity: https://youtu.be/2B0wWagE0dc
P.S. For more forward-looking Software Developerment topics, join ServerlessToronto.org Meetups, and if you have any questions about the Architectural Patterns discussed, reach out to me to chat.
Topics of this presentation:
- Basics and best practices of developing single-page applications (SPA) and Web API Services on Microsoft .NET -
- Core with Docker and Linux.
- PowerShell Core automated builds.
- Markdown/PDF documentation.
- Documentation of public interfaces with Swagger/OAS/YAML.
- Automated testing of SPA on Protractor and testing the Web API on Postman/Newman.
This presentation by Sergii Fradkov (Consultant, Engineering), Andrii Zarharov (Lead Software Engineer, Consultant), Igor Magdich (Lead Test Engineer, Consultant) was delivered at GlobalLogic Kharkiv .NET TechTalk #1 on May 24, 2019.
Fixing security by fixing software developmentNick Galbreath
Fixing Security by Fixing Software Development Using Continuous Deployment
Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.
SQL-RISC: New Directions in SQLi Prevention - RSA USA 2013Nick Galbreath
What if we could reduce SQLi attacks in your application by 90%? WIth little to no changes in your application, with no new hardware or firewalls?
First presentated at RSA Conference USA, 2013-02-27
Rebooting Software Development - OWASP AppSecUSA Nick Galbreath
If we are ever going to get ahead of the whack-a-mole security vulnerability game, we, as security professionals need to start getting involved more in the development of software. Let's review the origins of the traditional software development, and what assumptions are made. Then we'll review if those assumptions still hold for modern web applications, and what problems they cause, especially for security. Continuous deployment helps address these problems and allows for faster, more secure development. It's more than just "pushing code a lot", when done correctly it can be transformative to the organization. We'll discuss what continuous deployment is, how to get started, and what components are needed to make it successful, and secure.
libinjection and sqli obfuscation, presented at OWASP NYCNick Galbreath
SQL that isn't caught by WAFs but also isn't used (yet) by attackers! Why detecting SQLi is good, and why doing it with regular expressions is hard. And re-introducing libinjections which is a new way of detecting SQLi attacks.
This is a mashup of my Black Hat USA 2012 and DEFCON 20 talks, refreshed and updated.
Continuous Deployment - The New #1 Security Feature, from BSildesLA 2012Nick Galbreath
First presented at Security BSidesLA, Hermosa Beach, California, August 16, 2012
Continuous deployment is characters by a small and frequent changes to production. Find out why it's my #1 security feature. It's not just about pushing fast!
How do fonts look when uploaded onto slideshare when the presentation is of various sides? How does it look on a washed-out projector? For plain text? For computer-code?
This presentation provides a number of sans-serif and monospace fonts to help answer these questions.
Rate Limiting at Scale, from SANS AppSec Las Vegas 2012Nick Galbreath
Rate Limits at Scale SANS AppSec Las Vegas.
Rate Limit Everything All the time using a quantized time system with Memcache or Redis. Use this protect resources or discover anomalies.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
"Impact of front-end architecture on development cost", Viktor Turskyi
Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013
1. Nick Galbreath http://client9.com/20130501 @ngalbreath
Care and Feeding of
Large Scale
Graphite
InstallationsNick Galbreath ★ IPONWEB
DevOpsDays ★ Austin Texas ★ 2013-04-30
7. Nick Galbreath http://client9.com/20130501 @ngalbreath
Who cares?
• Making it easy to create, analyze and
share data can change your
organization
• Making a data-driven culture
• Empowering developers, operations,
qa, security and business to be more
confident in the changes they make.
8. Nick Galbreath http://client9.com/20130501 @ngalbreath
What is it you say
you do here?
• Your job is likely invisible to rest of the
organization
• invisible things aren't valued
• so make what you do visible
9. Nick Galbreath http://client9.com/20130501 @ngalbreath
Why Graphite?
• Many innovations in each part of the
stack
• But, it's the Full Stack that really makes
it special.
• On-disk layout to UI to API to... the
community around it.
10. Nick Galbreath http://client9.com/20130501 @ngalbreath
Sharing in Caring
• Allows data to be easily accessed
• And easily shared. This makes it
different than many monitoring
solutions.
• It's your own in-house mashup
generator.
13. Nick Galbreath http://client9.com/20130501 @ngalbreath
Installation
• 3 python-twistd servers
• "carbon-cache"
• "cache-aggregator"
• "carbon-relay"
• Apache / Django Web UI and API
• Uses SQLite3/MySQL for
dashboards / events
14. Nick Galbreath http://client9.com/20130501 @ngalbreath
Be Current
• Don't use the OS default version.
• Newer point releases of graphite have
significant improvements in storage
engine and webui/api
• It's 100% Python so
"building it yourself" shouldn't to hard.
• pip install works and is current.
15. Nick Galbreath http://client9.com/20130501 @ngalbreath
The Documentation
• everyone complains about it
• historically bad, but getting a lot better
• Switched locations, but not all
searchengines are updated to use:
http://graphite.readthedocs.org/
• Source code is quite good, so RTFS
17. Nick Galbreath http://client9.com/20130501 @ngalbreath
What is it?
• Storage engine
• Handles reads, writes and creates of a
single metric to a fixed size file.
• One file, kinda dumb (good).
• Here's the API:
https://github.com/graphite-project/
whisper/blob/master/whisper.py
18. Nick Galbreath http://client9.com/20130501 @ngalbreath
Graphite Math
• About 12 bytes per point.
• Store 1 minute points for 1 month and
15 minutes for 11 months.
• (60×24×30 + 4×24×30×11) ×12 =
878kB
• If you can keep all your points in
memory, then magic!
19. Nick Galbreath http://client9.com/20130501 @ngalbreath
Disk Layout
• Each metric create a directory tree
server123.myapp.logins.failed
• Makes 3 directories
• This creates a very branchy directory
structure
• This has good and bad points.
22. Nick Galbreath http://client9.com/20130501 @ngalbreath
Write Buffer
• Most important feature is write
buffering to protect the disk.
• Data is buffered and written out once
per minute (or so).
• But
23. Nick Galbreath http://client9.com/20130501 @ngalbreath
The Cache
• It's a write cache.
• Once data is written,
it's out of the cache
• In other words, the cache is
metrics not on disk.
• If the cache dies, you lose metrics
• (btw: the read cache is the os disk
cache)
24. Nick Galbreath http://client9.com/20130501 @ngalbreath
New Metrics
• New metrics are created automatically
• But, it is very expensive.
• MAX_CREATES_PER_MINUTE.=.50
• Saves your disk, but new metrics will
"pile up" in cache.
• May take 10m+ for your metrics to start
flowing....
25. Nick Galbreath http://client9.com/20130501 @ngalbreath
FALLOCATE
• WHISPER_FALLOCATE_CREATE.=.True
• Linux&Kernel&>=&2.6.23
• fallocate is used to preallocate blocks
to a file. For filesystems which
support the fallocate system call, this
is done quickly by allocating blocks
and marking them as uninitialized,
requiring no IO to the data blocks.
This is much faster than creating a
file by filling it with zeros.
• https://bugs.launchpad.net/whisper/
+bug/957827
26. Nick Galbreath http://client9.com/20130501 @ngalbreath
Limit the Size
Limit the size of the cache to avoid
swapping or becoming CPU bound.Sorts
and serving cache queries gets more
expensive as the cache grows.
Use the value "inf" (infinity) for an
unlimited cache size.
MAX_CACHE_SIZE = inf
No!.Infinity.does.not.exist.on.your.system!
.&
27. Nick Galbreath http://client9.com/20130501 @ngalbreath
Graphite for Graphite
By&default,&carbon&itself&will&log&
statistics&(such&as&a&count,
metricsReceived)&with&the&top&level&prefix&
of&'carbon'&at&an&interval&of&60
seconds.&Set&CARBON_METRIC_INTERVAL&to&0&to&
disable&instrumentation
CARBON_METRIC_PREFIX&=&carbon
CARBON_METRIC_INTERVAL&=&60
30. Nick Galbreath http://client9.com/20130501 @ngalbreath
Pre-Aggregation
• Sum or Average metrics based on
wildcards and regexps
• Helps eliminate very slow queries on
webui
• You can emit the final sum & all the
individual components or just the final
sum (via blacklists)
33. Nick Galbreath http://client9.com/20130501 @ngalbreath
Also...
• has support for broadcasting data to
multiple downstream caches
• but.. never used it.. and seems at odds
with the next middleware
35. Nick Galbreath http://client9.com/20130501 @ngalbreath
It's a Router!
• Consistent Hashing (Sharding)
• Or more rule-based routing
• Output to multiple carbon servers
have not really used it much, but should
work similarly to scale outs of
memcache, redis
37. Nick Galbreath http://client9.com/20130501 @ngalbreath
StatsD
• https://github.com/etsy/statsd/
• nodejs based but lots of other
implementations
• Receives UDP, send graphite-
compatible output, flushed periodically.
• Aggregation for all by default
• Beside sum, also can compute other
basic statistics (mean, 90% percentile),
do sampling, have counters, etc.
38. Nick Galbreath http://client9.com/20130501 @ngalbreath
StatsD use case
• It's UDP based, so it excels at
embedding a client inside the
application
• UDP can't block or break the sending
application
• Not so good for bulk metrics
• Use both! Can work together with
aggregator.
39. Nick Galbreath http://client9.com/20130501 @ngalbreath
Of Note
• https://github.com/armon/statsite
• Need to look at this more
• c + libev based
• modern time series algorithms
• very flexible output
41. Nick Galbreath http://client9.com/20130501 @ngalbreath
Backup
• Doing naive backup causes graphite
performance goes to crap.
• File system cache is trashed
• Metrics are not written to disk (lag)
• If OOM occurs then you lose metrics.
44. Nick Galbreath http://client9.com/20130501 @ngalbreath
ionice is better
IONICE(1) User Commands IONICE(1)
NAME
ionice - set or get process I/O scheduling class and priority
SYNOPSIS
ionice [-c class] [-n level] [-t] -p PID...
ionice [-c class] [-n level] [-t] command [argument...]
DESCRIPTION
This program sets or gets the I/O scheduling class and priority for a
program. If no arguments or just -p is given, ionice will query the current I/O
scheduling class and priority for that process.
When command is given, ionice will run this command with the given
arguments. If no class is specified, then command will be executed with the
"best-effort" scheduling class. The default priority level is 4.
NOTES
Linux supports I/O scheduling priorities and classes since 2.6.13 with the
CFQ I/O scheduler.
util-linux July 2011 IONICE(1)
45. Nick Galbreath http://client9.com/20130501 @ngalbreath
Even Better
• Just write the metrics to two graphite
servers in your client
• Script to copy / resync "holes" when
restoring.
47. Nick Galbreath http://client9.com/20130501 @ngalbreath
WebUI
• Hey, it's a web server
• do all the usual stuff
• Ask for known stats,
• check for 200
• check for valid json output
50. Nick Galbreath http://client9.com/20130501 @ngalbreath
MySQL
• If you use SQLite3 -- uhh nothing to
monitor
• If you use MySQL -- use the regular
suspects
• And don't forget to backup!!
56. Nick Galbreath http://client9.com/20130501 @ngalbreath
Tune Apache
• By default, your Apache install is likely
to be "unlimited" in CPU and Memory
usage.
• Select a wildcard metric for a long time
period can easily turn a httpd process
in 1GB. (this seems like a bug actually)
• OOM death.
59. Nick Galbreath http://client9.com/20130501 @ngalbreath
/events/
• Ad-Hoc Events that don't deserve their
own metric type.
• has tags, time, and text
• Stored in SQLite3 by default by the
webapp.
• Rest UI is primitive
60. Nick Galbreath http://client9.com/20130501 @ngalbreath
The WebUI
• it's "ok".. good for experiments
• You will want to make your own
dashboard.
• Good news! The API is a URL, so it's
very easy.
61. Nick Galbreath http://client9.com/20130501 @ngalbreath
WebUI Dashboards
• The WebUI has a dashboard feature for
loading and saving graphs
• It saves data in SQLite3 by default
• Since it's there people will use it
• So hack to remove it or, switch to
MySQL.
62. Nick Galbreath http://client9.com/20130501 @ngalbreath
Granularity
• Like RRDTool, the resolution of the
graph depends on number of pixels
used. No sub-pixel rendering!
• Rapid spikes can be "averaged away"
in week-long views in small graphs.
69. Nick Galbreath http://client9.com/20130501 @ngalbreath
Really Long URLs
• Making graph but the URL is so long
browsers are clipping them?
• Send query string data as a POST
70. Nick Galbreath http://client9.com/20130501 @ngalbreath
Client Side
Rendering
• yeah...
• works ok with a small number of points
• crashes existing browsers with large
number of points
• Server side faster in many cases!
• We'll try again in 2014
71. Nick Galbreath http://client9.com/20130501 @ngalbreath
Colors and
ChartJunk
• Default color scheme is gross
• Be kind to the handicapped (uhh, me)
http://colorbrewer2.org/
• Good overview here:
http://bit.ly/10Hu7zU
73. Nick Galbreath http://client9.com/20130501 @ngalbreath
Accelerate with PyPy
• JIT for Python
• ~ 5.9x performance improvement
• Actually works and is stable
• Compatible with twisted and Django
74. Nick Galbreath http://client9.com/20130501 @ngalbreath
Accelerate with numpy
• numpy provides fast vector
manipulation (C code)
• graphite web gui does a lot of vector
manipulation
• hmmmm.....
75. Nick Galbreath http://client9.com/20130501 @ngalbreath
Ceres Storage Engine
• "Eventually Fixed Size" storage
• More space efficient == more
performance
• see
http://blog.sweetiq.com/2013/01/
using-ceres-as-the-back-end-
database-to-graphite/
76. Nick Galbreath http://client9.com/20130501 @ngalbreath
OpenTSB
• Not Graphite, but similar in spirit
• Has "collectors" for basic ops stats
• Used by StumbleUpon, Box.net,
pintrest
• Good: Stores data in HBASE/Hadoop
• Bad: Stores data in HBASE/Hadoop
77. Nick Galbreath http://client9.com/20130501 @ngalbreath
Add More Functions
• coursen (I'm looking at you Ian Malpass,
that's useful for client-side rendering)
• Real vertical lines (our hacks are stupid)
• Better operators (would nice to know
easily how many metrics you have, e.g.
select count(*))
78. Nick Galbreath http://client9.com/20130501 @ngalbreath
Mine the Apache Log
• Which stats are used the most?
• What are really slow queries?
• Can you optimize them?
• What time frames are used?
• How much old data do you really need
to store?
it's in the
query string
79. Nick Galbreath http://client9.com/20130501 @ngalbreath
Add a TinyURL
Feature
• The URLs get really long and are hard
to put into email, etc.
• Make a tinyurl feature into the django
app and integrate into dashboard.
81. Nick Galbreath http://client9.com/20130501 @ngalbreath
Nick Galbreath
http://www.client9.com/
nickg@client9.com
http://www.iponweb.com/
ngalbreath@iponweb.net
Lets Make
Some Graphs!