The document discusses building real-time web applications. It addresses three hard problems: talking to the browser in real-time using techniques like polling, websockets, and flash sockets; handling high concurrency through non-blocking I/O; and scaling to support many users. It reviews common real-time frameworks like Twisted, Tornado, gevent, Node.js, and Erlang and recommends Twisted for Python and Node.js with Socket.IO for non-Python. It also discusses using Redis for a real-time backend pub/sub system and scaling strategies like adding frontend and backend nodes.
Resumable File Upload API using GridFS and TUSkhangtoh
TUS is a resumable file upload protocol and with MongoDB GridFS, we build an API for uploading files through a REST API and show how to scale this API horizontally using MongoDB as the storage for these files.
Singapore MongoDB User Group March Meetup
tus.io – Resumable file uploads for web and mobile apps by Felix GeisendörferCodemotion
File uploading is an incredibly annoying, yet very important feature to get right for your applications. Unfortunately time is usually short, so most applications handle network errors during file uploads very poorly. For this reason we have started tus.io, which aims to define an open protocol for resumable file uploads along with providing implementations for all plattforms and languages. This session will introduce you to the project, and show you how to add better file uploading to your web and mobile apps.
Resumable File Upload API using GridFS and TUSkhangtoh
TUS is a resumable file upload protocol and with MongoDB GridFS, we build an API for uploading files through a REST API and show how to scale this API horizontally using MongoDB as the storage for these files.
Singapore MongoDB User Group March Meetup
tus.io – Resumable file uploads for web and mobile apps by Felix GeisendörferCodemotion
File uploading is an incredibly annoying, yet very important feature to get right for your applications. Unfortunately time is usually short, so most applications handle network errors during file uploads very poorly. For this reason we have started tus.io, which aims to define an open protocol for resumable file uploads along with providing implementations for all plattforms and languages. This session will introduce you to the project, and show you how to add better file uploading to your web and mobile apps.
Cluster Fudge: Recipes for WordPress in the Cloud (WordCamp Austin 2014 Speaker)Grant Norwood
Your self-hosted WordPress site is quickly growing in popularity and page views. Or maybe you want to get away from that costly enterprise CMS currently on your plate and adopt a delectable, open-source platform. There are many reasons you might need the performance and redundancy of a clustered server solution, and I’ll show you how to mix up the ingredients needed to throw together a successful cloud-hosted WordPress environment that’s right for you.
We’ll talk about common multi-server configurations, from cheap and quick for the cost-conscious business, to robust and complex for the high level of control an enterprise demands.
Ruby developers need to stop using EventMachine. It's the wrong direction.
Lost in the "Threads vs Event Driven vs Process Spawning" debate is that you can combine them! Learn how Celluloid is improving thread programming by abstracting them using a higher level framework called Celluloid, how you can use Celluloid::IO to throw a reactor pattern into a thread. Using this approach, you can take advantage of threading and use all CPU power on a machine with JRuby or Rubinius. I also discuss the future of distributed objects and computing, and where I think things are going.
This deck represents our current thinking about the best way to build enterprise SaaS software in 2015 - using a variety of techniques from several disciplines.
Since I wrote this I have also become very interested in resilience engineering and the notion that web developers are primarily engaged in the construction of socio-technical systems. When I rewrite this I plan to talk about how we should try to minimize mean-time-to-recover (MTTR) instead of mean-time-between-failures (MTBF), and how continuous deployment grows a safety culture around your operations.
I redacted most of the examples that illustrate these points because they use sensitive code examples or URLs. If you want to see the rest of slides, join us!
Get Off My Thread! - keep your UI super-responsiveDroidConTLV
The importance of keeping the main thread free and ways to do it. We'll cover classic main thread busy pitfalls and classic workaround (IntentService, AsyncTask). We'll also cover useful techniques for getting back to the main thread and Simple synchronization strategies.
As presented in DroidCon Tel Aviv 2014 by:
Ori Peleg, Facebook
http://il.droidcon.com
Big Data! Great! Now What? #SymfonyCon 2014Ricard Clau
Big Data is one of the new buzzwords in the industry. Everyone is using NoSQL databases. MySQL is not cool anymore. But... do we really have big data? Where should we store it? Are the traditional RDBMS databases dead? Is NoSQL the solution to our problems? And most importantly, how can PHP and Symfony2 help with it?
My Stackato presentation given to the CopenhagenJS user group. Basic examples were implemented in Node.
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Posterous recently deployed Riak to serve as their content cache. In this talk, Julio Capote will cover why the engineering team chose Riak for the use case. He'll also share some details on the old post cache and its problems, what solutions they evaluated, and how they settled on Riak.
Phishing and client-side exploitation DevOps for all your needs. Combine BeEF, PhishingFrenzy and your fishy business to automate most of the usual phishing workflow while minimising human interaction.
A presentation made to the Bangalore Linux User Group in January 2002. Most people don't realize just how far ahead of its time Zope was as an application server and NoSQL database.
Html5 web sockets - Brad Drysdale - London Web 2011-10-20Nathan O'Hanlon
This month we focus in on WebSockets in HTML5 with Brad Drysdale.
You're probably aware of the various chat systems around the web (facebook chat for example) but recently there has been more of a desire for other types of interaction using these technologies. Many have requested talks on Node.js, Comet long-polling, or similar technologies. WebSockets are the evolution of these technologies, and a developing standard being progressed by the IETF.
The HTML 5 standard specifies new APIs for storage, drawing, drag-and-drop, and other areas that have made web programming painful. Browsers have already begun incorporating parts of HTML 5 (canvas, for example) even though the specification is far from complete. The HTML 5 Communication section includes two additional connectivity features: Server-Sent Events, a standardization of HTTP push, and Web Sockets, a cross-domain safe, full-duplex connection. Server-Sent Events will make real-time updates and notifications easy, and Web Sockets provide the functionality necessary to build chat for the web without the previously required hackery.
Come and hear the latest on what this great technology allows!
Brad Drysdale is the Technical Director in EMEA for Kaazing. Brad has also worked at Openwave and Netscape Communications in various pre-sales and technical evangelist roles.
Messaging, interoperability and log aggregation - a new frameworkTomas Doran
In this talk, I will talk about why log files are horrible, logging log lines, and more structured performance metrics from large scale production applications as well as building reliable, scaleable and flexible large scale software systems in multiple languages.
Why (almost) all log formats are horrible will be explained, and why JSON is a good solution for logging will be discussed, along with a number of message queuing, middleware and network transport technologies, including STOMP, AMQP and ZeroMQ.
The Message::Passing framework will be introduced, along with the logstash.net project which the perl code is interoperable with. These are pluggable frameworks in ruby/java/jruby and perl with pre-written sets of inputs, filters and outputs for many many different systems, message formats and transports.
They were initially designed to be aggregators and filters of data for logging. However they are flexible enough to be used as part of your messaging middleware, or even as a replacement for centralised message queuing systems.
You can have your cake and eat it too - an architecture which is flexible, extensible, scaleable and distributed. Build discrete, loosely coupled components which just pass messages to each other easily.
Integrate and interoperate with your existing code and code bases easily, consume from or publish to any existing message queue, logging or performance metrics system you have installed.
Simple examples using common input and output classes will be demonstrated using the framework, as will easily adding your own custom filters. A number of common messaging middleware patterns will be shown to be trivial to implement.
Some higher level use-cases will also be explored, demonstrating log indexing in ElasticSearch and how to build a responsive platform API using webhooks.
Interoperability is also an important goal for messaging middleware. The logstash.net project will be highlighted and we'll discuss crossing the single language barrier, allowing us to have full integration between java, ruby and perl components, and to easily write bindings into libraries we want to reuse in any of those languages.
Cluster Fudge: Recipes for WordPress in the Cloud (WordCamp Austin 2014 Speaker)Grant Norwood
Your self-hosted WordPress site is quickly growing in popularity and page views. Or maybe you want to get away from that costly enterprise CMS currently on your plate and adopt a delectable, open-source platform. There are many reasons you might need the performance and redundancy of a clustered server solution, and I’ll show you how to mix up the ingredients needed to throw together a successful cloud-hosted WordPress environment that’s right for you.
We’ll talk about common multi-server configurations, from cheap and quick for the cost-conscious business, to robust and complex for the high level of control an enterprise demands.
Ruby developers need to stop using EventMachine. It's the wrong direction.
Lost in the "Threads vs Event Driven vs Process Spawning" debate is that you can combine them! Learn how Celluloid is improving thread programming by abstracting them using a higher level framework called Celluloid, how you can use Celluloid::IO to throw a reactor pattern into a thread. Using this approach, you can take advantage of threading and use all CPU power on a machine with JRuby or Rubinius. I also discuss the future of distributed objects and computing, and where I think things are going.
This deck represents our current thinking about the best way to build enterprise SaaS software in 2015 - using a variety of techniques from several disciplines.
Since I wrote this I have also become very interested in resilience engineering and the notion that web developers are primarily engaged in the construction of socio-technical systems. When I rewrite this I plan to talk about how we should try to minimize mean-time-to-recover (MTTR) instead of mean-time-between-failures (MTBF), and how continuous deployment grows a safety culture around your operations.
I redacted most of the examples that illustrate these points because they use sensitive code examples or URLs. If you want to see the rest of slides, join us!
Get Off My Thread! - keep your UI super-responsiveDroidConTLV
The importance of keeping the main thread free and ways to do it. We'll cover classic main thread busy pitfalls and classic workaround (IntentService, AsyncTask). We'll also cover useful techniques for getting back to the main thread and Simple synchronization strategies.
As presented in DroidCon Tel Aviv 2014 by:
Ori Peleg, Facebook
http://il.droidcon.com
Big Data! Great! Now What? #SymfonyCon 2014Ricard Clau
Big Data is one of the new buzzwords in the industry. Everyone is using NoSQL databases. MySQL is not cool anymore. But... do we really have big data? Where should we store it? Are the traditional RDBMS databases dead? Is NoSQL the solution to our problems? And most importantly, how can PHP and Symfony2 help with it?
My Stackato presentation given to the CopenhagenJS user group. Basic examples were implemented in Node.
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Posterous recently deployed Riak to serve as their content cache. In this talk, Julio Capote will cover why the engineering team chose Riak for the use case. He'll also share some details on the old post cache and its problems, what solutions they evaluated, and how they settled on Riak.
Phishing and client-side exploitation DevOps for all your needs. Combine BeEF, PhishingFrenzy and your fishy business to automate most of the usual phishing workflow while minimising human interaction.
A presentation made to the Bangalore Linux User Group in January 2002. Most people don't realize just how far ahead of its time Zope was as an application server and NoSQL database.
Html5 web sockets - Brad Drysdale - London Web 2011-10-20Nathan O'Hanlon
This month we focus in on WebSockets in HTML5 with Brad Drysdale.
You're probably aware of the various chat systems around the web (facebook chat for example) but recently there has been more of a desire for other types of interaction using these technologies. Many have requested talks on Node.js, Comet long-polling, or similar technologies. WebSockets are the evolution of these technologies, and a developing standard being progressed by the IETF.
The HTML 5 standard specifies new APIs for storage, drawing, drag-and-drop, and other areas that have made web programming painful. Browsers have already begun incorporating parts of HTML 5 (canvas, for example) even though the specification is far from complete. The HTML 5 Communication section includes two additional connectivity features: Server-Sent Events, a standardization of HTTP push, and Web Sockets, a cross-domain safe, full-duplex connection. Server-Sent Events will make real-time updates and notifications easy, and Web Sockets provide the functionality necessary to build chat for the web without the previously required hackery.
Come and hear the latest on what this great technology allows!
Brad Drysdale is the Technical Director in EMEA for Kaazing. Brad has also worked at Openwave and Netscape Communications in various pre-sales and technical evangelist roles.
Messaging, interoperability and log aggregation - a new frameworkTomas Doran
In this talk, I will talk about why log files are horrible, logging log lines, and more structured performance metrics from large scale production applications as well as building reliable, scaleable and flexible large scale software systems in multiple languages.
Why (almost) all log formats are horrible will be explained, and why JSON is a good solution for logging will be discussed, along with a number of message queuing, middleware and network transport technologies, including STOMP, AMQP and ZeroMQ.
The Message::Passing framework will be introduced, along with the logstash.net project which the perl code is interoperable with. These are pluggable frameworks in ruby/java/jruby and perl with pre-written sets of inputs, filters and outputs for many many different systems, message formats and transports.
They were initially designed to be aggregators and filters of data for logging. However they are flexible enough to be used as part of your messaging middleware, or even as a replacement for centralised message queuing systems.
You can have your cake and eat it too - an architecture which is flexible, extensible, scaleable and distributed. Build discrete, loosely coupled components which just pass messages to each other easily.
Integrate and interoperate with your existing code and code bases easily, consume from or publish to any existing message queue, logging or performance metrics system you have installed.
Simple examples using common input and output classes will be demonstrated using the framework, as will easily adding your own custom filters. A number of common messaging middleware patterns will be shown to be trivial to implement.
Some higher level use-cases will also be explored, demonstrating log indexing in ElasticSearch and how to build a responsive platform API using webhooks.
Interoperability is also an important goal for messaging middleware. The logstash.net project will be highlighted and we'll discuss crossing the single language barrier, allowing us to have full integration between java, ruby and perl components, and to easily write bindings into libraries we want to reuse in any of those languages.
From a student to an apache committer practice of apache io tdbjixuan1989
This talk is introduce by Xiangdong Huang, who is a PPMC of Apache IoTDB (incubating) project, at Apache Event at Tsinghua University in China.
About the Event:
The open source ecosystem plays more and more important role in the world. Open source software is widely used in operating systems, cloud computing, big data, artificial intelligence, and industrial Internet. Many companies have gradually increased their participation in the open source community. Developers with open source experience are increasingly valued and favored by large enterprises. The Apache Software Foundation is one of the most important open source communities, contributing a large number of valuable open source software and communities to the world.
The invited guests of this lecture are all from ASF community, including the chairman of the Apache Software Foundation, three Apache members, Top 5 Apache code committers (according to Apache annual report), the first Committer in the Hadoop project in China, several Apache project mentors or VPs, and many Apache Committers. They will tell you what the open source culture is, how to join the Apache open source community, and the Apache Way.
Modern software architectures - PHP UK Conference 2015Ricard Clau
The web has changed. Users demand responsive, real-time interactive applications and companies need to store and analyze tons of data. Some years ago, monolithic code bases with a basic LAMP stack, some caching and perhaps a search engine were enough. These days everybody is talking about micro-services architectures, SOA, Erlang, Golang, message passing, queue systems and many more. PHP seems to not be cool anymore but... is this true? Should we all forget everything we know and just learn these new technologies? Do we really need all these things?
As a follow-up of the previous session about TFB, we will discuss what kind of tuning was made to the mORMot library, and its associated TFB sample implementation, to reach the top scores in charts. How can a pure Pascal project reach 7 millions of HTTP requests per seconds? How to scale and measure on high-end hardware? Are ORM frameworks damned to slow down everything? How to circumvent the lack of “async” programming at language level? How realistic is such a benchmark?
Cloud Connected Devices on a Global Scale (CPN303) | AWS re:Invent 2013Amazon Web Services
Increasingly, mobile and other connected devices are leveraging the scalability and capabilities of the cloud to deliver services to end users. However, connecting these devices to the cloud presents unique challenges. Resource constraints make it impossible to use many common frameworks and transport restrictions make it difficult to use dynamic cloud resources. In this session, learn how you can develop and deploy highly-scalable global solutions using Amazon Web Services (Amazon Virtual Private Cloud, Elastic IP addresses, Amazon Route 53, Auto Scaling) and tools like Puppet. Hear how Panasonic and Banjo architect their cloud infrastructure from both a start-up and enterprise perspective.
How do you continue to ship 50 times a day, when you're constantly hiring more engineers? How can you continue, when every day you write more tests that need to be run on every commit? This talk will cover how to scale up Continuous Integration and Continuous Deployment infrastracture, for teams as small as a handful of engineers and as large as hundreds of engineers.
Just In Time Scalability Agile Methods To Support Massive Growth PresentationTimothy Fitz
Eric Reis and Chris Hondl's MySQL conference presentation on Just In Time Scalability. http://startuplessonslearned.blogspot.com/2008/09/just-in-time-scalability.html
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
13. High Concurrency
• Blocking I/O
– Thread per process
– Tops out at 200 to 1k connections
• Non-blocking I/O
– One process, one thread
– 10k to 100k connections
18. Twisted
• Pro
– Can talk every protocol ever
– Oldest and most widely used in production
• Con
– Overkill for web-only tasks
– Not simple
19. Tornado
• Pro
– Simple
– Does HTTP stuff simply
• Con
– Might not interface with what you need
• Confusing
– You can run Tornado (HTTP layer) on top of
Twisted (networking layer)
20. gevent
• Pro
– Coroutines are a better model than callbacks
– As such, very easy to write complicated logic
• Con
– Least well documented
– Least consensus on best practices
– New, uncertain about production readiness
21. Node.js
• Pro
– Best documentation by far
– Socket.IO abstracts away browser communication
• Con
– Can’t share logic between Django app
– New, but has fairly large install base
22. Erlang
• Pro
– Hands down best for complex realtime tasks
– Forces you to think about concurrency/scale
– Abstracts away the network
– Old and reliable
• Con
– Forces you to think about concurrency/scale
– Can’t share logic between Django app
– High spin-up cost (functional, concurrency driven)
24. Just one
• Everything in memory
• Django nodes talk directly to box
• Spare for availability
• Failover = realtime data loss
– Make realtime 100% redundant
25. Probably good enough!
– WARNING: NAPKIN MATH
– 10k daily visits * 10.0min avg visit
= 70 average concurrent users
– One box can easily be built out to handle 3-5k
= Roughly 450k-700k daily visits
30. Roll your own
• Same language as your frontend
– (Twisted/Node/Whatever)
• Only do this if you have per-channel business
logic
– You probably don’t.
• Erlang maps really really well to this domain.
31. Full Stack Services
• REST APIs to push to the browser
• http://pusher.com
• http://beaconpush.com
33. Final Recommendations
• Need python? Twisted
• Don’t? Node.js/SocketIO
• Need scale/reliability? Redis backend.
• Complex? Going big? Erlang all the way.
35. Further Reading
• IMVU IMQ talk http://www.slideshare.net/JonWatte/message-queuing-
on-a-large-scale-imvus-stateful-realtime-message-queue
• Twilio talk on gevent + zeromq (given by Jeff Lindsay, highly recomended):
http://www.twilio.com/conference/video/distributed-systems-with-
gevent-and-zeromq
• Last.fm scaling Eralng/Mochiweb to 1 million concurrent connections on
one machine: http://www.metabrew.com/article/a-million-user-comet-
application-with-mochiweb-part-1
• The original Comet blog post: http://infrequently.org/2006/03/comet-low-
latency-data-for-the-browser/
• Django + Socket.IO + gevent:
http://codysoyland.com/2011/feb/6/evented-django-part-one-socketio-
and-gevent/
Editor's Notes
Also known as Comet (in response to AJAX)And before that, under the umbrella of “DHTML” (throwback to the late 90s!)
Latency often doesn’t matter at all (3-5s wouldn’t be noticed, for popular hashtags 1 minute wouldn’t make a difference)
Chat (which is pubsub on steroids)Presence (the fact that you’re connected is important)Latency matters some, but you wouldn’t notice 1s of lag.
Gaming, networked simulated physics / simulated spaces. Latency is critical in both directions (~200ms matters)
Also a dozen other methods, and aggregate methods that have built-in fall back semantics.
Supported absolutely everywhereIncredibly efficientIncredibly easy to implement, hard to get wrongRight for infrequent realtime, or tied to existing expensive operation (most common example: short poll Paypal/payment gateway for success confirmation)
Works everywhere (desktop and mobile)Supports most use cases (twitter, etc)
Requires flash support (user has it, no flashblock, desktop only for the most part)Bidirectional and binary.Bidirectional really only matters for realtime interactive apps (games, virtual spaces, motion is one of the few places where 200ms latency matters)Flash is dying, but if your app already requires (or if your UI is already in flash, hello vidya game) then this might be the best solution.
Works on Chrome, FF, Safari, iOS mobile, IE10 previews. Coming to Android Mobile soon.Bidirectional, but UTF-8 (probably doesn’t matter)Very new (RFC hit “Proposed Standard” in Dec 2011, which means the spec is solidified. “Internet Standard” is then next step, and reserved for two independent interoperable implementations, very close)Great but you’ll probably have to support fallback for a while
Super simplifying, lots of options exist including hybrids.Often run one non-blocking process per core (if you have to scale to multiple machines, using the same strategy for multiple processes is trivial)
Okay this is kind of a lie, there are hacky ways but you lose most of what makes Django, Django: sessions, users, auth, ORM, and most 3rd party libraries
There is no consensus. There are some good python options. There are a LOT of options I’m not even mentioning, almost every language has two or three non-blocking I/O webservers. Python might be important, especially if you have logic you want to reuse between your Django application and your non-blocking I/O app