Messaging becomes Data Distributions gets embedded event processing (not complex, made simple) - bending all the rules one benchmark at a time - Push Technology, Waratek and other things
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...JAX London
John Matthew Holt, Waratek CTO, explains how Waratek Cloud VM for Java transforms the JVM with key Cloud characteristics including genuine multitenancy, granular elasticity, instant scalability, realtime metering and prioritization of resources by application, to futureproof your Java investment in the age of Cloud computing.
Communication Patterns Using Data-Centric Publish/SubscribeSumant Tambe
Fundamental to any distributed system are communication patterns: point-to-point, request-reply, transactional queues, and publish-subscribe. Large distributed systems often employ two or more communication patterns. Using a single middleware that supports multiple communication patterns is a very cost-effective way of developing and maintaining large distributed systems. This talk will begin with an introduction of Data Distribution Service (DDS) – an OMG standard – that supports data-centric publish-subscribe communication for real-time distributed systems. DDS separates state management and distribution from application logic and supports discoverable data models. The talk will then describe how RTI Connext Messaging goes beyond vanilla DDS and implements various communication patterns including request-reply, command-response, and guaranteed delivery. You will also learn how these patterns can be combined to create interesting variations when the underlying substrate is as powerful as DDS. We’ll also discuss APIs for creating high-performance applications using the request-reply communication pattern.
New applications are increasingly built on distributed service architectures, including mobile and cloud-based services which increase the complexity and interdependency of the systems to be tested.
Teams are forced to do performance test earlier in the application lifecycle, before the application is complete or stable with the increased pressure to operate more efficiently, produce valuable results more quickly, and operate with fewer resources.
Learn how Service Virtualization is a critical enabler to allow testers to get started with performance testing from the first sprint.
We will demonstrate how Service Virtualization is an easy-to-use solution that integrated with multiple performance testing tools, resulting in an almost seamless solution for performance engineering and validation.
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...JAX London
John Matthew Holt, Waratek CTO, explains how Waratek Cloud VM for Java transforms the JVM with key Cloud characteristics including genuine multitenancy, granular elasticity, instant scalability, realtime metering and prioritization of resources by application, to futureproof your Java investment in the age of Cloud computing.
Communication Patterns Using Data-Centric Publish/SubscribeSumant Tambe
Fundamental to any distributed system are communication patterns: point-to-point, request-reply, transactional queues, and publish-subscribe. Large distributed systems often employ two or more communication patterns. Using a single middleware that supports multiple communication patterns is a very cost-effective way of developing and maintaining large distributed systems. This talk will begin with an introduction of Data Distribution Service (DDS) – an OMG standard – that supports data-centric publish-subscribe communication for real-time distributed systems. DDS separates state management and distribution from application logic and supports discoverable data models. The talk will then describe how RTI Connext Messaging goes beyond vanilla DDS and implements various communication patterns including request-reply, command-response, and guaranteed delivery. You will also learn how these patterns can be combined to create interesting variations when the underlying substrate is as powerful as DDS. We’ll also discuss APIs for creating high-performance applications using the request-reply communication pattern.
New applications are increasingly built on distributed service architectures, including mobile and cloud-based services which increase the complexity and interdependency of the systems to be tested.
Teams are forced to do performance test earlier in the application lifecycle, before the application is complete or stable with the increased pressure to operate more efficiently, produce valuable results more quickly, and operate with fewer resources.
Learn how Service Virtualization is a critical enabler to allow testers to get started with performance testing from the first sprint.
We will demonstrate how Service Virtualization is an easy-to-use solution that integrated with multiple performance testing tools, resulting in an almost seamless solution for performance engineering and validation.
Presentation of the DDS Interoperability demo performed in Washington DC between RTI, TwinOaks and PrismTech.
This demonstration shows the use of the DDS-RTPS interoperability protocol in 9 different scenarios.
WebSocket is a standardized interface for continuous, bi-directional and low-overhead communication between Web browser clients and back-end servers.
This presentation provides an overview of the WebSocket interface and descriptions of four WebSocket demonstration applications: i) a multi-user Facebook game; ii) a cloud server monitor; iii) a browser-based interface to LabVIEW; and iv) an interface to a MySQL database.
Presented Jan 26, 2011 at the HTML5 & CSS3 LA User Group Meetup, Los Angeles, CA
A Foundation for Success in the Information EconomyInside Analysis
The Briefing Room with Dr. Robin Bloor and Hewlett-Packard
Live Webcast on Oct. 30, 2012
Success in today's information economy rises and falls on the efficiency of data management. Companies that treat their information assets as mission-critical components of the business will find ways to better their competitors. The key is to ensure that the foundation of your information architecture can satisfy the wide range of user demands. Moreover, the ability to scale quickly and efficiently has become paramount.
Check out this episode of The Briefing Room to hear veteran Analyst Robin Bloor who will explain the benefits of embracing a modern Information Oriented Architecture (IOA). He'll also tout the purpose of using a flexible SQL engine in this era of NoSQL technologies. He will be briefed by Ajaya Gummadi of Hewlett-Packard, who will show how her company’s NonStop SQL has evolved to become a valuable solution for mission-critical data, mixed workloads and high volume databases. She will also explain how their integrated hardware and software stack can help reduce the cost of operations and management in a large-scale database environment.
Presentation of the DDS Interoperability demo performed in Washington DC between RTI, TwinOaks and PrismTech.
This demonstration shows the use of the DDS-RTPS interoperability protocol in 9 different scenarios.
WebSocket is a standardized interface for continuous, bi-directional and low-overhead communication between Web browser clients and back-end servers.
This presentation provides an overview of the WebSocket interface and descriptions of four WebSocket demonstration applications: i) a multi-user Facebook game; ii) a cloud server monitor; iii) a browser-based interface to LabVIEW; and iv) an interface to a MySQL database.
Presented Jan 26, 2011 at the HTML5 & CSS3 LA User Group Meetup, Los Angeles, CA
A Foundation for Success in the Information EconomyInside Analysis
The Briefing Room with Dr. Robin Bloor and Hewlett-Packard
Live Webcast on Oct. 30, 2012
Success in today's information economy rises and falls on the efficiency of data management. Companies that treat their information assets as mission-critical components of the business will find ways to better their competitors. The key is to ensure that the foundation of your information architecture can satisfy the wide range of user demands. Moreover, the ability to scale quickly and efficiently has become paramount.
Check out this episode of The Briefing Room to hear veteran Analyst Robin Bloor who will explain the benefits of embracing a modern Information Oriented Architecture (IOA). He'll also tout the purpose of using a flexible SQL engine in this era of NoSQL technologies. He will be briefed by Ajaya Gummadi of Hewlett-Packard, who will show how her company’s NonStop SQL has evolved to become a valuable solution for mission-critical data, mixed workloads and high volume databases. She will also explain how their integrated hardware and software stack can help reduce the cost of operations and management in a large-scale database environment.
Content Strategy: The Keys Unlocking Scalable Marketing ContentBuddy Scalera
CONTENT STRATEGY: Unlock Scalable Content to Ensure a Resonate Online Consumer Message - Presented by Buddy Scalera at Content Marketing World | Health Summit 2012. Explore the user journey and how content messaging can be provided at every step of the consumer continuum. Includes a case study with everybody's favorite comic book cave man Grok.
This is part of our presentation for the paper 'Understanding Consumers in India' taken at Amity Business School, Noida. Our Presentation was based on the study done by Deloitte India for the same.
The DMI Project Logical Data Model describes the sought after DMI Asset Metadata Structure represented
using UML Class Modelling diagrams. The model is describing the structure on a class level, but where
possible examples have been added to explain how the structure works.
The current version has not been fully populated with attributes but the required data structures should be in
place.
The model covers all types of media assets, not just linear television programmes. The structure of the
model is supporting Linear Television Programmes, Radio Programmes, Interactive Services, Promotions,
Web Pages, Interstitials and Scripts etc. Several areas will require further analysis; this will be highlighted in
the detail of the document. It is expected that this will be looked at in the detailed design phase.
One of the main focuses of the model is that the asset information should be held separately from
information about the usage of that asset; this will simplify the reuse of the assets, from an information point
of view. The concepts Publishable Asset and Media Asset have been introduced to the model to support this
separation of the data.
Databases create a real challenge for automation and dealing with database deployments is a complex process. Databases contain our most valuable information, business data, which must be preserved and protected at all costs and yet the automation processes for database deployment are not widely adopted.
Low Power Wireless Technologies and Standards for the Internet of ThingsDuncan Purves
Presentation on low power wireless technologies and standards for the Internet of Things (iOT), given at the BCS Communications Management Association AGM on 5th April 2016
Continuous Deployment Practices, with Production, Test and Development Enviro...Amazon Web Services
With AWS companies now have the ability to develop and run their applications with speed and flexibility like never before. Working with an infrastructure that can be 100% API driven enables businesses to use lean methodologies and realize these benefits. This in turn leads to greater success for those who make use of these practices. In this session we'll talk about some key concepts and design patterns for Continuous Deployment and Continuous Integration, two elements of lean development of applications and infrastructures.
UI for Big Data Visualization | Jonathan Stray | UX Week 2012Adaptive Path
Visualization is great way to understand data, but it breaks down when the data gets big. Simply plotting everything to the screen won’t work, because there isn’t enough screen real estate, interactions slow to a crawl, and human working memory isn’t up to the task anyway. Big data requires specific interaction techniques for visual exploration, such as filtering, summarization, and context. We’ll go over some basic principles, and I’ll show examples of recent systems, including our work on the Overview Project, a system for visual exploration of huge unstructured document sets.
Low Power Wireless Sensor Network Technologies and Standards for the Internet...Duncan Purves
Presentation on Low Power Wireless Sensor Network Technologies and Standards for the Internet of Things given at Institute of Physics, Sensors & their Applications XVIII Conference, 12 September 2016
This work focus on V2G technology as stated today and potential upgrades over the system in the near future. Particulalry investigating RES-E volatile production exploitation over congestion period and new frontieer of ancillary service market architecture
This talk will provide an in-depth treatment of satellite telephony networks from a security perspective. The overall system seems secure, but in reality, it cannot be expected to be fully reliable.
We will briefly cover the satellite mobile system architecture, then discuss GMR (GEO-Mobile Radio) system elements, e.g. GSS (Gateway Station Subsystem), MES (Mobile Earth Station), AOC (Advanced Operation Center), and TCS (Traffic Control Subsystem) for GMR-1 systems and NCC (Network Control Center), GW (Gateway), SCF (Satellite Control Facility) and CMIS (Customer Management Information System) for GMR-2 systems.
From there, we will discuss the security issues of GMR system as it shares similar vulnerabilities with GSM–GMR is derived from the terrestrial digital cellular standard GSM and support access to GSM core networks, along with some interesting demos.
Time permitting, a question and answer session at the end of the presentation will allow participants to cover any additional issues in satellite telephony system they’d like to discuss.
Combination of biology and electronics has led to many new inventions .these are useful to combat delay diseases .One such invention microelectronic pill. This phenomenon is used to detect diseases and abnormalities in the body. This is an indigestible pill and comprised of sensor. These sensor measures various body parameters like pH of stomach acid and intestinal .these measure temperature and dissolved oxygen. These sensors are mounted in the silicon chips. The microelectronics pill is completely harm-free to body.
Is your business NBN ready? – Developing a Digital Business Strategy: VELG Na...Vanguard Visions
The Australian Government is investing $43 billion over 8 years in fibre, wireless and satellite broadband infrastructure as part of the National Broadband Network (NBN). There is strong evidence that the NBN will be a key enabler of many economic, social, technological and environmental developments (Bowles, 2011). This new digital economy will have an impact on all Australian industries and will require every Australian to improve their information and communication technology (ICT) skills. Participants will take away tools and processes to develop a digital business strategy which will enable them to capitalise on the training and business opportunities being made possible through the roll out of the NBN.
The National Broadband Network (NBN) will contribute to Australia being among the world’s leading digital economies through improved productivity gains, global competitiveness and social well-being (National Digital Economy Strategy, 2011).
In order to ensure your business takes advantage of this improved connectivity it will need to make sure that is ‘NBN ready’.
This interactive session presented by Allison Miller of Vanguard Visions Consulting will provide an overview of:
What the NBN is and why it is important to your business
How your businesses can prepare itself to make the most of the NBN
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...Continuous Computing
Presentation delivered at Broadband Traffic Management Congress on November 2010 entitled, "Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway and Esuring Scalability"
P2P - Real Time Communications in the EnterpriseMead Eblan
"Mainstreaming Peer-to-Peer Connectivity:
Real-Time Communications in the Enterprise"
- Presented at Digital Government Institute Conference on IPv6
- September 2007
Smarter Test Automation for Web & Mobile AppsKeao Caindec
On any given day SOASTA may be looking for thousands of cloud servers to simulate millions of virtual mobile and web site visitors. As a result, they’ve done more cloud testing across more cloud providers than anyone else. SOASTA’s CloudTest platform depends on the swift provisioning and releasing of servers, while ensuring that each server plays its role in a distributed, multi-vendor architecture. Dimension Data is a key provider in SOASTA’s Global Test Cloud, giving our clients scalable access to cost effective servers from around the world.
ODCA Forecast 2012 Keynote: Curt Aubley, President, Open Data Center Alliance; VP/CTO NexGen Cyber Innovation & Technology; Lockheed Martin Information Systems & Global Services
The Essentials of Mobile App Performance Testing and MonitoringCorrelsense
Complexity across mobile carriers, locations and operating systems has made building mobile apps and monitoring their end user performance time consuming and expensive. The importance of testing mobile apps on iOS, Android and Windows Phone is increasing as more users embrace these devices. Join Correlsense and uTest for an online seminar which will teach you the steps to successful mobile application testing and performance management. We will discuss:
- The proliferation of mobile devices and the technical challenges they bring to end user experience monitoring
- Ways to prepare mobile applications for peak usage periods with the right load and performance testing techniques
- Tips and techniques for gaining visibility into the performance of mobile applications with the right monitoring tools
We will conclude with a discussion of the Correlsense and uTest solutions.
Slides from my talk at Philly ETE looking at the Lambda Architecture (originating at twitter) critically from the perspective of someone viewing it from the financial (faster, higher volume, spikier data) domain
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011darach
A presentation delivered to the Erlang User Group in London demonstrating how to embed the erjang implementation of erlang into the StreamBase CEP engine, enabling extending StreamBase with erlang based extensions.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Data distribution in the cloud with Node.js
1. Data Distribution
in the cloud with
Node.js
Copyright Push Technology 2012
2. • British startup. Founded in 2006.
• ‘Last mile’ data distribution specialist.
• Data-centric approach to
messaging/caching.
• Preferred by 6 of the top 10 online
eGaming exchanges.
• Growing fast. 400% year on year.
• Focus: Better bang for your bytes!
Introducing Push Technology
Copyright Push Technology 2012
Twitter: @push_technology
3. • Distributed Systems / HPC guy.
• Chief Scientist :- at Push Technology
• Alumnus of :-
Motorola, IONA, Betfair, JPMC, StreamBase.
• School: Trinity College Dublin.
- BA (Mod). Comp. Sci.+
- M.Sc. Networks & Distributed Systems
• Responds to: Guinness, Whisky
About me?
Copyright Push Technology 2012
Darach@PushTechnology.com
4. • Favorite language: Erlang
• Favorite bits?
• OTP – Behaviors
• Bit Syntax
• Least favorite language: Java
• Paid to write this stuff
• Love the JVM
• Liking Node a lot.
• Small fast data guy. I work in microseconds,
measure in nanoseconds. On my critical path
micro-benchmarking is a way of life.
About me?
Copyright Push Technology 2012
Darach@PushTechnology.com
5. 1st clean room certified JVM in 10
years. Built in Dublin! It rocks.
Tenant #1 Tenant #2 Tenant #N
(Diffusion) (Diffusion) (Diffusion)
Push Technology Diffusion
Waratek Cloud VM for Java
Benefits
• High density deployments
• Elastic. Scalable on demand
• Meterability: Bandwidth and compute utilization
• Multi-tenant. Each tenant fully isolated
Copyright Push Technology 2012
6. A US Cap Market second?
• 174 microseconds round trip
time rules out High
Frequency Trading
applications.
Not on the critical path!
Source: Me, former life
@StreamBase
• http://slidesha.re/guZOVe
8. Traditional Messaging
A
B
ba
bb
Producers ? Consumers
Pros Cons
• Loosely coupled. • No data model. Slinging blobs
• All you can eat messaging patterns • Fast producer, slow consumer? Ouch.
• Familiar • No data ‘smarts’. A blob is a blob.
Copyright Push Technology 2012
9. Invented yonks ago…
Before the InterWebs
For ‘reliable’ networks
For machine to machine
Remember DEC Message Queues?
- That basically. Vomit!
Copyright Push Technology 2012
10. When fallacies were simple
-The network is reliable
-Latency is zero
-Bandwidth is infinite
-There is one administrator
-The network is secure
-Transport cost is zero
-The network is homogeneous
Copyright Push Technology 2012
11. Then in 1992, this happened:
The phrase ‘surfing the internet’ was coined by Jean Poly.
First base.
Copyright Push Technology 2012
12. It grew, and it grew
Copyright Push Technology 2012
13. Then in 2007, this happened:
The god phone:
Surfing died. Touching happened.
Second base unlocked.
Copyright Push Technology 2012
14. Then in 2007, this happened:
So we took all the things and put them in the internet:
Cloud happened.
So we could touch all the things.
Messaging
Apps
Hardware
Virtualize all the things
Services
Skills,
Specialties
Copyright Push Technology 2012
15. Then in 2009, this happened:
Ryan Dahl, basically.
Tyrannically asynchronous.
Devilishly event oriented.
Amazoidingly non-blocking.
Copyright Push Technology 2012
16. It grew, and it grew
Like all the good things do.
Copyright Push Technology 2012
17. Stop. Fallacies? Reality:
-The network is not reliable
nor is it cost free.
-Latency is not zero
nor is it a democracy.
-Bandwidth is not infinite
nor predictable especially the last mile!
-There is not only one administrator
trust, relationships are key
-The network is not secure
nor is the data that flows through it
-Transport cost is not zero
but what you don’t do is free
-The network is not homogeneous
nor is it smart
Copyright Push Technology 2012
18. Look. What, How & Why?
-What and How are what geeks do.
-Why gets you paid
-Business Value and Trust dictate What and How
- Policies, Events and Content implements Business Value
-Science basically. But think like a carpenter:
-Measure twice. Cut once.
Copyright Push Technology 2012
19. The Problem: The bird, basically.
Immediately Inconsistent.
But, Eventually Consistent
… Maybe.
Copyright Push Technology 2012
20. Listen.
- Every nuance comes with a set of tradeoffs.
- Choosing the right ones can be hard, but it pays off.
- Context, Environment are critical
- Break all the rules, one benchmark at a time.
- Benchmark Driven Development FTW
Copyright Push Technology 2012
21. Act.
- You measured twice, right?
- So get cutting!
- Simples
Copyright Push Technology 2012
22. Act. Telepathy? Telemetry!
A
B
ba
bb
Buffer
Producers Bloat
Consumers
Virtualize client queues? Nuance: ‘See’ backlog, client affinity. Tradeoff GD harder :/
Copyright Push Technology 2012
23. Act. Stateless or Stateful Topics
A
B
ba x
bb x
Producers
Is it a Consumers
cache?
Data one hop closer to consumers. Good state? Touch it! Exploit it! Use it!
Copyright Push Technology 2012
24. Act. Finagle the data
A
B Snapshot Delta
ba x
bb x
Producers State! Consumers
Last value cached. Tradeoff? Memory. Snapshot on subscribe. Deltas thereafter
Copyright Push Technology 2012
25. Act. ‘Smart data’
A B C A C D
t0 t1
Don’t repeat yourself. Send the changes, not the whole list after initial ‘snapshot’.
Copyright Push Technology 2012
26. Act. Behaviors
A
B
ba x
bb x
X The
Producers
topic is Consumers
the
cloud!
Extensible. Nuance? Roll your own protocols. Tradeoff? 3rd party code in the engine :/
Copyright Push Technology 2012
27. Data Distribution
Messaging remixed around:
Relevance - Queue depth for conflatable data should be 0 or 1. No more
Responsiveness - Use HTTP/REST for things. Stream the little things
Timeliness - It’s relative. M2M != M2H.
Context - Packed binary, deltas mostly, snapshot on subscribe.
Environment- Don’t send 1M 1K events to a mobile phone with 0.5mbps.
Copyright Push Technology 2012
28. An Example
Operations:>
Tenants :>
Gaming Live Internet Apps Finance QA + Dev + UAT
Copyright Push Technology 2012
29. Either way?
It’s about the data.
Period.
The rest (analysis, storage, transformation) is sugar.
Copyright Push Technology 2012
30. Sugar? Streams
w w S
C Q
w w
Stream Operations
• Mapping. Change/enrich the data structurally.
• Aggregation. A ‘window of’ data. Eg. A seconds worth.
• Splitting & Filtering
• Combining multiple streams. Eg. Temporal pattern matching
• Access/Store. Eg: CRUD, variable, file, …
Copyright Push Technology 2012
31. Sugar? Streams
w w S
C Q
w w
Stream Operations
• Mapping. Just a function call in Node.js
• Aggregation. A ‘window of’ data. Eg. A seconds worth.
• Splitting & Filtering. An expression or a set thereof.
• Combining multiple streams. It depends. Can be ‘complex’
• Access/Store. Trivial.
Copyright Push Technology 2012
33. Introducing eep.js
w w S
C Q
w w
What is eep.js?
• Add aggregate functions and window operations to Node.js
• 4 window types: tumbling, sliding, periodic, monotonic
• Node makes evented IO easy. So just add windows.
• Fast. 8-40 million events per second (upper bound).
Copyright Push Technology 2012
34. eep.js: Tumbling Windows
x() x() x() x()
emit()
x() x() x() x() emit()
1 2 3 4
x() x() x() x()
emit()
2 3 4 5
init()
2 3 4 5
init()
init()
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ...
What is a tumbling window?
• Every N events, give me an average of the last N events
• Does not overlap windows
• ‘Closing’ a window, ‘Emits’ a result (the average)
• Closing a window, Opens a new window
Copyright Push Technology 2012
35. eep.js: Aggregate Functions
What is an aggregate function?
• A function that computes values over events.
• The cost of calculations are ammortized per event
• Just follow the above recipe
• Example: Aggregate 2M events (equity prices), send to GPU
on emit, receive 2M options put/call prices as a result.
Copyright Push Technology 2012
39. eep.js: Sliding Windows
init()
1 2 3 4 5 .. .. .. ..
x() 1 2 3 4 .. .. .. ..
x() 1 2 3 .. .. .. ..
x() 1 2 .. .. .. ..
t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 ...
What is a sliding window?
• Like tumbling, except can overlap. But O(N2), Keep N small.
• Every event opens a new window.
• After N events, every subsequent event emits a result.
• Like all windows, cost of calculation ammortized over events
Copyright Push Technology 2012
43. eep.js: Monotonic Windows
my my my
x() x() x() x()
emit()
x() x() x() x() emit()
1 2 3 4
x() x() x() x()
emit()
2 3 4 5
init()
2 3 4 5
init()
init()
t0 t1 t2 t3 ...
What is a monotonic window?
• Driven mad by ‘wall clock time’? Need a logical clock?
• No worries. Provide your own clock! Eg. Vector clock
Copyright Push Technology 2012
47. eep.js
Embedded Event Processing:
• Simple to use. Aggregates Functions and Windowed event processing.
• Get it from GitHub/npm soon. Use it. Fork it.
• Fast. CEP engines typically handle ~250K/sec.
• For small N (most common) is 34x - 200x faster than commercial CEP engines.
• But, at a small price. Simple. No multi-dimensional, infinite or predicate windows
• Reduces a flood of events into a few in near real time
• Can handle 8-40 million events per second (max, on my laptop). YMMV.
• Combinators may be added. [Ugh, if I need combinators]
Copyright Push Technology 2012
49. Performance? In perspective
• A 1 producer, 1 consumer lock-free wait-free full duplex queue implementation on
a 2.3GHz intel Sandybridge can:
• Distribute ~300M events between hyperthreads per second
• Distribute ~50M events between two hardware threads on two cores on the same physical die
• Distributed ~30M events between two hardware threads on two cores on separate physical dies
• You can, with a fully lock-free wait-free system (and you bypass the operating system
kernel), maybe, ~1M 1K events/second
• There’s no point being capable of > 30M events/second on a thread if you’re going over a wire.
• So, 8-40 million events/second in node is a pleasant sufficiency
• It’s not the algorithm. It’s the mechanical sympathy, stoopid!
• Lock free wait-free concurrency is easier than lock based concurrency. Try it.
Copyright Push Technology 2012
50. • Thank you for listening
• Thank you for having me
• Thank you Push for the beer budget
• Le twitter: @darachennis
• Expect eep.js in GitHub soon
• I’ll hashtag it #nodedublin
• Thank you @Waratek geeks.
About me?
Copyright Push Technology 2012
Darach@PushTechnology.com