This document discusses Boundary's real-time data streaming and visualization capabilities. It describes how lightweight collectors intercept and collect meter data via TLS authentication from multiple data sources. The data is stored and streamed in real-time at high resolution with sub-second latency. The streaming UI provides intuitive dashboards to view the continuously updating data. It also outlines the data structure and subscription process, and discusses strategies for optimizing large state dumps and resubscriptions to address data and subscription problems. Potential solutions and next steps are proposed, including stratified queries, top-N limitations, web workers, and using HTML5 local storage.
Couchbase Data Platform | Big Data DemystifiedOmid Vahdaty
Couchbase is a popular open source NoSQL platform used by giants like Apple, LinkedIn, Walmart, Visa and many others and runs on-premise or in a public/hybrid/multi cloud.
Couchbase has a sub-millisecond K/V cache integrated with a document based DB, a unique and many more services and features.
In this session we will talk about the unique architecture of Couchbase, its unique N1QL language - a SQL-Like language that is ANSI compliant, the services and features Couchbase offers and demonstrate some of them live.
We will also discuss what makes Couchbase different than other popular NoSQL platforms like MongoDB, Cassandra, Redis, DynamoDB etc.
At the end we will talk about the next version of Couchbase (6.5) that will be released later this year and about Couchbase 7.0 that will be released next year.
Slidedeck presented at http://devternity.com/ around MongoDB internals. We review the usage patterns of MongoDB, the different storage engines and persistency models as well has the definition of documents and general data structures.
This presentation will discuss implementing external authentication when using Percona Server for MongoDB and MongoDB Enterprise. It will review authentication using OpenLDAP or ActiveDirectory and ActiveDirectory with Kerberos.
The presentation will also include examples of the configurations required by these external directory services. It will also review the LDAP Authorization features introduced in MongoDB Enterprise 3.4.
Couchbase Data Platform | Big Data DemystifiedOmid Vahdaty
Couchbase is a popular open source NoSQL platform used by giants like Apple, LinkedIn, Walmart, Visa and many others and runs on-premise or in a public/hybrid/multi cloud.
Couchbase has a sub-millisecond K/V cache integrated with a document based DB, a unique and many more services and features.
In this session we will talk about the unique architecture of Couchbase, its unique N1QL language - a SQL-Like language that is ANSI compliant, the services and features Couchbase offers and demonstrate some of them live.
We will also discuss what makes Couchbase different than other popular NoSQL platforms like MongoDB, Cassandra, Redis, DynamoDB etc.
At the end we will talk about the next version of Couchbase (6.5) that will be released later this year and about Couchbase 7.0 that will be released next year.
Slidedeck presented at http://devternity.com/ around MongoDB internals. We review the usage patterns of MongoDB, the different storage engines and persistency models as well has the definition of documents and general data structures.
This presentation will discuss implementing external authentication when using Percona Server for MongoDB and MongoDB Enterprise. It will review authentication using OpenLDAP or ActiveDirectory and ActiveDirectory with Kerberos.
The presentation will also include examples of the configurations required by these external directory services. It will also review the LDAP Authorization features introduced in MongoDB Enterprise 3.4.
The integration between Spring Framework and MongoDB tends to be somewhat unknown. This presentation shows the different projects that compose Spring ecosystem, Springdata, Springboot, SpringIO etc and how to merge between the pure JAVA projects to massive enterprise systems that require the interaction of these systems together.
Conceptos básicos. Seminario web 5: Introducción a Aggregation FrameworkMongoDB
Este es el quinto seminario web de la serie Conceptos básicos, en la que se realiza una introducción a la base de datos MongoDB. En este seminario web, se analizan los aspectos básicos de Aggregation Framework.
Data Management 2: Conquering Data ProliferationMongoDB
Today's customers demand applications which integrate intelligently with data from mobile, social media and cloud sources. A system of engagement meets these expectations by applying data and analytics drawn from an array of master systems. The enormous scale and performance required overwhelm relational approaches, but we can use MongoDB to meet the challenge. We'll learn to capture and transmit data changes among disparate systems, expose batch data as interactive operational queries and build systems with strong division of concerns, agility and flexibility.
Data Management 3: Bulletproof Data ManagementMongoDB
"This session focuses on delivering operationally robust deployments of MongoDB via specific design capabilities and varying data feeds. Learn how to use services or driver wrappers to unify design patterns for managing data. This talk will address the following questions:
How do you enforce a schema?
How do you redact or remove sensitive data in queries and feeds?
How do you detect and police ""out of profile"" queries and make sure they do not threaten your system?"
Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB
Questo è il secondo webinar della serie Back to Basics che ti offrirà un'introduzione al database MongoDB. In questo webinar ti dimostreremo come creare un'applicazione base per il blogging in MongoDB.
Conceptos básicos. Seminario web 4: Indexación avanzada, índices de texto y g...MongoDB
Este es el cuarto seminario web de la serie Conceptos básicos, en la que se realiza una introducción a la base de datos MongoDB. Este seminario se ve en la compatibilidad con índices de texto libre y geoespaciales.
From qconsf 2010 - this presentation focuses on how the classic querying models like plain SQL and JPA map to distributed data stores. It first reviews the current distributed data stores landscape and its querying models, and then discuss the wide range of APIs for data extraction from these data stores. It then discusses the main challenges of mapping various APIs to a distributed data model and the trade offs to be aware off.
Since a couple of years, the NoSQL movement has developed a variety of open-source document stores. They are focused on high availability, horizontal scalability, and are designed to run on commodity hardware. These products have gained great traction in the industry to store large amounts of flexible data. Arguably, the next step for the NoSQL community is on harnessing flexible data processing.
The aim of this presentation is to introduce JSONiq: the SQL of NoSQL.
Each of us operates distributed systems. Some of us operate traditional infrastructure
with database, web, and load-balancing tiers. Others require infrastructure that is
more bespoke and may incorporate non-traditional storage solutions (such as Riak).
Regardless of where each of us falls on this spectrum, the network closely describes the
behavior of our applications. Furthermore, it is the only place we can look to understand
emergent behavior of applications working together in concert. In this talk, we take a
radiological view of network-derived imagery and discuss what it can tell us about our
systems as a whole.
The integration between Spring Framework and MongoDB tends to be somewhat unknown. This presentation shows the different projects that compose Spring ecosystem, Springdata, Springboot, SpringIO etc and how to merge between the pure JAVA projects to massive enterprise systems that require the interaction of these systems together.
Conceptos básicos. Seminario web 5: Introducción a Aggregation FrameworkMongoDB
Este es el quinto seminario web de la serie Conceptos básicos, en la que se realiza una introducción a la base de datos MongoDB. En este seminario web, se analizan los aspectos básicos de Aggregation Framework.
Data Management 2: Conquering Data ProliferationMongoDB
Today's customers demand applications which integrate intelligently with data from mobile, social media and cloud sources. A system of engagement meets these expectations by applying data and analytics drawn from an array of master systems. The enormous scale and performance required overwhelm relational approaches, but we can use MongoDB to meet the challenge. We'll learn to capture and transmit data changes among disparate systems, expose batch data as interactive operational queries and build systems with strong division of concerns, agility and flexibility.
Data Management 3: Bulletproof Data ManagementMongoDB
"This session focuses on delivering operationally robust deployments of MongoDB via specific design capabilities and varying data feeds. Learn how to use services or driver wrappers to unify design patterns for managing data. This talk will address the following questions:
How do you enforce a schema?
How do you redact or remove sensitive data in queries and feeds?
How do you detect and police ""out of profile"" queries and make sure they do not threaten your system?"
Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB
Questo è il secondo webinar della serie Back to Basics che ti offrirà un'introduzione al database MongoDB. In questo webinar ti dimostreremo come creare un'applicazione base per il blogging in MongoDB.
Conceptos básicos. Seminario web 4: Indexación avanzada, índices de texto y g...MongoDB
Este es el cuarto seminario web de la serie Conceptos básicos, en la que se realiza una introducción a la base de datos MongoDB. Este seminario se ve en la compatibilidad con índices de texto libre y geoespaciales.
From qconsf 2010 - this presentation focuses on how the classic querying models like plain SQL and JPA map to distributed data stores. It first reviews the current distributed data stores landscape and its querying models, and then discuss the wide range of APIs for data extraction from these data stores. It then discusses the main challenges of mapping various APIs to a distributed data model and the trade offs to be aware off.
Since a couple of years, the NoSQL movement has developed a variety of open-source document stores. They are focused on high availability, horizontal scalability, and are designed to run on commodity hardware. These products have gained great traction in the industry to store large amounts of flexible data. Arguably, the next step for the NoSQL community is on harnessing flexible data processing.
The aim of this presentation is to introduce JSONiq: the SQL of NoSQL.
Each of us operates distributed systems. Some of us operate traditional infrastructure
with database, web, and load-balancing tiers. Others require infrastructure that is
more bespoke and may incorporate non-traditional storage solutions (such as Riak).
Regardless of where each of us falls on this spectrum, the network closely describes the
behavior of our applications. Furthermore, it is the only place we can look to understand
emergent behavior of applications working together in concert. In this talk, we take a
radiological view of network-derived imagery and discuss what it can tell us about our
systems as a whole.
Real-Time Streaming: Move IMS Data to Your Cloud Data WarehousePrecisely
With over 22,000 transactions processed every second, your mainframe IMS is a critical source of data for the cloud data warehouses that feed analytics, customer experience or regulatory initiatives. However, extracting data from mainframe IMS can be time-consuming and costly, leading to the exclusion of IMS data from cloud data warehouses all together – and leaving valuable insights unseen.
Never ignore or manually extract mainframe IMS data again. In this on-demand webcast, you will learn how Connect CDC enables your team to develop integrations quickly and easily between mainframe IMS and cloud data warehouses in the most cost-effective way possible.
Streaming Operational Data with MariaDB MaxScaleMariaDB plc
MariaDB experts explain how to stream data using MariaDB MaxScale, a database proxy that can vastly improve your server's transactional data processing without sacrificing scalability, security or speed. In this webinar, learn how to use MaxScale to convert data to JSON documents or AVRO objects, and watch as MariaDB's senior software engineers do a live demo of how to use the Kafka producer.
Watch the webinar here: https://mariadb.com/resources/webinars/streaming-operational-data-mariadb-maxscale
Jump Start with Apache Spark 2.0 on DatabricksDatabricks
Apache Spark 2.0 has laid the foundation for many new features and functionality. Its main three themes—easier, faster, and smarter—are pervasive in its unified and simplified high-level APIs for Structured data.
In this introductory part lecture and part hands-on workshop you’ll learn how to apply some of these new APIs using Databricks Community Edition. In particular, we will cover the following areas:
What’s new in Spark 2.0
SparkSessions vs SparkContexts
Datasets/Dataframes and Spark SQL
Introduction to Structured Streaming concepts and APIs
Materials Project Validation, Provenance, and Sandboxes by Dan GunterDan Gunter
Summary of Goals, Progress, and Next steps for these three aspects of the Materials Project (materialsproject.org) infrastructure
* Validation: constantly guard against bugs in core data and imported data
* Provenance: know how data came to be
* Sandboxes: combine public and non-public data; "good fences make good neighbors"
Presenter: Dan Gunter, LBNL
MongoDB has taken a clear lead in adoption among the new generation of databases, including the enormous variety of NoSQL offerings. A key reason for this lead has been a unique combination of agility and scalability. Agility provides business units with a quick start and flexibility to maintain development velocity, despite changing data and requirements. Scalability maintains that flexibility while providing fast, interactive performance as data volume and usage increase. We'll address the key organizational, operational, and engineering considerations to ensure that agility and scalability stay aligned at increasing scale, from small development instances to web-scale applications. We will also survey some key examples of highly-scaled customer applications of MongoDB.
Analyzing big data is a challenge, requiring lots of processing power and storage.
Cloud Computing is an ideal platform to tackle this problem. HD Insight on Microsoft Azure deploys Hadoop and other open source big data tools to the cloud, making it easier to take advantage of the high scalability of this platform.
In this session, you will learn what tools are available in HD Insight and how to use them to store, process, and analyze large amounts of data.
Stratio Streaming is the result of combining the power of Spark Streaming as a continuous computing framework and Siddhi CEP engine as complex event processing engine.
Riga dev day: Lambda architecture at AWSAntons Kranga
My recent talk at Riga DevDay about Lambda architect at AWS. It illustrates few design simplifications that we can get when we implement Lambda Architecture in Cloud Native way
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...StampedeCon
At StampedeCon 2014, Ronald Indeck (VelociData), "Enabling Key Business Advantage from Big Data through Advanced Ingest Processing."
All too often we see critical data dumped into a “Data Lake” causing the data waters to stagnate and become a “Data Swamp”. We have found that many data transformation, quality, and security processes can be addressed a priori on ingest to enhance goodness and improve accessibility to the data. Data can still be stored in raw form if desired but this processing on ingest can unlock operational effectiveness and competitive advantage by integrating fresh and historical data and enable the full potential of the data. We will discuss the underpinnings of stream processing engines, review several relevant business use cases, and discuss future applications.
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...Databricks
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Comcast, GrubHub, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores.
Building a Scalable Real-Time Fleet Management IoT Data Tracker with Kafka St...HostedbyConfluent
In the IoT-enabled fleet management domain, real-time signal tracking is crucial. Signals refer to various datapoint readings from different sensors across the vehicle, like engine temperature, fuel level or braking force. Our solution processes batches of these signals, handling up to 8K batches (or 500K signals) per second in production every day. This talk explores our architectural journey, focusing on real-time, horizontal scalability, fault tolerance, monitoring and alerting. We utilized Kafka Streams' interactive queries API and a gRPC layer for Protobuf-formatted data storage and querying, achieving near-instantaneous data access. Key optimizations to both Kafka topology and cluster will be discussed, specifically aimed at reducing network overhead and controlling changelog size. These optimizations not only ensure resource efficiency but also enhance fault tolerance and rapid startups. Walk away with actionable insights for your own Kafka deployments.
Similar to Boundary Front end tech talk: how it works (20)
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
2. Tech Talk - Frontend
Stephen Boak (steve@boundary.com)
Mark Mahoney (mark@boundary.com)
Matt King (m@boundary.com)
3. When we say real-time we mean it: JSON being
pushed to the browser once a second and
visualizations built to constantly move and adapt
to streaming data
5. Meters !
Lightweight and highly scalable, these sit on the cloud as well as
private data centers across virtual or physical servers.
6. Intercept Meter data via Transport
Collectors Layer Security (TLS) Authentication.
7. Collect all of the data at high resolution to gain rich insight into complex
Data Store environments & problems before they can impact critical business services.
9. Stream data to your dashboard with sub-second latency,
Streaming UI providing you with intuitive, powerful dashboard.
10. How We Get Data
• CometD server (streaker)!
• Pulls in data from multiple backend services and
streams it out to clients!
• Can also aggregate and filter data on demand
11. Subscriptions
1 2 3 4 5
Make a Get back a Subscribe to Receive Receive
subscription request subscription ID new query state-dump add/remove
(query and filters) (Query ID) (unique channel) (inserts & schema) messages
!!
!
!
!
!
12. Data Structure
• Schema and key
(which fields are concatenated for each record)
• State dump with schema for requested time window
(up to 100kb state dumps)
• Save bandwidth with N-tuples indexed by the schema
• Adds/removes update the state with keys (subset of schema)
14. DataSource.js
• Negotiates between multiple data
sources and subscribers on a page Subscriptions
• One data source for N subscribers Data
Sources
• Abstracts CometD subscription
process for JS development
Subscribers
• Smart enough to resubscribe and
notifies subscribers
(so that state dumps aren’t doubled up)
17. Data and Subcription Problems
• Monolithic, Multi-Purpose Queries
• Large State Dumps cause CometD timeout
(5-10MBs per second)
• No Resubscription
• Aggregated all data on front-end
18. Data and Subcription Solutions
• Stratified, Filterable Queries and server-side aggregation
• Top-N Limitations
(constrained output)
• Resubscription & filtering
• Web workers haven’t helped much
• Time smoothing (1-second ticks)
20. What’s Next
• WebSockets
• Historical Data and long term data storage
• HTML5 local storage (store data)
• Machine-learning for real-time network visualization