How do NoSQL Document-Oriented Databases like Couchbase fit in with Apache Spark? This set of slides gives a couple of use cases, shows why Couchbase works great with Spark, and sets up a scenario for a demo.
Slides presented at SDBigData Meetup:
http://www.meetup.com/sdbigdata/events/225691323/
There was a request for more Couchbase use case information and NoSQL primer, so I added a number of slides to let me talk to those aspects right before doing the presentation.
ETL Made Easy with Azure Data Factory and Azure DatabricksDatabricks
Data Engineers are responsible for data cleansing, prepping, aggregating, and loading analytical data stores, which is often difficult and time-consuming. Azure Data Factory makes this work easy and expedites solution development. We’ll demonstrate how Azure Data Factory can enable a new UI-driven ETL design paradigm on top of Azure Databricks for building scaled-out data transformation pipelines.
Real-Time Data Pipelines with Kafka, Spark, and Operational DatabasesSingleStore
Eric Frenkiel, MemSQL CEO and co-founder and Gartner Catalyst. August 11, 2015, San Diego, CA. Watch the Pinterest Demo Video here: https://youtu.be/KXelkQFVz4E
Slides presented at SDBigData Meetup:
http://www.meetup.com/sdbigdata/events/225691323/
There was a request for more Couchbase use case information and NoSQL primer, so I added a number of slides to let me talk to those aspects right before doing the presentation.
ETL Made Easy with Azure Data Factory and Azure DatabricksDatabricks
Data Engineers are responsible for data cleansing, prepping, aggregating, and loading analytical data stores, which is often difficult and time-consuming. Azure Data Factory makes this work easy and expedites solution development. We’ll demonstrate how Azure Data Factory can enable a new UI-driven ETL design paradigm on top of Azure Databricks for building scaled-out data transformation pipelines.
Real-Time Data Pipelines with Kafka, Spark, and Operational DatabasesSingleStore
Eric Frenkiel, MemSQL CEO and co-founder and Gartner Catalyst. August 11, 2015, San Diego, CA. Watch the Pinterest Demo Video here: https://youtu.be/KXelkQFVz4E
Presentation on the struggles with traditional architectures and an overview of the Lambda Architecture utilizing Spark to drive massive amounts of both batch and streaming data for processing and analytics
Data Engineer's Lunch #55: Get Started in Data EngineeringAnant Corporation
In Data Engineer's Lunch #55, CEO of Anant, Rahul Singh, will cover 10 resources every data engineer needs to get started or master their game.
Accompanying Blog: Coming Soon!
Accompanying YouTube: Coming Soon!
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday:
https://www.meetup.com/Data-Wranglers-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
Powering Interactive BI Analytics with Presto and Delta LakeDatabricks
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources.
Customer Education Webcast: New Features in Data Integration and Streaming CDCPrecisely
View our quarterly customer education webcast to learn about the new advancements in Syncsort DMX and DMX-h data integration software and DataFunnel - our new easy-to-use browser-based database onboarding application. Learn about DMX Change Data Capture and the advantages of true streaming over micro-batch.
View this webcast on-demand where you'll hear the latest news on:
• Improvements in Syncsort DMX and DMX-h
• What’s next in the new DataFunnel interface
• Streaming data in DMX Change Data Capture
• Hadoop 3 support in Syncsort Integrate products
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA
In this talk, we will discuss how we use Spark as part of a hybrid RDBMS architecture that includes Hadoop and HBase. The optimizer evaluates each query and sends OLTP traffic (including CRUD queries) to HBase and OLAP traffic to Spark. We will focus on the challenges of handling the tradeoffs inherent in an integrated architecture that simultaneously handles real-time and batch traffic. Lessons learned include: - Embedding Spark into a RDBMS - Running Spark on Yarn and isolating OLTP traffic from OLAP traffic - Accelerating the generation of Spark RDDs from HBase - Customizing the Spark UI The lessons learned can also be applied to other hybrid systems, such as Lambda architectures.
Bio:-
John Leach is the CTO and Co-Founder of Splice Machine. With over 15 years of software experience under his belt, John’s expertise in analytics and BI drives his role as Chief Technology Officer. Prior to Splice Machine, John founded Incite Retail in June 2008 and led the company’s strategy and development efforts. At Incite Retail, he built custom Big Data systems (leveraging HBase and Hadoop) for Fortune 500 companies. Prior to Incite Retail, he ran the business intelligence practice at Blue Martini Software and built strategic partnerships with integration partners. John was a key subject matter expert for Blue Martini Software in many strategic implementations across the world. His focus at Blue Martini was helping clients incorporate decision support knowledge into their current business processes utilizing advanced algorithms and machine learning. John received dual bachelor’s degrees in biomedical and mechanical engineering from Washington University in Saint Louis. Leach is the organizer emeritus for the Saint Louis Hadoop Users Group and is active in the Washington University Elliot Society.
Delta Lake, an open-source innovations which brings new capabilities for transactions, version control and indexing your data lakes. We uncover how Delta Lake benefits and why it matters to you. Through this session, we showcase some of its benefits and how they can improve your modern data engineering pipelines. Delta lake provides snapshot isolation which helps concurrent read/write operations and enables efficient insert, update, deletes, and rollback capabilities. It allows background file optimization through compaction and z-order partitioning achieving better performance improvements. In this presentation, we will learn the Delta Lake benefits and how it solves common data lake challenges, and most importantly new Delta Time Travel capability.
Large Scale Lakehouse Implementation Using Structured StreamingDatabricks
Business leads, executives, analysts, and data scientists rely on up-to-date information to make business decision, adjust to the market, meet needs of their customers or run effective supply chain operations.
Come hear how Asurion used Delta, Structured Streaming, AutoLoader and SQL Analytics to improve production data latency from day-minus-one to near real time Asurion’s technical team will share battle tested tips and tricks you only get with certain scale. Asurion data lake executes 4000+ streaming jobs and hosts over 4000 tables in production Data Lake on AWS.
Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...Databricks
Building a curated data lake on real time data is an emerging data warehouse pattern with delta. However in the real world, what we many times face ourselves with is dynamically changing schemas which pose a big challenge to incorporate without downtimes.
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...StreamNative
Apache Hudi is an open data lake platform, designed around the streaming data model. At its core, Hudi provides a transactions, upserts, deletes on data lake storage, while also enabling CDC capabilities. Hudi also provides a coherent set of table services, which can clean, compact, cluster and optimize storage layout for better query performance. Finally, Hudi's data services provide out-of-box support for streaming data from event systems into lake storage in near real-time.
In this talk, we will walk through an end-end use case for change data capture from a relational database, starting with capture changes using the Pulsar CDC connector and then demonstrate how you can use the Hudi deltastreamer tool to then apply these changes into a table on the data lake. We will discuss various tips to operationalizing and monitoring such pipelines. We will conclude with some guidance on future integrations between the two projects including a native Hudi/Pulsar connector and Hudi tiered storage.
Presentation on the struggles with traditional architectures and an overview of the Lambda Architecture utilizing Spark to drive massive amounts of both batch and streaming data for processing and analytics
Data Engineer's Lunch #55: Get Started in Data EngineeringAnant Corporation
In Data Engineer's Lunch #55, CEO of Anant, Rahul Singh, will cover 10 resources every data engineer needs to get started or master their game.
Accompanying Blog: Coming Soon!
Accompanying YouTube: Coming Soon!
Sign Up For Our Newsletter: http://eepurl.com/grdMkn
Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday:
https://www.meetup.com/Data-Wranglers-DC/events/
Cassandra.Link:
https://cassandra.link/
Follow Us and Reach Us At:
Anant:
https://www.anant.us/
Awesome Cassandra:
https://github.com/Anant/awesome-cassandra
Email:
solutions@anant.us
LinkedIn:
https://www.linkedin.com/company/anant/
Twitter:
https://twitter.com/anantcorp
Eventbrite:
https://www.eventbrite.com/o/anant-1072927283
Facebook:
https://www.facebook.com/AnantCorp/
Join The Anant Team:
https://www.careers.anant.us
Powering Interactive BI Analytics with Presto and Delta LakeDatabricks
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources.
Customer Education Webcast: New Features in Data Integration and Streaming CDCPrecisely
View our quarterly customer education webcast to learn about the new advancements in Syncsort DMX and DMX-h data integration software and DataFunnel - our new easy-to-use browser-based database onboarding application. Learn about DMX Change Data Capture and the advantages of true streaming over micro-batch.
View this webcast on-demand where you'll hear the latest news on:
• Improvements in Syncsort DMX and DMX-h
• What’s next in the new DataFunnel interface
• Streaming data in DMX Change Data Capture
• Hadoop 3 support in Syncsort Integrate products
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA
In this talk, we will discuss how we use Spark as part of a hybrid RDBMS architecture that includes Hadoop and HBase. The optimizer evaluates each query and sends OLTP traffic (including CRUD queries) to HBase and OLAP traffic to Spark. We will focus on the challenges of handling the tradeoffs inherent in an integrated architecture that simultaneously handles real-time and batch traffic. Lessons learned include: - Embedding Spark into a RDBMS - Running Spark on Yarn and isolating OLTP traffic from OLAP traffic - Accelerating the generation of Spark RDDs from HBase - Customizing the Spark UI The lessons learned can also be applied to other hybrid systems, such as Lambda architectures.
Bio:-
John Leach is the CTO and Co-Founder of Splice Machine. With over 15 years of software experience under his belt, John’s expertise in analytics and BI drives his role as Chief Technology Officer. Prior to Splice Machine, John founded Incite Retail in June 2008 and led the company’s strategy and development efforts. At Incite Retail, he built custom Big Data systems (leveraging HBase and Hadoop) for Fortune 500 companies. Prior to Incite Retail, he ran the business intelligence practice at Blue Martini Software and built strategic partnerships with integration partners. John was a key subject matter expert for Blue Martini Software in many strategic implementations across the world. His focus at Blue Martini was helping clients incorporate decision support knowledge into their current business processes utilizing advanced algorithms and machine learning. John received dual bachelor’s degrees in biomedical and mechanical engineering from Washington University in Saint Louis. Leach is the organizer emeritus for the Saint Louis Hadoop Users Group and is active in the Washington University Elliot Society.
Delta Lake, an open-source innovations which brings new capabilities for transactions, version control and indexing your data lakes. We uncover how Delta Lake benefits and why it matters to you. Through this session, we showcase some of its benefits and how they can improve your modern data engineering pipelines. Delta lake provides snapshot isolation which helps concurrent read/write operations and enables efficient insert, update, deletes, and rollback capabilities. It allows background file optimization through compaction and z-order partitioning achieving better performance improvements. In this presentation, we will learn the Delta Lake benefits and how it solves common data lake challenges, and most importantly new Delta Time Travel capability.
Large Scale Lakehouse Implementation Using Structured StreamingDatabricks
Business leads, executives, analysts, and data scientists rely on up-to-date information to make business decision, adjust to the market, meet needs of their customers or run effective supply chain operations.
Come hear how Asurion used Delta, Structured Streaming, AutoLoader and SQL Analytics to improve production data latency from day-minus-one to near real time Asurion’s technical team will share battle tested tips and tricks you only get with certain scale. Asurion data lake executes 4000+ streaming jobs and hosts over 4000 tables in production Data Lake on AWS.
Designing and Implementing a Real-time Data Lake with Dynamically Changing Sc...Databricks
Building a curated data lake on real time data is an emerging data warehouse pattern with delta. However in the real world, what we many times face ourselves with is dynamically changing schemas which pose a big challenge to incorporate without downtimes.
Change Data Capture to Data Lakes Using Apache Pulsar and Apache Hudi - Pulsa...StreamNative
Apache Hudi is an open data lake platform, designed around the streaming data model. At its core, Hudi provides a transactions, upserts, deletes on data lake storage, while also enabling CDC capabilities. Hudi also provides a coherent set of table services, which can clean, compact, cluster and optimize storage layout for better query performance. Finally, Hudi's data services provide out-of-box support for streaming data from event systems into lake storage in near real-time.
In this talk, we will walk through an end-end use case for change data capture from a relational database, starting with capture changes using the Pulsar CDC connector and then demonstrate how you can use the Hudi deltastreamer tool to then apply these changes into a table on the data lake. We will discuss various tips to operationalizing and monitoring such pipelines. We will conclude with some guidance on future integrations between the two projects including a native Hudi/Pulsar connector and Hudi tiered storage.
ValueFrame - myynnistä toimitukseen -seminaari 17.11.2011 (id 2134) (id 2212)ValueFrame Oy
Esityksessä käydään läpi muutamia keskeisiä projektien epäonnistumiseen johtavia syitä sekä esitellään joitakin malleja näiden ehkäisyyn jo myyntivaiheen aikana.
www.valueframe.com
Guida di base alle Digital Arts: librerie, risorse, consigli e suggerimenti pratici per lo sviluppo di elaborati ad alto impatto visivo - documento a supporto dei webinar free & reserved dedicati al tema e disponibili su youtube all'indirizzo: http://youtube.com/artlandis77
#fotocasaResponde: ¿Cómo reclamar la devolución de las cláusulas suelo?fotocasa
¿En qué casos se puede reclamar? ¿Cuáles son las cantidades a devolver? ¿Cómo reclamar la devolución del dinero? Estas son sólo algunas de las preguntas que estos días nos han hecho llegar nuestros usuarios. Para dar respuesta a estas y otras dudas, contamos con la presencia de Miguel Muñoz, abogado experto en Derecho Inmobiliario de Legálitas.
Today, influence is determined by how high a social score you have. But that dilutes what true influence is, and places the attention on the wrong people.
By focusing on the customer and identifying who truly influencers their decisions at key times in the purchase life cycle, we can target better and gather lead generation, increase customer acquisition, and provide real ROI for influence marketing campaigns.
In 2010, I hit the jackpot when I got an internship to work at the Winter Olympics in Vancouver. Since then, I’ve been lucky enough to work at two more Olympics in London and Sochi. These experiences forever transformed my life both personally and professionally.
Over the years, I’ve been asked, “what did you actually do during the Olympics?” People often don’t realize that the Olympics event is the Mount Everest of the events industry. The athletes get the fame and glory, but behind the scenes there is an army of professionals who make it all come together. Many have made it their career to work in the sporting events industry. I was a tiny participant in this complex ecosystem – specifically in sports marketing and hospitality.
For a local Learning Night event, I created a presentation which explained my job and shared some of my personal reflections.
Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.
Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...Helena Edelson
O'Reilly Webcast with Myself and Evan Chan on the new SNACK Stack (playoff of SMACK) with FIloDB: Scala, Spark Streaming, Akka, Cassandra, FiloDB and Kafka.
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, QlikHostedbyConfluent
Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations.
When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce.
In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...DB Tsai
Apache Spark is a new cluster computing engine offering a number of advantages over its predecessor MapReduce. In-memory cache is utilized in Apache Spark to scale and parallelize iterative algorithms which makes it ideal for large-scale machine learning. It is one of the most active open source projects in big data, surpassing even Hadoop MapReduce. In this talk, DB will introduce Spark and show how to use Spark’s high-level API in Java, Scala or Python. Then, he will show how to use MLlib, a library of machine learning algorithms for big data included in Spark to do classification, regression, clustering, and recommendation in large scale.
This presentation contains following slides,
Introduction To OLAP
Data Warehousing Architecture
The OLAP Cube
OLTP Vs. OLAP
Types Of OLAP
ROLAP V/s MOLAP
Benefits Of OLAP
Introduction - Apache Kylin
Kylin - Architecture
Kylin - Advantages and Limitations
Introduction - Druid
Druid - Architecture
Druid vs Apache Kylin
References
For any queries
Contact Us:- argonauts007@gmail.com
The Future of Hadoop: A deeper look at Apache SparkCloudera, Inc.
Jai Ranganathan, Senior Director of Product Management, discusses why Spark has experienced such wide adoption and provide a technical deep dive into the architecture. Additionally, he presents some use cases in production today. Finally, he shares our vision for the Hadoop ecosystem and why we believe Spark is the successor to MapReduce for Hadoop data processing.
Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, PresetHostedbyConfluent
Streaming data systems have been growing rapidly in importance to the modern data stack. Kafka’s kSQL provides an interface for analytic tools that speak SQL. Apache Superset, the most popular modern open-source visualization and analytics solution, plugs into nearly any data source that speaks SQL, including Kafka. Here, we review and compare methods for connecting Kafka to Superset to enable streaming analytics use cases including anomaly detection, operational monitoring, and online data integration.
Sudhir Menon, Founder and COO of SnappyData explains how you can tackle Data Gravity, Kubernetes, and strategies/best practices to run, scale, and leverage stateful containers in production.
Vectorized Deep Learning Acceleration from Preprocessing to Inference and Tra...Databricks
This talk presents how we accelerated deep learning processing from preprocessing to inference and training on Apache Spark in SK Telecom. In SK Telecom, we have half a Korean population as our customers. To support them, we have 400,000 cell towers, which generates logs with geospatial tags.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
2. Agenda
Why integrate Spark and NoSQL?
Architectural alignment
Integration “Points of Interest”
Automatic sharding and data locality
Streams: Data Replication and Spark Streaming
Predicate pushdown and global indexing
Flexible schemas and schema inference
See it in action
4. NoSQL + Spark use cases
Operations Analysis
NoSQL
Recommendations
Next gen data warehousing
Predictive analytics
Fraud detection
Catalog
Customer 360 + IOT
Personalization
Mobile applications
5. Big Data at a Glance
Couchbase Spark Hadoop
Use cases
• Operational
• Web / Mobile
• Analytics
• Machine
Learning
• Analytics
• Machine
Learning
Processing
mode
• Online
• Ad Hoc
• Ad Hoc
• Batch
• Streaming (+/-)
• Batch
• Ad Hoc (+/-)
Low latency = < 1 ms ops Seconds Minutes
Performance Highly predictable Variable Variable
Users are
typically…
Millions of
customers
100’s of analysts or
data scientists
100’s of analysts or
data scientists
Memory-centric Memory-centric Disk-centric
Big data = 10s of Terabytes Petabytes Petabytes
ANALYTICALOPERATIONAL
6. Use Case: Operationalize Analytics / ML
Hadoop
Examples: recommend content and products, spot fraud or spam
Data scientists train machine learning models
Load results into Couchbase so end users can interact with them online
Machine
Learning
Models
Data
Warehouse
Historical Data
NoSQL
7. Use Case: Operationalize ML
Model
NoSQL
node(e.g.) Training Data
(Observations)
Serving
Predictions
8. Spark connects to everything…
DCP
KV
N1QL
Views
Adapted from: Databricks – Not Your Father’s Database https://www.brighttalk.com/webcast/12891/196891
9. Use Case #2: Data Integration
RDBMSs3hdfs
Elasticsearc
h
Data engineers query data in many systems w/ one language &
runtime
Store results where needed for further use
Late binding of schemas
NoSQL
11. Full Text Search
Search for and fetch
the most relevant
records given a
freeform text string
Key-Value
Directly fetch /
store a particular
record
Query
Specify a set of criteria
to retrieve relevant data
records.
Essential in reporting.
Map-Reduce
Views
Maintain materialized
indexes of data
records, with reduce
functions for
aggregation
Data Streaming
Efficiently, quickly
stream data records to
external systems for
further processing or
integration
12. Hash Partitioned Data
Auto Sharding – Bucket And vBuckets
A bucket is a logical, unique key space
Each bucket has active & replica data sets
Each data set has 1024 virtual buckets
(vBuckets)
Each vBucket contains 1/1024th of the data
set
vBuckets have no fixed physical server
location
Mapping of vBuckets to physical servers is
called the cluster map
Document IDs (keys) always get hashed to
the same vBucket
Couchbase SDK’s lookup the vBucket
server mapping
13. N1QL Query
N1QL, pronounced “nickel”, is a SQL service with extensions specifically for
JSON
Is stateless execution, however…
Uses Couchbase’s Global Secondary Indexes.
These are sorted structures, range partitioned.
Both can run on any nodes within the cluster. Nodes with differing
services can be added and removed as needed.
14. MapReduce Couchbase Views
A JavaScript based, incremental Map-Reduce
service for incrementally building sorted B+Trees.
Runs on every node, local to the data on that node, stored locally.
Automatically merge-sorted at query time.
15. Data Streaming with DCP
A general data streaming service, Database Change Protocol.
Allows for streaming all data out and continuing, or…
Stream just what is coming in at the time of connection, or…
Stream everything out for transfer/takeover…
17. Key-Value
Direct fetching/storing
of a particular record.
Query
Specifying a set of
criteria to retrieve
relevant data records.
Essential in reporting.
Map-Reduce
Views
Maintain materialized
indexes of data
records, with reduce
functions for
aggregation.
Data Streaming
Efficiently, quickly
stream data records to
external systems for
further processing or
integration.
Full Text Search
Search for, and allow
tuning of the system to
fetch the most relevant
records given a
freeform search string.
19. What happens in Spark Couchbase KV
When 1 Spark node per CB node, the connector will use the cluster map and
push down location hints
Helpful for situations where processing is intense, like transformation
Uses pipeline IO optimization
However, not available for N1QL or Views
Round robin - can’t give location hints
Back end is scatter gather with 1 node responding
21. SparkSQL on
N1QL with Global Secondary Indexes
TableScan
Scan all of the data and return it
PrunedScan
Scan an index that matches only relevant
data to the query at hand.
PrunedFilteredScan
Scan an index that matches only relevant
data to the query at hand.
25. Predicate pushdown
Notes from implementing:
Spark assumes it’s getting
all the data, applies the predicates
Future potential optimizations
Push down all the things!
Aggregations
JOINs
Looking at Catalyst engine extensions from SAP
But, it’s not backward compatible and…
…many data sources can only push down filters
image courtesy http://allthefreethings.com/about/
27. DCP and Spark Streaming
Many system architectures rely upon streaming from the ‘operational’ data
store to other systems
Lambda architecture => store everything and
process/reprocess everything based on access
Command Query Responsibility Segregation - (CQRS)
Other reactive pattern derived systems and frameworks
28. Documents flow into the
system from outside
Documents are then
streamed down to consumers
In most common cases, flows
memory to memory
DCP and Spark Streaming
Couchbase Node
DCP
Consumer
Other Cluster
Nodes
Spark Cluster
34. Deployment Topology
Couchbase
Spark
Worker
Couchbase
Couchbase
Many small gets
Streaming with low
mutation rate
Ad hoc
Couchbase
Data Node
Data Node
Spark
Worker
Couchbase
Couchbase
Medium processing
Predictable
workloads
Plenty of overhead
on machines
Couchbase
Couchbase
Couchbase
Couchbase
Couchbase
Couchbase
Data Node
Data Node
Spark
Worker
XDCR
Heaviest processing
Workload isolation
writes
Editor's Notes
CHECK THE ORDER
Analysis side – includes various types of machine learning and analytics
Often the data warehousing includes many different sources of data
This is a popular use case for Spark because it provides the ability to go big and serve your predictions to a lot of users
Generally speaking, there’s a thin layer like node.js that gets JSON from the NoSQL system and feeds it to the user.
Imagine that user is doing something like shopping or listening to music. As they use the App, you need ridiculously low latency because they system has to do something based on that data. But you also want to send that user’s actions back to the Spark system or wherever .
Couchbase advantages include:
Fast: memory-centric, integrated cache, implicit batching from the SDK with async & flat map
Dev Convenience: Native SDKs, automatic cluster management, code your app without reference to infrastructure
Sophisticated: Query using SQL for JSON (N1QL), supports JOINs
This is like federation
TODO: Come up with a better title here.
Goal is to lay out the services that are part of Couchbase and then talk about how they fit in with a spark deployment.
Add lots of notes
TODO: Merge ideas from the next slide (hidden)
TODO: fix the colors
On a given Spark worker node, we can optimize
Internal to the Couchbase JVM Core, we pipeline operations
Amortize the responses over many operations, meaning no cost for successful operations
Efficient scheduling
Future: use the union or differences of indexes for filtering down candidates.
More powerful than traditional relational databases owing to indexing architecture.
Surprise! We can push down predicates with an O(log n) lookup
2i – super awesome, that’s how awesome we are
Relational DBs do this, and they’re not expecting this to be good. Instead, go to every node and go full retard
Surprise! We can push down predicates with an O(log n) lookup
2i – super awesome, that’s how awesome we are
Relational DBs do this, and they’re not expecting this to be good. Instead, go to every node and go full retard
Spark always applies the predicates
Spark expects it will always get the data and apply the filters
If you’re building a similar system, turn off the flag so Spark doesn’t do needlessly re-apply the filters
Optional but cool
PERFORMANCE IMPLICATIONS!!! OMG
KAFKA is most common
Matt could show his mad science demo
Limitation - Only backfill works, can’t do from point in time. This is a bug.
Can’t currently shard across spark workers since there’s no way to see this topology
Fully transparent cluster and bucket management, including direct access if needed