My presentation from the NoSQL Now 2014 conference.
Abstract
NoSQL databases including Couchbase are increasingly being selected as the backend technology for web and mobile apps. Document databases in particular are well suited for a large number of different use cases as an operational datastore.
This session provides a brief overview of Couchbase Server, a document database and its underlying distributed architecture. In addition, Dipti will present some common use cases of Couchbase with a drill down into three specific customer use cases.
Paypal – A multi data center session store
LivePerson – A scalable, real time analytics system
Orbitz – A highly available cache solution
Introduction to NoSQL (Use case, considerations to NoSQL). Presentation of Couchbase with update on Couchbase 4.5 and Couchbase mobile 1.2. Special thanks to Dipti Borkar who inspired a large part of this presentation.
Rolling presentation during Couchbase Day. Including
Introduction to NoSQL
Why NoSQL?
Introduction to Couchbase
Couchbase Architecture
Single Node Operations
Cluster Operations
HA and DR
Availability and XDCR
Backup/Restore
Security
Developing with Couchbase
Couchbase SDKs
Couchbase Indexing
Couchbase GSI and Views
Indexing and Query
Couchbase Mobile
Introduction to NoSQL (Use case, considerations to NoSQL). Presentation of Couchbase with update on Couchbase 4.5 and Couchbase mobile 1.2. Special thanks to Dipti Borkar who inspired a large part of this presentation.
Rolling presentation during Couchbase Day. Including
Introduction to NoSQL
Why NoSQL?
Introduction to Couchbase
Couchbase Architecture
Single Node Operations
Cluster Operations
HA and DR
Availability and XDCR
Backup/Restore
Security
Developing with Couchbase
Couchbase SDKs
Couchbase Indexing
Couchbase GSI and Views
Indexing and Query
Couchbase Mobile
Hear Ryan Millay, IBM Cloudant software development manager, discuss what you need to consider when moving from world of relational databases to a NoSQL document store.
You'll learn about the key differences between relational databases and JSON document stores like Cloudant, as well as how to dodge the pitfalls of migrating from a relational database to NoSQL.
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...Data Con LA
NoSQL has exploded on the developer scene promising alternatives to RDBMS that make rapidly developing, Internet scale applications easier than ever. However, as a trade off to the ease of development and scale, some of the familiarity with other well-known query interfaces such as SQL, has been lost. Until now that is...N1QL (pronounced ‘N1QL’) is a SQL like query language for querying JSON, which brings the familiarity of RDBMS back to the NoSQL world. In this session you will learn about the syntax and basics of this new language as well as Integration with the Couchbase SDKs.
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetCloudera, Inc.
YapMap is a new kind of search platform that does multi-quanta search to better understand threaded discussions. This talk will cover how HBase made it possible for two self-funded guys to build a new kind of search platform. We will discuss our data model and how we use row based atomicity to manage parallel data integration problems. We’ll also talk about where we don’t use HBase and instead use a traditional SQL based infrastructure. We’ll cover the benefits of using MapReduce and HBase for index generation. Then we’ll cover our migration of some tasks from a message based queue to the Coprocessor framework as well as our future Coprocessor use cases. Finally, we’ll talk briefly about our operational experience with HBase, our hardware choices and challenges we’ve had.
Enterprise Architect's view of Couchbase 4.0 with N1QLKeshav Murthy
Enterprise architects have to decide on the database platform that will meet various requirements: performance and scalability on one side, ease of data modeling, agile development on the other, elasticity and flexibility to handle change easily, and a database platform that integrates well with tools and within ecosystem. This presentation will highlight the challenges and approaches to solution using Couchbase with N1QL.
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder HortonworksData Con LA
Arun Murthy will be discussing the future of Hadoop and the next steps in what the big data world would start to look like in the future. With the advent of tools like Spark and Flink and containerization of apps using Docker, there is a lot of momentum currently in this space. Arun will share his thoughts and ideas on what the future holds for us.
Bio:-
Arun C. Murthy
Arun is a Apache Hadoop PMC member and has been a full time contributor to the project since the inception in 2006. He is also the lead of the MapReduce project and has focused on building NextGen MapReduce (YARN). Prior to co-founding Hortonworks, Arun was responsible for all MapReduce code and configuration deployed across the 42,000+ servers at Yahoo!. In essence, he was responsible for running Apache Hadoop’s MapReduce as a service for Yahoo!. Also, he jointly holds the current world sorting record using Apache Hadoop. Follow Arun on Twitter: @acmurthy.
Relational databases vs Non-relational databasesJames Serra
There is a lot of confusion about the place and purpose of the many recent non-relational database solutions ("NoSQL databases") compared to the relational database solutions that have been around for so many years. In this presentation I will first clarify what exactly these database solutions are, compare them, and discuss the best use cases for each. I'll discuss topics involving OLTP, scaling, data warehousing, polyglot persistence, and the CAP theorem. We will even touch on a new type of database solution called NewSQL. If you are building a new solution it is important to understand all your options so you take the right path to success.
Harmonizing Multi-tenant HBase Clusters for Managing Workload DiversityHBaseCon
Speakers: Dheeraj Kapur, Rajiv Chittajallu & Anish Mathew (Yahoo!)
In early 2013, Yahoo! introduced multi-tenancy to HBase to offer it as a platform service for all Hadoop users. A certain degree of customization per tenant (a user or a project) was achieved through RegionServer groups, namespaces, and customized configs for each tenant. This talk covers how to accommodate diverse needs to individual tenants on the cluster, as well as operational tips and techniques that allow Yahoo! to automate the management of multi-tenant clusters at petabyte scale without errors.
Speakers: Lars George and Jon Hsieh (Cloudera)
Today, there are hundreds of production HBase clusters running a multitude of applications and use cases. Many well-known implementations exercise opposite ends of the HBase's capabilities emphasizing either entity-centric schemas or event-based schemas. This talk presents these archetypes and others based on a use-case survey of clusters conducted by Cloudera's development, product, and services teams. By analyzing the data from the nearly 20,000 HBase cluster nodes Cloudera has under management, we'll categorize HBase users and their use cases into a few simple archetypes, describe workload patterns, and quantify the usage of advanced features. We'll also explain what an HBase user can do to alleviate pressure points from these fundamentally different workloads, and use these results will provide insight into what lies in HBase's future.
CBDW2014 - NoSQL Development With Couchbase and ColdFusion (CFML)Ortus Solutions, Corp
NoSQL document stores are reinventing the way we design our databases and cache layers. Couchbase server is a unique offering with unparalleled performance, automatic replication and failover. In this session, we'll talk about how to get started with Couchbase using the open source CFML SDK as well as native caching via the Railo Couchbase Extension.
Before joining Couchbase Phil has been a consultant on many different node.js and NoSQL projects working with many different languages and databases. By helping clients solve problems regarding scalability as well building completely new APIs he gained a broad knowledge of the available platforms and their tradeoffs in the big and small. He's a Developer Evangelist for Couchbase where he works to educate developers on the different parts of using a NoSQL database from mobile to big iron servers.
Hear Ryan Millay, IBM Cloudant software development manager, discuss what you need to consider when moving from world of relational databases to a NoSQL document store.
You'll learn about the key differences between relational databases and JSON document stores like Cloudant, as well as how to dodge the pitfalls of migrating from a relational database to NoSQL.
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...Data Con LA
NoSQL has exploded on the developer scene promising alternatives to RDBMS that make rapidly developing, Internet scale applications easier than ever. However, as a trade off to the ease of development and scale, some of the familiarity with other well-known query interfaces such as SQL, has been lost. Until now that is...N1QL (pronounced ‘N1QL’) is a SQL like query language for querying JSON, which brings the familiarity of RDBMS back to the NoSQL world. In this session you will learn about the syntax and basics of this new language as well as Integration with the Couchbase SDKs.
HBaseCon 2012 | Building a Large Search Platform on a Shoestring BudgetCloudera, Inc.
YapMap is a new kind of search platform that does multi-quanta search to better understand threaded discussions. This talk will cover how HBase made it possible for two self-funded guys to build a new kind of search platform. We will discuss our data model and how we use row based atomicity to manage parallel data integration problems. We’ll also talk about where we don’t use HBase and instead use a traditional SQL based infrastructure. We’ll cover the benefits of using MapReduce and HBase for index generation. Then we’ll cover our migration of some tasks from a message based queue to the Coprocessor framework as well as our future Coprocessor use cases. Finally, we’ll talk briefly about our operational experience with HBase, our hardware choices and challenges we’ve had.
Enterprise Architect's view of Couchbase 4.0 with N1QLKeshav Murthy
Enterprise architects have to decide on the database platform that will meet various requirements: performance and scalability on one side, ease of data modeling, agile development on the other, elasticity and flexibility to handle change easily, and a database platform that integrates well with tools and within ecosystem. This presentation will highlight the challenges and approaches to solution using Couchbase with N1QL.
The Future of Hadoop by Arun Murthy, PMC Apache Hadoop & Cofounder HortonworksData Con LA
Arun Murthy will be discussing the future of Hadoop and the next steps in what the big data world would start to look like in the future. With the advent of tools like Spark and Flink and containerization of apps using Docker, there is a lot of momentum currently in this space. Arun will share his thoughts and ideas on what the future holds for us.
Bio:-
Arun C. Murthy
Arun is a Apache Hadoop PMC member and has been a full time contributor to the project since the inception in 2006. He is also the lead of the MapReduce project and has focused on building NextGen MapReduce (YARN). Prior to co-founding Hortonworks, Arun was responsible for all MapReduce code and configuration deployed across the 42,000+ servers at Yahoo!. In essence, he was responsible for running Apache Hadoop’s MapReduce as a service for Yahoo!. Also, he jointly holds the current world sorting record using Apache Hadoop. Follow Arun on Twitter: @acmurthy.
Relational databases vs Non-relational databasesJames Serra
There is a lot of confusion about the place and purpose of the many recent non-relational database solutions ("NoSQL databases") compared to the relational database solutions that have been around for so many years. In this presentation I will first clarify what exactly these database solutions are, compare them, and discuss the best use cases for each. I'll discuss topics involving OLTP, scaling, data warehousing, polyglot persistence, and the CAP theorem. We will even touch on a new type of database solution called NewSQL. If you are building a new solution it is important to understand all your options so you take the right path to success.
Harmonizing Multi-tenant HBase Clusters for Managing Workload DiversityHBaseCon
Speakers: Dheeraj Kapur, Rajiv Chittajallu & Anish Mathew (Yahoo!)
In early 2013, Yahoo! introduced multi-tenancy to HBase to offer it as a platform service for all Hadoop users. A certain degree of customization per tenant (a user or a project) was achieved through RegionServer groups, namespaces, and customized configs for each tenant. This talk covers how to accommodate diverse needs to individual tenants on the cluster, as well as operational tips and techniques that allow Yahoo! to automate the management of multi-tenant clusters at petabyte scale without errors.
Speakers: Lars George and Jon Hsieh (Cloudera)
Today, there are hundreds of production HBase clusters running a multitude of applications and use cases. Many well-known implementations exercise opposite ends of the HBase's capabilities emphasizing either entity-centric schemas or event-based schemas. This talk presents these archetypes and others based on a use-case survey of clusters conducted by Cloudera's development, product, and services teams. By analyzing the data from the nearly 20,000 HBase cluster nodes Cloudera has under management, we'll categorize HBase users and their use cases into a few simple archetypes, describe workload patterns, and quantify the usage of advanced features. We'll also explain what an HBase user can do to alleviate pressure points from these fundamentally different workloads, and use these results will provide insight into what lies in HBase's future.
CBDW2014 - NoSQL Development With Couchbase and ColdFusion (CFML)Ortus Solutions, Corp
NoSQL document stores are reinventing the way we design our databases and cache layers. Couchbase server is a unique offering with unparalleled performance, automatic replication and failover. In this session, we'll talk about how to get started with Couchbase using the open source CFML SDK as well as native caching via the Railo Couchbase Extension.
Before joining Couchbase Phil has been a consultant on many different node.js and NoSQL projects working with many different languages and databases. By helping clients solve problems regarding scalability as well building completely new APIs he gained a broad knowledge of the available platforms and their tradeoffs in the big and small. He's a Developer Evangelist for Couchbase where he works to educate developers on the different parts of using a NoSQL database from mobile to big iron servers.
Couchbase Chennai Meetup: Developing with Couchbase- made easyKarthik Babu Sekar
This session provided an overview of Couchbase Solutions and whats latest and greatest in the new release. This session also talks about how easy is to develop with Couchbase and query the database
Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !! Karthik Babu Sekar
This session provided an overview of Couchbase Solutions and whats latest and greatest in the new release. This session also talks about how easy is to develop with Couchbase and query the database
Webinar: Emerging Trends in Data Architecture – What’s the Next Big Thing?DATAVERSITY
The migration to cloud-based data architectures continues at a rapid pace, including databases and data management. Oracle databases are part of this trend, and during this webinar you will learn how to automate the provisioning and management of Oracle databases so that you can deliver an “as-a-service” experience with 1-click simplicity. Experts will walk you through the process of:
· Using Kubernetes to deliver a production-ready
solution for your Oracle-based applications
· Turbocharging your data infrastructure using
cloud-native architecture
· Improving the agility and efficiency of your BI
and Data Operation teams, Developers, and Data Scientists
· Defining the business impact and benefits of
cloud-based Oracle solutions
We are proud to announce the release of Neo4j 3.2. This version marks an expansion in global scale, performance and refinement. It signals that the next generation of graph-powered internet applications, generating personalized content or finding coordinated malfeasance, will span the globe. This webinar detailing the themes behind Neo4j version 3.2, including: enterprise scale for global internet applications, while refining its enterprise governance capabilities and investing in performance improvements up and down the native graph stack.
Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)VMware Tanzu
Recorded at SpringOne2GX 2013 in Santa Clara, CA
Speaker: Adam Shook
This session assumes absolutely no knowledge of Apache Hadoop and will provide a complete introduction to all the major aspects of the Hadoop ecosystem of projects and tools. If you are looking to get up to speed on Hadoop, trying to work out what all the Big Data fuss is about, or just interested in brushing up your understanding of MapReduce, then this is the session for you. We will cover all the basics with detailed discussion about HDFS, MapReduce, YARN (MRv2), and a broad overview of the Hadoop ecosystem including Hive, Pig, HBase, ZooKeeper and more.
Learn More about Spring XD at: http://projects.spring.io/spring-xd
Learn More about Gemfire XD at:
http://www.gopivotal.com/big-data/pivotal-hd
Presto – Today and Beyond – The Open Source SQL Engine for Querying all Data...Dipti Borkar
Born at Facebook, Presto is an open source high performance, distributed SQL query engine. With the disaggregation of storage and compute, Presto was created to simplify querying of all data lakes - cloud data lakes like S3 and on premise data lakes like HDFS. Presto's high performance and flexibility has made it a very popular choice for interactive query workloads on large Hadoop-based clusters as well as AWS S3, Google Cloud Storage and Azure blob store. Today it has grown to support many users and use cases including ad hoc query, data lake house analytics, and federated querying. In this session, we will give an overview on Presto including architecture and how it works, the problems it solves, and most common use cases. We'll also share the latest innovation in the project as well as the future roadmap.
Silicon Valley NoSQL Meetup - Nov 2012. View with animations: video version here: https://vimeo.com/54691785
http://www.meetup.com/Silicon-Valley-NoSQL/events/88257222/
For more information visit: www.couchbase.com
Navigating the Transition from relational to NoSQL - CloudCon Expo 2012Dipti Borkar
For more deep NoSQL content from Couchbase, check out http://www.couchbase.com/webinars
NoSQL databases have emerged as a better match than relational systems for modern interactive applications, offering cost-effective data management at “Big Data” scale. But there are significant differences between structured and schema-less database technology. What should architects and technical managers know as they explore NoSQL solutions for their teams?
In this workshop you will learn:
- How to evaluate NoSQL (both technical advantages and limitations) as a potential data management approach
- Critical differences between NoSQL and RDBMS for designing, building and running production applications
- Ideal use cases for NoSQL technology and sample reference architectures
Couchbase Server and IBM BigInsights: One + One = ThreeDipti Borkar
Session presented at CouchConf San Francisco
http://www.couchbase.com/couchconf-san-francisco
Frequently the terms NoSQL and Big Data are used as synonyms. While both technologies divert from the traditional RDBMS data model and spread data across clusters of servers, the “problems” these technologies address are quite different. Hadoop, is focused on data analysis – gleaning insights from large volumes of data. NoSQL databases, focus on interactive applications – delivering high-performance, cost-effective data management for massive number of users. In this session, we share how IBM BigInsights and Couchbase Server can used together to build better applications.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
3. Overview
Couchbase
offers
a
full
range
of
Data
Management
solu7ons
High
Availability
Cache
Key
Value
Document
Mobile
device
SSN:
400
658
9993
Pass:
******
Pass:
******
4. NoSQL
Database
Considera7ons
Easy
Scalability
Consistent
High
Performance
Flexible
Data
Model
Always
On
24x7x365
Grow
cluster
without
applica<on
changes,
without
down<me
when
needed
Always
awesome
experience
for
your
applica<on
users
The
sun
never
sets
on
the
Internet,
your
applica<on
needs
the
database
to
always
serve
data
Keep
developers
produc<ve
and
allow
fast
and
easy
addi<on
of
new
features
JSON
JSON
JSON
JSONJSON
PERFORMANCE
6. 3
3
2
Single
node
–
Couchbase
Write
Opera7on
Managed
Cache
Disk
Queue
Disk
Replica<on
Queue
App
Server
Couchbase
Server
Node
To
other
node
Doc
1
Doc
1
Doc
1
7. 3
3
2
Single
node
–
Couchbase
Read
Opera7on
Managed
Cache
Disk
Queue
Disk
Replica<on
Queue
App
Server
Couchbase
Server
Node
To
other
node
Doc
1
Get
Doc
1
Doc
1
Doc
1
8. Auto
Sharding
and
Cluster
Map
Hash
func7on
(KEY)
vB1
vB2
vB3
vB4
vB5
vB6
Physical
servers
A
B
C
More
scalability
required
Add
node
Logical
Par77ons
Cluster
Map
New
Cluster
Map
9. Couchbase
Server
Cluster
Basic
Opera7on
User
Configured
Replica
Count
=
1
Read/write/update
Ac<ve
SERVER
1
Ac<ve
SERVER
2
Ac<ve
SERVER
3
App
Server
1
COUCHBASE
Client
Library
CLUSTER
MAP
COUCHBASE
Client
Library
CLUSTER
MAP
App
Server
2
Doc
5
Doc
2
Doc
9
Doc
Doc
Doc
Doc
4
Doc
7
Doc
8
Doc
Doc
Doc
Doc
1
Doc
3
Doc
6
Doc
Doc
Doc
Replica
Replica
Replica
Doc
4
Doc
1
Doc
8
Doc
Doc
Doc
Doc
6
Doc
3
Doc
2
Doc
Doc
Doc
Doc
7
Doc
9
Doc
5
Doc
Doc
Doc
• Docs
distributed
evenly
across
servers
• Each
server
stores
both
ac7ve
and
replica
docs
Only
one
server
ac<ve
at
a
<me
• Client
library
provides
app
with
simple
interface
to
database
• Cluster
map
provides
map
to
which
server
doc
is
on
App
never
needs
to
know
• App
reads,
writes,
updates
docs
• Mul7ple
app
servers
can
access
same
document
at
same
7me
10. Add
Nodes
to
Cluster
SERVER
4
SERVER
5
Replica
Ac<ve
Replica
Ac<ve
Read/write/update
App
Server
1
COUCHBASE
Client
Library
CLUSTER
MAP
COUCHBASE
Client
Library
CLUSTER
MAP
App
Server
2
User
Configured
Replica
Count
=
1
Couchbase
Server
Cluster
Ac<ve
SERVER
1
Doc
5
Doc
2
Doc
9
Doc
Doc
Doc
Replica
Doc
4
Doc
1
Doc
8
Doc
Doc
Doc
Ac<ve
SERVER
2
Doc
4
Doc
7
Doc
8
Doc
Doc
Doc
Replica
Doc
6
Doc
3
Doc
2
Doc
Doc
Doc
Ac<ve
SERVER
3
Doc
1
Doc
3
Doc
6
Doc
Doc
Doc
Replica
Doc
7
Doc
9
Doc
5
Doc
Doc
Doc
Read/write/update
• Two
servers
added
with
one-‐click
opera7on
• Docs
automa7cally
rebalance
across
cluster
Even
distribu<on
of
docs
Minimum
doc
movement
• Cluster
map
updated
• App
database
calls
now
distributed
over
larger
number
of
servers
11. Fail
Over
Node
User
Configured
Replica
Count
=
1
SERVER
4
SERVER
5
Replica
Ac<ve
Replica
Ac<ve
App
Server
1
COUCHBASE
Client
Library
CLUSTER
MAP
COUCHBASE
Client
Library
CLUSTER
MAP
App
Server
2
Couchbase
Server
Cluster
Ac<ve
SERVER
1
Doc
5
Doc
2
Doc
9
Doc
Doc
Doc
Replica
Doc
4
Doc
1
Doc
8
Doc
Doc
Doc
Ac<ve
SERVER
2
Doc
4
Doc
7
Doc
8
Doc
Doc
Doc
Replica
Doc
6
Doc
3
Doc
2
Doc
Doc
Doc
Ac<ve
SERVER
3
Doc
1
Doc
3
Doc
6
Doc
Doc
Doc
Replica
Doc
7
Doc
9
Doc
5
Doc
Doc
Doc
• App
servers
accessing
docs
• Requests
to
Server
3
fail
• Cluster
detects
server
failed
– Promotes
replicas
of
docs
to
ac<ve
– Updates
cluster
map
• Requests
for
docs
now
go
to
appropriate
server
• Typically
rebalance
would
follow
Doc
1
Doc
3
Doc
12. Couchbase
Server
Cluster
Indexing
and
Querying
User
Configured
Replica
Count
=
1
Ac<ve
SERVER
1
SERVER
3
App
Server
1
COUCHBASE
Client
Library
CLUSTER
MAP
COUCHBASE
Client
Library
CLUSTER
MAP
App
Server
2
Doc
5
Doc
2
Doc
9
Doc
Doc
Doc
Ac<ve
Doc
1
Doc
3
Doc
6
Doc
Doc
Doc
Replica
Doc
4
Doc
1
Doc
8
Doc
Doc
Doc
Ac<ve
SERVER
2
Doc
4
Doc
7
Doc
8
Doc
Doc
Doc
Replica
Doc
6
Doc
3
Doc
2
Doc
Doc
Doc
Replica
Doc
7
Doc
9
Doc
5
Doc
Doc
Doc
• Indexing
work
is
distributed
amongst
nodes
• Large
data
set
possible
• Parallelize
the
effort
• Each
node
has
index
for
data
stored
on
it
• Queries
combine
the
results
from
required
nodes
Query
13.
ACTIVE
SERVER
1
RAM
DISK
Doc
Doc
2
Doc
9
Doc
Doc
Doc
ACTIVE
SERVER
2
RAM
DISK
Doc
Doc
Doc
Doc
Doc
Doc
ACTIVE
SERVER
3
RAM
DISK
Doc
Doc
Doc
Doc
Doc
Doc
Cross
Data
Center
Replica7on
(XDCR)
COUCHBASE
SERVER
CLUSTER
NYC
DATA
CENTER
COUCHBASE
SERVER
CLUSTER
SF
DATA
CENTER
ACTIVE
SERVER
1
RAM
DISK
Doc
Doc
2
Doc
9
Doc
Doc
Doc
ACTIVE
SERVER
2
RAM
DISK
Doc
Doc
Doc
Doc
Doc
Doc
ACTIVE
SERVER
3
RAM
DISK
Doc
Doc
Doc
Doc
Doc
Doc
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
15. High-‐Availability
Caching
RDBMS
Applica7on
Layer
User
Requests
Cache
Misses
and
Write
Requests
Read-‐Write
Requests
Couchbase
Distributed
Cache
Use
Case
1
16. • Applica<on
objects
• Popular
search
query
results
• Session
informa<on
• Heavily
accessed
web
landing
pages
High-‐Availability
Caching
• Speed
up
RDBMS
• Consistently
low
response
<mes
for
document
/
key
lookups
• High-‐availability
24x7x365
• Replacement
for
en<re
caching
<er
Data
cached
in
Couchbase?
Applica7on
characteris7c
Use
Case
1
hap://www.Look.PopularSearchWuerycom
Look
Something
Search
WEB
%
of
clicks
%
of
clicks
something
56.3
28
DoSomething.com
13.4
25.08
SomethingFishy.org
9.8
14.68
Popular
Couchbase,
Inc.
Confiden<al
17. High-‐Availability
Caching
• Low
latency
in
sub-‐milliseconds
with
consistently
high
read
/
write
throughput
using
built-‐in
cache
• Always-‐on
opera7ons
even
for
database
upgrades
and
maintenance
with
zero
down
7me
Why
NoSQL
and
Couchbase?
Use
Case
1
Couchbase,
Inc.
Confiden<al
19. Session
Store
• Extremely
fast
access
to
session
data
using
unique
session
ID
• Easy
scalability
to
handle
fast
growing
number
of
users
and
user-‐generated
data
• Always-‐on
func<onality
for
global
user
base
Applica7on
characteris7c
Use
Case
2
• Session
values
or
Cookies
(stored
as
key-‐value
pairs)
• Examples
include:
items
in
a
shopping
cart,
flights
selected,
search
results,
etc.
Data
stored
in
Couchbase?
Couchbase,
Inc.
Confiden<al
20. Session
Store
• Low
latency
in
sub-‐milliseconds
with
consistently
high
read
/
write
throughput
for
session
data
via
the
built-‐in
object-‐level
cache
• Linear
throughput
scalability
to
grow
the
database
as
user
and
data
volume
grow
• Always-‐on
opera7ons
even
par7cularly
high
availability
using
Couchbase
replica7on
and
failover
• Intra
cluster
and
cross
cluster
(XDCR)
replica7on
for
globally
distributed
ac7ve-‐ac7ve
plagorm
Why
NoSQL
and
Couchbase?
Use
Case
2
Couchbase,
Inc.
Confiden<al
22. hap://www.ProfileStore.com
e
enim
nec
felis
rhoncus,
ac
volutpat
magna
blandit.
Nunc
facilisis
turpis
eget
dolor
mollis,
id
<ncidunt
dui
mais.
Nunc
sodales
elementum
turpis,
vel
interdum
ante
congue
quis.
Pellentesque
habitant
morbi
tris<que
senectus
et
netus
et
malesuada
fames
ac
turpis
egestas.
Aliquam
erat
volutpat.
Nullam
suscipit
diam
nec
tortor
pharetra,
vitae
adipiscing
dolor
pre<um.
Integer
ac
porta
tortor.
Ves<bulum
imperdiet
quam
laoreet
nisl
scelerisque,
a
tempus
tortor
<ncidunt.
Mauris
suscipit
dui
ac
urna
dignissim,
vitae
aliquet
velit
convallis.
Phasellus
lobor<s
felis
eu
magna
vulputate
dapibus.
Ut
ornare
ut
quam
a
vulputat
ullam
et
dui
odio.
Nulla
pharetra,
velit
ac
convallis
semper,
dolor
turpis
porta
nunc,
in
egestas
mauris
leo
a
nisi.
Pellentesque
fringilla
sagiis
magna
vitae
imperdiet.
Mauris
ac
leo
ut
tellus
aliquet
interdum.
Interdum
et
malesuada
fames
ac
ante
ipsum
primis
in
faucibus.
Nunc
cursus
odio
sit
amet
elit
mollis,
et
sollicitudin
lacus
accumsan.
Nulla
facilisi.
Fusce
et
vehicula
sem.
Curabitur
interdum
ves<bulum
nulla
id
accumsan.
Integer
ut
tortor
in
ligula
semper
vehicula.
Ves<bulum
ut
nibh
ultrices,
venena<s
metus
at,
adipiscing
ipsum.
Donec
quis
consequat
lectus.
Class
aptent
taci<
sociosqu
ad
litora
torquent
per
conubia
nostra,
per
inceptos
himenaeos.
Donec
a
diam
tempus,
aliquet
ipsum
eu,
ves<bulum
sapien.
Donec
eleifend
lectus
sit
amet
luctus
facilisis.
Morbi
poritor,
orci
sit
amet
placerat
tempus,
nisi
justo
dictum
augue,
ac
dignissim
elit
enim
eget
dolor.
Praesent
pulvinar
ipsum
arcu,
eu
posuere
eros
luctus
nec.
Ves<bulum
odio
eros,
ultrices
non
metus
sit
amet,
tris<que
malesuada
augue.
Pellentesque
lacinia
dolor
nec
diam
eleifend
mollis.
Ves<bulum
sit
amet
ultrices
diam.
Aliquam
lacinia
accumsan
eros
id
hendrerit.
Cras
placerat
laoreet
urna
scelerisque
rutrum.
Duis
ornare
mi
ac
augue
varius,
sit
amet
accumsan
leo
lacinia.
Vivamus
nec
egestas
neque.
Quisque
interdum
enim
moles<e
urn.
turpis
eget
dolor
mollis,
id
<ncidunt
dui
mais.
Nunc
sodales
elementum
turpis,
vel
interdum
ante
congue
quis.
Pellentesque
habitant
morbi
tris<que
senectus
et
netus
et
malesuada
Welcome
back
Laura!
You
have
3
items
in
your
shopping
cart
wai<ng
for
you.
LOGIN
ID:
PASS:
Globally
Distributed
User
Profile
Store
• Extremely
fast
access
to
individual
profiles
• Always
online
system
as
mul<ple
applica<ons
access
user
profiles
• Flexibility
to
add
and
update
user
aaributes
• Easy
scalability
to
handle
fast
growing
number
of
users
• User
profile
with
unique
ID
• User
seing
/
preferences
• User’s
network
• User
applica<on
state
Data
stored
in
Couchbase?
Applica7on
characteris7c
Use
Case
3
Laura930
********
23. Globally
Distributed
User
Profile
Store
• Low
latency
and
high
throughput
for
very
quick
lookups
for
millions
of
concurrent
users
using
built-‐in
cache
• Intra
cluster
and
cross
cluster
(XDCR)
replica7on
for
high
availability
and
disaster
recovery
• Ac7ve-‐ac7ve
geo-‐distributed
system
to
handle
globally
distributed
user
base
• Online
admin
opera7ons
eliminate
system
down7me
Why
NoSQL
and
Couchbase?
Use
Case
3
24. Data
Aggrega7on
• Flexibility
to
store
any
kind
of
content
• Flexibility
to
handle
schema
changes
• Full-‐text
Search
across
data
set
• High
speed
data
inges<on
• Scales
horizontally
as
more
content
gets
added
to
the
system
• Social
media
feeds:
Twiaer,
Facebook,
LinkedIn
• Blogs,
news,
press
ar<cles
• Data
service
feeds:
Hoovers,
Reuters
• Data
form
other
systems
Data
stored
in
Couchbase?
Applica7on
characteris7c
Use
Case
4
in
F
t
NEWS
Blog
25. Data
Aggrega7on
• JSON
provides
schema
flexibility
to
store
all
types
of
content
and
metadata
• Fast
access
to
individual
documents
via
built-‐in
cache,
high
write
throughput
• Indexing
and
querying
provides
real-‐7me
analy7cs
capabili7es
across
dataset
• Integra7on
with
Elas7cSearch
for
full-‐text
search
• Ease
of
scalability
ensures
that
the
data
cluster
can
be
grown
seamlessly
as
the
amount
of
user
and
ad
data
grows
Why
NoSQL
and
Couchbase?
Use
Case
4
26. Content
and
Metadata
Store
Use
Case
5
Content
and
Metadata
Nature,
Field,
Summer,
Farm,
Sky,
Environment,
Landscaped,
Gr
ass,
Green,Blue,
Oilseed,
Rape,
Agriculture,
Scenics,
Land,
Spring,
Non-‐Urban
Scene,Environmental,
Conserva<on,
Sun,
Meadow,
Horizon,
Season,
Cloud,
Landscapes,
Travel
Loca<ons,
Pasture,
Cul<vated
Land,
Stratoshpere,
cloudy
day,
Oliseed
Rape,
Rural
Scene,
Vibrant
Color,
No
People,
Beauty
In
Nature,Gold,
Color
Image,
Beauty,
Idyllic,
Mul<colored,
Yellow,
Colors,
Cloudscape,
Outdoors,
Plant,
Sunlight,
Horizon
Over
Land
27. Content
and
Metadata
Store
• Flexibility
to
store
any
kind
of
content
• Fast
access
to
content
metadata
(most
accessed
objects)
and
content
• Full-‐text
Search
across
data
set
• Scales
horizontally
as
more
content
gets
added
to
the
system
• Content
metadata
• Content:
Ar<cles,
text
• Landing
pages
for
website
• Digital
content:
eBooks,
magazine,
research
material
Data
stored
in
Couchbase?
Applica7on
characteris7c
Use
Case
5
hap://www.LandingPage.com
ebook
Mag
28. Content
and
Metadata
Store
• Fast
access
to
metadata
and
content
via
object-‐managed
cache
• JSON
provides
schema
flexibility
to
store
all
types
of
content
and
metadata
• Indexing
and
querying
provides
real-‐7me
analy7cs
capabili7es
across
dataset
• Integra7on
with
Elas7cSearch
for
full-‐text
search
• Ease
of
scalability
ensures
that
the
data
cluster
can
be
grown
seamlessly
as
the
amount
of
user
and
ad
data
grows
Why
NoSQL
and
Couchbase?
Use
Case
5
30.
User
Profile,
Ad
Targe2ng
&
Real-‐Time
Analy2cs
• Company
Global
Leader
in
Online
Payments
132m
Ac<ve
Accounts,
193
Markets,
25
Currencies
• Scalability
and
Performance
Requirements
300m
to
1bn
documents
with
3
Tb
to
10TB
Billions
of
requests
and
sub
200ms
response
<mes
access
to
JSON
documents
Read/write
mix
50/50
with
5ms
latency
• Exis7ng
Database
Infrastructure
Mul<ple
Tiers
–
Separate
caching
and
durable
store
MySQL,
Oracle,
Terracoaa,
Coherence
• Pain
Real-‐Time
Access
to
Iden<ty
Mapping
–
eBay
ID,
PayPal
ID,
Social
ID,
3rd
Party
ID,
Email
Performance
–
Ad
needs
to
be
served
in
200ms
Cost
–
Mul<ple
<ers
for
caching
and
durability
Highly
Available
–
Across
large
clusters
and
across
data
centers
• Couchbase
Benefits
Performance
–
Reduced
latency
with
5ms
access
<mes
Cost
–
Consolida<on
of
database
and
cache
layers
Cross
Data
Center
Availability
+
+
+
31. Why
couchbase?
§ Data
volume
• Online
system
;
300M
–
1B
documents
@
10k
value
size
;
3-‐10TB
total
storage
§ Data
Access
• Distributed
caching
• Persistence
§ Data
Structure
• Flexible
&
Schemaless
§ Read/Write
• 50%
read/50%
write
• Low
latency
<
10
msec
§ Par77oning
§ Replica<on
§ Auto
Healing
§ Availability
and
scalability
• Resilient
• Mul<
data
center
–
DR/BCP
• Linearly
Scalable
32. Use
cases
at
PayPal
• Ad
Tech
targe7ng
• Cookie
infrastructure
• Real
7me
analy7cs
33. Cookie
architecture
CookieService
Couchbase
DC
A
Couchbase
DC
B
Front
Tier
Interac<on
Channels
Applica7on
Cookie
Libraries
Mid
Tier
Data
Service
-‐ Key
Value
-‐ Cache
Interface
-‐ Couchbase
Client
Data
Tier
XDCR
35. DEPLOYMENT MODEL
A CB
Cookie
Service
Cookie
Service
Cookie
Service
XDCR
ACTIVE
ACTIVE
PASSIVE
AVAILABILITY
REDUNDANCY
DISASTER
RECOVERY
WRITE
READ
36.
High
Performance
Caching
• Company
Leading
online
travel
company
• Scalability
and
Performance
Requirements
11
Clusters/100
Nodes
Over
3TB
of
Data
149,000
Ops/
sec
• Exis7ng
Database
Infrastructure
Rela<onal
Database
technology,
Terracota
• Pain
Scalability/Capacity
Planning
–
Cannot
be
planned.
Dependent
onexternal
factors
Scalability
–
Complex
and
<me
consuming
scaleout
Performance
–
Caching
too
complex.
Weeks
of
planning/hours
of
down<me
Cost
–
Mul<ple
<ers
of
hardware
for
database
and
caching
• Couchbase
Benefits
Scalability
–
Over
70
Nodes
with
simple
scaleout
in
minutes
not
hours
Performance
–
Improved
response
<mes
by
up
to
47%
with
consistent
3ms
to
4ms
response
Cost
-‐
Consolidate
caching
and
database
<ers
–
less
machines,
power,
cooling,
footprint
–
drama<c
savings
Dynamic
schema
change
–
Drama<cally
reduced
down<me
37. High
Availability
Cache
• 11
Clusters
(4
mirrored)
100
nodes
• >
3
TB
of
data
• ~430m
objects
(146m
in
largest)
• Total
ops/sec
~
75k
*149k
with
HA
38. Use
Case
#1
• Content
HTML
Image
Links
HA
caches
XDCR
40.
Real
7me
analy7cs
• Company
Leading
cloud
company
–
allows
enterprises
to
connect
in
real-‐<me
with
their
customers
via
chat,
voice,
and
content
delivery
• Scalability
and
Performance
Requirements
13TB/Month
20m
engagements/month
1.8bn
sessions/month
• Exis7ng
Database
Infrastructure
MySQL
• Pain
Scalability
Performance
–
Batch
analy<cs
and
real-‐<me
access
to
customer
profiles
Cross
Data
Center
Replica<on
–
4
data
centers
• Couchbase
Benefits
Scalability
Performance
–
Mixed
read/
write
with
very
high
throughput
Document
Store
–
Ease
of
Development
+
41. Use
Case:
3rd
party
data
aggrega7on
with
analy7cs
Real
<me
Analy<cs
for
LivePerson's
customers
LiveEngage
DASHBOARD
43. Requirements Requirements Requirements
• High
throughput,
really
fast
• Linear
scale
• Searchable
(Views
and
M/R)
• Supports
both
K/V
&
Document
store
• Cross
data
center
replica<on
• “Always
on”,
Resilience
solu<on
The Problem
13
TB
per
month
~1
PB
In
total
1.8
B
Visits
per
month
VOLUME