SlideShare a Scribd company logo
Cloud-Based Architecture and Big Data
Philip Balinov,
DevOps Engineer
SOME DEFINITIONS
►
def. CLOUD COMPUTING
l
- distributed computing over a network
l
- the ability to run a program or application on many connected computers at
the same time.
►
def. BIG DATA
l
data sets so large and complex that it becomes difficult to process using
traditional data processing applications
Retail
Finance
E-Commerce
Telecommunication
B2B
Publishing/Media
Government & NGO
Automotive
Travel
KOMFO'S CLIENTS
KOMFO PLATFORM WORKFLOW OVERVIEW
EXTERNAL PROVIDERS
MODERN COMPANY STRUCTURE
►
Agile software development: Scrum, Kanban
►
Short product feature lifecycle
►
Continuous delivery
►
Self-organizing teams
►
Cross-functional teams
AGILE DEVELOPMENT, CONTD.
►
Scrum
►
Excellent for product development
►
Short, iterative cycles (sprints), commit to deadlines
►
Self-organizing teams
►
Kanban
►
Developed at Toyota to optimize supply-chain
►
Good for ad-hoc tasks, e.g. support, bugfixing
AGILE DEVELOPMENT, CONTD.
►
Agile development requires agile operations
►
Underlying architecture must scale together with the product
►
Continuous integration
►
Quality assurance
►
Deployment
►
The solution: de-couple development and operations
►
DevOps
USE CASE
►
Fast and dependable communication with external providers
►
On-demand resource scaling
►
Flexibility
►
Storage, indexing & analysis of huge amounts of data
►
Dependability
CLOUD CONCEPTS
►
Three main service models
►
IaaS – EC2, Rackspace, Azure, HP, Oracle
►
PaaS – Google App Engine, Heroku
►
SaaS – Gmail, Wordpress.com, Salesforce, Office 365
CLOUD CONCEPTS, CONTD
►
Four deployment models
►
Public
►
Private
►
Community
►
Hybrid
CLOUD INTERNALS
SERVERS
STORAGE
EXTERNAL SERVICES
(CDN, NETWORKING, SaaS)
VMs VMs
PROS AND CONS
Dedicated Cloud
+ simplicity
+ performance
+ predictability
+ tried and tested
+ agility
+ ease of use
+ scalability
+ stability
- inflexible
- ineffective
- nightmare in case of disaster
- vendor lock-in
- blackboxed
- fluctuations
- helpless in case of disaster
CLOUD ARCHITECTURE
►
Application servers scale up and down based on load
►
Application software written for parallelism
►
Communication between nodes via messaging service
►
Write for eventual consistency
OK, so we have an (endlessly) scalable cloud app
now.
But didn't we just create a bottleneck elsewhere?
MIX & MATCH
►
Crunch numbers in the cloud
►
Application servers
►
Slow running tasks
►
Temporary services
►
Test servers
MIX & MATCH, CONTD.
►
Traditional servers for:
►
Incompatible apps (single-threaded, memory, disk intensive, specialized
hardware) do not work well in cloud environments
►
Database servers are best kept on dedicated machines in our experience
DATABASES, NOSQL
►
def. NoSQL
l
a mechanism for storage and retrieval of data that is modeled in means other
than the tabular relations used in relational databases.
DATABASES
• Use the best tool for the job depending on the task
• NoSQL Advantages
►
Social networks generate a lot of data
►
Complex interconnections, cyclical dependencies
►
Aggregations must be performed on both new and old data
►
Structure of foreign sources may change on short notice
DATABASES, NOSQL CONTD
►
Riak, Hadoop, MongoDB, Cassandra, Redis
►
In-memory dataset for faster operation
►
No predefined structure
►
Integrated sharding, load-balancing and failover
►
Versatility - can be used for anything from data storage to real-time messaging to search
indexes
DATABASES, LONG TERM STRATEGY
►
Data quickly becomes irrelevant
►
Archive it, but still be accessible
►
Online Data Warehouse solutions
►
Amazon Redshift
►
Keep Everything
►
Terabytes for pennies
WE ARE LOOKING FOR…
Senior Web Developer
Junior Web Developer
Junior QA
DevOps Engineer
Questions?

More Related Content

What's hot

Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public CloudScylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
ScyllaDB
 
From Local to Global
From Local to Global From Local to Global
From Local to Global
AWS Germany
 
Cosmos db made easy with azure functions
Cosmos db made easy with azure functionsCosmos db made easy with azure functions
Cosmos db made easy with azure functions
Luis Beltran
 
IronSource Atom - Redshift - Lessons Learned
IronSource Atom -  Redshift - Lessons LearnedIronSource Atom -  Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons Learned
Idan Tohami
 
Clustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityClustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityConSanFrancisco123
 
Shift: Real World Migration from MongoDB to Cassandra
Shift: Real World Migration from MongoDB to CassandraShift: Real World Migration from MongoDB to Cassandra
Shift: Real World Migration from MongoDB to Cassandra
DataStax
 
Build A Scalable Mobile App
Build A Scalable Mobile App Build A Scalable Mobile App
Build A Scalable Mobile App
Mohamed Aboul-Fotouh
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native Platform
Sunil Govindan
 
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
iguazio
 
Migrating to Cassandra
Migrating to CassandraMigrating to Cassandra
Migrating to Cassandra
Instaclustr
 
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
iguazio
 
The Journey To Serverless At Home24 - reflections and insights
The Journey To Serverless At Home24 - reflections and insights The Journey To Serverless At Home24 - reflections and insights
The Journey To Serverless At Home24 - reflections and insights
AWS Germany
 
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud EngineJelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Ruslan Synytsky
 
Building big data applications on AWS by Ran Tessler
Building big data applications on AWS by Ran TesslerBuilding big data applications on AWS by Ran Tessler
Building big data applications on AWS by Ran Tessler
Idan Tohami
 
An Introduction To Space Based Architecture
An Introduction To Space Based ArchitectureAn Introduction To Space Based Architecture
An Introduction To Space Based Architecture
Amin Abbaspour
 
Caching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session ICaching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session I
VMware Tanzu
 
Kafka: Legacy microservices
Kafka: Legacy microservicesKafka: Legacy microservices
Kafka: Legacy microservices
bleporini
 
RedisConf18 - Video Experience Operational Insights in Real Time.
RedisConf18 - Video Experience Operational Insights in Real Time.RedisConf18 - Video Experience Operational Insights in Real Time.
RedisConf18 - Video Experience Operational Insights in Real Time.
Redis Labs
 
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDBScylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
ScyllaDB
 
20160309-VDI for Under $100 / User
20160309-VDI for Under $100 / User20160309-VDI for Under $100 / User
20160309-VDI for Under $100 / User
Atlantis Computing
 

What's hot (20)

Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public CloudScylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
Scylla Summit 2022: ScyllaDB Cloud: Simplifying Deployment to the Public Cloud
 
From Local to Global
From Local to Global From Local to Global
From Local to Global
 
Cosmos db made easy with azure functions
Cosmos db made easy with azure functionsCosmos db made easy with azure functions
Cosmos db made easy with azure functions
 
IronSource Atom - Redshift - Lessons Learned
IronSource Atom -  Redshift - Lessons LearnedIronSource Atom -  Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons Learned
 
Clustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And AvailabilityClustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And Availability
 
Shift: Real World Migration from MongoDB to Cassandra
Shift: Real World Migration from MongoDB to CassandraShift: Real World Migration from MongoDB to Cassandra
Shift: Real World Migration from MongoDB to Cassandra
 
Build A Scalable Mobile App
Build A Scalable Mobile App Build A Scalable Mobile App
Build A Scalable Mobile App
 
Big Data on Cloud Native Platform
Big Data on Cloud Native PlatformBig Data on Cloud Native Platform
Big Data on Cloud Native Platform
 
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...
 
Migrating to Cassandra
Migrating to CassandraMigrating to Cassandra
Migrating to Cassandra
 
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018
 
The Journey To Serverless At Home24 - reflections and insights
The Journey To Serverless At Home24 - reflections and insights The Journey To Serverless At Home24 - reflections and insights
The Journey To Serverless At Home24 - reflections and insights
 
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud EngineJelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
Jelastic (PaaS + IaaS) Virtual Cluster on Google Cloud Engine
 
Building big data applications on AWS by Ran Tessler
Building big data applications on AWS by Ran TesslerBuilding big data applications on AWS by Ran Tessler
Building big data applications on AWS by Ran Tessler
 
An Introduction To Space Based Architecture
An Introduction To Space Based ArchitectureAn Introduction To Space Based Architecture
An Introduction To Space Based Architecture
 
Caching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session ICaching for Microservices Architectures: Session I
Caching for Microservices Architectures: Session I
 
Kafka: Legacy microservices
Kafka: Legacy microservicesKafka: Legacy microservices
Kafka: Legacy microservices
 
RedisConf18 - Video Experience Operational Insights in Real Time.
RedisConf18 - Video Experience Operational Insights in Real Time.RedisConf18 - Video Experience Operational Insights in Real Time.
RedisConf18 - Video Experience Operational Insights in Real Time.
 
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDBScylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
Scylla Summit 2022: Multi-cloud State for k8s: Anthos and ScyllaDB
 
20160309-VDI for Under $100 / User
20160309-VDI for Under $100 / User20160309-VDI for Under $100 / User
20160309-VDI for Under $100 / User
 

Viewers also liked

Human-powered Javascript Compression for Fun and Gummy Bears
Human-powered Javascript Compression for Fun and Gummy BearsHuman-powered Javascript Compression for Fun and Gummy Bears
Human-powered Javascript Compression for Fun and Gummy BearsRui Lopes
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
Amazon Web Services
 
AWS October Webinar Series - Introducing AWS Import / Export Snowball
AWS October Webinar Series - Introducing AWS Import / Export SnowballAWS October Webinar Series - Introducing AWS Import / Export Snowball
AWS October Webinar Series - Introducing AWS Import / Export Snowball
Amazon Web Services
 
Analyzing Mixpanel Data into Amazon Redshift
Analyzing Mixpanel Data into Amazon RedshiftAnalyzing Mixpanel Data into Amazon Redshift
Analyzing Mixpanel Data into Amazon Redshift
George Psistakis
 
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
Amazon Web Services
 
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
Amazon Web Services
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
Amazon Web Services
 

Viewers also liked (7)

Human-powered Javascript Compression for Fun and Gummy Bears
Human-powered Javascript Compression for Fun and Gummy BearsHuman-powered Javascript Compression for Fun and Gummy Bears
Human-powered Javascript Compression for Fun and Gummy Bears
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
 
AWS October Webinar Series - Introducing AWS Import / Export Snowball
AWS October Webinar Series - Introducing AWS Import / Export SnowballAWS October Webinar Series - Introducing AWS Import / Export Snowball
AWS October Webinar Series - Introducing AWS Import / Export Snowball
 
Analyzing Mixpanel Data into Amazon Redshift
Analyzing Mixpanel Data into Amazon RedshiftAnalyzing Mixpanel Data into Amazon Redshift
Analyzing Mixpanel Data into Amazon Redshift
 
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
High Performance Big Data Loading for AWS: Deep Dive and Best Practices from ...
 
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS P...
 
Crypto Options in AWS
Crypto Options in AWSCrypto Options in AWS
Crypto Options in AWS
 

Similar to Cloud & Big Data: Lessons Learnt

NoSQL and Cloud Services - Philip Balinow, Comfo
NoSQL and Cloud Services -  Philip Balinow, ComfoNoSQL and Cloud Services -  Philip Balinow, Comfo
NoSQL and Cloud Services - Philip Balinow, ComfobeITconference
 
Migrate to Microservices Judiciously!
Migrate to Microservices Judiciously!Migrate to Microservices Judiciously!
Migrate to Microservices Judiciously!
pflueras
 
Building FoundationDB
Building FoundationDBBuilding FoundationDB
Building FoundationDB
FoundationDB
 
IMS01 IMS Keynote
IMS01   IMS KeynoteIMS01   IMS Keynote
IMS01 IMS Keynote
Robert Hain
 
How to Choose a Host for a Big Data Project
How to Choose a Host for a Big Data ProjectHow to Choose a Host for a Big Data Project
How to Choose a Host for a Big Data ProjectPeak Hosting
 
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
DataStax
 
Data management in cloud computing trainee
Data management in cloud computing  traineeData management in cloud computing  trainee
Data management in cloud computing trainee
Damilola Mosaku
 
A scalable server environment for your applications
A scalable server environment for your applicationsA scalable server environment for your applications
A scalable server environment for your applicationsGigaSpaces
 
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...Splunk
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
confluent
 
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStax
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStaxWebinar | From Zero to 1 Million with Google Cloud Platform and DataStax
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStax
DataStax
 
Seamless, Real-Time Data Integration with Connect
Seamless, Real-Time Data Integration with ConnectSeamless, Real-Time Data Integration with Connect
Seamless, Real-Time Data Integration with Connect
Precisely
 
The Last Frontier- Virtualization, Hybrid Management and the Cloud
The Last Frontier-  Virtualization, Hybrid Management and the CloudThe Last Frontier-  Virtualization, Hybrid Management and the Cloud
The Last Frontier- Virtualization, Hybrid Management and the Cloud
Kellyn Pot'Vin-Gorman
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
confluent
 
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
Inside Analysis
 
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfWhy Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
DATAVERSITY
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
SpringPeople
 
Modernize Your Banking Platform with Temenos and NuoDB
Modernize Your Banking Platform with Temenos and NuoDBModernize Your Banking Platform with Temenos and NuoDB
Modernize Your Banking Platform with Temenos and NuoDB
NuoDB
 
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
Srijan Technologies
 
Public Cloud Workshop
Public Cloud WorkshopPublic Cloud Workshop
Public Cloud Workshop
Amer Ather
 

Similar to Cloud & Big Data: Lessons Learnt (20)

NoSQL and Cloud Services - Philip Balinow, Comfo
NoSQL and Cloud Services -  Philip Balinow, ComfoNoSQL and Cloud Services -  Philip Balinow, Comfo
NoSQL and Cloud Services - Philip Balinow, Comfo
 
Migrate to Microservices Judiciously!
Migrate to Microservices Judiciously!Migrate to Microservices Judiciously!
Migrate to Microservices Judiciously!
 
Building FoundationDB
Building FoundationDBBuilding FoundationDB
Building FoundationDB
 
IMS01 IMS Keynote
IMS01   IMS KeynoteIMS01   IMS Keynote
IMS01 IMS Keynote
 
How to Choose a Host for a Big Data Project
How to Choose a Host for a Big Data ProjectHow to Choose a Host for a Big Data Project
How to Choose a Host for a Big Data Project
 
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
Webinar: Dyn + DataStax - helping companies deliver exceptional end-user expe...
 
Data management in cloud computing trainee
Data management in cloud computing  traineeData management in cloud computing  trainee
Data management in cloud computing trainee
 
A scalable server environment for your applications
A scalable server environment for your applicationsA scalable server environment for your applications
A scalable server environment for your applications
 
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...
SplunkLive! Nutanix Session - Turnkey and scalable infrastructure for Splunk ...
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
 
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStax
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStaxWebinar | From Zero to 1 Million with Google Cloud Platform and DataStax
Webinar | From Zero to 1 Million with Google Cloud Platform and DataStax
 
Seamless, Real-Time Data Integration with Connect
Seamless, Real-Time Data Integration with ConnectSeamless, Real-Time Data Integration with Connect
Seamless, Real-Time Data Integration with Connect
 
The Last Frontier- Virtualization, Hybrid Management and the Cloud
The Last Frontier-  Virtualization, Hybrid Management and the CloudThe Last Frontier-  Virtualization, Hybrid Management and the Cloud
The Last Frontier- Virtualization, Hybrid Management and the Cloud
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
 
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?The Crown Jewels: Is Enterprise Data Ready for the Cloud?
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
 
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfWhy Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
 
SpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud ComputingSpringPeople - Introduction to Cloud Computing
SpringPeople - Introduction to Cloud Computing
 
Modernize Your Banking Platform with Temenos and NuoDB
Modernize Your Banking Platform with Temenos and NuoDBModernize Your Banking Platform with Temenos and NuoDB
Modernize Your Banking Platform with Temenos and NuoDB
 
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...
 
Public Cloud Workshop
Public Cloud WorkshopPublic Cloud Workshop
Public Cloud Workshop
 

Recently uploaded

GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
Aftab Hussain
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pierluigi Pugliese
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 

Recently uploaded (20)

GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 

Cloud & Big Data: Lessons Learnt

  • 1. Cloud-Based Architecture and Big Data Philip Balinov, DevOps Engineer
  • 2. SOME DEFINITIONS ► def. CLOUD COMPUTING l - distributed computing over a network l - the ability to run a program or application on many connected computers at the same time. ► def. BIG DATA l data sets so large and complex that it becomes difficult to process using traditional data processing applications
  • 6. MODERN COMPANY STRUCTURE ► Agile software development: Scrum, Kanban ► Short product feature lifecycle ► Continuous delivery ► Self-organizing teams ► Cross-functional teams
  • 7. AGILE DEVELOPMENT, CONTD. ► Scrum ► Excellent for product development ► Short, iterative cycles (sprints), commit to deadlines ► Self-organizing teams ► Kanban ► Developed at Toyota to optimize supply-chain ► Good for ad-hoc tasks, e.g. support, bugfixing
  • 8. AGILE DEVELOPMENT, CONTD. ► Agile development requires agile operations ► Underlying architecture must scale together with the product ► Continuous integration ► Quality assurance ► Deployment ► The solution: de-couple development and operations ► DevOps
  • 9. USE CASE ► Fast and dependable communication with external providers ► On-demand resource scaling ► Flexibility ► Storage, indexing & analysis of huge amounts of data ► Dependability
  • 10.
  • 11.
  • 12. CLOUD CONCEPTS ► Three main service models ► IaaS – EC2, Rackspace, Azure, HP, Oracle ► PaaS – Google App Engine, Heroku ► SaaS – Gmail, Wordpress.com, Salesforce, Office 365
  • 13. CLOUD CONCEPTS, CONTD ► Four deployment models ► Public ► Private ► Community ► Hybrid
  • 15. PROS AND CONS Dedicated Cloud + simplicity + performance + predictability + tried and tested + agility + ease of use + scalability + stability - inflexible - ineffective - nightmare in case of disaster - vendor lock-in - blackboxed - fluctuations - helpless in case of disaster
  • 16. CLOUD ARCHITECTURE ► Application servers scale up and down based on load ► Application software written for parallelism ► Communication between nodes via messaging service ► Write for eventual consistency
  • 17. OK, so we have an (endlessly) scalable cloud app now. But didn't we just create a bottleneck elsewhere?
  • 18. MIX & MATCH ► Crunch numbers in the cloud ► Application servers ► Slow running tasks ► Temporary services ► Test servers
  • 19. MIX & MATCH, CONTD. ► Traditional servers for: ► Incompatible apps (single-threaded, memory, disk intensive, specialized hardware) do not work well in cloud environments ► Database servers are best kept on dedicated machines in our experience
  • 20. DATABASES, NOSQL ► def. NoSQL l a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.
  • 21. DATABASES • Use the best tool for the job depending on the task • NoSQL Advantages ► Social networks generate a lot of data ► Complex interconnections, cyclical dependencies ► Aggregations must be performed on both new and old data ► Structure of foreign sources may change on short notice
  • 22. DATABASES, NOSQL CONTD ► Riak, Hadoop, MongoDB, Cassandra, Redis ► In-memory dataset for faster operation ► No predefined structure ► Integrated sharding, load-balancing and failover ► Versatility - can be used for anything from data storage to real-time messaging to search indexes
  • 23. DATABASES, LONG TERM STRATEGY ► Data quickly becomes irrelevant ► Archive it, but still be accessible ► Online Data Warehouse solutions ► Amazon Redshift ► Keep Everything ► Terabytes for pennies
  • 24. WE ARE LOOKING FOR… Senior Web Developer Junior Web Developer Junior QA DevOps Engineer