The document discusses the pros and cons of using public cloud computing services versus hosting infrastructure internally for a new startup. Some advantages mentioned include flexibility, avoiding large upfront capital expenditures, and the ability to scale resources up and down as needed. Disadvantages include public cloud services becoming more expensive than internal hosting at large scale, inefficient resource ratios for some workloads, and high costs for intensive disk and SSD usage. The document aims to provide considerations for a startup evaluating whether to use public cloud services.
What if you could get blazing fast queries on your data without having to be on call for a giant, expensive database? By picking the right file format for your data, you can store your data on disk in the cloud and still get the performance you need for modern analytics. We'll discuss benchmarks of four different data storage formats: Parquet, ORC, Avro, and traditional character-separated files like CSV. We'll cover what they are, how they work at a bits-and-bytes level, and why you might choose each one for your use case.
FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...Mohamed Sayed
Using Open Source and Cloud Computing principles, these slides walk through the architectural patterns for building scalable cloud services. The second part of the presentation focuses on profiling common geolocation tasks like importing large datasets and rendering map tiles.
Top 5 mistakes when writing Spark applicationsmarkgrover
This is a talk given at Advanced Spark meetup in San Francisco (http://www.meetup.com/Advanced-Apache-Spark-Meetup/events/223668878/). It focusses on common mistakes when writing Spark applications and how to avoid them.
AWS vs Azure vs Google Cloud Storage Deep DiveRightScale
Cloud services keep evolving, and cloud storage is no different. It can be difficult to keep up to date with the latest from each cloud provider and understand how they compare. We’ll drill down on object, block, archival, and file storage for the leading public clouds. We’ll also compare prices for a variety of storage scenarios.
What if you could get blazing fast queries on your data without having to be on call for a giant, expensive database? By picking the right file format for your data, you can store your data on disk in the cloud and still get the performance you need for modern analytics. We'll discuss benchmarks of four different data storage formats: Parquet, ORC, Avro, and traditional character-separated files like CSV. We'll cover what they are, how they work at a bits-and-bytes level, and why you might choose each one for your use case.
FOSS4G In The Cloud: Using Open Source to build Cloud based Spatial Infrastru...Mohamed Sayed
Using Open Source and Cloud Computing principles, these slides walk through the architectural patterns for building scalable cloud services. The second part of the presentation focuses on profiling common geolocation tasks like importing large datasets and rendering map tiles.
Top 5 mistakes when writing Spark applicationsmarkgrover
This is a talk given at Advanced Spark meetup in San Francisco (http://www.meetup.com/Advanced-Apache-Spark-Meetup/events/223668878/). It focusses on common mistakes when writing Spark applications and how to avoid them.
AWS vs Azure vs Google Cloud Storage Deep DiveRightScale
Cloud services keep evolving, and cloud storage is no different. It can be difficult to keep up to date with the latest from each cloud provider and understand how they compare. We’ll drill down on object, block, archival, and file storage for the leading public clouds. We’ll also compare prices for a variety of storage scenarios.
AWS Summit London 2014 | Maximising EC2 and EBC Performance (400)Amazon Web Services
This advanced technical session is ideal for customers that are looking to maximise the performance of AWS Elastic Block Store (EBS) storage to support workloads with demanding IO performance requirements. If you need to run high IO workloads on EBS such as NoSQL or RBDMS systems then attend this session to find out how to optimise your EBS configuration to enable this.
All these large data sets are so big its difficult to manage with traditional tools. Distributing computing is an approach to solve that problem! First the data needs to be mapped, then it can be analyzed or reduced.
Day 3 - Maintaining Performance & Availability While Lowering Costs with AWSAmazon Web Services
AWS provides you several pricing options that can help you significantly reduce your overall IT cost, including On-Demand Instances, Spot Instances, and Reserved Instances. This session covers high-level architectures and when to use and not to use each of the pricing models for components of those architectures. We walk through several customer examples to illustrate when to use each pricing option. Additionally, we walk through tools that may be useful to determine when to use each pricing model. This session is aimed at technically savvy managers and engineers who need to reduce their cloud spending.
Reasons to attend:
- Learn about Reserved Instances, On-Demand Instances and Spot Instances.
- Discover ways of running more for less in Amazon EC2.
- If you are already running a workload in AWS, attend this webinar to learn how to run the same workload at reduced costs.
WKS404 7 Things You Must Know to Build Better Alexa SkillsAmazon Web Services
As we add thousands of skills to the skills store our developers have uncovered some basic and more complex tips for building better skills. Whether you are new to Alexa Skill development or if you have created skills that are live today, this session will help you understand and learn best practices. During this session, you’ll build an Alexa skill using more advanced VUI concepts and we’ll cover how to use AWS services like dynamoDB and S3 to implement the best practices we cover.
How to scale up, out or down in Windows Azure - WebinarCommon Sense
Webinar presented on Jan 26th 2011 by Juan De Abreu.
Learn how to achieve:
• Scalability linear scale, scale up vs. scale out, choose VM sizes
• Storage Cache
• Elasticity, scale out, scale back and automation of scaling
Intended for:CIOs, CTOs, IT Managers, IT Developers, Lead Developers
More Nines for Your Dimes: Improving Availability and Lowering Costs using Au...Amazon Web Services
Running your Amazon EC2 instances in Auto Scaling groups allows you to improve your application's availability right out of the box. Auto Scaling replaces impaired or unhealthy instances automatically to maintain your desired number of instances (even if that number is one). You can also use Auto Scaling to automate the provisioning of new instances and software configurations as well as to track of usage and costs by app, project, or cost center. Of course, you can also use Auto Scaling to adjust capacity as needed - on demand, on a schedule, or dynamically based on demand. In this session, we show you a few of the tools you can use to enable Auto Scaling for the applications you run on Amazon EC2.
Peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns of our Memcached and Redis offerings and how customers have used them for in-memory operations and achieved improved latency and throughput for applications. During this session, we review best practices, design patterns, and anti-patterns related to Amazon ElastiCache.
(SDD403) Amazon RDS for MySQL Deep Dive | AWS re:Invent 2014Amazon Web Services
Learn about architecting a highly available RDS MySQL implementation to support your high-performance applications and production workloads. We will also talk about best practices in the areas of security, storage, compute configurations, and management that will contribute to your success with Amazon RDS for MySQL. In addition, you will learn about how to effectively move data between Amazon RDS and on-premises instances.
Planning a successful private cloud - CloudStack Collaboration Europe 2013Tim Mackey
So your boss just asked you to build a private cloud. Now what? Successful private clouds require a bit of planning, and your existing best practices may need to be adjusted. This deck covers some of the issues you'll face, or be aware of, as you migrate from an existing data center operation to one which is more "cloud-like". Some things may seem obvious, but there are aspects to network and storage design which impact success. This deck draws from my experience in building my first CloudStack cloud in early 2012 and has applicability to anyone seeking to deliver cloud services.
AWS Summit London 2014 | Maximising EC2 and EBC Performance (400)Amazon Web Services
This advanced technical session is ideal for customers that are looking to maximise the performance of AWS Elastic Block Store (EBS) storage to support workloads with demanding IO performance requirements. If you need to run high IO workloads on EBS such as NoSQL or RBDMS systems then attend this session to find out how to optimise your EBS configuration to enable this.
All these large data sets are so big its difficult to manage with traditional tools. Distributing computing is an approach to solve that problem! First the data needs to be mapped, then it can be analyzed or reduced.
Day 3 - Maintaining Performance & Availability While Lowering Costs with AWSAmazon Web Services
AWS provides you several pricing options that can help you significantly reduce your overall IT cost, including On-Demand Instances, Spot Instances, and Reserved Instances. This session covers high-level architectures and when to use and not to use each of the pricing models for components of those architectures. We walk through several customer examples to illustrate when to use each pricing option. Additionally, we walk through tools that may be useful to determine when to use each pricing model. This session is aimed at technically savvy managers and engineers who need to reduce their cloud spending.
Reasons to attend:
- Learn about Reserved Instances, On-Demand Instances and Spot Instances.
- Discover ways of running more for less in Amazon EC2.
- If you are already running a workload in AWS, attend this webinar to learn how to run the same workload at reduced costs.
WKS404 7 Things You Must Know to Build Better Alexa SkillsAmazon Web Services
As we add thousands of skills to the skills store our developers have uncovered some basic and more complex tips for building better skills. Whether you are new to Alexa Skill development or if you have created skills that are live today, this session will help you understand and learn best practices. During this session, you’ll build an Alexa skill using more advanced VUI concepts and we’ll cover how to use AWS services like dynamoDB and S3 to implement the best practices we cover.
How to scale up, out or down in Windows Azure - WebinarCommon Sense
Webinar presented on Jan 26th 2011 by Juan De Abreu.
Learn how to achieve:
• Scalability linear scale, scale up vs. scale out, choose VM sizes
• Storage Cache
• Elasticity, scale out, scale back and automation of scaling
Intended for:CIOs, CTOs, IT Managers, IT Developers, Lead Developers
More Nines for Your Dimes: Improving Availability and Lowering Costs using Au...Amazon Web Services
Running your Amazon EC2 instances in Auto Scaling groups allows you to improve your application's availability right out of the box. Auto Scaling replaces impaired or unhealthy instances automatically to maintain your desired number of instances (even if that number is one). You can also use Auto Scaling to automate the provisioning of new instances and software configurations as well as to track of usage and costs by app, project, or cost center. Of course, you can also use Auto Scaling to adjust capacity as needed - on demand, on a schedule, or dynamically based on demand. In this session, we show you a few of the tools you can use to enable Auto Scaling for the applications you run on Amazon EC2.
Peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns of our Memcached and Redis offerings and how customers have used them for in-memory operations and achieved improved latency and throughput for applications. During this session, we review best practices, design patterns, and anti-patterns related to Amazon ElastiCache.
(SDD403) Amazon RDS for MySQL Deep Dive | AWS re:Invent 2014Amazon Web Services
Learn about architecting a highly available RDS MySQL implementation to support your high-performance applications and production workloads. We will also talk about best practices in the areas of security, storage, compute configurations, and management that will contribute to your success with Amazon RDS for MySQL. In addition, you will learn about how to effectively move data between Amazon RDS and on-premises instances.
Planning a successful private cloud - CloudStack Collaboration Europe 2013Tim Mackey
So your boss just asked you to build a private cloud. Now what? Successful private clouds require a bit of planning, and your existing best practices may need to be adjusted. This deck covers some of the issues you'll face, or be aware of, as you migrate from an existing data center operation to one which is more "cloud-like". Some things may seem obvious, but there are aspects to network and storage design which impact success. This deck draws from my experience in building my first CloudStack cloud in early 2012 and has applicability to anyone seeking to deliver cloud services.
Slides from the second meeting of the Toronto High Scalability Meetup @ http://www.meetup.com/toronto-high-scalability/
-Basics of High Scalability and High Availability
-Using a CDN to Achieve 99% Offload
-Caching at the Code Layer
Mapping Life Science Informatics to the CloudChris Dagdigian
Infrastructure cloud platforms such as those offered by Amazon Web Services are not designed and built with scientific research as the primary use case. These presentation slides cover the current state of mapping life science research and HPC technique onto “the cloud” and how to work around the common engineering, orchestration and data movement problems.
[Note: I've replaced the 2011 version of this talk deck with a slightly updated version as delivered at the AIRI Petabyte Challenge Meeting]
Leveraging Databricks for Spark PipelinesRose Toomey
How Coatue Management saved time and money by moving Spark pipelines to Databricks.
Talk given at AWS + Databricks ML Dev Day workshop in NYC on 27 February 2020.
Leveraging Databricks for Spark pipelinesRose Toomey
How Coatue Management saved time and money by moving Spark pipelines to Databricks.
Talk given at AWS + Databricks ML Dev Day workshop in NYC on 27 February 2020.
You host Relativity data in SQL Server, and you can't just go buy a new server. You need to figure out where your server's bottleneck is, and find the easiest workaround to make it go faster. Microsoft Certified MasterBrent Ozar will show you how to check your SQL Server's vital stats with free tools, and then determine whether you need indexes, memory, or different config settings.
Amazon Redshift in Action: Enterprise, Big Data, and SaaS Use Cases (DAT205) ...Amazon Web Services
Since Amazon Redshift launched last year, it has been adopted by a wide variety of companies for data warehousing. In this session, learn how customers NASDAQ, HauteLook, and Roundarch Isobar are taking advantage of Amazon Redshift for three unique use cases: enterprise, big data, and SaaS. Learn about their implementations and how they made data analysis faster, cheaper, and easier with Amazon Redshift.
My Site is slow - Drupal Camp London 2013hernanibf
Drupal is a powerful and flexible tool to create web applications without building everything from scratch. This ability can drive developers to build complex websites without understanding what is Drupal doing behind the scenes.
The majority of Drupal performance talks mostly focus in aspects like infrastructure changes, caching strategies or comparisons between modules and architectures. Unfortunately when performance problems occur, development teams also follow strategies to replace different aspects of the platform looking only to standard aspects like slow queries without understanding and profiling the real problem.
The majority of times it is fundamental to measure and analyze what is the application is actually doing to understand te real problems. Drupal is a platform used by million of websites worlwide and its performance can in most cases be compared after measured.
In Acquia we do dozens of performance assessments per year, and even in most clients we find the same problems, often we find situations that only can be detected when measured and analized when looking to a profiler report.
In this session, I will explain how to detect performance problems looking to simple data, from logs to profiler data and providing some nice targets that can be analyzed to understand what is causing the uncommon bad performance of a site.
Mtc learnings from isv & enterprise interactionGovind Kanshi
This is one of the dated presentation for which I keep getting requests for, please do reach out to me for status on various things as Azure keeps fixing/innovating whole of things every day.
There are bunch of other things I can help you on to ensure you can take advantage of Azure platform for oss, .net frameworks and databases.
Mtc learnings from isv & enterprise (dated - Dec -2014)Govind Kanshi
This is little dated deck for our learnings - I keep getting multiple requests for it. I have removed one slide for access permissions (RBAC -which are now available).
Drupal is a powerful and flexible platform to build websites with rich funcionalities without building almost anything from scratch. This flexibility brought by the usage of a powerful framework and the work of a super active community can abstract people to understand what is Drupal doing behind the scenes.
Most of performance talks regarding Drupal focus on aspects like infrastructure changes, caching strategies, and comparison of performance between modules or platforms. Unfortunately when performance problems occur, development teams also follow several strategies to replace several aspects in their platforms, jump directly to look for slow queries before trying really to understand where is the bottleneck.
However, most of the times what really needs to be done is to look to what the application is doing and understanding why is it taking so long to do it. Drupal is a platform used by million of websites worldwide and its performance is easy to measure and compare.
At Acquia we have done dozens of performance assessments, and even if we usually face the same problems, sometimes we found weird situations that are only possible to be detected when measured. Measuring and profiling is the only way to understand performance problems in a site and provide valid fixes.
In this talk I will explain how to detect problems regarding performance in Drupal, using simple modules like devel, profilers like XhProf and looking to logs to understand the impact done on the application.
Bursting into the public Cloud - Sharing my experience doing it at large scal...Igor Sfiligoi
When compute workflow needs spike well in excess of the capacity of a local compute resource, capacity should be temporarily provisioned from somewhere else to both meet deadlines and to increase scientific output. Public Clouds have become an attractive option due to their ability to be provisioned with minimal advance notice. I have recently helped IceCube expand their resource pool by a few orders of magnitude, first to 380 PFLOP32s for a few hours and later to 170 PFLOP32s for a whole workday. In the process we moved O(50 TB) of data to and from the clouds, showing that networking is not a limiting factor, either. While there was a non-negligible dollar cost involved with each, the effort involved was quite modest. In this session I will explain what was done and how, alongside an overview of why IceCube needs so much compute.
Where to start? - the first 2 hours of performance troubleshooting
• The performance cheat sheet: cover all the basics before you start
• Data collections and mining the logs
• Common techniques to improve performance
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
To Cloud or Not To Cloud?
1. To Cloud
or Not to Cloud?
Greg Lindahl, CTO
@glindahl – greg@blekko.com"
2. About
Us
• Web-‐scale
search
engine
with
our
own
crawl
&
index
• Public
launch,
November
2010
• $60
M
raised
• 800
servers,
16
PB
spinning
rust,
½
PB
flash
disk
5. The
wiring
diagram
Web
Crawler
Extractor
Ranker
Indexer
Lookup
Query
Analyzer
Front
End
Query
SERP
DIG
KB
6. Hijacking
a
meetup
topic
• Original
topic
was
“virtualizaUon
or
not”
• But
really,
virtualizaUon
is
an
implementaUon
detail
these
days
– cloud
=>
virtual
– virtual
=>
public
or
private
cloud
(probably)
• This
talk:
Public
cloud
vs.
not
• I’m
trying
to
list
a
bunch
of
things
that
you
should
think
about
…
your
situaUon
probably
differs
from
mine
7. The
quesUon
• It’s
2007,
and
your
CEO
asks
you:
Should
our
new
startup
use
this
newfangled
cloud
compuUng
stuff
or
not?
8. Why
cloud
at
all?
• Flexible
– prototyping
&
development
– tesUng
at
scale
– scale
up
for
high
usage
and
back
down
later
• Turns
CapEx
into
OpEx
– startups
prefer
paying
over
Ume
– “money
tomorrow
is
cheaper
than
money
today”,
if
you’re
successful
{btw,
plenty
of
banks
will
loan
against
equipment.}
9. Cloud
win
examples
• CommonCrawl.org
has
a
web
crawl
dataset
on
EC2
– Map/Reduce
job
to
read
the
whole
thing
is
~
$50
• Fewer
ops
people
is
actually
true
• Your
company
can
change
direcUon
10. OK,
so
what’s
bad?
• Examine
the
curve
of
Amazon’s
pricing
over
Ume
and
per
volume
• People
think
it’s
a
low-‐priced
product,
but
it’s
not.
• It’s
value
priced.
• Not
enough
compeUUon,
yet,
to
really
drive
Amazon’s
margins
down
• This
is
good
for
Amazon,
maybe
not
for
you.
11. 6
Reasons
to
not
use
Amazon
• Economy
of
scale
in
your
favor?
• Your
max::min
raUo
is
not
large
enough
• Cloud
IOPs
are
expensive
• Data
is
heavy
if
you
use
a
lot
of
local
disk
• SSDs
are
overpriced
• RaUo
of
disk
capacity
or
bandwidth
::
ssd
::
memory
::
compute
may
not
be
ideal
for
you
12. Economy
of
scale
• “Amazon
has
100s
of
thousands
of
servers,
so
they
can
run
them
cheaper
than
I
can.”
• But:
– you
pay
retail,
not
wholesale
price
– there
are
diminishing
returns
with
size
• At
some
point,
it’s
cheaper
to
do
it
yourself
• 100
servers?
50
servers?
{
blekko
had
700
at
launch…
}
13. Your
max::min
raUo
is
not
big
enough
• Maybe
you
use
100x
as
many
servers
some
days?
– Cloud
is
for
you!
• How
long
do
your
usage
spikes
last?
• Can
you
predict
them
far
enough
in
advance?
• How
long
does
it
take
you
to
spin
up
a
new
node?
{blekko’s
day::night
is
only
2x}
14. Cloud
IOPs
are
expensive
• I/O
OperaUons
are
expensive
to
start
with
– “spinning
rust”
disks
only
seek
so
much
• Networked
storage
has
low
bandwidth
compared
to
10
apached
disks
– 1
Gbyte/sec
sustained
–
woah!
• Networked
disks
are
more
expensive
than
local
– beper
failure
behavior,
whether
I
want
it
or
not
15. Data
is
heavy
if
you
use
a
lot
of
local
disk
• I
mean:
it
takes
a
loooooong
Ume
to
copy
a
few
tbytes
of
data
onto
your
local
disk
over
the
network
– 1
gigabit:
½
tbyte/hour
– 10
gigabit:
5
tbytes/hour
– even
filling
your
½
tbyte
SSD
is
kinda
slow
• Slow
spin-‐up/down
of
nodes
hurts
your
ability
to
flex
up
and
down
16. SSDs
are
overpriced
(by
cloud
providers)
• SSDs
are
completely
awesome
for
read-‐heavy
analyUcs
queries
• SSDs
wear
out
with
writes
• No
cloud
provider
charges
a
fee
for
writes?
• Instead,
they
assume
all
their
customers
are
average
• …
and
so
they
charge
way
too
much
to
customers
who
are
smart
about
not
wriUng
too
much
{
blekko
is
great
at
not
wriUng
to
our
SSDs
}
17. RaUos
available
might
not
fit
your
usage
• Amazon
tries
prepy
hard:
– high
memory,
high-‐CPU,
GPU,
high
I/O,
high-‐storage
– weirder
ones
are
less
flexible
• It’s
sUll
easy
to
not
fit
into
that
set
of
cookie
cupers
• Not
firng
==
wasted
money
– idle
resources
that
you’ve
paid
for
– moves
the
break-‐even
point
to
smaller
node
count
{
blekko
crawler
nodes:
10
local
disks
(capacity,
bandwidth,
seeks),
2
ssds,
96
gigs
ram}
18. So…
• For
us,
it
was
easy
to
predict
the
right
answer
• Our
SWAG
for
launch
day
was
600
servers
– and
our
enUre
index
in
SSD
– and
we
can’t
scale
down
from
that
• Amazon
wasn’t
renUng
SSDs
yet
• If
you’re
going
to
run
your
own
servers,
you
need
to
start
early
19. How
about
you?
• RT
analyUcs
is
a
complicated
subject
• Two
main
thrusts
– Pre:
pre-‐compute
aggregate
numbers,
query
those
– Mem:
sUck
a
subset
of
your
big
data
that
fits
into
ram
or
ssd,
do
complicated
queries
against
those
{
blekko
only
does
Pre
}
20. Pre
• Needs
to
be
wired
into
your
stream
of
data
generaUon,
e.g.
your
webserver
• Summary
data
can
be
prepy
small
• Doesn’t
really
maper
where
you
put
it
• Not
much
impact
on
the
cloud/no-‐cloud
decision
{
blekko
pre-‐computes
a
lot
of
things
using
“combinators”
in
our
home-‐grown
NoSQL,
opUonally
stuffing
them
into
our
SSD
caching
system
}
21. SERVER 1
PROCESS 1 PROCESS 2
SERVER 2
PROCESS 1 PROCESS 2
DISK 1 DISK 2 DISK 3
+4 +3 +4 +7
+11
+11+11
+7
+7+7
+18 +18 +18
Combinators
reduce
the
total
work
22. Mem
• Even
a
decimated
subset
of
your
fresh
data
can
involve
a
lot
of
write
bandwidth
– SomeUmes
referred
to
as
“high
velocity”
• High
BW
probably
needs
to
go
nearby
your
big
data
store
• AnalyUcs
probably
isn’t
going
to
influence
the
cloud/not-‐cloud
decision
23. Discuss!
• Discuss
• For
more
about
blekko’s
setup:
– 3
part
blog
series
at
highscalability.com
– Please
search
[high
scalability
blekko]
in
your
search
engine
of
choice
– greg@blekko.com
-‐-‐-‐
@glindahl