Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Modern Software Infrastructures Revisited
1. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Modern Software Infrastructures Revisited
Igor Bogicevic (igor.bogicevic@sbgenomics.com)
January 29, 2011
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
2. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
3. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Features, robustness of code, scalable architecture are indeed increasing the
quality of software, but are also increasing complexity.
Refactoring, as a good practice, does not reduce complexity, but it merely pays
for a technical debt.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
4. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Features, robustness of code, scalable architecture are indeed increasing the
quality of software, but are also increasing complexity.
Refactoring, as a good practice, does not reduce complexity, but it merely pays
for a technical debt.
In environment where multiple components mutually depend complexity can
explode.
Moreover, complexity is not related to a sandbox of mutual dependencies, but
also on the production environment as well.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
5. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Features, robustness of code, scalable architecture are indeed increasing the
quality of software, but are also increasing complexity.
Refactoring, as a good practice, does not reduce complexity, but it merely pays
for a technical debt.
In environment where multiple components mutually depend complexity can
explode.
Moreover, complexity is not related to a sandbox of mutual dependencies, but
also on the production environment as well.
Embracing and controlling the complexity might be a better way.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
6. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Features, robustness of code, scalable architecture are indeed increasing the
quality of software, but are also increasing complexity.
Refactoring, as a good practice, does not reduce complexity, but it merely pays
for a technical debt.
In environment where multiple components mutually depend complexity can
explode.
Moreover, complexity is not related to a sandbox of mutual dependencies, but
also on the production environment as well.
Embracing and controlling the complexity might be a better way.
Integration complexity != Code complexity
Integration complexity Code complexity
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
7. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Introduction and Axiomatics
Complexity of a software does not reduce over a time.
Features, robustness of code, scalable architecture are indeed increasing the
quality of software, but are also increasing complexity.
Refactoring, as a good practice, does not reduce complexity, but it merely pays
for a technical debt.
In environment where multiple components mutually depend complexity can
explode.
Moreover, complexity is not related to a sandbox of mutual dependencies, but
also on the production environment as well.
Embracing and controlling the complexity might be a better way.
Integration complexity != Code complexity
Integration complexity Code complexity
Infrastructure complexity != Code complexity
Infrastructure complexity Code complexity
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
8. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
9. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
10. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Yes, I made up this one...
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
11. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Yes, I made up this one...
In my defense, I needed a glue term that will somehow connect software
architecture, infrastructure, build and deploy process
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
12. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Yes, I made up this one...
In my defense, I needed a glue term that will somehow connect software
architecture, infrastructure, build and deploy process
... configuration distribution, monitoring/alerting
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
13. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Yes, I made up this one...
In my defense, I needed a glue term that will somehow connect software
architecture, infrastructure, build and deploy process
... configuration distribution, monitoring/alerting
... dispatching of system updates, server orchestration
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
14. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
What is the ”Software Infrastructure”
infrastructure - noun: the basic, underlying framework or features of a system or
organization.
software - noun: ...
Yes, I made up this one...
In my defense, I needed a glue term that will somehow connect software
architecture, infrastructure, build and deploy process
... configuration distribution, monitoring/alerting
... dispatching of system updates, server orchestration
No, that’s not ”Information System”.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
15. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
16. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
17. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
18. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
19. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
After certain point, you need to re-think how are you storing and processing your
data.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
20. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
After certain point, you need to re-think how are you storing and processing your
data.
Once you get to the ”internet scale” of data, this becomes inevitable.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
21. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
After certain point, you need to re-think how are you storing and processing your
data.
Once you get to the ”internet scale” of data, this becomes inevitable.
Scaling up the data processing heavily depends on the both nature and
operations on top of data.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
22. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
After certain point, you need to re-think how are you storing and processing your
data.
Once you get to the ”internet scale” of data, this becomes inevitable.
Scaling up the data processing heavily depends on the both nature and
operations on top of data.
That also applies to information retrieval, search on top of data, etc.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
23. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scope Of The Problem
Hardware didn’t became cheap, however it became cheap enough.
Operating system became a commodity.
What was sometimes a privilege of the few, became approachable to a larger
audience.
An organic transformation - with more traffic and content, there was more data
available to use.
After certain point, you need to re-think how are you storing and processing your
data.
Once you get to the ”internet scale” of data, this becomes inevitable.
Scaling up the data processing heavily depends on the both nature and
operations on top of data.
That also applies to information retrieval, search on top of data, etc.
This problem was not new, and concepts and solutions existed on the market -
but not as a commodity.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
24. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
25. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
26. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
27. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Products and business are starting to be driven by available capacity.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
28. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Products and business are starting to be driven by available capacity.
On-demand infrastructure, in the matter of minutes.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
29. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Products and business are starting to be driven by available capacity.
On-demand infrastructure, in the matter of minutes.
Dynamic allocation of resources - AWS, Google App Engine, Rackspace Cloud,
Joyent SDC, etc.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
30. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Products and business are starting to be driven by available capacity.
On-demand infrastructure, in the matter of minutes.
Dynamic allocation of resources - AWS, Google App Engine, Rackspace Cloud,
Joyent SDC, etc.
As proven again - when you have a hammer, everything looks like a nail.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
31. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Paradigm Shift
Complexity moved from the code and software architecture to the infrastructure
as a whole.
New resources are driving both new patterns and paradigm shift.
Instead of infrastructure being driven by business requirements, in last couple of
years we have an opposite situation:
Products and business are starting to be driven by available capacity.
On-demand infrastructure, in the matter of minutes.
Dynamic allocation of resources - AWS, Google App Engine, Rackspace Cloud,
Joyent SDC, etc.
As proven again - when you have a hammer, everything looks like a nail.
Paradigm shift also leads to a new set of pointy-hair semantic and yugioh grade
conversations with appropriate keywords (Cloud, SaaS, PaaS, ...).
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
32. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Comic Relief
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
33. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
34. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
35. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
36. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
37. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Distributed everything?
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
38. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Distributed everything?
Maybe you don’t need to distribute your problem after all.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
39. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Distributed everything?
Maybe you don’t need to distribute your problem after all.
In reality we’re struggling with infrastructure design as we’re pioneering good
practices in the various scenarios.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
40. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Distributed everything?
Maybe you don’t need to distribute your problem after all.
In reality we’re struggling with infrastructure design as we’re pioneering good
practices in the various scenarios.
This means we have various tools, platforms, paradigms at our disposal, yet we’re
hitting various nails with various hammers.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
41. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Infrastructure Specialization
We’ve got to the point where we have multitude of paradigms to choose from.
Many theoretical CS paradigms in distributed computing got to a stage of
transformation to a production ready software.
Many of them are open.
To number some: distributed map/reduce, distributed stream computing,
distributed analytical search, distributed graph databases, distributed
synchronization.
Distributed everything?
Maybe you don’t need to distribute your problem after all.
In reality we’re struggling with infrastructure design as we’re pioneering good
practices in the various scenarios.
This means we have various tools, platforms, paradigms at our disposal, yet we’re
hitting various nails with various hammers.
And this is a call for infrastructure specialization.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
42. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
43. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
44. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
45. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
46. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Horizontal scaling is completely driven by the domain of the problem and data.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
47. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Horizontal scaling is completely driven by the domain of the problem and data.
Some of the problems and datasets are trivial to scale (sharding), most of them
are not - especially from the standpoint of fault tolerance.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
48. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Horizontal scaling is completely driven by the domain of the problem and data.
Some of the problems and datasets are trivial to scale (sharding), most of them
are not - especially from the standpoint of fault tolerance.
We are living victims of CAP theorem...
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
49. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Horizontal scaling is completely driven by the domain of the problem and data.
Some of the problems and datasets are trivial to scale (sharding), most of them
are not - especially from the standpoint of fault tolerance.
We are living victims of CAP theorem...
And large corpus of papers on distributed computing in general.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
50. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Scaling Up
Scaling should be avoided if possible (as premature optimization).
... and in ideal world you would be able to ”throw some money on it” and solve
the problem with vertical scaling.
... however that scales only up to certain level.
Horizontal scaling imposes a new set of problems.
Horizontal scaling is completely driven by the domain of the problem and data.
Some of the problems and datasets are trivial to scale (sharding), most of them
are not - especially from the standpoint of fault tolerance.
We are living victims of CAP theorem...
And large corpus of papers on distributed computing in general.
Design for a failure - you’ll appreciate when you see your system running.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
51. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
52. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
53. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
54. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
55. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Either you can’t attract great engineers, or you can’t build them.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
56. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Either you can’t attract great engineers, or you can’t build them.
Problem is always in you - figure out what you’re doing wrong.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
57. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Either you can’t attract great engineers, or you can’t build them.
Problem is always in you - figure out what you’re doing wrong.
No-Ops - BOFH syndrome has no place in an environment with complex
infrastructures.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
58. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Either you can’t attract great engineers, or you can’t build them.
Problem is always in you - figure out what you’re doing wrong.
No-Ops - BOFH syndrome has no place in an environment with complex
infrastructures.
Same standards should apply for System Engineers/DevOps as for Engineers.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
59. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
Organizational Changes
No-Product, this is a red area for Product Management.
Engineering should be empowered to drive the core product.
Product Management should move back to where it started - driving business
requirements and articulating BizDev requests.
”We can’t hire sufficiently good engineers” is a lousy excuse.
Either you can’t attract great engineers, or you can’t build them.
Problem is always in you - figure out what you’re doing wrong.
No-Ops - BOFH syndrome has no place in an environment with complex
infrastructures.
Same standards should apply for System Engineers/DevOps as for Engineers.
What is traditionally considered to be Engineering and Operations should be one
organizational group.
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
60. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
We are hiring!
We are fresh and well funded startup from Boston, MA.
We are building a first open on-demand scalable platform for distributed genomic
data processing.
We’re in a BigData business (10GB-2Tb per genome).
We have an ambition of becoming a de-facto platform for Bioinformatic data
processing.
We are looking for a talent across several categories:
- Core Engineering
- Frontend Application Engineers (python/django)
- Frontend Developers (html5/js/css)
- System Engineers/DevOps
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited
61. Outline
Introduction and Axiomatics
What is the ”Software Infrastructure”
Scope Of The Problem
Paradigm Shift
Comic Relief
Infrastructure Specialization
Scaling Up
Organizational Changes
We are hiring!
We are hiring!
We are fresh and well funded startup from Boston, MA.
We are building a first open on-demand scalable platform for distributed genomic
data processing.
We’re in a BigData business (10GB-2Tb per genome).
We have an ambition of becoming a de-facto platform for Bioinformatic data
processing.
We are looking for a talent across several categories:
- Core Engineering
- Frontend Application Engineers (python/django)
- Frontend Developers (html5/js/css)
- System Engineers/DevOps
If you’d like to work on a problems of large scale genomics data processing, please
shoot me an email: igor.bogicevic@sbgenomics.com
Igor Bogicevic (igor.bogicevic@sbgenomics.com) Modern Software Infrastructures Revisited