Models vs Reality: Quest for the Roots of Complexity
•Download as PPTX, PDF•
0 likes•94 views
In this talk we will get familiar with the concept of Model Dependent Realism, coined by Stephen Hawking and Leonard Mlodinow. We will then apply it to reason about Software Development as operating hierarchy of languages and abstractions. Equipped with the new mental model, we will try to disect and understand the phenomenon of complexity - concept so ill-defined and misused that it has grown an aura of mysticism around. There is an uncompensated bias among software engineers to address the known problems - the technicalities. This bias is reflected by topics of most software conference talks, where technical solutions are presented addressing narrow groups of problems. This bias puts evolution of our discipline at risk. The main goal is to induce particular way of thinking and give some guidance for engineers. To even out the bias and have more engineering time and energy devoted for the most crucial subject - how to develop interfaces that can be understood by layman. How to develop systems that perfectly match the business they represent. How to reduce the complexity of our products and processes and how to be better engineers.
Recommended
Recommended
More Related Content
Similar to Models vs Reality: Quest for the Roots of Complexity
Similar to Models vs Reality: Quest for the Roots of Complexity (20)
Recently uploaded
Recently uploaded (20)
Models vs Reality: Quest for the Roots of Complexity
- 1. Models vs Reality Quest for the roots of complexity @hundredmondays
- 2. Exercise: draw a car
- 3. Domain Driven Design Test Driven Development SOLID Principles Functional Programming Stop coding, learn the above, continue coding
- 10. Model Dependent Realism reality should be interpreted based upon models… where several models overlap… multiple, equally valid, realities exist.
- 12. "Today the software comprising the carefully planned interpretive routines, compilers, and other aspects of automative programming are at least as important to the modern electronic calculator as its hardware of tubes, transistors, wires, tapes and the like" John Tuckey - 1958
- 14. Programming constructs you use are not fundamental to computer. They are models built on top of more primitive paradigms. They hide irrelevant complexity so you can... communicate better
- 15. Abstraction
- 16. Software is an abstract landscape built of abstract components
- 18. Why do we need abstractions? Pass and move Give and go Long through ball Triangular movement Swapping the wing men Strong side overloads Target man Envelopment Wide play Man-to-man marking Catenaccio [page 1 of 100]
- 19. Abstractions over abstractions Star: self-luminous gaseous spheroidal celestial body Planet: round celestial body which has removed debris and small objects from the area around its orbit Asteroid: small rocky body orbiting a star Comet: celestial object consisting of a nucleus of ice and dust Orbit: elliptical path of a planet, asteroid, or comet Perihelion: point in the orbit closest to the star
- 20. /inline all Perihelion: point in the elliptical path of a round celestial body which has removed debris and small objects from the area around its path, small rocky celestial body or celestial object consisting of a nucleus of ice and dust closest to a self- luminous gaseous spheroidal celestial body.
- 21. Is the language I use really that abstract?
- 22. Virtual memory, Floating point arithmetic, Garbage collection, Concurrency controls, Inter process communication, Platform independence, Flow control, Process isolation, Device drivers, Streaming, Lambda abstraction, Object orientation, Exceptions, Nested functions, Lexical scope, Type inference, Template system, Namespaces...
- 24. V => T needs null check can throw exception can mutate V can require global state can modify global state can take time
- 25. (V, Context) -> Future[ Try[ T ] ] can take time can failno side effects dependencies
- 26. The world of computers is full of things that should have been strict from the beginning, and became ridiculously complex because they weren't. --Understanding Software by Max Kanat-Alexander
- 28. ...the other two drugs might be particularly effective during cell division. If so, a drug promoting cancer cell division might be a good thing to have in combination with the other two.
- 29. What abstractions have You provided?
- 33. Boundary between the business and the software is just a model und das model ist...
- 34. The Ultimate Stack Business Software Hardware End of physics End of formality
- 35. there is no “Turing effective procedure” to list all the uses of a screwdriver complexity -> specificity simplicity -> adaptability
- 37. ROI industry growth customer funnel account npv product user test deployment environment system protocol class data feature insight logic experience behavior architecture Design pattern function expression role persona acquisition margins
- 38. ROI industry growth customer funnel account npv product user test deployment environment system protocol class data feature insight logic experience behavior architecture Design pattern function expression role persona acquisition margins
- 41. Delivery speed of a system always decreases Complexity of a system always increases Entropy of a system always increases
- 42. Software is like entropy. difficult to grasp, weighs nothing, and obeys the Second Law of Thermodynamics; it always increases. Norman Ralph Augustine Law Number XVII
- 43. Yea, yea, can we formalize it?
- 44. Cyclomatic Complexity CC = E - N + 2 x P E = number of edges in the flow graph. N = number of nodes in the flow graph. P = number of nodes that have exit points
- 47. Cyclomatic complexity only indicates complexity at the unit level. What about the structure of units collaborate? Do we assume the structure is perfect?
- 51. Component structure analysis only indicates complexity at the system level. What about how the system expresses the business idea? Should we assume this expression is perfect?
- 52. The Pyramid of Wrong
- 53. Wrong idea Wrong specification Wrong code
- 54. How about writing proper spec? How about writing proper code?
- 55. How about we help each other
- 56. Experimentation = testing the business idea feature v2 feature v1 control group experiment group
- 57. How do we turn an idea into architecture? experimentation Domain driven design specification as code software engineering
- 60. Euclidean distance in multidimensional system between current solution and an ideal solution which is unknown and always changing
- 61. moving target, ideal today is not ideal tomorrow! not able to define ideal if we could, we would build it! ideal solution current solution infinite dimensions!
- 64. It is not the way things ARE in software that is complex. It is the language we choose to SPEAK about them. We choose that language with every line of code we write and every line of code we decide not to write.
- 65. What’s in it for… anyone?
- 68. You don’t remove complexity You direct its growth by models Model is a domain specific language Language = hierarchy of abstractions All we do as devs is developing languages Let’s do it consciously!
- 70. Julian Warszawski from Łódź fintech is my battleground LendUp is my favorite startup spits sarcasm at @hundredmondays
Editor's Notes
- just in case you decided to leave early or fall asleep in 5 mins, here is the message of the day if you are a software engineer and don’t know one of these concepts, learn them immediately notice I mentioned solid rather than OO, cuz there is some confusion in the industry on what OO really means and SOLID is like an essence of OOP principles
- here is a model of Chevrolet Corvette this model serves a particular purpose what is the purpose? it shows exact proportions, shapes, you could put it into a design software and have a 3d model if dimensions were given, you could basically build the chassis
- this is also a model but it serves a different purpose it shows less details, emphasizes a few lines it is to generate a certain impression of dynamism probably an early concept to drive further refinements of design it would fail miserably if we wanted to build a car based on it but the other one is too concrete, exposes irrelevant details if we only want to discuss design wireframe vs mockup
- “For what purpose”
- The best solution is dependent upon the purpose. Without a purpose it is not possible which model is ‘the simplest’ Elementary particles vs atoms
- model is not a simplification of reality in its particular bounded context, it is the reality the concept car drawing was drawn before a car was manufactured maybe tht car will never be manufactured, so the drawing itself is the reality give example of model for tooth fairy
- what is the difference between the first set of terms and the second? the first ones are not physical, you can touch a transistor but cannot touch a compiler
- does it matter that these concepts don’t exist? what is even the purpose of concepts that are not physical?
- literally most of the programming concepts we learned are built on top of typical OS layer OS is a model, there are different OS models, but the one we use (UNiX) is most widely adopted
- Primitive languages were mostly concrete fire, man, woman, spear, tiger Abstractions were built on top of concrete terms and other abstractions family, danger, hunting
- this guy has 2 minutes to lay out a game plan although he talks english, you wouldn’t understand a word he says because in fact he talks a domain specific language - very effective dsl built on top of English (gpl)
- This is how programming in assembly would look like
- Titanous amount of work has been put in over almost a century to deliver these abstractions Given any computer program could theoretically be written without a single one of them, what is the value Again, these concepts are not physical. They model the way we think about execution of a computer program so the cognitive load on us when writing a program is reduced, so we can write more complex programs Computer Science has been overly generous with us providing high grade, very useful abstractions, so we can do more And what have we done with all of this?
- to be fair there is still more work to do with generic abstractions even worse in dynamic languages
- we need to use language that allow us to introduce relevant constraints expressive type system is superior to duck typing functional programming is superior to imperative programming, etc.
- you probably already see that I’m trying to relate the concepts of models and complexity for inspiration I was looking for other contexts where complexity plays an important role a book I can definitely refer to that changed my view of complexity in summary, complexity is not a force to fight with, but you rather direct it complex systems, whether business, life or society, they act a bit like living things they respond to change, they grow, they compete for resources and they cannot be fully controlled or even understood
- we talk about constraints and strictness, but is there a limit can we overconstrain, so that our code is so specific that it cannot be used effectively?
- what do you mean? my clients are not programmers give examples: GraphQL, SQL, Sequence Diagram, Workflows, etc.
- this relies on outdated arbitrary division of labor that is a mix with computer science and civil engineering. someone drew these lines long ago and we stick to them we borrow metaphors because of lack of new ones we dismiss problems of upper layers. thinking like computer scientists. there is a need for the new kind of engineer. one that can communicate through the boundaries, understand the boundaries and change them.
- typical model with boundary between business and software (engineering) where business provides description of what should be developed and engineering delivering it
- how it really looks like
- Spectrum from absolute order to absolute chaos software can’t be touched business can’t be proven or can it?
- complexity yields specificity simplicity yields adaptability SQL is a screwdriver of software - simple enough to reason about for non programmers powerful enough to for very advanced usage when SQL was developed, it’s all possible usages were not known
- some of these concepts may even overlap speak about the product, user, account who of you have User object in the domain what about Account what about Product
- here are some means we use to communicate across the boundary go through each of them stop at DSL and express how it fulfills each of the needs of the other ones but it is still a niche (APIs, executable specifications, BDD, ATDD)
- but generally this is how we speak Metaphor has higher chance to be understood by a human Metaphor is also a model, very context specific Impossible to codify, but helps trigger some intuition about the problem In those without capacity of understanding it otherwise Formal model is easier to translate into code
- infinite number of forces acting on the system the system has infinite number of attributes is there self similarity between components as between particles? No how do we even identify elementary components? Impossible metaphors from physics are compelling, because if we formalized them, we would be able to prove them, but it might be based on false assumptions mention a metaphor of a tech debt metaphor is a model, what is it useful for? limited context how I use tech debt
- let’s go back to the basics here is one definition of complexity by Thomas J. McCabe it is fully formal (you can write a program to calculate it)
- looked at some academic papers that deal with system complexity the problem seems to still be open there is controversy around if the intuitive understanding of complexity is even possible to describe due to subjective nature of it
- apparently if one starts reasoning about the system of components as a graph we can develop a plethora of useful metrics derived from graph theory which is scientific (formal) field
- dependency circle there were suggestions to use this type of diagram to break monoliths into microservices
- you can notice some recursive self-similarity if we see that, what do we do? build a pyramid
- the wrong code is the area that we talk about most test automation, formally provable programs, expressive type systems we dismiss problems of upper layers. thinking like computer scientists. like the problem we are solving is well defined mostly it is not what if we are writing the right code for wrong specification or if we have right code, right specification but wrong idea there is a need for the new kind of engineer. one that can communicate through the boundaries, understand the boundaries and change them. the complexity that we focus on has the least roi
- Experimentation Feature toggles A/B testing Continuous delivery testing in production feedback feedback feedback!
- 1 crunching knowledge 2 formal Specification (User Stories, Use Cases, UML) Behavior Driven Development Specification as Code Domain Driven Design Ubiquitous Language Change Management Process SDLC
- dimensions throughput extensibility cost testability scalability experimentation analytics etc.
- how useful is that?
- Should the complexity measure include business cost? what if complex solution has little to no cost? Should the complexity express transition cost? descending off local maximum Infinite dimensions of solution space design decisions lay on multiple orthogonal axes are we labelling disparate concepts with single name? business complexity, technical complexity, brittleness, security
- fog of war two categories of problems one can be proven, other cannot where is the boundary say there are systems that are complex and systems that are simple simple system can be analyzed deterministically meaning that we know exactly how each of its elements behaves complex process can only be analyzed stochastically its internals is just statistics, effectively black box
- but if what you really need is to calculate the temperature of the container you could either use thermodynamics equations or measure velocity vector of all particles and maybe do some event driven simulation, good luck with that. one is simple, the other one is complex two models describing the same phenomenon two realities, only one is simple
- that language is not Java, JS or Python folks complain that there are too many programming languages nonsense there should be a lot more programming languages every system that we build should provide a formal language to describe it we should explore more semantic paradigms, syntax experiments not in isolated fashion, languages could be interoperable idea from book that software evolution is not Newtonian process most popular tool on earth - screwdriver one cannot list all use cases of screwdriver exploration and readiness to fill new niches accidental vs business complexity junk dna vs dna good complexity = specificity how specific