Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Coding For Cloud
Harjindersingh Mistry
Bargava Subramanian
Red Hat
Grace
● Smart individual
● Just finished MBA
Mark
● A geek
● Just completed engineering
Grace & Mark
Online Store
Hey Mark,
We don’t have enough money to
buy our own servers !
So, can we rent some ?
That sounds like a good idea, Grace !...
Cloud Computing
● Computers on rent
● Some guy has lot of computer infrastructure
○ He wants to give it on rent to make mo...
Alright, I have finished the
implementation !
Cloud, Take it and go !
Hey dude, customers are
reporting that our service is very
slow :(
Please check what is wrong
there.
Oh! is it ? But I had...
Horizontal Scalability
● Think of data in terms of standard ‘collections’
● Write functional as much as possible
● Let the...
Alright, I have done cool
clustering and my architecture is
reactive now!
Cloud, …. take it and go !
Wow Mark, the customers seem
to be happy with performance
now :)
But one customer has reported a
bug ! Can you please fix ...
Ok ! So, I have fixed the
“data-access” module.
Cloud, …. take it and go one
more time !
Hey dude, what are your
programming ?!?
There is another bug reported :(
Oh! is it ? Let me check that.
Oh man, this debugging is taking
forever :(
I need to configure the whole
system to repro the scenario !
...
Finally, I go...
Ok ! So, the tests passed. Cloud, ….
take it and go one more time !
Wait, now I remember what Prof
said about Micro Servic...
Micro Services
● Divide the whole application into smaller services, called
Micro Services
● It gives flexibility in testi...
Ok ! So, I have done Micro
Services :)
Cloud, …. take it and go !
Awesome Mark ! We are doing
good !
How about adding one new
feature into our store !
Oh yes ! That will be fun :)
It has b...
Alright ! The new feature is ready !
But I somehow don’t like all micro
services sharing same computing
resources :(
Let m...
Containers
● Deploy services by using Containers
● Containers increase speed of deployment
● Developer can simulate produc...
So, now I have each micro
service containerized separately
! That looks good.
Cloud, take it and go !
Crux of the Story
● Current business model demands speed
○ Need to experiment: It’s ok to fail, but fail fast
○ Need the a...
Cloud Native Application
● It helps us experiment with an idea
● We can achieve those attributes of good software as follo...
Cloud Native Application: Example
● Movie Recommendation Service
● Cloud Native Machine Learning Application in Python
● T...
Layers of Cloud Native Application
Deployment
Output Channel
Functionality
Data Access
S3 Local Files ...
Functionality Layer
● Contains core business logic
● As much ‘functional’ as possible i.e. immutability and statelessness
...
Data Access Layer
● Provides abstraction on various data sources
● Helps in testing the core logic locally
Abstract Class ...
Output Channel Layer
● Different channels for different consumers
● For micro service, it will have REST component
Functio...
Deployment Layer
● Contains the scripts for deployments
○ Docker files and/or YAML files for containers
○ Python scripts w...
Demo
● Functionality Layer: Training
● Functionality Layer: Scoring
● Deployment Layer: OpenShift
Input Data Read/Split Tr...
Not Covered
● Continuous Integration / Continuous Delivery
● Security
● Monitoring
● Vendor APIs for deployment
Code, Slides and Contact
● Code:
○ https://github.com/harjinder-hari/cloud_native_analytics
● Slides:
○ https://speakerdec...
Conclusion
● Current business model demands speed
○ Need to experiment: It’s ok to fail, but fail fast
○ Need the ability ...
Images Courtesy
[1] Grace and Mark
https://image.freepik.com/free-vector/happy-university-students_23-2147531065.jpg
[2] P...
Upcoming SlideShare
Loading in …5
×

Designing and coding for cloud-native applications using Python, Harjinder Mistry

PyParis 2017
http://pyparis.org

  • Be the first to comment

  • Be the first to like this

Designing and coding for cloud-native applications using Python, Harjinder Mistry

  1. 1. Coding For Cloud Harjindersingh Mistry Bargava Subramanian Red Hat
  2. 2. Grace ● Smart individual ● Just finished MBA Mark ● A geek ● Just completed engineering
  3. 3. Grace & Mark Online Store
  4. 4. Hey Mark, We don’t have enough money to buy our own servers ! So, can we rent some ? That sounds like a good idea, Grace ! That’s called Cloud Computing. I remember what Prof told about it ...
  5. 5. Cloud Computing ● Computers on rent ● Some guy has lot of computer infrastructure ○ He wants to give it on rent to make money ● Very similar to car-rental service like Europcar ● Cloud-native Application is a program that is designed and developed specifically for cloud computing architecture
  6. 6. Alright, I have finished the implementation ! Cloud, Take it and go !
  7. 7. Hey dude, customers are reporting that our service is very slow :( Please check what is wrong there. Oh! is it ? But I had used best algo! Ahha, there seems to be a scalability issue. Let me call my Prof and get his suggestions ...
  8. 8. Horizontal Scalability ● Think of data in terms of standard ‘collections’ ● Write functional as much as possible ● Let the data flow through functions: ○ If data is static then we can do Parallel Processing ○ If data size is not fully known then we can be Reactive ○ There is also a possibility of mixing them together by using Stream Processing
  9. 9. Alright, I have done cool clustering and my architecture is reactive now! Cloud, …. take it and go !
  10. 10. Wow Mark, the customers seem to be happy with performance now :) But one customer has reported a bug ! Can you please fix it ? Oh! is it ? Let me check that. It might be something related to “data-access” module.
  11. 11. Ok ! So, I have fixed the “data-access” module. Cloud, …. take it and go one more time !
  12. 12. Hey dude, what are your programming ?!? There is another bug reported :( Oh! is it ? Let me check that.
  13. 13. Oh man, this debugging is taking forever :( I need to configure the whole system to repro the scenario ! ... Finally, I got it !!! Bug is inside “business-logic” module. I fixed it. Let me test the whole app. Come on Mark ! How much more time do you need ?
  14. 14. Ok ! So, the tests passed. Cloud, …. take it and go one more time ! Wait, now I remember what Prof said about Micro Services ...
  15. 15. Micro Services ● Divide the whole application into smaller services, called Micro Services ● It gives flexibility in testing and deploying in small independent chunks ● This improves maintainability a lot
  16. 16. Ok ! So, I have done Micro Services :) Cloud, …. take it and go !
  17. 17. Awesome Mark ! We are doing good ! How about adding one new feature into our store ! Oh yes ! That will be fun :) It has been while since I coded something meaningful !
  18. 18. Alright ! The new feature is ready ! But I somehow don’t like all micro services sharing same computing resources :( Let me talk to Prof ...
  19. 19. Containers ● Deploy services by using Containers ● Containers increase speed of deployment ● Developer can simulate production environment locally ● Moreover, each service runs in its own environment ○ So if one service goes down, others are still safe
  20. 20. So, now I have each micro service containerized separately ! That looks good. Cloud, take it and go !
  21. 21. Crux of the Story ● Current business model demands speed ○ Need to experiment: It’s ok to fail, but fail fast ○ Need the ability to react: Everything cannot be planned ● But, the software has to be good in terms of: ○ Correctness ○ Performance ○ Maintainability ○ Operability
  22. 22. Cloud Native Application ● It helps us experiment with an idea ● We can achieve those attributes of good software as follows: ○ Correctness: Test cases. This cannot change :) ○ Performance: Horizontal Scalability ○ Maintainability: Micro Services ○ Operability: Containers
  23. 23. Cloud Native Application: Example ● Movie Recommendation Service ● Cloud Native Machine Learning Application in Python ● There are two parts: ○ Offline Training Job: ○ Online Scoring Service: Movie Data Training Job ML Model Cloud Storage User Input + Model Scoring Service Recommendations
  24. 24. Layers of Cloud Native Application Deployment Output Channel Functionality Data Access S3 Local Files ...
  25. 25. Functionality Layer ● Contains core business logic ● As much ‘functional’ as possible i.e. immutability and statelessness ● Data flows through pipeline of functions: ● For example: Input Data Function-1 ... Function-N Output Result Input Data Read/Split Transform Train ML Model
  26. 26. Data Access Layer ● Provides abstraction on various data sources ● Helps in testing the core logic locally Abstract Class / Interface S3 Local Files ...
  27. 27. Output Channel Layer ● Different channels for different consumers ● For micro service, it will have REST component Functionality CLI REST ...
  28. 28. Deployment Layer ● Contains the scripts for deployments ○ Docker files and/or YAML files for containers ○ Python scripts with calls to vendor-specific APIs e.g. AWS API run_job_flow()
  29. 29. Demo ● Functionality Layer: Training ● Functionality Layer: Scoring ● Deployment Layer: OpenShift Input Data Read/Split Transform Train ML Model User Input + Model Scoring Service Recommendations
  30. 30. Not Covered ● Continuous Integration / Continuous Delivery ● Security ● Monitoring ● Vendor APIs for deployment
  31. 31. Code, Slides and Contact ● Code: ○ https://github.com/harjinder-hari/cloud_native_analytics ● Slides: ○ https://speakerdeck.com/harjinderhari/coding-for-cloud ● Contact: ○ Harjinder email: harjinder.v2@gmail.com ○ Bargava twitter: @bargava
  32. 32. Conclusion ● Current business model demands speed ○ Need to experiment: It’s ok to fail, but fail fast ○ Need the ability to react: Everything cannot be planned ● Cloud Native Applications is a way to achieve it ○ We saw ■ Layers involved in a cloud native application ■ Techniques to achieve the required speed ■ Everything is written in Python ! :)
  33. 33. Images Courtesy [1] Grace and Mark https://image.freepik.com/free-vector/happy-university-students_23-2147531065.jpg [2] Professor https://image.freepik.com/free-vector/science-teacher_23-2147509483.jpg [3] Rocket https://image.freepik.com/free-vector/launching-business-concept_23-2147504957.jpg

×