Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Проекты и Процессы
Next

3

Share

Data stream processing and micro service architecture

Data stream processing and micro service architecture.
English version

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Data stream processing and micro service architecture

  1. 1. Data Stream processing and Micro-service architecture © 2015 BY VYACHESLAV BENEDICHUK
  2. 2. Vyacheslav Benedichuk Software development professional Overall IT experience spanning 15 years. Project management experience 9+ years. Biggest team managed - 25 members. Biggest project developed – over 15 man-years https://ru.linkedin.com/in/vbenedichuk
  3. 3. Presentation plan History Today Challenges Solutions Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  4. 4. History
  5. 5. History Application Business logic DTO DAL UI Service Service Service Service Service History Today Challenges Solutions Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  6. 6. Today Challenges Application complexity grows •Complexity of interactions between layers is growing •Layers interaction complexity growing •Intensity of communications between developers is growing •More complex test cases Data size growth •Classic applications require a lot of resources in a single physical node. Such applications are not intended for horizontal scalability. •Single issue can break the whole application and may lead to data loss Instability of data flows •Levels of user activity in a social network very susceptible to the time of day •Buyer activity pm an online shopping site very susceptible to the time of day and season •Hardware purchased for peak workload is under-used outside peak times •Hardware purchased for medium workload is inadequate for peak workload History Today Challenges Solutions Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  7. 7. Solution Build the application as a set of small (50-300 lines of code) loosely coupled modules. • Each micro-service works independently • Data exchange between layers is made with standard messages • Services linked with message queues. Other data interaction protocols may be used (for example REST), however message queues increase overall system stability. History Today Challenges Solution Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  8. 8. Advantages overview Fault tolerance Scalability Better performance evaluation and optimization of the cost of ownership Ease of development Updates do not require downtime Simplified A/B and regression testing History Today Challenges Solution Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  9. 9. Advantages Fault tolerance The number of single points of failure is minimal or none. Failure of a single service instance, or one of the nodes does not lead to failure of the entire system. It only affects performance in a short term. Service 1 SaaS Queue Service 2 Service 3 Service 3 SaaS Queue Service 2 Service 2 Service 1 Service 1 Service 3 SaaS Data Storage Fault tolerance
  10. 10. Advantages Scalability Scalability Local load spikes (seconds) are flattered by message queues. Response time grows but no requests are denied. Global load spikes (tens of seconds and more) are handled by starting new worker instances. Fault tolerance in case of DDoS attack is limited by finances only. If provided in the cloud, it is much cheaper than buying in-house infrastructure designed for excessive load. Service 1 SaaS queue Service 2 Service 3 Service 3 SaaS queue Service 1 Service 1 Service 3 SaaS Data Storage 5 messages in the queue 100500 messages in the queue Service 3 (запускается) Service 2 (stopping) Service 3 (запускается)Service 3 (Starting)
  11. 11. Advantages Ease performance evaluation and optimization of the cost of ownership Ease performance evaluation Maximum load capacity of each particular service can be evaluated independently. On the basis of the "laboratory" measurement of performance and the projected load, optimal network topology can be constructed. It allows the minimum cost of ownership to be determined. Different combinations of infrastructure solutions can be compared on the basis of actual performance indexes rather than "by eye". Infrastructure investments are rising in proportion with the growing business without spikes. You can freely distribute the components of the system between in-house and cloud datacenters. SaaS queue Service 2 Measured load capacity is 300 messages per second. SaaS queue Test data stream 100500000 messages accumulated 1080000 after the 1 hour of processing With a minimum flow of 450 messages per second with hour peaks of up to 1,500 messages per second, we need 2 instances of pre-paid (or in house) and up to 3-4 on- demand services.
  12. 12. Advantages Ease of development Ease of development Micro-services code can be easily maintained due to its size. Micro-service independence allows services to be created using different technology stacks and with minimal or no integration overhead. Service 1 (php) SaaS queue Service 2 (c#) Service 3 (Java) Service 3 (Java) SaaS queue Service 2 (c#) Service 1 (php) SaaS Data Storage 100 lines of code vs 1000 lines of code
  13. 13. Advantages Update does not require the downtime New version can be launched by sequential launch of new micro- services. Accumulated requests will be processed by the old infrastructure. New requests will be processed by the new infrastructure. Old and new versions can work simultaneously. SaaS queue (accumulated data) Service 2 V1 Service 3 V1 SaaS queue Service 2 V1 Service 2 V2 Service 1 V2 SaaS Data Storage Updates do not require downtime Service 3 V1 SaaS queue Service 3 V1
  14. 14. Advantages Simplified A/B and regression testing A/B testing and regress testing can be implemented by running 2 micro service instances in parallel. Service 1 V1 Service 2 V1 Service 2 V2 Service 1 V2Simplified A/B and regression testing Comparator SaaS queue SaaS queue Duplicator
  15. 15. Disadvantages The complexity of the initial deployment • Each micro-service has a separate installer. Manual deployment of large systems requires considerable time. It should be solved by automated deployment. For example using the Docker. The complexity of the manual update • Many small services distributed across different physical devices or datacenters require automated upgrades. Additional efforts required for monitoring • By default, a single event log / error is not available. Industrial solutions such as Zabbix should be used. This may require additional programming effort to transmit service state. History Today Challenges Solution Advantages Disadvantages Application samples Implementation suggestions Related design patterns Questions
  16. 16. Application Samples Financial Industry. Debtor reliability evaluation. High flow of applications due to business size. Multi-stage evaluation process. The system consists of several independent jobs. The system makes a large number of long-running queries to external systems (BCI). The evaluation process is changed regularly. The system can operate several parallel processes. History Today Challenges Solution Advantages Disadvantages Application Samples Implementation suggestions Related design patterns Questions
  17. 17. Application Samples SMM. Audience analysis. Huge data flow. (billions of messages per day). Users activity varies during the day. It has several high spikes. Each message affects several parameters which should be counted independently. Each message should be counted. Statistics should be calculated in real time. Additional long-running requests to the external systems are required. History Today Challenges Solution Advantages Disadvantages Application Samples Implementation suggestions Related design patterns Questions
  18. 18. Application Samples Online ticket sales. Order processing. Large number of users. Users activity varies during the day. The number of users is large, but the number of tickets is limited, you need to avoid overbooking. There are a lot of independent sales channels, so only the service provider has an accurate information on ticket availability. History Today Challenges Solution Advantages Disadvantages Application Samples Implementation suggestions Related design patterns Questions
  19. 19. Implementation Suggestions It is very useful to define a single message format for all the services even if each service will use only part of the data. It permits new services to be added easily. Data processing and data persistence should .separate into different services. It gives more flexibility and simplifies refactoring in the case where data storage is inadequate for the growth in load. In the case where you have a workplace that requires user interaction, it makes sense to minimize data storage. In the case where you are planning to use XaaS infrastructure do not forget to estimate costs of additional services (queues, storages, etc.). Architecture is a representation of the needs of the business in the form of technological solutions and the cost of ownership is one of the most important business parameters. History Today Challenges Solution Advantages Disadvantages Application Samples Implementation Suggestions Related design patterns Questions
  20. 20. Related architectural templates CQRS (Command and Query Responsibility Separation) Event Sourcing Key-Value storage NoSQL Memory Cache History Today Challenges Solution Advantages Disadvantages Application Samples Implementation Suggestions Related Design Patterns Questions
  21. 21. Questions? If you have any questions feel free to contact me. Email: vbenedichuk@gmail.com LinkedIn: https://ru.linkedin.com/in/vbenedichuk History Today Challenges Solution Advantages Disadvantages Application Samples Implementation Suggestions Related Design Patterns Questions
  • omidm2012

    May. 7, 2018
  • AliKiani2

    Mar. 23, 2018
  • HenriqueMartinsRober

    Nov. 16, 2017

Data stream processing and micro service architecture. English version

Views

Total views

1,199

On Slideshare

0

From embeds

0

Number of embeds

21

Actions

Downloads

1

Shares

0

Comments

0

Likes

3

×