SlideShare a Scribd company logo
1 of 42
Amir Zuker
Software Architecture
in the
Multi-Cloud Era
Barak Mor Rotem Barda
CTO Software
Architect
Fancy Books
Typical API-over-Data System Architecture (AWS)
Compute Platform
API
Gateway
CloudFront CDN S3
Messaging
Cognito
Storage and
Data Stores
CloudFormation
CI/CD
Monitoring /
Traceability
CloudWatch
Agenda
 What is Multi-Cloud
 Why Multi-Cloud
 Workload Portability
 Abstraction Strategies
5
What is Multi-cloud?
6
Private | Public | Hybrid | Multi
7
Company Grounds
Private | Public | Hybrid | Multi
8
Company Grounds
Private | Public | Hybrid | Multi
9
Company Grounds
Private | Public | Hybrid | Multi
10
Company Grounds
Private | Public | Hybrid | Multi
11
Company Grounds
AWS | Azure | GCP | etc.
Why Multi-cloud?
13
Why Multi-Cloud?
14
BUSINESS COST ARCHITECTURE
Price
Lock-in Benefits Optimize
Regulations
Merges
Product
Customers
QoS
Tech
Rotem Barda
15
One
sensor
Infinite opportunities
https://www.youtube.com/watch?v=m2lTF1GEY6Q
► Unique spatial
signal processing
algorithms
► Most advanced RFIC
for mmWave 4D
imaging
► Patented antennas
and 4D electronic
structures
Algorithm
/
software
Hardwar
e
► Machine learning over
4D imaging radar
signals
AI-
based
service
s
Chi
p
Connecti
ng all
product
aspects
17 Vayyar Imaging Confidential
14/1/22
Vayya
r
Care
Contactless and
private fall
detection
► Robust
Works in steam,
humidity, and all lighting
conditions
► Real-Time Alerts
VOIP call initiated
to caregiver
► Penetration
RF signals pass
through obstacles
► Private
RF-based, no optics
► Passive
No need to wear a
pendant
► Easily installed
No need for a
professional
18 Vayyar Imaging Confidential
14/1/22
Vayyar: Business Requirements
 Global Reach and Customer Needs
 Flexible support of multiple verticals at Vayyar
 Enable different business models (service, integration, etc.)
 Maintain operation alignment
 Costs
19
Vayyar: Challenges
 Multiple divisions and teams
 Ability to innovate
 Reusability across clouds
 Testability across clouds
 Different pricing models
20
Vayyar
Focusing
On
22
Workload Portability
Workload Portability
 Move workloads across clouds
 "Write once, run anywhere"
23
App
Cloud B
App
Cloud A Cloud A
Component
Cloud B
Component System
But How?
Abstraction
25
Dependencies
Abstraction
Strategies
27
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Dependencies
Abstraction
Strategies
28
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Demo
Rituals
API's
Prayers
Repository
[azure]
Prayers
Repository
[aws]
Dependencies
Abstraction
Strategies
29
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Standards
Dependencies
Abstraction
Strategies
30
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Standards
Demo
Rituals
API's
Gremlin
Dependencies
Abstraction
Strategies
31
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Near
Service
Near
Service
Gateway
Gateway
Standards
Dependencies
Abstraction
Strategies
32
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Near
Service
Near
Service
Gateway
Gateway
Standards
Demo
Rituals
API's
Dependencies
Abstraction
Strategies
33
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Near
Service
Near
Service
Gateway
Gateway
Platform
Standards
Dependencies
Abstraction
Strategies
34
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Near
Service
Near
Service
Gateway
Gateway
Platform
Standards
Dependencies
Abstraction
Strategies
35
System
Component
Module
(micro)Service
Accessors
Component
Module
(micro)Service
Accessors
Near
Service
Near
Service
Gateway
Gateway
Platform
Standards
Various Integrative
Tools & Solutions
Azure Arc
Tanzu
Packer
AWS Outposts
GCP Anthos
HPE Ezmeral
SDDC
Abstraction Strategies
36
Abstraction Strategies
 Application level
 Interoperable standards and protocols
 Near-service and gateway components
 Platform-level
 External SaaS
 Various integrative tools and solutions
37
Mix and match
Should everything
be portable?
Specifics still apply
Duplication or rewrite
Progress at Vayyar
39
Multiple portable workloads:
Containerized
Serverless
via enabler platforms
(e.g., CaaS / k8s)
Leverage app-level abstractions
Pluggable dependency components
(e.g., ADX / BigQuery)
Choosing standards and
high-level API's
(e.g., Gremlin / MongoAPI)
Unit, Integration, and
multi-cloud system tests!
..including simulators and load tests
A unified release management with
streamlined CI/CD pipelines
Architecture methodology,
Documentation standards,
Automation and tools
Pave The Way
 Build a multi-cloud strategy
 ..CloudOps
 ..Architecture and methodology
 ..Automation and control plane
 ..Monitoring and observability
 Cost especially
40
41
Multi-Cloud
A journey,
not a destination
Key Takeaways
42
Key Takeaways
 Really understand needs
 Choose Open / Common Platforms
 Integrative tools and solutions
 Big landscape
 IaaS, Private / Hybrid Clouds
 Abstraction | Rewrite strategies
 Single-Cloud?
 Still make considerations
 Establish a (Multi-)Cloud Strategy
43
Amir Zuker Barak Mor Rotem Barda
CTO Software
Architect
45
Slides:
https://www.slideshare.net/amirzuker/
software-architecture-in-the-multi-cloud-era-az
Demos:
https://github.com/barak-codevalue/multi-cloud-demo
Software Architecture in the
Multi-Cloud Era

More Related Content

Similar to Software Architecture in The Multi-Cloud Era AZ

Introducing rubrik a new approach to data protection
Introducing rubrik   a new approach to data protectionIntroducing rubrik   a new approach to data protection
Introducing rubrik a new approach to data protectionDatabarracks
 
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...Amazon Web Services
 
Cloud computing Basics
Cloud computing BasicsCloud computing Basics
Cloud computing BasicsSagar Sane
 
The Evolution of Cloud Architectures: Focusing More on the Business Logic
The Evolution of Cloud Architectures: Focusing More on the Business LogicThe Evolution of Cloud Architectures: Focusing More on the Business Logic
The Evolution of Cloud Architectures: Focusing More on the Business LogicScott Weber
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.pptMunmunSaha7
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.pptahmedraed19
 
The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?Codit
 
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...WSO2
 
webMethods 10.5 and webMethods.io Integration: Everything You Must Know
webMethods 10.5 and webMethods.io Integration: Everything You Must KnowwebMethods 10.5 and webMethods.io Integration: Everything You Must Know
webMethods 10.5 and webMethods.io Integration: Everything You Must KnowKellton Tech Solutions Ltd
 
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
Cloud lockin and interoperability v2   indic threads cloud computing conferen...Cloud lockin and interoperability v2   indic threads cloud computing conferen...
Cloud lockin and interoperability v2 indic threads cloud computing conferen...IndicThreads
 
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
Cloud lockin and interoperability v2   indic threads cloud computing conferen...Cloud lockin and interoperability v2   indic threads cloud computing conferen...
Cloud lockin and interoperability v2 indic threads cloud computing conferen...IndicThreads
 
CCI2017 - Azure Virtual Machine & Networking - Marco Gumini
CCI2017 - Azure Virtual Machine & Networking - Marco GuminiCCI2017 - Azure Virtual Machine & Networking - Marco Gumini
CCI2017 - Azure Virtual Machine & Networking - Marco Guminiwalk2talk srl
 
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...Kangaroot
 
nullcon 2011 - Security and Forensic Discovery in Cloud Environments
nullcon 2011 - Security and Forensic Discovery in Cloud Environmentsnullcon 2011 - Security and Forensic Discovery in Cloud Environments
nullcon 2011 - Security and Forensic Discovery in Cloud Environmentsn|u - The Open Security Community
 
Software Architecture Conference - Monitoring Microservices - A Challenge
Software Architecture Conference -  Monitoring Microservices - A ChallengeSoftware Architecture Conference -  Monitoring Microservices - A Challenge
Software Architecture Conference - Monitoring Microservices - A ChallengeAdrian Cockcroft
 
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUD
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUDTHE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUD
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUDRadu Vunvulea
 

Similar to Software Architecture in The Multi-Cloud Era AZ (20)

Introducing rubrik a new approach to data protection
Introducing rubrik   a new approach to data protectionIntroducing rubrik   a new approach to data protection
Introducing rubrik a new approach to data protection
 
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...
Re-Architecting a Banking Application for Scale and Reliability (SRV220-R1) -...
 
Cloud computing Basics
Cloud computing BasicsCloud computing Basics
Cloud computing Basics
 
Cloud computing
Cloud computingCloud computing
Cloud computing
 
The Evolution of Cloud Architectures: Focusing More on the Business Logic
The Evolution of Cloud Architectures: Focusing More on the Business LogicThe Evolution of Cloud Architectures: Focusing More on the Business Logic
The Evolution of Cloud Architectures: Focusing More on the Business Logic
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.ppt
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.ppt
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.ppt
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.ppt
 
cloudintro-lec01.ppt
cloudintro-lec01.pptcloudintro-lec01.ppt
cloudintro-lec01.ppt
 
The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?The Ideal Approach to Application Modernization; Which Way to the Cloud?
The Ideal Approach to Application Modernization; Which Way to the Cloud?
 
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...
[WSO2 Summit EMEA 2020] Automating an Integrated API Supply Chain Using a Clo...
 
webMethods 10.5 and webMethods.io Integration: Everything You Must Know
webMethods 10.5 and webMethods.io Integration: Everything You Must KnowwebMethods 10.5 and webMethods.io Integration: Everything You Must Know
webMethods 10.5 and webMethods.io Integration: Everything You Must Know
 
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
Cloud lockin and interoperability v2   indic threads cloud computing conferen...Cloud lockin and interoperability v2   indic threads cloud computing conferen...
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
 
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
Cloud lockin and interoperability v2   indic threads cloud computing conferen...Cloud lockin and interoperability v2   indic threads cloud computing conferen...
Cloud lockin and interoperability v2 indic threads cloud computing conferen...
 
CCI2017 - Azure Virtual Machine & Networking - Marco Gumini
CCI2017 - Azure Virtual Machine & Networking - Marco GuminiCCI2017 - Azure Virtual Machine & Networking - Marco Gumini
CCI2017 - Azure Virtual Machine & Networking - Marco Gumini
 
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...
Red Hat OpenShift & CoreOS by Ludovic Aelbrecht, Senior Solution Architect at...
 
nullcon 2011 - Security and Forensic Discovery in Cloud Environments
nullcon 2011 - Security and Forensic Discovery in Cloud Environmentsnullcon 2011 - Security and Forensic Discovery in Cloud Environments
nullcon 2011 - Security and Forensic Discovery in Cloud Environments
 
Software Architecture Conference - Monitoring Microservices - A Challenge
Software Architecture Conference -  Monitoring Microservices - A ChallengeSoftware Architecture Conference -  Monitoring Microservices - A Challenge
Software Architecture Conference - Monitoring Microservices - A Challenge
 
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUD
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUDTHE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUD
THE MONSTER UNDER THE BED – OVERENGINEERING THE CLOUD
 

Recently uploaded

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusZilliz
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbuapidays
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesrafiqahmad00786416
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 

Recently uploaded (20)

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 

Software Architecture in The Multi-Cloud Era AZ

Editor's Notes

  1. Test length: רגע לפני שמסכמים, אין לנו הרבה זמן, אז חשוב לי רק לציין את הדברים הבאים. כפי שאמרנו תמיד סביב בניית תרבות ענן, אז בMC זה יותר (יותר vendors) וגם יותר קריטי *כל הקשור לCloudOps – DevOps, SecOps, FinOps *בנוסף, לדאוג למתודולוגיה, סטנדרטיים ארכיטקטוניים, ולהשקיע בכלים שיתמכו *אוטומציה – workflows (oam) / control plane – איך להעביר, איך לדעת מה נמצא *מוניטורינג – cost בעיקר *אם צריכים עזרה, תשיגו עזרה ...אז כן..
  2. שלום לכולם, ברוכים הבאים לtalk – SW arch in the multi-cloud era אני אמיר צוקר, הCTO של קוד ווליו לצידי ברק מור, ארכיטקט תוכנה מקוד ווליו שיעזור לי עם Demos אני שמח לומר שהtalk משלב סיפור לקוח, ולטובת כך, יעלה בהמשך רותם ברדה מויאר
  3. Eg: fancy books / SaaS B2B / שירותי ספריה לעובדים --- Let's say we're a book store company Fancy Books! This talk: SaaS that can be integrated with businesses to allow their employees to request and receive books and such --- We sell books, but have additional cool features We augment books, add narration and visualizations, and we integrate ML/AI processes and pipelines to make it happen. MVP - We need to build a simple system, webapp + backend We go with a cloud-based system, because, well.. why not
  4. מבוססת AWS --- AWS-based --- HTTP/Rest | WebSocket north south gRPC east west --- Load balancer (network load balancer NLB) Compute – lambda Messaging – SQS, SNS, AmazonMQ Storage – Auroa, Dynamo, Timestream DevOps – CloudFormation -- Customer: we need it be in Azure CFO: we received million$ credits from GCP Mgmt: we acquire another start-up which is Alibaba Cloud Etc.. -- Not be in a place where this is catastrophic – re-do everything, re-build skillset, ecosystem, workflow, etc.
  5. סקירה מהירה על מודלים של פתרונות ענן
  6. סקירה מהירה על מודלים של פתרונות ענן
  7. הבדלים בין IT מסורתי, עולמות on-prem, hosted clouds צד שלישי, פתרונות מבוססות stack וכו' לא חולקים שלנו, עבורנו ובשליטתנו
  8. לצד זה יש את הPubC / AWS.. משתמשים בענן ציבורי, חולקים סביבה עם עסקים אחרים, לא מתחזקים תשתיות --- Offered by providers Less control of what is exposed to internet (in terms of the entire cloud) Scalable (Flexible and elastic scale) Convenient, cost effective, easy to manage Secured Different offerings (IaaS, PaaS, SaaS) Not in your complete control, not owned by you, serves others
  9. בנוסף, יש לנו את הHC – מצב שכולל גם Priv/Pub Cloud נמצא את עצמינו לרוב ב2 מצבים *מיגרציה לענן (נישאר הרבה זמן) *רגולציות, אבטחה וcompliance --- Enjoy best of both worlds Enable gradual migration to the cloud application evolution (enrich via cloud services – e.g. ML/AI) Increased control over data and process Shared security responsibility
  10. ואז יש לנו את הMC בפשטות – מצב בו אנחנו משתמשים בכמה עננים ציבוריים יכול לכלול גם PrivC (או כמה) ואת עולם הHybC אבל בבסיס, שוב, אנחנו צורכים כמה עננים ציבוריים --- As if Hybrid Cloud wasn't hard enough.. Multi-cloud – be able to use multiple cloud providers simultaneously Do we have a private-cloud – COULD BE. In some sense, multi-cloud can include private/on-prem (bloated term) Why? We'll talk about it in a second Wiki: Multicloud (also spelled multi-cloud or multi cloud) is a company's use of multiple cloud computing and storage services from different vendors in a single heterogeneous architecture to improve cloud infrastructure capabilities and cost. It also refers to the distribution of cloud assets, software, applications, etc. across several cloud-hosting environments. With a typical multicloud architecture utilizing two or more public clouds as well as multiple private clouds, a multicloud environment aims to eliminate the reliance on any single cloud provider. DigitalOcean | Oracle | IBM
  11. שימו לב, MC זה לא משהו Far Fetched סקר מHashiCorp, אלפי משיבים, 76% המספרים יכולים להיות שונים, הרבה מחברות גדולות מה שחשוב לקחת מכאן – לא דבר עתידני, מציאות של הרבה מאיתנו --- https://www.hashicorp.com/blog/hashicorp-state-of-cloud-strategy-survey-welcome-to-the-multi-cloud-era Of the more than 3,000 global respondents to HashiCorp’s first State of Cloud Strategy Survey, 76% already work in multi-cloud environments. This overwhelming majority of respondents reported using more than one public or private cloud, making multi-cloud no longer some vague or aspirational goal, but rather their everyday reality. And multi-cloud adoption looks to keep growing from there — 86% of respondents expect to be using multi-cloud within two years. Not surprisingly, multi-cloud adoption skews heavily toward larger enterprises (90% of organizations with more than 5,000 employees are already multi-cloud), but 60% of small businesses (<100 employees) are already multi-cloud, and 81% expect to be multi-cloud within two years.
  12. B: רכישות, מוצר, רגולציות, לקוחות (צרכים/דרישות) C: price-lock / חבילות הטבות / cost opti. A: unique tech / QoS, SLA – HA, DR, Latency --- B: acquisitions, merges | product | regulation | customer C: price lock-in | credits, benefits | cost optimizations A: sla, qos, qa | unique tech
  13. עכשיו שאנחנו יודע מה ולמה MC אני רוצה לנצל את ההזדמנות להזמין את רותם ברדה מויאר לבמה רותם יספר לנו על ויאר (כמובן) וישתף תובנות לגבי המסע שלנו יחד סביב MC --- I'd like to invite Rotem Barda from Vayyar to the stage. We have been working extensively with Vayyar in their journey into the multi-cloud. Let's hear as he takes us through this initiative, process and on-going journey. Mention that they're hybrid?
  14. ויאר באמת פתחו מוצר מדהים, ובבסיסו – הסנסור מחובר לענן, IoT – prov, security, pipelines, messaging, updates, uploads, etc. מסביב כל האפליקציות, מבוסס אנליטיקות, דשבורדינג *חטיבות שונות, מפתחות את המערכת e2e, קורה הרבה כולן פוגשות את אתגר הMC כחלק מהמהלך, מפתחים את VaaP והפלטפורמה הזו גם צריכה להיות MC כמובן *פרויקט מאוד מעניין.. --- radar device Install on a wall or other place מזהה אוביקטים לעקוב אחריהם גלי רדיו Radar vs camera כמה מוצרים נפרדים -- Retail – know more, sell more Public safety – detecting weapons, border keeping Automotive – alert left babies Senior care – falls Medical – Screening easrly stage breast cancer
  15. עושים מעבר חד לWorkload Portability אנחנו מדברים בעצם על MC והיכולת להשתמש בכמה עננים ז"א.. --- To make use of multiple cloud providers Our components might need to run and operate on different environments ..Enters Workload Portability!
  16. שאנחנו רוצים להיות מסוגלים להזיז workloads בין עננים Write once run anywhere שימו לב, Full/Partial יש הבדלים בין הפתרונות אבל זה לא רלוונטי כרגע כי אנחנו מדברים על עקרון שרלוונטי לשניהם -Data – מוסיף סיבוכיות, יכול ליצור lock. Porting של מערכת אקטיבית, לא from scratch – צריך לשלב יחד, מחוץ לסקופ ..אז עולה השאלה.. --- Full vs. Partial Involving data portability – complicates things obviously (continuous/other/etc.) https://thenewstack.io/the-4-definitions-of-multicloud-part-3-workload-portability/ Multicloud workload portability means you can push a button and move a workload from one cloud or on-premises data center to another. Sort of like the idea: “Write it once, and run it anywhere.” Unfortunately, it’s very difficult to write an app once for one cloud and still be able to run that app on other clouds with no code modifications. Different vendors have different APIs, semantics, capabilities, syntax and other nuances that make workload portability, in reality, one of the most challenging forms of multicloud portability. Achieving workload portability isn’t as simple as write once run everywhere, but it is doable. By its nature, it’s more complicated because it is a superset of data and workflow portability, meaning both of them are required in order for this type of portability to work. It is a viable strategy and depending on the business requirements, might be required. Companies may need to implement workload portability for compliance and regulatory reasons to enable failover between multiple cloud vendors. Others might do it for cost savings. One example is a large hedge fund that used HashiCorp’s workload orchestrator, Nomad, to schedule portable workloads onto the cheapest cloud vendor and instance type each day, leveraging things like spot instance pricing. There are three types of workload portability: Full workload portability Partial workload portability Dataless workload portability Some of these types may make sense for your use case if the challenges of enablement are worth the returns in cost or capability.
  17. איך? נושא נרחב, ויש את הפרקטיקה של קוד ווליו, במסגרת הזמן שיש לנו, חשוב לי להתרכז בדבר אחד וזה.. --- But how? Both Full or Partial
  18. Abstraction! הטכניקה מאפשרת לנו לקחת את התלות בdeps ולבצע generalization אפקטיבית ובנוסף גם encapsulation כדי שלא יהיה leakage לשאר אזורי המערכת. Abstraction זהו שם המשחק וכלי מהותי להפחתה של צמידות ויצירת גמישות סביב portability. יש מצבים Too many or bad – role of architect לדעת כיצד לעשות את האבסטרקציות הנכונות. אז בואו ניגע בכמה מהאסטרטגיות אבסטרקציה העיקריות.. --- The process of taking away or removing characteristics from something in order to reduce it to a set of essential characteristics Generalize, minimize specifics, then encapsulate -- DRY – don't repeat yourself WET - “write everything twice”, “we enjoy typing” or “waste everyone’s time” AHA – avoid hasty abstraction / abstraction hell --- The wrong abstraction can and will become far more complex and harder to read or maintain than a code duplication, over time. --- The role of the architect How can one achieve it? Let's focus on that
  19. יש מערכת (לא משנה..), משתמשת בdeps, רוצים לעשות אבס. *app-level – יכול כשכבה (לא משנה..), יכול ע"י להוצאה למיקרוסרביס דמו * Standards (mongo-api דוגמא) * near-service/gateway – kong restify, db restify, dapr, serverless fw * Platform – סביבה דומה למערכת / סביבה שהיא בעצמה interoperable, k8s, ecosystem *SaaS – co-location – MongoAtlas, CloudAmqp, Auth0, Cloudinary IaaS פתרונות מגוונים ומוצרי מדף, יכולות שונות של remote או פתרונות הוליסטיים ואינטגרטיביים – control plane, tools מי שצריך את הramp-up ו"בטריות בפנים" – יכול להתאים, שווה לבדוק, למרות שזו נעילה לכלי או נותן שירות אחר בעצם --- Platform: Gcp cloud run / aws fargate / azure container apps --- Near-service – service-mesh or even gateways that do protocol translation (e.g. IoT edge), dapr, serverless fw, Kong – de-graphql, restify dbs App-template standards – OAM (open application model) Service-Standards – HTTP, REST, gRPC, AMQP, MQTT Dep-Standards – Kafka API with EventHub, CosmosDb MongoAPI, Aurora Postgres, Gremlin with AWS Neptune and Azure CosmosDb – FINE PRINT however! (not all is supported) Platform: K8s/EKS/AKS/microk8s/etc., ECS Anywhere, AppService Anywere, Logic App Anywhere, Azure Fn as container, etc. - Supported across environments (e.g. k8s) - Provides similar runtime characteristics and capabilities for SW components (e.g., Azure Container Apps/AWS ECS & Fargate/GCP Cloud Run) - Commonality: Popular target for third party platforms/tools/services/technologies where these support deployment over this platform - e.g. k8s - MinIO, RMQ, Mongo, etc. SaaS – Auh0, Mongo Atlas, CloudAMQP, Cloudinary IaaS – servers, RAM, disks, network, etc. Platform – driven by containerization Infra (physical) – driven by virtualization AWS Outposts The concept of AWS Outposts, geared toward customers with hybrid cloud architectures, is rather simple. The service enables customers to use AWS compute and storage services (AWS EC2 and EBS) within their own data centers but with hardware provided by AWS. Google Cloud Anthos With Google Cloud Anthos, customers can take advantage of their existing on-premises or public cloud investments by allowing them to modernize their applications and run them anywhere. At its core, Anthos uses the Google Kubernetes Engine (GKE) and other existing GCP services to provide an easy hybrid pathway and familiar development experience for engineering teams already using Google Cloud. Microsoft Azure Arc Microsoft Azure Arc allows customers to simplify the use of different environments by extending Azure management capabilities to any infrastructure, regardless of its location. By using Kubernetes, Azure Arc provides the ability to deploy and manage container-based applications. However, Azure Arc can also organize other types of resources, such as Windows and Linux services.
  20. יש מערכת (לא משנה..), משתמשת בdeps, רוצים לעשות אבס. *app-level – יכול כשכבה (לא משנה..), יכול ע"י להוצאה למיקרוסרביס
  21. DEMO: App-level
  22. Standards (mongo-api דוגמא) --- App-template standards – OAM (open application model) Service-Standards – HTTP, REST, gRPC, AMQP, MQTT Dep-Standards – Kafka API with EventHub, CosmosDb MongoAPI, Aurora Postgres, Gremlin with AWS Neptune and Azure CosmosDb – FINE PRINT however! (not all is supported)
  23. Demo - standards
  24. near-service/gateway – kong restify, db restify, Protocol translation and bridging dapr, service mesh, serverless fw --- Near-service – service-mesh or even gateways that do protocol translation (e.g. IoT edge), dapr, serverless fw, Kong – de-graphql, restify dbs
  25. Demo – dapr
  26. Platform – סביבה דומה למערכת – runtime features and capabilities דומים לcompute שלנו Gcp cloud run / aws fargate / azure container apps סביבה שהיא קומונלית, שאנחנו יכולים להעביר את הפלטפורמה בין פרוביידרס ברמות שונות חשוב להזכיר את k8s De-facto standard, ecosystem אדיר – managed offerings, מוצרי מדף / כלים וטכנולוגיות – targeting k8s k8s - ops is different - can be complex- במיוחד בין private לpublic אם תקראו, הרבה מוביל לk8s, קפיצה גדולה קדימה, אבל זה לא לכולם --- Platform: K8s/EKS/AKS/microk8s/etc., ECS Anywhere, AppService Anywere, Logic App Anywhere, Azure Fn as container, etc. - Supported across environments (e.g. k8s) - Provides similar runtime characteristics and capabilities for SW components (e.g., Azure Container Apps/AWS ECS & Fargate/GCP Cloud Run) - Commonality: Popular target for third party platforms/tools/services/technologies where these support deployment over this platform - e.g. k8s - MinIO, RMQ, Mongo, etc.
  27. External SaaS Co-location --- Auh0, Mongo Atlas, CloudAMQP, Cloudinary
  28. IaaS – מחוץ לסקופ, עולם גדול, כלים סביב virtualization (sddc לדוגמא), אפשר לעשות בעצמכם סביב תרבות Cloud IT, או להשתמש במוצרי.. פתרונות מגוונים ומוצרי מדף - יכולות שונות של remote או פתרונות הוליסטיים ואינטגרטיביים – control plane, tools שווה להזכיר – open-shift, פתרון מעל k8s שנותן יכולות כפי שציינו מי שצריך את הramp-up ו"בטריות בפנים" ולהגביר את רמת הבטחון כי יש מישהו מאחורי המוצר (סוג של פרטנר) יכול להתאים, שווה לבדוק, למרות שזה אומר להכניס משהו לא קטן, וליצור lock במקומות אחרים בעצם --- IaaS – servers, RAM, disks, network, etc. – virtualization! AWS Outposts The concept of AWS Outposts, geared toward customers with hybrid cloud architectures, is rather simple. The service enables customers to use AWS compute and storage services (AWS EC2 and EBS) within their own data centers but with hardware provided by AWS. Google Cloud Anthos With Google Cloud Anthos, customers can take advantage of their existing on-premises or public cloud investments by allowing them to modernize their applications and run them anywhere. At its core, Anthos uses the Google Kubernetes Engine (GKE) and other existing GCP services to provide an easy hybrid pathway and familiar development experience for engineering teams already using Google Cloud. Microsoft Azure Arc Microsoft Azure Arc allows customers to simplify the use of different environments by extending Azure management capabilities to any infrastructure, regardless of its location. By using Kubernetes, Azure Arc provides the ability to deploy and manage container-based applications. However, Azure Arc can also organize other types of resources, such as Windows and Linux services.
  29. אז נגענו בלא מעט אסטרטגיות.. --- App: Module / component / service / subsystem / etc. – Microservice-based or Modular Monolith Az – iot-hub / aws – iot / gcp – iot core Edge: az – iot-edge / aws – greengrass / gcp – iot core
  30. App, סטנדרטים, near-service gw Platform-level, SaaS חיצוני וכלים מגוונים, הוליסטיים, אינטגרטיבים וכדומה *שימו לב, הכוונה.. --- App: Module / component / service / subsystem / etc. – Microservice-based or Modular Monolith Standards: App-template – OAM (open application model) Service – HTTP, REST, gRPC, AMQP, MQTT Dep – Kafka API with MongoDb, CosmosDb MongoAPI, Aurora Postgres, Gremlin with AWS Neptune and Azure CosmosDb – FINE PRINT however! (not all is supported) Near-service: Service mesh, Dapr, API Gateway request transformation, serverless fw, gateways that do protocol translation (e.g. IoT edge), Kong – de-graphql, , restify dbs Platform: K8s/EKS/AKS/microk8s/etc., ECS Anywhere, AppService Anywere, Logic App Anywhere, Azure Fn as container, etc. Driven by containerization - Supported across environments (e.g. k8s) - Provides similar runtime characteristics and capabilities for SW components (e.g., Azure Container Apps/AWS ECS & Fargate/GCP Cloud Run) - Commonality: Popular target for third party platforms/tools/services/technologies where these support deployment over this platform - e.g. k8s - MinIO, RMQ, Mongo, etc. SaaS: Auth0, Mongo Atlas, CloudAMQP, Cloudinary, etc. Holistic: Platform + control plane + Workflow/RnD/Business integrative Azure Arc, OpenShift, Tanzu, OpenStack, Packer, etc. Might diminish fear, transfer responsibility, have something to talk to Can help greatly in case of large companies and involving hybrid cloud (+on-prem) AWS Outposts The concept of AWS Outposts, geared toward customers with hybrid cloud architectures, is rather simple. The service enables customers to use AWS compute and storage services (AWS EC2 and EBS) within their own data centers but with hardware provided by AWS. Google Cloud Anthos With Google Cloud Anthos, customers can take advantage of their existing on-premises or public cloud investments by allowing them to modernize their applications and run them anywhere. At its core, Anthos uses the Google Kubernetes Engine (GKE) and other existing GCP services to provide an easy hybrid pathway and familiar development experience for engineering teams already using Google Cloud. Microsoft Azure Arc Microsoft Azure Arc allows customers to simplify the use of different environments by extending Azure management capabilities to any infrastructure, regardless of its location. By using Kubernetes, Azure Arc provides the ability to deploy and manage container-based applications. However, Azure Arc can also organize other types of resources, such as Windows and Linux services.
  31. הכוונה היא שלא תשתמשו באחד Mix and match תפקיד שלנו ארכיטקטים (too many, bad) *בנוסף, עוד אסטרטגיה – לא לעשות (dup, rewrite) *specifics - cost of abstraction - miss "managed features" (DPS - Device Twin Desired Properties / GCP - Device Config) גם לא בהכרח הכול צריך להיות portable בכללותו, לדוגמא unique tech --- Mix and match – arsenal, utility belt. Also – rewrite (WET) E.g. glue type functions – abstraction isn't necessarily worth it. Plus, should everything be portable? Perhaps not. Some vendors might have specific advantages. Sometimes abstractions to run everywhere could be too complex to build and maintain. Multi-cloud is not about running everything everywhere. Some parts could utilize specific environments for an optimal outcome.
  32. Workloads – caas/k8s (plat) | serverless – entry/glue/dapr/knative App-level – pluggable, ADX/BigQuery Standards – protocols, higher-level APIs, gremlin/mongo-api Tests Releast mgmt. Method, automation, tools (cli's, doc as code, devcontainers, etc)
  33. רגע לפני שמסכמים, אין לנו הרבה זמן, אז חשוב לי רק לציין את הדברים הבאים. כפי שאמרנו תמיד סביב בניית תרבות ענן, אז בMC זה יותר (יותר vendors) וגם יותר קריטי *כל הקשור לCloudOps – DevOps, SecOps, FinOps *בנוסף, לדאוג למתודולוגיה, סטנדרטיים ארכיטקטוניים, ולהשקיע בכלים שיתמכו *אוטומציה – workflows (oam) / control plane – איך להעביר, איך לדעת מה נמצא *מוניטורינג – cost בעיקר *אם צריכים עזרה, תשיגו עזרה ...אז כן.. --- Strategy <-> Culture: CloudOps, Architecture, Methodology, Standards, and Practice Monitoring, Automation and Control Plane Diversity (autonomy vs. uniformity) / Skillset Shape your multi-cloud strategy/culture Lengths: Mileage may vary Continuous investment - Every service, platform, approach - Carefully examine in multi-cloud glasses Unit of Reuse: Code – reuse as much as possible "Make it an Ops issue" | Ops – still need to account for multi-cloud, be strong and innovating Dev: Effective decomposition and encapsulation Regardless micro/macro/nano-services, serverless, modular monolith, etc. Need something specific to a cloud vendor? See how you can encapsulate it Abstractions – still something to consider Open / interoperable / standard / common "Quick wins" CI/CD: Release management Workflow portability Use enabler tools and technologies Tools: OAM, Terraform, Pulumi, Ansible
  34. A journey, not a destination Investment, and continuous --- Every service, platform, approach - Carefully examine in multi-cloud glasses Continuous investment Requires control and discipline Maintain a list, document, etc. CLI's to create Streamline, etc.
  35. בואו נסכם, key takeaways
  36. בראשונה – להבין צרכים, משפיע פלטפורמה – תבחרו דברים פתוחים וקומונלים – k8s long way אבל לא לכולם יש עולם ענק של מוצרים ושירותים שיכולים לעשות (במיוחד Iaas, Priv/Hyb-Cloud) תשתמשו באבסטרקציות נכונות נמצאים בSingle – הכל טוב, עדיין תשקלו quick wins לגמישות עתידית (Arch/Ops-oam) תשקיעו באסטרטגיה MC שמתאימה לכם ...אני רוצה להודות --- - Understand needs: Determine desired lengths, solution, etc. Holistic: Big landscape, do pilots, ok not to marry K8s – a big step forward, but not for everyone --- Really understand your needs and the motivation behind it Determine strategy and length -- Hybrid / IaaS Look into holistic and integrative virtualization solutions, big landscape -- Platform Leverage containerization Choose supporting ecosystem (open, interoperable, common) -- Abstraction | Rewrite strategies App-level, Near-service, Standards, Platform, SaaS, Holistic Currently going Single-Cloud? Still consider: Effective architecture Modern workflows with multi-cloud enabler ecosystem Invest in (Multi-)Cloud Culture Cost, Security, Monitoring, Automation and Control Plane --
  37. אני רוצה להודות לברק מור על הקלידים, לרותם ברדה שהעשיר את כולנו מזוויתה של ויאר. וכמובן – לכולכם על ההקשבה.. ...זהו, (לינקים)
  38. לינקים זמן לשאלות?