FinOps for private cloud
•Download as PPTX, PDF•
2 likes•161 views
1. The document proposes a low-code solution for billing in a private cloud using open-source tools like KillBill and Prometheus. 2. It outlines an initial architecture that would ingest usage metrics from products, aggregate the data, and publish billing events to KillBill for invoicing and payments. 3. Exporters would collect metrics from products like S3 and ingress and expose them in a format readable by Prometheus for long-term storage and analysis by the billing system.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to FinOps for private cloud
Similar to FinOps for private cloud (20)
More from Alexander Tokarev
More from Alexander Tokarev (20)
Recently uploaded
Recently uploaded (20)
FinOps for private cloud
- 1. Low-Code-биллинг для private cloud Back to FinOps Александр Токарев
- 2. Who am I Head of RnD Platform V Sbertech – platformv.sber.ru AWS and GCP expert The man who cares about money
- 4. 1. In-house FinOps platforms 2. Billing reference architecture 3. Opensource BSS solutions 4. Hand-made billing for private cloud What about now
- 5. Low-Code • YAML/TOML/XML configuration • Zero line on any programming language • Opensource tools only
- 6. What is FinOps – management point “FinOps” is a movement that advocates: 1. a collaborative working relationship between DevOps and Finance data-driven management of infrastructure spending 2. Transparency between IT and finance 3. Cost efficiency, profitability and product delivery pace
- 7. What is FinOps – IT point Near real time reporting + Just-in-time processes + IT and finance teams work together + Shared cloud dictionary = FinOps + Trust = Balance between speed of changes, availability of services and cloud costs
- 8. Opensource FinOps for public cloud Public cloud $$$ Mock API https://cutt.us/tKkrA $$$
- 9. Opensource FinOps for private cloud Private cloud Doesn’t exist!!! Neither IAAS nor PAAS! $$$
- 10. IAAS billing • The best part is dead • Ceilometer is not the best + deprecated • Alive projects aren’t opensource ISP solutions!
- 11. Private cloud FinOps tools 1 relevant link on 5th page! 2 relevant link on 2nd page!
- 12. FinOps platforms
- 13. Exivity
- 14. Exivity
- 15. Exivity
- 16. • Cloud/Virtualization/Backup/Storage software Extractor Templates – AWS, GCP, Openstack, Vmware, Veam, NetApp… • API for custom extractors • Near real-time extraction/billing tools • Lookup management – organization hierarchy • Rate management • Budget management, chargeback/showback • Account management with CMDB integration • Reporting with drilldown • RBAC + SSO Exivity features Very small dev team but perfect in-house FinOps platform! From organization to VM Not a good option for Sber + no Open-Source!
- 17. Full billing cycle reference architecture All IAAS/PAAS products Billing event ingestion User/product onboarding ingestion database Event aggregation Publish to billing API Billing
- 18. Billing features • Extensible product catalog • Extensible product pricing models • Extensible product cancelation policies • Invoicing • API • Proven performance • ………………………….. Too hard to implement from scratch!
- 19. Opensource BSS Language License Community Features set API Documentation Huge customers
- 20. Opensource BSS Language License Community Features set API Documentation Huge customers Java JBL dead rich Decent Bad No Java Apache 2.0 alive huge Decent Good Yes PHP AGPL-3 small rich Good Not bad Yes
- 21. KillBill services architecture Services + plugin API + Internal queue interaction
- 22. KillBill deployment architecture UI and analytic schema PG or MySQL OLTP schema PG or MySQL K8s cluster KillBill KillBill KillBill KaUI Analytic plugin 4 core, 16 Gb 4 core, 16 Gb
- 23. KillBill features • Flat and hierarchical account • Tiered plans, multi-phase plans • Plans versioning + deferred execution time • Prepaid/postpaid billing • Usage/subscription billing • Invoicing • Payment/taxation engine • Entitlement and overdue decoupled engines • Each service could be extended by plugins + set of plugins in the box Very cool for complicated enterprises
- 24. KillBill used features • Flat and hierarchical account • Tiered plans, multi-phase plans • Plans versioning + deferred execution time • Prepaid/postpaid billing • Usage/subscription billing • Invoicing • Payment/taxation engine • Entitlement and overdue decoupled engines • Each service could be extended by plugins + set of plugins in the box • …
- 25. Documentation
- 27. O v e r d u e
- 28. In-house billing implementation steps • Plan catalog population • Usage metrics delivery • Warning about not-paid products
- 29. KillBill product catalog All products All plans
- 30. KillBill plan for S3
- 31. KillBill plan for S3 First Gb is free 5 Gb - 1 RUB More 5 Gb – 0,71 RUB
- 32. In-house billing API calls • Create user billing account • Install product • Send usage • Product removal-related boring stuff…
- 33. KillBill send user registration IAAS /PAAS product Billing event ingestion User/product onboarding ingestion database Event aggregation Publish to billing API Billing
- 34. KillBill send user registration
- 35. KillBill send user registration IAAS /PAAS product Billing event ingestion User /product onboarding ingestion database Event aggregation Publish to billing API Billing
- 37. KillBill send user registration IAAS /PAAS product Billing event ingestion User /product onboarding ingestion database Event aggregation Publish to billing API Billing
- 38. KillBill send S3 usage Decrease API calls count
- 39. IIAS full billing cycle initial architecture IAAS /PAAS product Billing event ingestion User/product onboarding ingestion database Event aggregation Publish to billing API Billing
- 40. IIAS full billing cycle initial architecture IAAS /PAAS product Billing event ingestion User/product onboarding ingestion database Event aggregation Publish to billing API Billing XML with plans
- 41. In-house billing implementation steps • Plan catalog population • Usage metrics delivery • Warning about not-paid products
- 42. In-house billing implementation steps • Plan catalog population • Usage metrics delivery • Warning about not-paid products • Usage metrics population
- 43. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event ingestion User /product onboarding ingestion database Event aggregation Publish to billing API Billing XML with plans Metrics
- 44. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event ingestion User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Metrics
- 45. Metrics 1. Timeseries data 2. Value = usage value 3. Mandatory labels: • Unique private cloud resource name – RN • Metric type: counter or gauge • Unit type from KillBill plan
- 47. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event ingestion User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Metrics
- 48. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics VMAgent YAML with scrape endpoint
- 49. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint
- 50. Billing data population concept • Easier for IAAS/PAAS products • Harder for IAAS/PAAS products Pull billing data Push metrics Metrics exposition endpoint Billing API interaction
- 52. Billing data population • Easier for IAAS/PAAS products • Harder for IAAS/PAAS products Pull billing data Push metrics Single metrics exposition endpoint Billing API interaction Developers are not happy with any approach! State DB Survive reboot Buffer flow consumer
- 53. Generic connector Что мы хотим? Биллинг! Что мы сделаем для этого? Ничего! Кто поработает за нас? Биллинг! •Inbound traffic •Outbound traffic •API calls count
- 55. NGINX ingress metrics labels RN!
- 56. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint
- 57. Ingress exporter duties 1. Scan all ingresses labeled for billing 2. Extract RN from ingress path 3. Extract nginx_ingress_controller_bytes_sent_sum + nginx_ingress_controller_request_size_sum + nginx_ingress_controller_request into download- bytes_total, upload-bytes_total, api_call_count_total counter metrics 4. Label metrics with RN, unit and metric type 5. Write logs 6. Expose metrics in PEF via endpoint Lines: 105 Тут скрин настройки nginx
- 58. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint
- 59. Billing importer duties 1. Extract data from VictoriaMetrics just after previous successful KillBill interaction 2. Aggregate event by RN, unit type and metric type – 24 hour interval 3. Write delta for counter and gauge usage to KillBill API in bulk mode 4. Write self-healing data 5. Write logs 6. Repeat each 60 seconds Lines: 112 Thank you VM for INCREASE function
- 60. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User /product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint
- 61. Portal connector duties 1. Connect to portal Kafka 2. Read tenant create, resource add events 3. Create KillBill accounts, create KillBill subscriptions 4. Write logs Lines: 90
- 62. IIAS full billing cycle initial architecture IAAS/PAAS product Billing event consumption User/product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint
- 63. Common product billing initial architecture IAAS/PAAS product Billing event consumption User/product onboarding ingestion Timeseries database Event aggregation Publish to billing API Billing XML with plans Prometheus Exposition Format Metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation VMAgent YAML with scrape endpoint Self-service portal Event log New/Remove user/product events New/Remove product events New/Remove user/product events Portal connector accounts and subscriptions Ingress API calls Metrics API call count Inbound traffic Outbound traffic Ingress exporter Billing importer Usage metrics Aggregated usage metrics
- 64. 1. Scan all ingresses labeled for billing 2. Extract RN from S3 service tenant namespace name 3. Extract get, head, post, put requests information from S3 /metrics endpoint 4. Merge get+head and put+post into get-head-requests_total and put-post-requests_total counter metrics 5. Count delete requests just in case 6. Extract nginx_ingress_controller_bytes_sent_sum into download-bytes_total counter metrics 7. Extract from S3 CLI current used space for tenant into bytes_sum to gauge metric 8. Label metrics with RN, unit and metric type 9. Write logs 10. Expose metrics in PEF via endpoint Lines: 112 S3 exporter duties Тут скрин настройки nginx No changes in S3 service source code! Mediation microservice!
- 65. Common product billing initial architecture IAAS/PAAS product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics API call count Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events Usage metrics Ingress exporter Ingress API calls New/Remove product events VMAgent YAML with scrape endpoint
- 66. Advanced product billing initial architecture IAAS/PAAS product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics API call count Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events Usage metrics Ingress exporter Ingress API calls New/Remove product events VMAgent YAML with scrape endpoint
- 67. Advanced product billing initial architecture Timeseries database Billing XML with plans Prometheus Exposition Format Metrics API call count Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events Usage metrics Ingress exporter Ingress New/Remove product events VMAgent YAML with scrape endpoint S3 product S3 API calls S3 exporter S3 CLI logs Metrics storage size Metrics API calls per group Metrics Outbound traffic
- 68. Billing product vision из Виктории!!! • Take some data from log/endpoint • Transform to consumption metrics • Expose to Prometheus 90% of code!
- 70. Vector sources
- 71. Vector sources
- 72. Vector sinks
- 73. Vector transform
- 75. Ingress controller metrics configuration Вычитываем Фильтруем Проставляем единицы измерения Проставляем тип метрики Извлекаем RN Очищаем Profit! Python lines: 105 TOML lines: 57 Real TOML lines: 40
- 76. Ingress controller metrics configuration Вычитываем Фильтруем Проставляем единицы измерения Проставляем тип метрики Извлекаем RN Очищаем Profit! Python lines: 105 TOML lines: 57 Real TOML lines: 40 Embedded Vector logging, monitoring, tracing, HA, unit tests!
- 77. Advanced product billing initial architecture S3 product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics S3 exporter S3 API calls S3 CLI logs Ingress Metrics API calls per group Metrics storage size New/Remove product events VMAgent YAML with scrape endpoint
- 78. Low-Code billing architecture S3 product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics S3 exporter S3 API calls S3 CLI logs Ingress Metrics API calls per group Metrics storage size New/Remove product events VMAgent YAML with scrape endpoint
- 79. Low-Code billing architecture Any product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Any traffic metrics Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics Any API calls Any logs Ingress Metrics Any product metric Metrics Any logs metric mediation TOML with pipeline New/Remove product events VMAgent YAML with scrape endpoint S3 exporter S3 CLI logs Metrics storage size Metrics API calls per group Metrics Inbound traffic Outbound traffic
- 80. Consumption quotas • Yet another Low-Code solution • DSL for defining quotas
- 81. Low-Code billing architecture Any product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics Any API calls Any logs Ingress Metrics Any product metric Metrics Any logs metric mediation TOML with pipeline New/Remove product events VMAgent YAML with scrape endpoint
- 82. Final Low-Code billing architecture Any product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics Any API calls Any logs Ingress Metrics Any product metric Metrics Any logs metric mediation TOML with pipeline New/Remove product events VMAgent YAML with scrape endpoint YAML Quotas Definition Language Quotas enforcer
- 83. Final Low-Code billing architecture Any product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics Any API calls Any logs Ingress Metrics Any product metric Metrics Any logs metric mediation TOML with pipeline New/Remove product events VMAgent YAML with scrape endpoint YAML Quotas Definition Language Quotas enforcer
- 84. Final Low-Code billing architecture Any product Timeseries database Billing XML with plans Prometheus Exposition Format Metrics Inbound traffic Outbound traffic Retention – 1 month Timeseries database Retention – 2 years History and disputes investigation Self-service portal Event log New/Remove user/product events Portal connector Billing importer accounts and subscriptions Aggregated usage metrics New/Remove user/product events usage metrics Any API calls Any logs Ingress Metrics Any product metric Metrics Any logs metric mediation TOML with pipeline New/Remove product events VMAgent YAML with scrape endpoint YAML Quotas Definition Language Quotas enforcer Ответственности Команда биллинга – код! Продуктовые команды – Low-Code
- 85. Pains • Generic connector is K8S-tailored • Analytic reports are too slow without KillBill analytics plugin • KillBill is slow for balance requests – 2-3 seconds delay • KillBill is Java-based • Metrics history has gaps • VRL hard for disputes
- 86. Pros • Basic billing metrics – no efforts from dev teams • All SRE are familiar with used software • 80% of the solution – YAML + XML + TOML • KillBill community is alive
- 87. Small remark from Prometheus team
- 88. SAAS and public cloud billing 1. Add push metrics API with Kafka/Vector aggregator buffer layer 2. Add to metricsQL interval queries 3. Improve observability 4. Add more high-availability 5. Replace PG database 6. Invent sophisticated plans 7. Implement hierarchical account
- 89. Sbertech vs Exivity Current solution KillBill overdue/entitlement KillBill payment
- 90. Conclusion 1. Generic private cloud billing solution 2. Pull billing model 3. IAAS/PAAS billing project – 2 SRE 1 month part-time 4. 3 opensource projects 5. 380 lines of Python, 12 lines Rego – generic part, 112 – S3 part 6. 1 XML config, 1 YAML files per product*, 1 TOML file per product 7. KillBill code is intact 8. Customization – code is still required *without K8S deploy manifests
- 91. Q&A We are hiring Александр Токарев: https://www.slideshare.net/AlexanderTokarev4 Your feedback is appreciated!
Editor's Notes
- Шутка про скорость доклада
- Сказать про qr код – видео + слайдшаре
- Добавить слайд вообще что такое лоу код. Задать вектор что код будет но не на языкке программирования.
- Что будет много ямла
- Дать акцент почему отказались
- Голосом говорим, что поэтому не берём
- А вообще почему так?
- Инойсинг – выставление счетов в разные электронные системы
- Для экономии трафика – у меня ошибка. Типа приз. Уточнить у комитета есть ли смысл искать синтаксис в джейсоне.
- Поныть что выбор rn это отдельное искусство
- А что неправильно
- И как вы думаете что это из меток метрик? Правильно – RN.
- OpenEvent
- Непонятно откуда физически тащатся метрики. Может есть смысл вообще тащить их с http
- Норм скрин с цифрами
- Норм скрин с цифрами
- В томл всё равно есть язык, но базовые вещи делаются в нём очень просто
- Gaps - промежутки
- E2e ackowledge
- Идея для стартапа