Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scaling Micro- 
Services in Go 
Matt Heath, Hailo
PHP 
API 
Load Balancer 
PHP 
API 
PHP 
API 
PHP 
API 
Java 
App 
Load Balancer 
PHP 
API 
PHP 
API 
MySQL 
Redis MySQL Sl...
ALL 
HAIL 
THE 
MONOLITH
PHP 
API 
Load Balancer 
PHP 
API 
PHP 
API 
PHP 
API 
Java 
App 
Load Balancer 
PHP 
API 
PHP 
API 
MySQL 
Redis MySQL Sl...
PHP 
API 
MySQL 
PHP 
API 
PHP 
Service 
Java 
Service 
ELB 
ELB ELB 
PHP 
API 
PHP 
API 
per city multi-region 
ELB 
Cass...
PHP 
API 
MySQL 
PHP 
Service 
Java 
Service 
ELB 
ELB ELB 
PHP 
API 
PHP 
API 
per city multi-region 
ELB 
Cassandra 
PHP...
PHP 
API 
MySQL 
PHP 
Service 
Java 
Service 
ELB 
ELB ELB 
PHP 
API 
PHP 
API 
per city multi-region 
ELB 
Cassandra 
PHP...
Communication paths 
with two programmers 
Communication paths 
with three programmers 
Communication paths 
with four pro...
DEVELOPING NEW FEATURES 
UNCLEAR RESPONSIBILITIES 
SLOW FAILOVER 
LACK OF AUTOMATION
“construct a highly agile and 
highly available service from 
ephemeral and assumed broken 
components” - Adrian Cockcroſt
Cassandra 
Go 
Service 
Go 
Service 
Java 
Service 
ELB 
Go “Thin” API 
Go 
Service 
us-east-1 eu-west-1 
Go 
Service 
Jav...
Logic 
Service 
Handler 
Storage
Handler 
Library for building services that talk Protobuf via RMQ 
Logic 
Storage 
Service 
go-platform-layer
package handler 
import ( 
"..." 
) 
func Foo(req *server.Request) (proto.Message, errors.Error) { 
}
service.Register( 
&service.Endpoint{ 
Name: "foo", 
Mean: 100, 
Upper95: 200, 
Handler: handler.Foo, 
} 
)
Handler 
Library for building services that talk Protobuf via RMQ 
Logic 
Storage go-service-layer 
Self-configuring exter...
package handler 
import ( 
"github.com/hailocab/gomemcache/memcache" 
) 
func Foo(req *server.Request) (proto.Message, err...
Handler go-platform-layer 
Library for building services that talk Protobuf via RMQ 
Logic 
Storage go-service-layer 
Self...
CI Pipeline (Janky/Jenkins) 
Amazon S3 
Provisioning Service Provisioning Service Provisioning Service 
Provisioning Manag...
CI Pipeline (Janky/Jenkins) 
Amazon S3 
Provisioning Service Provisioning Service Provisioning Service 
Provisioning Manag...
AUTOMATIC SERVICE DISCOVERY 
Provisioning Service 
Rabbit MQ 
Provisioning Service 
Discovery 
Service 
New 
Service
AUTOMATIC SERVICE DISCOVERY 
Provisioning Service 
Rabbit MQ 
Discovery 
Service 
Binding 
Service 
Provisioning Service 
...
SMALL INDEPENDENT SERVICES 
SINGLE RESPONSIBILITY 
EASE PAIN, SCALE RAPIDLY
CLOUD NATIVE / ANTIFRAGILE 
EXPECT FAILURE 
AUTOMATE EVERYTHING
TESTING
LOAD 
FAILURE 
DEGRADATION
15,000 JOBS/HOUR 
50,000 DRIVERS 
30,000+ REQ/S
CONTINUOUS 
PRODUCTION 
TESTING
MONITORING
AUTOMATIC HEALTHCHECK REGISTRATION 
Provisioning Service 
Rabbit MQ 
Monitoring 
Service 
Provisioning Service 
Service 
P...
DISTRIBUTED 
TRACING
hailo~2~api api.v1.customer service.customer 
hailo~2~api api.v1.customer service.customer
hailo~2~api api.v1.customer service.customer 
REQ 
REP 
REQ 
REP 
IN 
OUT 
IN 
OUT 
hailo~2~api api.v1.customer service.cu...
{ 
"timestamp": 1410262798427145176, 
"traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", 
"type": "OUT", 
"messageId": "a6...
Phosphor 
Host Instances 
Publish 
Service A 
Trace Library 
chan 
goroutine 
UDP 
Service B 
Trace Library 
chan 
gorouti...
var traceChan chan []byte 
func init() { 
// Use a buffered channel 
traceChan = make(chan []byte, 200) 
// Fire off a bac...
Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6 
2014-09-07 02:20:19.867 [/] [START] → - 
2014-09-07 02:20:19.867 [eu-west-1...
SO, GO?
SMALL 
SIMPLE 
EASY TO READ & LEARN
CONCURRENCY PRIMITIVES 
INTERFACE SUPPORT
EASY DEPLOYMENT & 
MANAGEMENT IN PRODUCTION 
AND AT SCALE
СПАСИБО!
IMAGES 
HMS President: Roger Marks 
Orbital Ion Cannon: www.rom.ac 
Go Gophers: Renee French 
Utopia/Dystopia: Dylan Glynn...
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Upcoming SlideShare
Loading in …5
×

Mасштабирование микросервисов на Go, Matt Heath (Hailo)

2,279 views

Published on

Доклад Мэтта Хиза на HighLoad++ 2014.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Mасштабирование микросервисов на Go, Matt Heath (Hailo)

  1. 1. Scaling Micro- Services in Go Matt Heath, Hailo
  2. 2. PHP API Load Balancer PHP API PHP API PHP API Java App Load Balancer PHP API PHP API MySQL Redis MySQL Slave eu-west-1
  3. 3. ALL HAIL THE MONOLITH
  4. 4. PHP API Load Balancer PHP API PHP API PHP API Java App Load Balancer PHP API PHP API MySQL Redis MySQL Slave eu-west-1
  5. 5. PHP API MySQL PHP API PHP Service Java Service ELB ELB ELB PHP API PHP API per city multi-region ELB Cassandra PHP Service MySQL MySQL Java App Java App Java App
  6. 6. PHP API MySQL PHP Service Java Service ELB ELB ELB PHP API PHP API per city multi-region ELB Cassandra PHP Service MySQL MySQL Java App Java App Java App PHP API
  7. 7. PHP API MySQL PHP Service Java Service ELB ELB ELB PHP API PHP API per city multi-region ELB Cassandra PHP Service MySQL MySQL Java App Java App Java App PHP API
  8. 8. Communication paths with two programmers Communication paths with three programmers Communication paths with four programmers Communication paths with five programmers Communication paths with ten programmers
  9. 9. DEVELOPING NEW FEATURES UNCLEAR RESPONSIBILITIES SLOW FAILOVER LACK OF AUTOMATION
  10. 10. “construct a highly agile and highly available service from ephemeral and assumed broken components” - Adrian Cockcroſt
  11. 11. Cassandra Go Service Go Service Java Service ELB Go “Thin” API Go Service us-east-1 eu-west-1 Go Service Java Service ELB Go “Thin” API RabbitMQ Message Bus (federated clusters per AZ) RabbitMQ Message Bus (federated clusters per AZ) Cassandra
  12. 12. Logic Service Handler Storage
  13. 13. Handler Library for building services that talk Protobuf via RMQ Logic Storage Service go-platform-layer
  14. 14. package handler import ( "..." ) func Foo(req *server.Request) (proto.Message, errors.Error) { }
  15. 15. service.Register( &service.Endpoint{ Name: "foo", Mean: 100, Upper95: 200, Handler: handler.Foo, } )
  16. 16. Handler Library for building services that talk Protobuf via RMQ Logic Storage go-service-layer Self-configuring external service adapters Service go-platform-layer
  17. 17. package handler import ( "github.com/hailocab/gomemcache/memcache" ) func Foo(req *server.Request) (proto.Message, errors.Error) { item, err := memcache.Get("key") if err != nil { // handle error } // ... }
  18. 18. Handler go-platform-layer Library for building services that talk Protobuf via RMQ Logic Storage go-service-layer Self-configuring external service adapters Services get for free: • Provisioning • Service discovery • Configuration • Monitoring • Authentication/authorisation • AB testing • Self configuring connectivity to third-party services Service
  19. 19. CI Pipeline (Janky/Jenkins) Amazon S3 Provisioning Service Provisioning Service Provisioning Service Provisioning Manager / Scheduler
  20. 20. CI Pipeline (Janky/Jenkins) Amazon S3 Provisioning Service Provisioning Service Provisioning Service Provisioning Manager / Scheduler
  21. 21. AUTOMATIC SERVICE DISCOVERY Provisioning Service Rabbit MQ Provisioning Service Discovery Service New Service
  22. 22. AUTOMATIC SERVICE DISCOVERY Provisioning Service Rabbit MQ Discovery Service Binding Service Provisioning Service New Service
  23. 23. SMALL INDEPENDENT SERVICES SINGLE RESPONSIBILITY EASE PAIN, SCALE RAPIDLY
  24. 24. CLOUD NATIVE / ANTIFRAGILE EXPECT FAILURE AUTOMATE EVERYTHING
  25. 25. TESTING
  26. 26. LOAD FAILURE DEGRADATION
  27. 27. 15,000 JOBS/HOUR 50,000 DRIVERS 30,000+ REQ/S
  28. 28. CONTINUOUS PRODUCTION TESTING
  29. 29. MONITORING
  30. 30. AUTOMATIC HEALTHCHECK REGISTRATION Provisioning Service Rabbit MQ Monitoring Service Provisioning Service Service Publish Healthchecks
  31. 31. DISTRIBUTED TRACING
  32. 32. hailo~2~api api.v1.customer service.customer hailo~2~api api.v1.customer service.customer
  33. 33. hailo~2~api api.v1.customer service.customer REQ REP REQ REP IN OUT IN OUT hailo~2~api api.v1.customer service.customer
  34. 34. { "timestamp": 1410262798427145176, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "OUT", "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "parentMessageId": "", "from": "com.hailocab.hshell", "to": "com.hailocab.service.nearest-driver.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a", "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”, "duration": 11222094 } { "timestamp": 1410262798416053450, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "REQ", "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf", "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "from": "com.hailocab.service.nearest-driver", "to": "com.hailocab.service.zoning.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a" }
  35. 35. Phosphor Host Instances Publish Service A Trace Library chan goroutine UDP Service B Trace Library chan goroutine UDP Trace Service Dashboards Monitoring In-memory Aggregates Optional persistant storage
  36. 36. var traceChan chan []byte func init() { // Use a buffered channel traceChan = make(chan []byte, 200) // Fire off a background worker for this channel defaultClient = NewClient(traceChan) go defaultClient.publisher() } // Send, drops trace if the backend is at capacity func Send(msg []byte) { select { case traceChan <- msg: // Success default: // Default case fired if channel is full // Ensures this is non blocking } }
  37. 37. Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6 2014-09-07 02:20:19.867 [/] [START] → - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms 2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms 2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms 2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms 2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 ms ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms 2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms 2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms 2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
  38. 38. SO, GO?
  39. 39. SMALL SIMPLE EASY TO READ & LEARN
  40. 40. CONCURRENCY PRIMITIVES INTERFACE SUPPORT
  41. 41. EASY DEPLOYMENT & MANAGEMENT IN PRODUCTION AND AT SCALE
  42. 42. СПАСИБО!
  43. 43. IMAGES HMS President: Roger Marks Orbital Ion Cannon: www.rom.ac Go Gophers: Renee French Utopia/Dystopia: Dylan Glynn Control Room: NASA

×