This document discusses the use of Go at Badoo for backend services. It begins by describing Badoo's backend technologies in 2013 and 2014, which included PHP, C/C++, Python, and the introduction of Go in 2014. It then covers Badoo's infrastructure and practices for Go projects, such as logging, testing, and dependencies. Specific Go services developed at Badoo are then summarized, including "Bumped" which was completed in a week by three people. Metrics are provided showing initial performance of ~2800 requests/second but with long GC pauses. The document concludes with tips and examples for profiling and optimizing memory usage in Go.
Building a queueing system in MongoDB and monitoring your cluster. Presentation by David Mytton at MongoSF May 2011 and MongoDB London User Group July 2011.
Everything you wanted to know about Stack Traces and Heap DumpsAndrei Pangin
Stack traces and heap dumps are not just debugging tools; they open a doorway to the very heart of the Java Virtual Machine. This session is focused on peculiarities of JDK somehow related to heap walking and thread stacks.
• How to create dump in production without side effects?
• What is the performance impact of stack traces?
• How do jmap and jstack work under the hood, and what is special about 'forced' mode?
• Why do all profilers lie, and how to deal with it?
• How to scan heap using JVMTI?
The presentation supported by the live demos will address all these questions. It will also introduce new Java 9 Stack-Walking API (JEP 259) and reveal some useful but little known features of HotSpot JVM.
There is hardly a Senior Java developer who has never heard of sun.misc.Unsafe. Though it has always been a private API intended for JDK internal use only, the popularity of Unsafe has grown too fast, and now it is used in many open-source projects. OK.RU is not an exception: its software also heavily relies on Unsafe APIs.
During this session we'll try to understand what is so attractive about Unsafe. Why do people keep using it regardless the warnings of removal from future JDK releases? Are there any safe alternatives to private API or is it absolutely vital? We will review the typical cases when Java developers prefer to go unsafe and discuss major benefits and the drawbacks of it. The report will be supported by the real examples from OK.RU experience.
Odoo Online platform: architecture and challengesOdoo
A short introduction to the technical architecture of the Odoo Online platform, including the advanced integrated features (instant DNS, email gateways, etc.), and the technical aspect of the SLA.
By Olivier Dony - Lead Developer & Community Manager, OpenERP
Down to Stack Traces, up from Heap DumpsAndrei Pangin
Глубже стек-трейсов, шире хип-дампов
Stack trace и heap dump - не просто инструменты отладки; это потайные дверцы к самым недрам виртуальной Java машины. Доклад будет посвящён малоизвестным особенностям JDK, так или иначе связанным с обоходом хипа и стеками потоков.
Мы разберём:
- как снимать дампы в продакшне без побочных эффектов;
- как работают утилиты jmap и jstack изнутри, и в чём хитрость forced режима;
- почему все профилировщики врут, и как с этим бороться;
- познакомимся с новым Stack-Walking API в Java 9;
- научимся сканировать Heap средствами JVMTI;
- узнаем о недокументированных функциях Хотспота и других интересных штуках.
Building a queueing system in MongoDB and monitoring your cluster. Presentation by David Mytton at MongoSF May 2011 and MongoDB London User Group July 2011.
Everything you wanted to know about Stack Traces and Heap DumpsAndrei Pangin
Stack traces and heap dumps are not just debugging tools; they open a doorway to the very heart of the Java Virtual Machine. This session is focused on peculiarities of JDK somehow related to heap walking and thread stacks.
• How to create dump in production without side effects?
• What is the performance impact of stack traces?
• How do jmap and jstack work under the hood, and what is special about 'forced' mode?
• Why do all profilers lie, and how to deal with it?
• How to scan heap using JVMTI?
The presentation supported by the live demos will address all these questions. It will also introduce new Java 9 Stack-Walking API (JEP 259) and reveal some useful but little known features of HotSpot JVM.
There is hardly a Senior Java developer who has never heard of sun.misc.Unsafe. Though it has always been a private API intended for JDK internal use only, the popularity of Unsafe has grown too fast, and now it is used in many open-source projects. OK.RU is not an exception: its software also heavily relies on Unsafe APIs.
During this session we'll try to understand what is so attractive about Unsafe. Why do people keep using it regardless the warnings of removal from future JDK releases? Are there any safe alternatives to private API or is it absolutely vital? We will review the typical cases when Java developers prefer to go unsafe and discuss major benefits and the drawbacks of it. The report will be supported by the real examples from OK.RU experience.
Odoo Online platform: architecture and challengesOdoo
A short introduction to the technical architecture of the Odoo Online platform, including the advanced integrated features (instant DNS, email gateways, etc.), and the technical aspect of the SLA.
By Olivier Dony - Lead Developer & Community Manager, OpenERP
Down to Stack Traces, up from Heap DumpsAndrei Pangin
Глубже стек-трейсов, шире хип-дампов
Stack trace и heap dump - не просто инструменты отладки; это потайные дверцы к самым недрам виртуальной Java машины. Доклад будет посвящён малоизвестным особенностям JDK, так или иначе связанным с обоходом хипа и стеками потоков.
Мы разберём:
- как снимать дампы в продакшне без побочных эффектов;
- как работают утилиты jmap и jstack изнутри, и в чём хитрость forced режима;
- почему все профилировщики врут, и как с этим бороться;
- познакомимся с новым Stack-Walking API в Java 9;
- научимся сканировать Heap средствами JVMTI;
- узнаем о недокументированных функциях Хотспота и других интересных штуках.
Presentation by David Mytton about monitoring MongoDB at the MongoSV conference 3rd Dec 2010.
A full blog series covering everything in this presentation is at http://blog.boxedice.com/mongodb-monitoring/
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.
Puppet Module Reusability - What I Learned from Shipping to the ForgePuppet
A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.
Gareth Rushgrove
Technical Architect, Government Digital Service
Gareth Rushgrove is now a technical architect at the Government Digital Service, part of the UK Government. He is mainly interested in configuration management, infrastructure and platform as a service, deployment and monitoring tooling and the whole devops community. He thinks when used well together these allow you to move really fast, even in tightly controlled environments like Government. When not working, Gareth can be found blogging over on morethanseven.net or uploading code to GitHub. He also curates the Devops Weekly newsletter and occasionally organises community events.
Практический опыт профайлинга и оптимизации производительности Ruby-приложенийOlga Lavrentieva
Алексей Туля, Senior Software Developer в Sam Solutions
«Практический опыт профайлинга и оптимизации производительности Ruby-приложений»
В своем докладе Алексей сделает краткий обзор различных реализаций Ruby, попытается найти причины, почему Ruby медленный. Рассмотрит вопрос сборки мусора в Ruby и вызова методов – почему в Ruby это дорого. Расскажет и покажет, что делать, чтобы поднять производительность, проведет обзор утилит для поиска проблемных мест, обзор профайлеров и расскажет, как интерпретировать результаты.
Доклад в основном нацелен на практический подход по поиску проблем. Материал предназначен для пользователей Linux, поэтому все практические советы будут для ОС Linux.
Presentation by David Mytton about monitoring MongoDB at the MongoSV conference 3rd Dec 2010.
A full blog series covering everything in this presentation is at http://blog.boxedice.com/mongodb-monitoring/
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
Gisting is an implementation of Google\'s MapReduce framework for processing and extracting useful information from very large data sets. At the time of this writing, the code is available for PREVIEW at http://github.com/mchung/gisting. I am currently working to release this framework for general usage.
Puppet Module Reusability - What I Learned from Shipping to the ForgePuppet
A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.
Gareth Rushgrove
Technical Architect, Government Digital Service
Gareth Rushgrove is now a technical architect at the Government Digital Service, part of the UK Government. He is mainly interested in configuration management, infrastructure and platform as a service, deployment and monitoring tooling and the whole devops community. He thinks when used well together these allow you to move really fast, even in tightly controlled environments like Government. When not working, Gareth can be found blogging over on morethanseven.net or uploading code to GitHub. He also curates the Devops Weekly newsletter and occasionally organises community events.
Практический опыт профайлинга и оптимизации производительности Ruby-приложенийOlga Lavrentieva
Алексей Туля, Senior Software Developer в Sam Solutions
«Практический опыт профайлинга и оптимизации производительности Ruby-приложений»
В своем докладе Алексей сделает краткий обзор различных реализаций Ruby, попытается найти причины, почему Ruby медленный. Рассмотрит вопрос сборки мусора в Ruby и вызова методов – почему в Ruby это дорого. Расскажет и покажет, что делать, чтобы поднять производительность, проведет обзор утилит для поиска проблемных мест, обзор профайлеров и расскажет, как интерпретировать результаты.
Доклад в основном нацелен на практический подход по поиску проблем. Материал предназначен для пользователей Linux, поэтому все практические советы будут для ОС Linux.
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
Speaker: Andrzej Dyjak
Language: English
In recent years security industry started to grow fond of Apple’s iOS and OS X platforms. This talk will cover one of XNU's flagship debugging utilities: DTrace, a dynamic tracing framework for troubleshooting kernel and application problems on production systems in real time. It will be shown how it can be used in order to ease various tasks within the realm of dynamic binary analysis and beyond.
CONFidence: http://confidence.org.pl/
DSLing your System For Scalability Testing Using Gatling - Dublin Scala User ...Aman Kohli
The power of Gatling is the DSL it provides to allow writing meaningful and expressive tests. We provide an overview of the framework, a description of their development environment and goals, and present their test results.
Source code available https://github.com/lawlessc/random-response-time
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Yauheni Akhotnikau
Description of a demo project for serving images by using Actor Model and embedded HTTP-server. This project is implemented in C++17 with SObjectizer and RESTinio (OpenSource products from stiffstream).
Docker Logging and analysing with Elastic StackJakub Hajek
Collecting logs from the entire stateless environment is challenging parts of the application lifecycle. Correlating business logs with operating system metrics to provide insights is a crucial part of the entire organization. What aspects should be considered while you design your logging solutions?
Docker Logging and analysing with Elastic Stack - Jakub Hajek PROIDEA
Collecting logs from the entire stateless environment is challenging parts of the application lifecycle. Correlating business logs with operating system metrics to provide insights is a crucial part of the entire organization. We will see the technical presentation on how to manage a large amount of the data in a typical environment with microservices.
Golang Performance : microbenchmarks, profilers, and a war storyAerospike
Slides for Brian Bulkowski's talk about Golang performance:
microbenchmarks, profilers, and a war story about optimizing the Aerospike Database Go client.
http://www.meetup.com/Go-lang-Developers-NYC/events/216650022/
StHack 2013 - Florian "@agixid" Gaultier No SQL injection but NoSQL injectionStHack
La mouvance NoSQL fait de plus en plus parler d'elle. La plupart du temps open source, les implémentations sont nombreuses et offrent des alternatives intéressantes à la rigidité du SQL. Malheureusement ces diverses solutions NoSQL (MongoDB, CouchDB, Cassandra...) débarquent avec NoSecurity. Nous verrons que, tout comme le SQL, une mauvaise utilisation des clients/drivers peut avoir des conséquences tout aussi critique, si ce n'est plus...
GDG Cloud Taipei meetup #50 - Build go kit microservices at kubernetes with ...KAI CHU CHUNG
Gokit is microservice tookit and use Service/Endpoint/Transport to strict separation of concerns design. This talk to use go-kit develop microservice application integrate with istio, jaeger prometheus, etc service and deploy on Kubernetes.
5. INFRASTRUCTURE
• Same config, same logging, same directory structure
• Same protocol, including JSON to Protobuf conversion
• Build and testing inTeamCity
• QA team should not know this is a Go project
Go is not special in any way
6. INFRASTRUCTURE
• Logs go to syslog and eventually to Splunk
• Metrics are collected with home-grown system based on
RRD (you will see some examples)
• HTTP based profiling is always on
7. INFRASTRUCTURE
• As of now we do not use vendoring
• We use go get for dependencies
• Sometimes we just fork projects (as with rocksdb lib)
• We use Make for building
15. DO NOT GIVE UP
for {
generateSomeLoad()
go tool pprof -alloc_objects http://.../debug/pprof/heap
go tool pprof -inuse_objects http://.../debug/pprof/heap
think()
go build -gcflags=-m foobar.go
thinkAndFix()
}
17. MEMORY PROFILING CRASH COURSE
Allocating on stack
!
• Fast
• No GC pressure
• Stack size is not fixed in Go
• Not always possible
Allocating on heap
!
• Slower
• GC pressure
• Always possible
18. type Person struct {
Name string
Age uint
}
var People []*Person
const PeopleCount = 10000000
func allocateInitial() { ...
func allocateMore() { ...
!
!
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
allocateInitial()
go allocateMore()
http.ListenAndServe("localhost:8080", nil)
}
MEMORY PROFILING CRASH COURSE
19. func allocateMore() {
for {
for i := 0; i < PeopleCount; i++ {
People = append(People, &Person{"marko", 29})
}
People = People[0:PeopleCount]
time.Sleep(10 * time.Second)
}
}
func allocateInitial() {
for i := 0; i < PeopleCount; i++ {
People = append(People, &Person{"marko", 29})
}
}
MEMORY PROFILING CRASH COURSE
20. ESCAPE ANALYSIS
$ go build -gcflags=-m
# github.com/mkevac/test002
./test.go:20: &Person literal escapes to heap
./test.go:27: &Person literal escapes to heap
31. MEMORY PROFILING CRASH COURSE (FIXED)
type Person struct {
Name [6]byte // was string
Age uint
}
var People []Person // was []*Person
const PeopleCount = 10000000
func allocateMore() {
for {
for i := 0; i < PeopleCount; i++ {
People = append(People,
Person{[6]byte{'m', 'a', 'r', 'k', 'o', 0}, 29})
}
People = People[0:PeopleCount]
time.Sleep(10 * time.Second)
}
}
32. $ go tool pprof —inuse_objects ./test003 http:/…/debug/pprof/heap
(pprof) top10
1820 of 1824 total (99.78%)
Dropped 8 nodes (cum <= 9)
flat flat% sum% cum cum%
1820 99.78% 99.78% 1820 99.78% mcommoninit
0 0% 99.78% 1820 99.78% runtime.rt0_go
0 0% 99.78% 1820 99.78% runtime.schedinit
37. GO IS NOT C
• Often it is preferable to copy a little bit,
but avoid using pointers
• We had to thoroughly inspect our code
and remove pointers everywhere we
could
38. $ go tool pprof --alloc_objects ./heaptest /tmp/…/mem.pprof
Adjusting heap profiles for 1-in-4096 sampling rate
Welcome to pprof! For help, type 'help'.
(pprof) top
Total: 29161720 objects
22648298 77.7% 77.7% 22648298 77.7% newselect
6513152 22.3% 100.0% 6513152 22.3% main.main
256 0.0% 100.0% 256 0.0% runtime.mallocinit
14 0.0% 100.0% 14 0.0% allocg
0 0.0% 100.0% 270 0.0% _rt0_go
0 0.0% 100.0% 22648298 77.7% main.loop
0 0.0% 100.0% 14 0.0% mcommoninit
NOT AN ALLOCATION
49. $ cat test.c
!
int get_error(char **error) {
*error = "error";
return 0;
}
$ cat test.go
[…skipped…]
!
func main() {
var errStr *C.char
C.get_error(&errStr)
s := C.GoString(errStr)
fmt.Println(s)
}
$ go build -gcflags=-m
# github.com/mkevac/test001
[…skipped…]
./test.go:13: moved to heap: errStr
./test.go:14: &errStr escapes to heap
./test.go:16: main ... argument does not escape
go 1.3.0
50. $ cat test.c
!
int get_error(char **error) {
*error = "error";
return 0;
}
$ cat test.go
[…skipped…]
!
func main() {
var errStr *C.char
C.get_error(&errStr)
s := C.GoString(errStr)
fmt.Println(s)
}
$ go build -gcflags=-m
# github.com/mkevac/test001
./test.go:14: main &errStr does not escape
./test.go:16: main ... argument does not escape
go 1.4.2
52. db.Put() can cause data to escape!
func SaveCoord(c Coord) error {
key := GetKey(c)
data := make([]byte, c.Size())
n, _ := c.MarshalTo(data)
return db.Put(key, data[:n])
}
59. MAPS
var m = make(map[int]int)
!
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
!
for i := 0; i < 10000000; i++ {
m[i] = i
}
!
for {
runtime.GC()
time.Sleep(5 * time.Second)
}
}
GC Pause:
500 ms
60. commit 85e7bee19f9f26dfca414b1e9054e429c448b14f
Author: Dmitry Vyukov <dvyukov@google.com>
Date: Mon Jan 26 21:04:41 2015 +0300
!
runtime: do not scan maps when k/v do not contain pointers
!
Currently we scan maps even if k/v does not contain pointers.
This is required because overflow buckets are hanging off the main table.
This change introduces a separate array that contains pointers to all
overflow buckets and keeps them alive. Buckets themselves are marked
as containing no pointers and are not scanned by GC (if k/v does not
contain pointers).
!
This brings maps in line with slices and chans -- GC does not scan
their contents if elements do not contain pointers.
!
Currently scanning of a map[int]int with 2e8 entries (~8GB heap)
takes ~8 seconds. With this change scanning takes negligible time.
!
Update #9477.
!
Change-Id: Id8a04066a53d2f743474cad406afb9f30f00eaae
Reviewed-on: https://go-review.googlesource.com/3288
Reviewed-by: Keith Randall <khr@golang.org>
expected in 1.5
61. GOTIP
gc #11 @11.995s 0%: 0+0+0+0+3 ms clock, 0+0+0+0+25 ms
cpu, 304->304->304 MB, 8 P (forced)
62. GC SUMMARY & LINKS
• If you want to write low latency apps, you have to fight GC :-(
• Debugging performance issues: http://goo.gl/jRIyGg
• Go Escape Analysis Flaws: http://goo.gl/U1wkvy
• Go execution tracer: http://goo.gl/KHLBQN
• Interface type conversion: http://goo.gl/oJDYPa
• GC debugcharts: https://github.com/mkevac/debugcharts
• GC visualisation (davecheney): http://goo.gl/ubz5DL