Slides for Brian Bulkowski's talk about Golang performance:
microbenchmarks, profilers, and a war story about optimizing the Aerospike Database Go client.
http://www.meetup.com/Go-lang-Developers-NYC/events/216650022/
Rapid Application Design in Financial ServicesAerospike
Applying internet NoSQL design patterns to fraud detection and risk scoring, including when to use SQL and when to use NoSQL. The state of NAND Flash and NVMe is also discussed, as well as storage class memory futures with Intel's 3D Xpoint technology.
This talk was presented in LA at the following meetup:
http://www.meetup.com/scalela/events/233396111/
Go debugging and troubleshooting tips - from real life lessons at SignalFxSignalFx
Exploring tips and advice on writing production Go systems that are easy to debug and troubleshoot. Jack Lindamood from SignalFx presents patterns that facilitate this process.
Jack addresses tools built into Go you can take advantage of, build process techniques they've learned over time, and open source tools and libraries you can use that help troubleshoot your production code when things go wrong.
Read more here: http://blog.signalfx.com/a-pattern-for-optimizing-go
Developing High Performance Application with Aerospike & GoChris Stivers
In this presentation, Chris Stivers, introduces the audience to Aerospike and provides tips on improving performance of Application written in Go. Tips include how to use memory more effectively in Go, and using Aerospike for high throughput / low latency transactions.
Mongo db - How we use Go and MongoDB by Sam HelmanHakka Labs
In this talk, we'll hear from Sam Helman, Software Engineer at 10gen, on how 10gen is integrating Go into our new and existing cloud tools. Some of the tools leveraging Go include the backup capabilities in MongoDB Management Service and a continuos integration tool. This was an opportunity to experiment with new technologies and create a better product for end users.
Programming the in Go and the mgo driver has been extremely satisfying. Between the lightweight syntax, the first-class concurrency and the well documented, idiomatic libraries such as mgo, Go is a great choice for writing anything from small scripts to large distributed applications. In this talk, Sam will go through how the team has integrated Go and why Go and MongoDB are a great match for cloud services.
PyCon AU 2012 - Debugging Live Python Web ApplicationsGraham Dumpleton
Monitoring tools record the result of what happened to your web application when a problem arises, but for some classes of problems, monitoring systems are only a starting point. Sometimes it is necessary to take more intrusive steps to plan for the unexpected by embedding mechanisms that will allow you to interact with a live deployed web application and extract even more detailed information.
Rapid Application Design in Financial ServicesAerospike
Applying internet NoSQL design patterns to fraud detection and risk scoring, including when to use SQL and when to use NoSQL. The state of NAND Flash and NVMe is also discussed, as well as storage class memory futures with Intel's 3D Xpoint technology.
This talk was presented in LA at the following meetup:
http://www.meetup.com/scalela/events/233396111/
Go debugging and troubleshooting tips - from real life lessons at SignalFxSignalFx
Exploring tips and advice on writing production Go systems that are easy to debug and troubleshoot. Jack Lindamood from SignalFx presents patterns that facilitate this process.
Jack addresses tools built into Go you can take advantage of, build process techniques they've learned over time, and open source tools and libraries you can use that help troubleshoot your production code when things go wrong.
Read more here: http://blog.signalfx.com/a-pattern-for-optimizing-go
Developing High Performance Application with Aerospike & GoChris Stivers
In this presentation, Chris Stivers, introduces the audience to Aerospike and provides tips on improving performance of Application written in Go. Tips include how to use memory more effectively in Go, and using Aerospike for high throughput / low latency transactions.
Mongo db - How we use Go and MongoDB by Sam HelmanHakka Labs
In this talk, we'll hear from Sam Helman, Software Engineer at 10gen, on how 10gen is integrating Go into our new and existing cloud tools. Some of the tools leveraging Go include the backup capabilities in MongoDB Management Service and a continuos integration tool. This was an opportunity to experiment with new technologies and create a better product for end users.
Programming the in Go and the mgo driver has been extremely satisfying. Between the lightweight syntax, the first-class concurrency and the well documented, idiomatic libraries such as mgo, Go is a great choice for writing anything from small scripts to large distributed applications. In this talk, Sam will go through how the team has integrated Go and why Go and MongoDB are a great match for cloud services.
PyCon AU 2012 - Debugging Live Python Web ApplicationsGraham Dumpleton
Monitoring tools record the result of what happened to your web application when a problem arises, but for some classes of problems, monitoring systems are only a starting point. Sometimes it is necessary to take more intrusive steps to plan for the unexpected by embedding mechanisms that will allow you to interact with a live deployed web application and extract even more detailed information.
Developing high-performance network servers in LispVladimir Sedach
Overview of current high-performance Common Lisp web servers and implementation techniques, and description of a new hybrid approach to asynchronous I/O based on separate racing accept() and epoll() thread pools.
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
A Recovering Java Developer Learns to GoMatt Stine
As presented at OSCON 2014.
The Go programming language has emerged as a favorite tool of DevOps and cloud practitioners alike. In many ways, Go is more famous for what it doesn’t include than what it does, and co-author Rob Pike has said that Go represents a “less is more” approach to language design.
The Cloud Foundry engineering teams have steadily increased their use of Go for building components, starting with the Router, and progressing through Loggregator, the CLI, and more recently the Health Manager. As a “recovering-Java-developer-turned-DevOps-junkie” focused on helping our customers and community succeed with Cloud Foundry, it became very clear to me that I needed to add Go to my knowledge portfolio.
This talk will introduce Go and its distinctives to Java developers looking to add Go to their toolkits. We’ll cover Go vs. Java in terms of:
* type systems
* modularity
* programming idioms
* object-oriented constructs
* concurrency
Groovy & Grails: Scripting for Modern Web Applicationsrohitnayak
Dynamic scripting languages are a powerful addition to a software designer’s toolbox. Rails/Ruby and Python have not gained much acceptance in the enterprise. Grails and Groovy are an attempt to bridge the gap between the modern scripting world and the Enterprise Java world.
This talk is an introduction towards building web applications in Grails. First we will go about creating a REST based webservice. We will also show how to replace the default database backend of Grails with MySQL.
We will then build a web application that consumes this webservice. The emphasis will be on the design patterns and idioms in Grails that address the web application development lifecycle.
From zero to hero - Easy log centralization with Logstash and ElasticsearchRafał Kuć
Presentation I gave during DevOps Days Warsaw 2014 about combining Elasticsearch, Logstash and Kibana together or use our Logsene solution instead of Elasticsearch.
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Tim Bunce
Slides of my talk on Devel::NYTProf and optimizing perl code at the Italian Perl Workshop (IPW09). It covers the new features in NYTProf v3 and a new section outlining a multi-phase approach to optimizing your perl code.
30 mins long plus 10 mins of questions. Best viewed fullscreen.
Although we don't use it for the core web application, most other places in Launchpad that have to deal with concurrency issues do it using Twisted. This talk will survey these areas and talk about issues we've found and design patterns we've found helpful.
Android is going to Go! Android and GolangAlmog Baku
I recently talked in Droidcon about building using Go with Android
We talked about what is Go? And why it's so awesome? And how can we use it with Android?
On this session we've learnt a little bit about Go, and how can we use it with Android- about the use cases, and the secret
tricks to use it with mobile applications.
Developing high-performance network servers in LispVladimir Sedach
Overview of current high-performance Common Lisp web servers and implementation techniques, and description of a new hybrid approach to asynchronous I/O based on separate racing accept() and epoll() thread pools.
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
A Recovering Java Developer Learns to GoMatt Stine
As presented at OSCON 2014.
The Go programming language has emerged as a favorite tool of DevOps and cloud practitioners alike. In many ways, Go is more famous for what it doesn’t include than what it does, and co-author Rob Pike has said that Go represents a “less is more” approach to language design.
The Cloud Foundry engineering teams have steadily increased their use of Go for building components, starting with the Router, and progressing through Loggregator, the CLI, and more recently the Health Manager. As a “recovering-Java-developer-turned-DevOps-junkie” focused on helping our customers and community succeed with Cloud Foundry, it became very clear to me that I needed to add Go to my knowledge portfolio.
This talk will introduce Go and its distinctives to Java developers looking to add Go to their toolkits. We’ll cover Go vs. Java in terms of:
* type systems
* modularity
* programming idioms
* object-oriented constructs
* concurrency
Groovy & Grails: Scripting for Modern Web Applicationsrohitnayak
Dynamic scripting languages are a powerful addition to a software designer’s toolbox. Rails/Ruby and Python have not gained much acceptance in the enterprise. Grails and Groovy are an attempt to bridge the gap between the modern scripting world and the Enterprise Java world.
This talk is an introduction towards building web applications in Grails. First we will go about creating a REST based webservice. We will also show how to replace the default database backend of Grails with MySQL.
We will then build a web application that consumes this webservice. The emphasis will be on the design patterns and idioms in Grails that address the web application development lifecycle.
From zero to hero - Easy log centralization with Logstash and ElasticsearchRafał Kuć
Presentation I gave during DevOps Days Warsaw 2014 about combining Elasticsearch, Logstash and Kibana together or use our Logsene solution instead of Elasticsearch.
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Tim Bunce
Slides of my talk on Devel::NYTProf and optimizing perl code at the Italian Perl Workshop (IPW09). It covers the new features in NYTProf v3 and a new section outlining a multi-phase approach to optimizing your perl code.
30 mins long plus 10 mins of questions. Best viewed fullscreen.
Although we don't use it for the core web application, most other places in Launchpad that have to deal with concurrency issues do it using Twisted. This talk will survey these areas and talk about issues we've found and design patterns we've found helpful.
Android is going to Go! Android and GolangAlmog Baku
I recently talked in Droidcon about building using Go with Android
We talked about what is Go? And why it's so awesome? And how can we use it with Android?
On this session we've learnt a little bit about Go, and how can we use it with Android- about the use cases, and the secret
tricks to use it with mobile applications.
Lua: the world's most infuriating languagejgrahamc
Slides from a talk I gave at the Lua London Meetup on October 17: "When first confronted with Lua an experienced programmer (like me!) finds themselves infuriated by the languages little differences and hopes that can dismiss it as not worth learning :-) Later they find themselves infuriated to learn that they can't dismiss it: Lua is just too fast, too useful and too flexible. This talk will look at my experience of learning Lua and using it to send a high-altitude balloon into the stratosphere and build CloudFlare's new low latency WAF."
Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.
Quantifying Container Runtime Performance: OSCON 2017 Open Container DayPhil Estes
A talk given at Open Container Day at O'Reilly's OSCON convention in Austin, Texas on May 9th, 2017. This talk describes an open source project, bucketbench, which can be used to compare performance, stability, and throughput of various container engines. Bucketbench currently supports docker, containerd, and runc, but can be extended to support any container runtime. This work was done in response to performance investigations by the Apache OpenWhisk team in using containers as the execution vehicle for functions in their "Functions-as-a-Service" runtime. Find out more about bucketbench here: https://github.com/estesp/bucketbench
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
As part of the Tungsten project, Spark has started an ongoing effort to dramatically improve performance to bring the execution closer to bare metal. In this talk, we’ll go over the progress that has been made so far and the areas we’re looking to invest in next. This talk will discuss the architectural changes that are being made as well as some discussion into how Spark users can expect their application to benefit from this effort. The focus of the talk will be on Spark SQL but the improvements are general and applicable to multiple Spark technologies.
UKOUG version of a presentation trying to establish the sensible limits of parallelism on a couple of hardware configurations. Detailed white paper is at http://oracledoug.com/px_slaves.pdf
Amazon EC2 may offer the possibility of high performance computing to programmers on a budget. Instead of building and maintaining a permanent Beowulf cluster, we can launch a cluster on-demand using Python and EC2. This talk will cover the basics involved in getting your own cluster running using Python, demonstrate how to run some large parallel computations using Python MPI wrappers, and show some initial results on cluster performance.
Apache Beam (formerly Google Cloud Dataflow SDK) is an unified model and set of language-specific SDKs for defining and executing data processing workflows. You design pipelines, simplifying the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service).
This presentation introduces the Beam programming model, and how you can use it to design your pipelines, transporting PCollection and applying some PTransforms. You will see how the same code will be "translated" to a target runtimes thanks to a specific runner. You will also have an overview of the current roadmap, with the new interesting features.
introduction to data processing using Hadoop and PigRicardo Varela
In this talk we make an introduction to data processing with big data and review the basic concepts in MapReduce programming with Hadoop. We also comment about the use of Pig to simplify the development of data processing applications
YDN Tuesdays are geek meetups organized the first Tuesday of each month by YDN in London
Apache Spark 2.0: Faster, Easier, and SmarterDatabricks
In this webcast, Reynold Xin from Databricks will be speaking about Apache Spark's new 2.0 major release.
The major themes for Spark 2.0 are:
- Unified APIs: Emphasis on building up higher level APIs including the merging of DataFrame and Dataset APIs
- Structured Streaming: Simplify streaming by building continuous applications on top of DataFrames allow us to unify streaming, interactive, and batch queries.
- Tungsten Phase 2: Speed up Apache Spark by 10X
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...Flink Forward
The Apache Beam programming model is designed to support several advanced data processing features such as autoscaling and dynamic work rebalancing. In this talk, we will first explain how dynamic work rebalancing not only provides a general and robust solution to the problem of stragglers in traditional data processing pipelines, but also how it allows autoscaling to be truly effective. We will then present how dynamic work rebalancing works as implemented in the Google Cloud Dataflow runner and which path other Apache Beam runners link Apache Flink can follow to benefit from it.
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopDatabricks
Tech-talk at Bay Area Apache Spark Meetup.
Apache Spark 2.0 will ship with the second generation Tungsten engine. Building upon ideas from modern compilers and MPP databases, and applying them to data processing queries, we have started an ongoing effort to dramatically improve Spark’s performance and bringing execution closer to bare metal. In this talk, we’ll take a deep dive into Apache Spark 2.0’s execution engine and discuss a number of architectural changes around whole-stage code generation/vectorization that have been instrumental in improving CPU efficiency and gaining performance.
Code Examples: https://github.com/nolanerck/modern-cfml-demos
The recent versions of ColdFusion have added many new language features. We’ve now got closures, functional programming constructs, QueryExecute, the safe navigation operator, the Elvis operator, and even more coming in future releases. For people new to functional programming or object oriented programming, it can be hard to see where these features can be beneficial. Let’s fix that!
In this talk we’ll discuss many of the new language enhancements in CFML. But rather than stopping with an “x = 1” example, we’ll look at blocks of code that are more real-world in nature. The code demos will all be based on scenarios you’ll run across in your daily development. That's right, we'll run real live code! You'll get to see these language features in action, right before your very eyes!
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Golang Performance : microbenchmarks, profilers, and a war story
1. The fastest NoSQL database!
!
Talking about Go Performance!
!
Try it while I blab !!
github.com/aerospike/aerospike-server!
github.com/aerospike/aerospike-client-go!
2. Who am I ?
Brian Bulkowski!
brian@bulkowski.org!
brian@aerospike.com!
@bbulkow!
TRS-80, PC, Apple II, Vax 11/70, Wang
First product: lightpen university teaching kiosk
Palo Alto High School ( ‘85 )
Liberate / NetComputer through the boom
10B market cap in 1999, employee 32
2003-2007 “time off” ( startups )
Citrusleaf / Aerospike history
42 year old first-time CEO (me)
2008 Prototype
2010 First sales “get the band back together”
2011+ 3 rounds of funding (Draper, ALP, NEA, CNTP)
70 employees, 2 offices
3. Does brian know performance?
Brian Bulkowski!
brian@bulkowski.org!
brian@aerospike.com!
@bbulkow!
Undergrad project: image converter
Single pass arbitrary scale and rotate w/ nyquist filters
Novell
Fastest Appletalk server + router available
Starlight Networks
150Mb/sec video server on P133
Liberate
HTML technology for embedded systems
Aggregate Knowledge
Realtime reccommendations: 2x faster in first week
Aerospike
10x faster than existing NoSQL, 100x faster than RDBMs
4. Internet Technology Stack
MILLIONS OF CONSUMERS
BILLIONS OF DEVICES
APP SERVERS
DATA
WRITE CONTEXT
INSIGHTS WAREHOUSE
In-memory NoSQL
WRITE REAL-TIME CONTEXT
READ RECENT CONTENT
PROFILE STORE
Cookies, email, deviceID, IP address, location,
segments, clicks, likes, tweets, search terms...
REAL-TIME ANALYTICS
Best sellers, top scores, trending tweets
BATCH ANALYTICS
Discover patterns,
segment data: location
patterns, audience
affinity
9. If it is so good, why haven't I heard of it?
Established in 2009 (newer than most)
Used in Advertising – ad exchanges, data exchanges,
targeting, real-time bidding, real-time attribution.
Open Sourced in June 2014
10. When should I use Aerospike?
Redis, but with scale & flash
Cassandra, but fast
User data, session data, behavior, fraud…
API billing ~ retail actions ~ recommendations
Up and running in 10 minutes!
( vagrant, EC2 …)!
11. Why does Aerospike care about Go?
It’s cool !
Promises performance with expressive
( as an old C guy, Go is aimed at me )
Our customers are diving in, deploying
What about (other versions of other languages)…
( sure, they’re cool too! )
Go!
12. Let’s talk about….
Some old microbenchmarks
Profilers, how to run it
War story: optimizing our Go client
( sure, we know Go isn’t JUST about performance )
14. Old Microbenchmark
Seconds (Nov 2009)
1.1 -
python (CPython 2.6.2, the distro release with no tweaks) "
4.6 -
go (current hg release) "
4.2 -
ruby 1.8 (distro release) "
1.1 -
ruby 1.9 (distro release)
Pike said: "
I suspect the great majority of the time in your benchmark is due to Go's current
rudimentary garbage collector. Tests like this generate a lot of garbage that is
collected slowly. From experiments I've done, a better implementation can make a
huge difference. Profiling this test shows at least 50% of the time is in the allocator
and collector, as opposed to about 5% printing the string and less than 15% in the
map code. A better allocator and collector would make a dramatic change. "
"
The short answer: the Go runtime is new and completely untuned. The libraries
need work too.
15. Microbenchmark
“T1”
for i := 0; i < 1000000; i++ {
x = ( 2 * x ) + x + 1
}
1.96 s (big integer only) Python
1.04 ms (2.17s big.Int) Go
5 ms (2.15s BigNum) Java
Good news: go is right in the hunt, but easier to code
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
16. Microbenchmarks
T5 – the 2009 benchmark
12.5 sec Python
12.56 sec Go
2.56 sec Java
Good news: not slower than python!
Bad news: Holy Crap compared to Java
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
17. Microbenchmarks – the old code
T5 – the 2009 benchmark (slower CPU)
for x := 0; x < 1000000; x++ {
a := make(map[int] string);
for a1 := 0; a1 < 50; a1++ {
a[a1] = strconv.Itoa(a1);
}
}
12.56 seconds
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
18. Microbenchmarks – tune the map
T5 – the 2009 benchmark
for x := 0; x < 1000000; x++ {
a := make(map[int] string, 50);
for a1 := 0; a1 < 50; a1++ {
a[a1] = strconv.Itoa(a1);
}
}
7.80 seconds
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
19. Microbenchmarks – remove the Itoa
T5 – the 2009 benchmark
for x := 0; x < 1000000; x++ {
a := make(map[int] string, 50);
for a1 := 0; a1 < 50; a1++ {
a[a1] = "123456”;
}
}
5.45 seconds
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
20. Microbenchmarks – singleton Map
T5 – the 2009 benchmark
a := make(map[int] string, 50);
for x := 0; x < 1000000; x++ {
// a := make(map[int] string, 50);
for a1 := 0; a1 < 50; a1++ {
a[a1] = "123456”;
}
}
2.03 seconds ! Finally better than Java !
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
21. Microbenchmarks – Java
T5 – the 2009 benchmark
for (int x=0; x < 1000000; x++) {
HashMap<Integer, String> a = new HashMap<Integer, String>();
for (int a1=0; a1 < 50; a1++) {
a.put(a1, Integer.toString(a1) );
}
}
2.56 seconds
Amazon m3.xlarge (4 core E3@2.5Ghz)"
Python 2.6.9"
Go 1.3.3"
Java 1.7.0_71"
Amazon Linux (3.16)
23. Next microbenchmarks !
Float, String
Go Channels vs Java Futures
… couldn’t code the java part in time!
Simple TCP echo, but with transactions
Log processing
Ruby 2.1, Go 1.4…
Your votes ?
24. Profilers
pprof is pretty great!
Import in all your main’s, does not seem to hurt
import _ "net/http/pprof”
Add the HTTP listener ( only on flag )
// launch http pprof listener if in profile mode
if *profileMode {
go func() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}()
}
28. Profilers
Good old ‘oprofile’, let’s not forget it –---
( especially if you can get kernel symbols, hard )
sudo yum -y install oprofile
Start capturing
sudo opcontrol --reset
sudo opcontrol --no-vmlinux
sudo opcontrol –start
Run your program
sudo opcontrol --dump
sudo opcontrol --shutdown
Dump your result
sudo opreport -l --demangle=smart --debug-info
Cheat Sheet http://www.bonsai.com/wiki/howtos/tuning/oprofile/
30. Tuning the Aerospike Client
What does the client do?!
!
Maintain the DHT state!
!
Keep a connection pool!
!
Make requests to the right servers!
!
Box / unbox to wire protocol…!
SIMPLE
31. Tuning the Aerospike Client
Attempt 1: run pprof!
!
The usual dance of making life!
easy for the garbage collector !
(just like java)!
!
pprof worked!!
the hot objects showed up!
!
Cache easily with Sized Channels !!!!
32. Tuning the Aerospike Client
Attempt 2: oprofile!
!
oprofile found rand() taking time!
!
Optimization gave nothing!
!
… not sure why not …!
!
Currently happy with throughput!
33. Tuning the Aerospike Client
Latency problem at customer site !!
!
User validating a server install with a quick Go client!
“17 ms average latency @ 20K TPS” --- terrible!!
!
Server measured at 0.4 ms @ 40k TPS, !
-- ping ok!
-- it’s the client!
!
Where’s the latency source? GC? Green Threads? Network?!
-- Profile shows low GC load!
-- Hard to measure thread latency!
EC2 m3.xlarge ($0.05/hr)!
4 core E5-2670 @ 2.5 Ghz!
Bare metal vs Virtual!
Centos 6 vs Latest Kernel!
Intel SSDs vs RAM!
35. What happened?
• Not sure what happened at deployment !
(yet, suspect old kernel)!
• A week lost by developers using MacOS, Laptop!
(MacOS is showing bad latency)!
• C code is running slower – we think it’s random fill of buffer!
• Lesson: just switch to Linux 3.12-ish kernels!
• Lesson: fewer lines ~ 11k Go, 17k Java!
• Lesson: for network / IO, these languages are THE SAME !