This document discusses Uber's transition from a monolithic architecture to a microservices architecture and the adoption of Go as a primary programming language. It provides examples of some key Go services at Uber including Geofences, an early service, and Geobase, a more recent service. It also discusses Uber's development of open source Go libraries and tools like Ringpop, TChannel, go-torch, and others to help establish Go as a first-class language at Uber.
This presentation walks through essential points for developing and working with REST APIs or web services to communicate through various platforms. This also explains HTTP methods.
This presentation walks through essential points for developing and working with REST APIs or web services to communicate through various platforms. This also explains HTTP methods.
In this webinar, Postman Developer Educator Sue Smith walks you through the basics of the Postman API Platform and what you can do with it. Topics include:
- How to get started with Postman
- Key tips and tricks every student should know, including how to build basic request configurations and which Postman features are most likely to be useful to you
- Best practices for using Postman to further your education
2020년 서울시에서 주최한 강소기업탐방 프로그램에서 발표한 자료 입니다.
학교를 졸업하고 software engineer로 취직을 하기까지의 여정을 다뤘습니다
1. 개발자가 나에게 맞을지 고민하기 위한 방법
2. 개발자로 취직하기 (이력서/면접 준비 팁)
3. 개발자로 취직한 후 우리가 하는 일
How Green are Java Best Coding Practices? - GreenDays @ Rennes - 2014-07-01Jérôme Rocheteau
This work investigates if best coding practices in Java can stand for eco-design rules as they deal with software performance. It focuses on how validating such an hypothesis for consumed energy, spent execution time and peak allocated memory. It leads to this silent feedback: no need to carry on many measures.
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
This presentation is an overview of the API design and management solutions suitable for Cloud Native Environments. This main focus lies on synchronous API design and micro services.
In this webinar, Postman Developer Educator Sue Smith walks you through the basics of the Postman API Platform and what you can do with it. Topics include:
- How to get started with Postman
- Key tips and tricks every student should know, including how to build basic request configurations and which Postman features are most likely to be useful to you
- Best practices for using Postman to further your education
2020년 서울시에서 주최한 강소기업탐방 프로그램에서 발표한 자료 입니다.
학교를 졸업하고 software engineer로 취직을 하기까지의 여정을 다뤘습니다
1. 개발자가 나에게 맞을지 고민하기 위한 방법
2. 개발자로 취직하기 (이력서/면접 준비 팁)
3. 개발자로 취직한 후 우리가 하는 일
How Green are Java Best Coding Practices? - GreenDays @ Rennes - 2014-07-01Jérôme Rocheteau
This work investigates if best coding practices in Java can stand for eco-design rules as they deal with software performance. It focuses on how validating such an hypothesis for consumed energy, spent execution time and peak allocated memory. It leads to this silent feedback: no need to carry on many measures.
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
This presentation is an overview of the API design and management solutions suitable for Cloud Native Environments. This main focus lies on synchronous API design and micro services.
Bighead: Airbnb’s End-to-End Machine Learning Platform with Krishna Puttaswa...Databricks
Airbnb has a wide variety of ML problems ranging from models on traditional structured data to models built on unstructured data such as user reviews, messages and listing images. The ability to build, iterate on, and maintain healthy machine learning models is critical to Airbnb’s success. Many ML Platforms cover data collection, feature engineering, training, deploying, productionalization, and monitoring but few, if any, do all of the above seamlessly.
Bighead aims to tie together various open source and in-house projects to remove incidental complexity from ML workflows. Bighead is built on Python and Spark and can be used in modular pieces as each ML problem presents unique challenges. Through standardization of the path to production, training environments and the methods for collecting and transforming data on Spark, each model is reproducible and iterable.
This talk covers the architecture, the problems that each individual component and the overall system aims to solve, and a vision for the future of machine learning infrastructure. It’s widely adapted in Airbnb and we have variety of models running in production. We have seen the overall model development time go down from many months to days on Bighead. We plan to open source Bighead to allow the wider community to benefit from our work.
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthNicolas Brousse
TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.
2017 Microservices Practitioner Virtual Summit: The Mechanics of Deploying En...Ambassador Labs
Abstract: The idea of the "service mesh" is becoming very popular in microservice design circles. However, the mechanics of deploying one into an existing infrastructure are far from simple. In this talk we will cover the logistical details of how Envoy was developed and deployed incrementally at Lyft, focusing primarily on the evolution of service mesh configuration management. We will also discuss why high level systems such as Istio are likely to be the main mechanism by which most customers ultimately get access to service mesh technology.
This talk was presented as part of the Microservices Practitioner Virtual Summit, https://www.microservices.com/summit/
Not my problem - Delegating responsibility to infrastructureYshay Yaacobi
Slides for for my talk, appeared on Code-Europe Poznan 12.06.2018
(https://www.codeeurope.pl/en/speakers/yshay-yaacobi)
https://github.com/yshayy/not-my-problem-talk
https://github.com/Yshayy/not-my-problem-talk/blob/master/slides/demo.md
In the last years we have seen huge changes in IT infrastructures and concepts. VoIP architectures too are evolving towards Software Defined Telecoms. In this talk we'll see how VoIP solutions are being shaped by the Cloud, the open points and share some thoughts about its future.
This is co-authored by Giacomo Vacca and Federico Cabiddu.
“Serverless” is the hottest ticket in town right now.
But many serverless platforms restrict your choice of language and / or dictate where your code runs.
In this talk, I’ll describe how we can go to the serverless ball with open source and the Fn project in particular.
“Serverless” aims to improve developer productivity by abstracting, underlying infrastructure layers. The servers are still there, but you just can’t see them.
This abstraction allows the developer to focus solely on the functions that deliver value to the business and not on the plumbing.
The economics of serverless are also interesting since you only consume resources when your functions run, rather than having applications running continually waiting to server requests.
Sadly some leading serverless platforms are not open and restrict choice in terms of: - language - deployment
In this talk, I want to show how you can do serverless development with your choice of language, and deployment location.
Presentation Summary
The evolution of “serverless”
Functions as a Service (FaaS)
open source serverless frameworks
The Fn project (see http://fnproject.io)
Fn functions
building
managing state
logging
FDKs (Function Development Kits)
How we can link individual functions together to create serverless applications
Building an example serverless application with Fn.
Fighting legacy with hexagonal architecture and frameworkless phpFabio Pellegrini
Molto spesso capita di venire a contatto con applicazioni legacy piuttosto datate, i classici monoliti che sono cresciuti a dismisura nel tempo accumulando debito tecnologico.
A causa delle priorità di business delle aziende, non sempre si riesce ad allocare il budget e il tempo necessario per iniziare subito il processo di ristrutturazione architetturale e di rimodellazione dei dati che servirebbe.
In questo talk presenterò una soluzione che mi è capitato di adottare recentemente per iniziare a ridefinire la struttura di un progetto legacy, utilizzando un approccio basato su Domain Driven Design, architettura esagonale e l’utilizzo di PHP senza framework.
Vedremo come è stato creato un nuovo servizio “satellite” da zero, come sono state implementate le componenti principali, come si è tenuto il codice legacy ai margini dell’applicazione, come si è approcciato il testing, il tutto nell’ottica di poter spacchettare il monolite in microservizi in un secondo momento.
Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control.
In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including
A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain
A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxR&R Consult
CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems!
Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected.
R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production.
An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred.
R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance.
Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production.
It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call!
Work done in cooperation with James Malloy and David Moelling from Tetra Engineering.
More examples of our work https://www.r-r-consult.dk/en/cases-en/
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
Immunizing Image Classifiers Against Localized Adversary Attacksgerogepatton
This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks
(CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We
introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations.
When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves
the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach
using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10
and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing
accuracy improvements over previous techniques. The results indicate that the combination of the volumetric
input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating
adversary training.
Overview of the fundamental roles in Hydropower generation and the components involved in wider Electrical Engineering.
This paper presents the design and construction of hydroelectric dams from the hydrologist’s survey of the valley before construction, all aspects and involved disciplines, fluid dynamics, structural engineering, generation and mains frequency regulation to the very transmission of power through the network in the United Kingdom.
Author: Robbie Edward Sayers
Collaborators and co editors: Charlie Sims and Connor Healey.
(C) 2024 Robbie E. Sayers
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
2. ● Previously at Google — my first production experience with Go
● At Uber, I work on the RPC team
○ Support the Go client library for TChannel, our open source RPC framework
○ Building out the new fleetwide routing and load balancing sidecar
● One of the core reviewers for internal Go libraries
● Maintainer of the internal Go build infrastructure, go-build
About me
Java-free for 2 years
3. ● Traditionally used Python and Node.js
○ Node.js is popular in our marketplace
and dispatch services
○ Python is popular for business logic
as well as data analysis
● Almost all the business logic was
behind a monolithic Python API server
About Uber
From a large monolithic service...
The Uber Monolith
Users
Products
Background
Checks
Trips
Cities
Vehicles
Payments
Documents
Promos
4. ● As Uber grew (in features and
engineers), continuous integration
turned into a liability as deploying the
codebase meant deploying everything
● We followed the lead of other hyper-
growth companies (i.e., Twitter, Netflix)
and broke up the monolith into many
microservices.
● This allowed more flexibility in
languages
About Uber
...to over one thousand microservices
Users
ProductsTrips
Cities
Communication
between services
uses HTTP/JSON or
TChannel/Thrift
5. ● In 2014, Aiden Scandella, one of our early
engineers, experimented with Go
○ Deployed a sandbox marketplace for the Uber
External API
● By early 2015, Go was used across the
company,
○ Data Engineering: A cross-cluster query cache
○ Marketplace: The Geofences service
● Go was selected for the following characteristics:
○ Simple language, high developer productivity
○ Strong concurrency and parallelism built in
Introduction of Go
Cross-language microservices
7. ● No standardized support for Go (build, lint, coverage) in our internal
infrastructure
○ Custom Makefiles integrated with go-junit-report and gocov
● Dependency management, used godep
○ Builds passed locally but not on build machines due to missing deps
○ Vendored dependencies hard to manage and review
● Lack of libraries for internal infrastructure
○ E.g., Kafka logging, dynamic configuration, background tasks
Making Go a first-class citizen
9. ● Before the GO15VENDOREXPERIMENT, we started with godep
○ Vendored dependencies in every repo
○ Even internal dependencies were vendored as godep restore was flaky
○ Noisy code reviews, hard to update dependencies,
● The glide migration
○ Users can choose to check in their dependencies in the vendor folder
○ Or check in glide.yaml and glide.lock and have dependencies
installed before build
○ Use an internal Github mirror to avoid outages affecting internal builds
Dependency Management
From godep to glide
10. ● Base libraries used by almost all services: config, metrics, logging, tracing,
RPC
● Utility and extension libraries:
○ Transport helpers, JSON + HTTP, TChannel + Thrift
○ Backoff, LRU cache, flags, worker pools
● Libraries for internal infrastructure built up over time
○ Storage (schemaless -- internal key-value store built on MySQL)
○ Translations, AVRO-encoded logging, dynamic config, experiments
Libraries for internal infrastructure
Monorepo of Go libraries
11. ● Now have over a hundred services written in Go
● Two years of production experience
○ Integrated runtime metrics (#goroutines, GC stats)
○ Easy profiling of services running in production
○ Investigated and fixed file descriptor leaks in open
source libraries
○ Even found a compiler bug that led to stack corruption
● Strong library and infrastructure support
● Lots of resources: documentation, classes, mailing list
● Now the recommended language for new services
Experience with Go
Writing code is fun again
13. Goal: given a lat/lng pair, find the list of geofences this location falls in
● HTTP/JSON interface
● High throughput, low latency (P99 < 100ms)
● No infrastructure support, so project handled everything:
○ Config: Used the standard “encoding/json” package
○ Logging: wrapped the standard “log” package, and reported errors to
Sentry
○ Metrics: Wrapped open source StatsD client
Blog Post
Geofences
One of the earliest Go services at Uber
14. Goal: match riders to drivers, sharding the matching across machines
● Ringpop for sharding the matching by location
● TChannel / Thrift for RPC interface
● Uses our internal libraries for:
○ Config (both static, and dynamic configuration)
○ Logging (includes logging to disk, Kafka, and Sentry)
○ Metrics (reported to M3, our internal Metrics infrastructure)
● Much more infrastructure support for Go services
Geobase
One of the more recent Go services at Uber
15. ● Embeddable application level sharding:
○ Fault detection: provides membership list of alive
nodes using a variation of SWIM (similar to Serf)
○ Consistent hashing: keys are hashed to a node,
and failures will not create load imbalances or
redistribute every key
○ Forwarding: provides forward of HTTP or
TChannel requests
● Used for sharding work, sharded cache, serializing
requests to a resource (in-order)
Ringpop
Scalable, fault-tolerant application-layer sharding. Open source!
16. ● Strong forwarding performance, as it was built for Ringpop forwarding
● Built on top of TCP, provides multiplexing and framing
○ Supports out of order responses, no head of line blocking
○ Very simple protocol, easy to implement in multiple languages
● Provides high-level RPC features:
○ Timeouts, consistent retry semantics
○ Connection management and smart peer selection
● Integrates with Thrift as a first-class citizen
TChannel
Intra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python
17. ● Strong integration with Thrift
○ Uses net/context
○ Integrates with error values
val, err := client.Get(ctx, key)
if err != nil {
switch err := err.(type) {
case *keyvalue.KeyNotFound:
// Handle Thrift exception
default:
// Unknown error
}
}
TChannel + Thrift
Intra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python
18. ● Often used internally with Python and Node.js
● Uses pprof output to generate the flamegraph
go-torch
Visualization of profiling output. Open source!
19. ● Powers the majority of high QPS services at Uber
● Already the most popular language for new services
● Open source our core libraries and tools
■ TChannel, Ringpop, go-torch, yab (beta), zap (beta)
■ Working with open source community on opentracing
● Check out our engineering blog and Github page for more information
Go at Uber