0
Assorted Topics on Scalable
Distributed Services
- Sharma Podila
Senior Software Engineer
PS: Lots of things can, and will, fail at that scale
Topics
Managing Service Dependencies
Asynchronous Event Processing (RxJava)
Async IO
Mesos - DataCenter OS
Deployment and ...
Managing Service Dependencies
Distributed architectures can have dozens of dependencies
Each can fail independently
Even 0...
A Healthy Request Flow
When One Backend Service Becomes
Latent
Threads and other resources can exhaust
with high volume
Hystrix to the Rescue
Wrap calls to external systems in a
dependency command object, run in
separate thread
Timeout calls ...
Taming Tail Latencies of Service Calls
Real time metrics show problems as they occur
Trends help configure timeouts
Set ti...
Reactive Data/Event Stream
Processing
Processing a Data/Event Stream
Iterator<T> iterator = dataStream.iterator();
while(iterator.hasNext()) { process(iterator....
RxJava
Java impl for Reactive Extensions
A library for composing asynchronous and event-based programs by using
observable...
RxJava Operator Examples
Example Code: Iterable and Observable
getDataFromLocalMemory()
.skip(10)
.take(5)
.map({ s -> return s + "
transformed" })...
Async IO
Async IO with Netty, RxNetty
Netty is an NIO client server framework
(see Java IO Vs. NIO)
Supports non-blocking IO
High t...
RxNetty Server Example
public static void main(final String[] args) {
final int port = 8080;
RxNetty.createHttpServer(port...
RxNetty Server Example (Cntd.)
return request.getContent().materialize()
.flatMap(new Func1<Notification<ByteBuf>, Observa...
Mesos
Mesos Cluster Manager
Resource allocation across distributed applications (aka
Frameworks) on shared pool of nodes.
Akin t...
Mesos Architecture
Mesos Resource Offers
Mesos Framework Development
Implement Framework Scheduler and Executor
Scheduler:
resourceOffers(SchedulerDriver driver, j...
Mesos Framework Fault Tolerance
Mesos task reconciliation
Periodic heartbeats from executors
State engine
taskStatesStream...
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Upcoming SlideShare
Loading in...5
×

Prezo tooracleteam (2)

274

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
274
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Prezo tooracleteam (2)"

  1. 1. Assorted Topics on Scalable Distributed Services - Sharma Podila Senior Software Engineer
  2. 2. PS: Lots of things can, and will, fail at that scale
  3. 3. Topics Managing Service Dependencies Asynchronous Event Processing (RxJava) Async IO Mesos - DataCenter OS Deployment and Build Automation
  4. 4. Managing Service Dependencies Distributed architectures can have dozens of dependencies Each can fail independently Even 0.01% downtime on each of dozens of services equates to potentially hours a month of downtime if not engineered for resilience Service A Service B Service C request Dependency Dependency
  5. 5. A Healthy Request Flow
  6. 6. When One Backend Service Becomes Latent
  7. 7. Threads and other resources can exhaust with high volume
  8. 8. Hystrix to the Rescue Wrap calls to external systems in a dependency command object, run in separate thread Timeout calls after time ~ >99.5th of all latencies Control #threads w/ a pool or semaphore Measure success, trip circuit if needed Perform fallback logic Monitor metrics and change in real time Hystrix Wiki
  9. 9. Taming Tail Latencies of Service Calls Real time metrics show problems as they occur Trends help configure timeouts Set timeouts based on histogram data 99.5th + buffer is a good start Tier timeouts for retrials on other servers (e.g., 5, 15, 30)
  10. 10. Reactive Data/Event Stream Processing
  11. 11. Processing a Data/Event Stream Iterator<T> iterator = dataStream.iterator(); while(iterator.hasNext()) { process(iterator.next()); } What if dataStream represents an unbounded stream? What if data comes over the network? With latencies, failures. What if data comes from multiple sources? How would you manage concurrency? Threads? Semaphores? RxJava implementation of reactive extensions addresses these questions “...provides a collection of operators with which you can filter, select, transform, combine, and compose Observables. This allows for efficient execution and composition…”
  12. 12. RxJava Java impl for Reactive Extensions A library for composing asynchronous and event-based programs by using observable sequences Extends the observer pattern to support sequences of data/events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety, concurrent data structures, and non-blocking I/O Event Iterable (pull) Observable (push) retrieve data T next() onNext(T) discover error throws Exception onError(Exception) complete returns onComplete()
  13. 13. RxJava Operator Examples
  14. 14. Example Code: Iterable and Observable getDataFromLocalMemory() .skip(10) .take(5) .map({ s -> return s + " transformed" }) .forEach({ println "next => " + it }) getDataFromNetwork() .skip(10) .take(5) .map({ s -> return s + " transformed" }) .subscribe({ println "onNext => " + it }) Data can be pushed from multiple sources No need to block for result availability RxJava is a tool to react to push data Java Futures as an alternative are non-trivial with nested async execution
  15. 15. Async IO
  16. 16. Async IO with Netty, RxNetty Netty is an NIO client server framework (see Java IO Vs. NIO) Supports non-blocking IO High throughput, low latency, less resource consumption RxNetty is Reactive Extensions adaptor for Netty When using something like Netty, Total #threads in app = Total #cores in the system
  17. 17. RxNetty Server Example public static void main(final String[] args) { final int port = 8080; RxNetty.createHttpServer(port, new RequestHandler<ByteBuf, ByteBuf>() { @Override public Observable<Void> handle(HttpServerRequest<ByteBuf> request, final HttpServerResponse<ByteBuf> response) { System.out.println("New request recieved"); System.out.println(request.getHttpMethod() + " " + request.getUri() + ' ' + request.getHttpVersion()); for (Map.Entry<String, String> header : request.getHeaders().entries()) { System.out.println(header.getKey() + ": " + header.getValue()); } <continued…>
  18. 18. RxNetty Server Example (Cntd.) return request.getContent().materialize() .flatMap(new Func1<Notification<ByteBuf>, Observable<Void>>() { @Override public Observable<Void> call(Notification<ByteBuf> notification) { if (notification.isOnCompleted()) { return response.writeStringAndFlush("Welcome!!!"); } else if (notification.isOnError()) { return Observable.error(notification.getThrowable()); } else { ByteBuf next = notification.getValue(); System.out.println(next.toString(Charset.defaultCharset())); return Observable.empty(); } } }); } }).startAndWait(); }
  19. 19. Mesos
  20. 20. Mesos Cluster Manager Resource allocation across distributed applications (aka Frameworks) on shared pool of nodes. Akin to Google Borg Plugable isolation for CPU, I/O, etc. via Linux CGroups, Docker, etc. Fault tolerant leader election via ZooKeeper Used at Twitter, AirBnB, etc.
  21. 21. Mesos Architecture
  22. 22. Mesos Resource Offers
  23. 23. Mesos Framework Development Implement Framework Scheduler and Executor Scheduler: resourceOffers(SchedulerDriver driver, java.util.List<Offer> offers) executorLost(SchedulerDriver driver, ExecutorID executorId, SlaveID slaveId, int status) statusUpdate(SchedulerDriver driver, TaskStatus status) … and more Executor: launchTask(ExecutorDriver driver, TaskInfo task) killTask(ExecutorDriver driver, TaskID taskId) … and more
  24. 24. Mesos Framework Fault Tolerance Mesos task reconciliation Periodic heartbeats from executors State engine taskStatesStream .groupBy(JobId) .flatmap(groupedObservable) { groupedObservable.takeWhile(state != TerminalState) .debounce(2000) .doOnNext(state) { schedule(taskStuckInState(state), stateTimeout); } } RxJava Style
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×