BranchReduce Distributed Branch-and-Bound on YARN

BranchReduce Distributed Branch-‐and-‐Bound on YARN June 14, 2012

A Terrifyingly Accurate Paraphrasing of JWZ Some people, when confronted with a tedious problem, say, “I know, I’ll write a framework.” Now they have two tedious problems. Copyright 2012 Cloudera Inc. All rights reserved

Leverage ExisUng Frameworks •  Popular RPC libraries with support for mul@ple languages •  C++, Java, Python •  We need to make it easy to deploy exisUng applicaUons on YARN Copyright 2012 Cloudera Inc. All rights reserved

Design PaXern: The Uniﬁed ApplicaUon Master •  Contains business logic and YARN logic •  Primary reason: Communica@on •  Also: dynamic resource alloca@on •  Develop our master/ worker applicaUons locally and then deploy them on YARN Copyright 2012 Cloudera Inc. All rights reserved

YARN Lifecycle Management as a Service •  Speciﬁcally, extensions of Guava’s Service interface •  YarnClientService •  AppMasterService •  Contains all of the logic for crea@ng applica@ons and keeping an eye on them Copyright 2012 Cloudera Inc. All rights reserved

Lua as a ConfiguraUon Language •  Small and Simple •  Looks like a configura@on file •  Func@ons are there when/if you need them •  Inheritance •  Don’t Repeat Yourself •  Forgiving of undefined values •  Java/C++ Integra@on Copyright 2012 Cloudera Inc. All rights reserved

The Challenge of Parallel Branch and Bound: Unbalanced Search Space •  Some branches are pruned quickly •  Can be diﬃcult to determine the best splits a priori •  Easy to revert to a de facto single-‐threaded search Copyright 2012 Cloudera Inc. All rights reserved

You Write Three Classes •  A Task class that implements Writable •  A GlobalState class that implements Writable and has a mergeWith(GlobalState other) method •  A Processor class that deﬁnes: •  execute(T task, BranchReduceContext<T, GlobalState> ctxt); •  With op@onal iniUalize and cleanup methods •  Conﬁgura@on is done via BranchReduceJob Copyright 2012 Cloudera Inc. All rights reserved

QuesUons? @josh_wills

http://marilson.pbworks.com	32
http://eventifier.co	14
http://www.twylah.com	1

BranchReduce Distributed Branch-and-Bound on YARN

by Hadoop_Summit on Jun 20, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 47

Statistics