0
HBase Client API (for webapps?)         Nick Dimiduk   Seattle Scalability Meetup          2013-03-27                     ...
2
3
What are my choices?   switch (technology) {       case ‘    ’:         ...       case ‘    ’:         ...       case ‘   ...
Apache HBase               5
Java client Interfaces•   Configuration holds details where to find the cluster and tunable    settings. Roughly equivalent...
Java client Examplepublic static final byte[] TABLE_NAME = Bytes.toBytes("twits");public static final byte[] TWITS_FAM = B...
Java client Exampleprivate static class Twit {   private Twit(Result r) {     this(           r.getColumnLatest(TWITS_FAM,...
Java client Example       private static Get mkGet(String user, DateTime dt) {         Get g = new Get(mkRowKey(user, dt))...
Ruby, Python client Interface                                10
Ruby, Python client Interface        Jyth onJRu by,             : (                                11
Thrift client Interface1. Generate bindings2. Run a “Gateway” between clients and cluster3. ... profit? code!        w rite...
HBase Cluster HBase ClientsSidebar: Architecture Recap                                 13
Thrift                    Gateway     HBase ClusterThrift Clients                 Thrift Architecture                     ...
Thrift client Interface•   Thrift gateway exposes a client to RegionServers•   stateless :D•   ... except for scanners :( ...
Thrift client Exampletransport = TSocket.TSocket(host, port)transport = TTransport.TBufferedTransport(transport)protocol =...
Thrift client Examplecolumns = [info:user,info:name,info:email]scanner = client.scannerOpen(users, , columns)row = client....
Thrift client Exampledef user_from_row(row):    user = {}    for col,cell in row.columns.items():        user[col[5:]] = c...
REST client Interface1. Stand up a "REST Gateway" between your application and the cluster2. HTTP verbs translate (roughly...
HBase Cluster                   RESTREST Clients                  Gateway               REST Architecture                 ...
REST client Interface•   REST gateway exposes a client to RegionServers•   stateless :D•   ... except for scanners :(     ...
REST client Example$ curl -H "Accept: application/json" http://host:port/{  "table": [ {               "name": "followers"...
REST client Example$ curl -H ... http://host:port/table/row [/family:qualifier]{    "Row": [        {             "key": "...
REST client Example<Rows>  <Row key="VGhlUmVhbE1U">    <Cells>       <Cell column="aW5mbzplbWFpbA==" timestamp="1338701491...
Beyond Apache                25
asynchbase•   Asynchronous non-blocking interface.•   Inspired by Twisted Python.•   Partial implementation of HTableInter...
asynchbase                                                                UpdateResult                                    ...
asynchbase Example       final Scanner scanner = client.newScanner(TABLE_NAME);       scanner.setFamily(INFO_FAM);       s...
OthersReduce day-to-day                                                        Full-blown schema developer pain           ...
Apache Futures•   Protobuf wire messages (0.96)•   C client (TBD, HBASE-1015)•   HBase Types (TBD, HBASE-8089)            ...
So, Webapps?http://www.amazon.com/Back-Point-Rapiers/dp/B0000271GC                                                        ...
Software Architecture•   Isolate DAO from app logic, separation of concerns, &c.•   Separate environment configs from code....
Deployment Architecture•   Cache everywhere.•   Know your component layers.                                   33
HBase Warts•   Know thy (HBase) version 0.{92,94,96} !•   long-running client bug (HBASE-4805).•   Gateway APIs only as up...
Thanks!                                         Nick Dimiduk                                              github.com/ndimi...
Upcoming SlideShare
Loading in...5
×

HBase Client APIs (for webapps?)

6,123

Published on

This talk examines HBase client options available to application developers working with HBase. The focus is framed on, but not limited to, building webapps.

1 Comment
9 Likes
Statistics
Notes
  • Hi Nick, Indeed great presentation.

    I have following query and need your help ...

    I am trying to insert a row in HBase using curl and getting an error. I am unable to locate what am I doing wrong?

    Scan of 'testable' table output:
    ---------------------------
    hbase(main):003:0> scan 'testtable'
    ROW COLUMN+CELL
    firstrow column=columnfamily:message, timestamp=1403601833682, value=hello world
    firstrow column=columnfamily:name, timestamp=1403843584253, value=Agrawal
    firstrow column=columnfamily:number, timestamp=1403765553459, value=43
    secondrow column=columnfamily:number, timestamp=1403601842745, value=42
    test-key column=columnfamily:q, timestamp=1403785401271, value=value
    3 row(s) in 0.1690 seconds

    Curl command:
    -------------------
    curl -X POST -H 'Content-Type: application/json' -d '[{'key' : '4throw', 'columnfamily':[{'name' : 'Cool' , 'number' : 2500 }] }]' http://localhost:9000/testtable/

    Error when run with curl:
    ---------------------------------
    Error 405 Method Not AllowedHTTP ERROR 405Problem accessing /testtable/. Reason:
    Method Not AllowedPowered by Jetty://



    Thanks
    -Agrawal
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
6,123
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
60
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "HBase Client APIs (for webapps?)"

  1. 1. HBase Client API (for webapps?) Nick Dimiduk Seattle Scalability Meetup 2013-03-27 1
  2. 2. 2
  3. 3. 3
  4. 4. What are my choices? switch (technology) { case ‘ ’: ... case ‘ ’: ... case ‘ ’: ... } 4
  5. 5. Apache HBase 5
  6. 6. Java client Interfaces• Configuration holds details where to find the cluster and tunable settings. Roughly equivalent to JDBC connection string.• HConnection represents connections to to the cluster.• HBaseAdmin handles DDL operations (create, list, drop, alter, &c.)• HTablePool connection pool for table handles.• HTable (HTableInterface) is a handle on a single HBase table. Send "commands" to the table (Put, Get, Scan, Delete, Increment) 6
  7. 7. Java client Examplepublic static final byte[] TABLE_NAME = Bytes.toBytes("twits");public static final byte[] TWITS_FAM = Bytes.toBytes("twits");public static final byte[] USER_COL = Bytes.toBytes("user");public static final byte[] TWIT_COL = Bytes.toBytes("twit");private HTablePool pool = new HTablePool(); https://github.com/hbaseinaction/twitbase/blob/master/src/main/java/HBaseIA/TwitBase/hbase/TwitsDAO.java#L23-L30 7
  8. 8. Java client Exampleprivate static class Twit { private Twit(Result r) { this( r.getColumnLatest(TWITS_FAM, USER_COL).getValue(), Arrays.copyOfRange(r.getRow(), Md5Utils.MD5_LENGTH, Md5Utils.MD5_LENGTH + longLength), r.getColumnLatest(TWITS_FAM, TWIT_COL).getValue()); } private Twit(byte[] user, byte[] dt, byte[] text) { this( Bytes.toString(user), new DateTime(-1 * Bytes.toLong(dt)), Bytes.toString(text)); }https://github.com/hbaseinaction/twitbase/blob/master/src/main/java/HBaseIA/TwitBase/hbase/TwitsDAO.java#L129-L143 8
  9. 9. Java client Example private static Get mkGet(String user, DateTime dt) { Get g = new Get(mkRowKey(user, dt)); g.addColumn(TWITS_FAM, USER_COL); g.addColumn(TWITS_FAM, TWIT_COL); return g; }https://github.com/hbaseinaction/twitbase/blob/master/src/main/java/HBaseIA/TwitBase/hbase/TwitsDAO.java#L60-L65 9
  10. 10. Ruby, Python client Interface 10
  11. 11. Ruby, Python client Interface Jyth onJRu by, : ( 11
  12. 12. Thrift client Interface1. Generate bindings2. Run a “Gateway” between clients and cluster3. ... profit? code! w rite 12
  13. 13. HBase Cluster HBase ClientsSidebar: Architecture Recap 13
  14. 14. Thrift Gateway HBase ClusterThrift Clients Thrift Architecture 14
  15. 15. Thrift client Interface• Thrift gateway exposes a client to RegionServers• stateless :D• ... except for scanners :( 15
  16. 16. Thrift client Exampletransport = TSocket.TSocket(host, port)transport = TTransport.TBufferedTransport(transport)protocol = TBinaryProtocol.TBinaryProtocol(transport)client = Hbase.Client(protocol)transport.open() https://github.com/hbaseinaction/twitbase.py/blob/master/TwitBase.py#L17-L21 16
  17. 17. Thrift client Examplecolumns = [info:user,info:name,info:email]scanner = client.scannerOpen(users, , columns)row = client.scannerGet(scanner)while row: yield user_from_row(row[0]) row = scannerGet(scanner)client.scannerClose(scanner) https://github.com/hbaseinaction/twitbase.py/blob/master/TwitBase.py#L33-L39 17
  18. 18. Thrift client Exampledef user_from_row(row): user = {} for col,cell in row.columns.items(): user[col[5:]] = cell.value return "<User: {user}, {name}, {email}>".format(**user) https://github.com/hbaseinaction/twitbase.py/blob/master/TwitBase.py#L26-L30 18
  19. 19. REST client Interface1. Stand up a "REST Gateway" between your application and the cluster2. HTTP verbs translate (roughly) into table commands3. decent support for basic DDL, HTable operations 19
  20. 20. HBase Cluster RESTREST Clients Gateway REST Architecture 20
  21. 21. REST client Interface• REST gateway exposes a client to RegionServers• stateless :D• ... except for scanners :( 21
  22. 22. REST client Example$ curl -H "Accept: application/json" http://host:port/{ "table": [ { "name": "followers" }, { "name": "twits" }, { "name": "users" } ]} 22
  23. 23. REST client Example$ curl -H ... http://host:port/table/row [/family:qualifier]{ "Row": [ { "key": "VGhlUmVhbE1U", "Cell": [ { "$": "c2FtdWVsQGNsZW1lbnMub3Jn", "column": "aW5mbzplbWFpbA==", "timestamp": 1338701491422 }, { "$": "TWFyayBUd2Fpbg==", "column": "aW5mbzpuYW1l", "timestamp": 1338701491422 }, ] } ] } 23
  24. 24. REST client Example<Rows> <Row key="VGhlUmVhbE1U"> <Cells> <Cell column="aW5mbzplbWFpbA==" timestamp="1338701491422"> c2FtdWVsQGNsZW1lbnMub3Jn </Cell> <Cell ...> ... </Cells> </Row></Rows> 24
  25. 25. Beyond Apache 25
  26. 26. asynchbase• Asynchronous non-blocking interface.• Inspired by Twisted Python.• Partial implementation of HTableInterface.• HBaseClient provides entry-point to data. https://github.com/OpenTSDB/asynchbase http://tsunanet.net/~tsuna/asynchbase/api/org/hbase/async/HBaseClient.html 26
  27. 27. asynchbase UpdateResult object output to => [next state] 3 / Interpretinput => [this state] response => [error state] Exception Boolean Put response UpdateFailed Exception 27
  28. 28. asynchbase Example final Scanner scanner = client.newScanner(TABLE_NAME); scanner.setFamily(INFO_FAM); scanner.setQualifier(PASSWORD_COL); ArrayList<ArrayList<KeyValue>> rows = null; ArrayList<Deferred<Boolean>> workers = new ArrayList<Deferred<Boolean>>(); while ((rows = scanner.nextRows(1).joinUninterruptibly()) != null) { for (ArrayList<KeyValue> row : rows) { KeyValue kv = row.get(0); byte[] expected = kv.value(); String userId = new String(kv.key()); PutRequest put = new PutRequest( TABLE_NAME, kv.key(), kv.family(), kv.qualifier(), mkNewPassword(expected)); Deferred<Boolean> d = client.compareAndSet(put, expected) .addCallback(new InterpretResponse(userId)) .addCallbacks(new ResultToMessage(), new FailureToMessage()) .addCallback(new SendMessage()); workers.add(d); } }https://github.com/hbaseinaction/twitbase-async/blob/master/src/main/java/HBaseIA/TwitBase/AsyncUsersTool.java#L151-L173 28
  29. 29. OthersReduce day-to-day Full-blown schema developer pain management [Orderly] Phoenix Spring-Data Hadoop Kiji.org https://github.com/ndimiduk/orderly http://www.springsource.org/spring-data/ https://github.com/forcedotcom/phoenix http://www.kiji.org/ 29
  30. 30. Apache Futures• Protobuf wire messages (0.96)• C client (TBD, HBASE-1015)• HBase Types (TBD, HBASE-8089) 30
  31. 31. So, Webapps?http://www.amazon.com/Back-Point-Rapiers/dp/B0000271GC 31
  32. 32. Software Architecture• Isolate DAO from app logic, separation of concerns, &c.• Separate environment configs from code.• Watch out for resource contention. 32
  33. 33. Deployment Architecture• Cache everywhere.• Know your component layers. 33
  34. 34. HBase Warts• Know thy (HBase) version 0.{92,94,96} !• long-running client bug (HBASE-4805).• Gateway APIs only as up to date as the people before you require.• REST API particularly unpleasant for “Web2.0” folk. 34
  35. 35. Thanks! Nick Dimiduk github.com/ndimiduk @xefyrNick DimidukAmandeep Khurana n10k.com FOREWORD BY Michael Stack MANNINGhbaseinaction.com 35
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×