Neptune<br />hjkim<br />http://www.openneptune.com<br />http://dev.naver.com/projects/neptune(korean)<br />babokim@gmail.c...
Neptune<br />Distributed Data Storage<br />semi-structured data store(not file system)<br />Use Distributed File System fo...
Features<br />Schema Management<br />Create, drop, modify table schema<br />Real-time Transaction<br />Single row operatio...
Architecture<br />User Application<br />MapReduce<br />Neptune<br />Master<br />Neptune<br />TabletServer #1<br />TabletSe...
Components<br />Master<br />Lock Server<br />Neptune Client<br />Neptune Master<br />ZooKeeper<br />Pleidas<br />Neptune M...
Data Model<br />Table<br />Column#1<br />Column#n<br />Rowkey<br />row #1<br />ck-1         v1, t1<br />TabletA-1<br />rk-...
Index<br />Root Index<br />M.T1.1000:M1<br />…<br />Max:mn<br />M.T1.2000:M2<br />index of Meta Index<br />Meta Tablet<br ...
Data/index file in HDFS<br />TabletA<br />TabletB<br />Column<br />Data/index file<br />
TabletServer<br />Minor <br />Compaction<br />MemoryTable<br />ChangeLogServer<br />Data Operation<br />put(key, value)<br...
Failover<br />Master fail<br />disabled only Table Schema Management and Tablet Split<br />can execute Multi-Master<br />T...
MapReduce with Neptune<br />Hadoop<br />TaskTracker<br />TableA<br />Map Task<br />Map Task<br />TabletInputFormat<br />Ma...
Client<br />Client API<br />Single row operation: put/get<br />Multi row operation: like, between<br />Batch operation: sc...
Client API Example<br />TableShematableSchema = <br />                           new TableSchema(“T_TEST”, new String[]{“c...
Neptune Shell<br />Data Definition<br />CREATE TABLE<br />DROP TABLE<br />SHOW TABLES<br />DESC<br />Data Manipulation<br ...
Web Console<br />
Performance<br />Number of 1000-byte values read/written per second<br />
HBase/Bigtable Comparison<br />
Powered by Neptune<br />http://searcus.com/nosql<br />twitter search service<br />
Upcoming SlideShare
Loading in...5
×

Neptune Distributed Data System

2,150

Published on

Neptune is Distributed Large scale Structured Data Storage, and open source project implementing Google's Bigtable.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,150
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Neptune Distributed Data System

  1. 1. Neptune<br />hjkim<br />http://www.openneptune.com<br />http://dev.naver.com/projects/neptune(korean)<br />babokim@gmail.com<br />
  2. 2. Neptune<br />Distributed Data Storage<br />semi-structured data store(not file system)<br />Use Distributed File System for Data file<br />Supports real time and batch processing<br />Google Bigtable clone<br />Data Model, Architecture, Features<br />Open source<br />http://dev.naver.com/projects/neptune(korean)<br />http://www.openneptune.com<br />Goal<br />500 nodes<br />200 GB 이상/per node, Peta bytes<br />
  3. 3. Features<br />Schema Management<br />Create, drop, modify table schema<br />Real-time Transaction<br />Single row operation(no join, group by, order by)<br />Multi row operation: like, between<br />Batch Transaction<br />Scanner, Direct Uploader, MapReduce Adapter<br />Scalability<br />Automatic table split & re-assignment<br />Reliability<br />Data file stored in Distributed File System<br />Commit log stored in ChangeLog Cluster<br />Failover<br />Tablet takeover time: max 1 min.<br />Utility<br />Web Console, Shell(simple query), Data Verifier<br />
  4. 4. Architecture<br />User Application<br />MapReduce<br />Neptune<br />Master<br />Neptune<br />TabletServer #1<br />TabletServer #2<br />TabletServer #n<br />Table<br />Distributed File System(Hadoop or other)<br />Physical storage<br />
  5. 5. Components<br />Master<br />Lock Server<br />Neptune Client<br />Neptune Master<br />ZooKeeper<br />Pleidas<br />Neptune Master<br />NTable<br />Scanner<br />Shell<br />NChubby<br />failover<br />/ event<br />failover<br />/ event<br />Data/Control<br />Control<br />TabletServer #1<br />(Neptune)<br />TabletServer #2<br />(Neptune)<br />TabletServer #n<br />(Neptune)<br />LogServer<br />#1<br />LogServer<br />#2<br />LogServer<br />#n<br />DFS #1<br />(DataNode)<br />Computing #1<br />(Map&Reduce)<br />DFS #2<br />(DataNode)<br />Computing #2<br />(Map&Reduce)<br />DFS #n<br />(DataNode)<br />Computing #n<br />(Map&Reduce)<br />Local disk<br />Local disk<br />Local disk<br />
  6. 6. Data Model<br />Table<br />Column#1<br />Column#n<br />Rowkey<br />row #1<br />ck-1 v1, t1<br />TabletA-1<br />rk-1<br /> v2, t2<br />ck-2 v3, t2<br /> v4, t3<br />row #k<br /> v5, t4<br />row #k+1<br />ck-n vn, tn<br />TabletA-2<br />- Sorted by rowKey<br />- Sorted by columnKey<br />row #m<br />Row#1<br />Cell<br />row #m+1<br />Row.Key<br />TabletA-n<br />Column1<br />Column2<br />Column-n<br />Cell.Key<br />Cell1<br />Cell1<br />Cell1<br />Cell.Value(t1)<br />Cell2<br />Cell2<br />Cell2<br />…<br />…<br />row #n<br />Cell.Value(t2)<br />Cell3<br />…<br />…<br />Cell-k<br />Cell-m<br />Cell.Value(tn)<br />Cell-n<br />
  7. 7. Index<br />Root Index<br />M.T1.1000:M1<br />…<br />Max:mn<br />M.T1.2000:M2<br />index of Meta Index<br />Meta Tablet<br />m2<br />m1<br />mn<br />T1.100:U1<br />…<br />xx<br />xx<br />…<br />n<br />T1.200:U2<br />T1.1000:UN<br />…<br />T1.2000:UN<br />T1.1100:U1<br />T1.1200:U2<br />…<br />index of User Tablet<br />User defined Tablet<br />U1<br />U2<br />10<br />20<br />…<br />100<br />110<br />120<br />…<br />200<br />…<br />Index of TableMapFile’sblock(max-key, file-offset)<br />Index Record format: . Key - TableName.MaxRowKey<br /> . Value – Tablet Name, assigned host<br />scan<br />64KB<br />TableMapFile(Physical file,sortedby rowkey, columnkey)<br />
  8. 8. Data/index file in HDFS<br />TabletA<br />TabletB<br />Column<br />Data/index file<br />
  9. 9. TabletServer<br />Minor <br />Compaction<br />MemoryTable<br />ChangeLogServer<br />Data Operation<br />put(key, value)<br />ChangeLog<br />Searcher<br />get(key)<br />Merged<br />MapFile<br />(HDFS)<br />MapFile#2<br />(HDFS)<br />MapFile#1<br />(HDFS)<br />MapFile #n<br />(HDFS)<br />Major Compaction<br />
  10. 10. Failover<br />Master fail<br />disabled only Table Schema Management and Tablet Split<br />can execute Multi-Master<br />TabletServer fail<br />assign to other TabletServer by master<br />within 2 minutes<br />
  11. 11. MapReduce with Neptune<br />Hadoop<br />TaskTracker<br />TableA<br />Map Task<br />Map Task<br />TabletInputFormat<br />Map Task<br />TabletA-1<br />TaskTracker<br />Partitioned <br />by key<br />TableB<br />Reduce<br />Task<br />TaskTracker<br />Tablet A-2<br />Tablet B-1<br />Map Task<br />Map Task<br />Map Task<br />Tablet A-3<br />TaskTracker<br />Tablet B-2<br />…<br />Reduce<br />Task<br />TaskTracker<br />Map Task<br />Tablet A-N<br />Map Task<br />Map Task<br />DBMS<br />or HDFS<br />META Table<br />
  12. 12. Client<br />Client API<br />Single row operation: put/get<br />Multi row operation: like, between<br />Batch operation: scanner/uploader<br />MapReduce: TabletInputFormat<br />Command line Shell<br />NQL(Neptune Query Language)<br />JDBC support<br />Web Console<br />
  13. 13. Client API Example<br />TableShematableSchema = <br /> new TableSchema(“T_TEST”, new String[]{“col1”, “col2”});<br />NTable.createTable(tableSchema);<br />NTablentable = Ntable.openTable(“T_TEST”);<br />Row row = new Row(new Row.Key(“RK1”));<br />Row.addCell(“col1”, new Cell(new Cell.Key(“CK1”), “test_value”.getBytes()));<br />ntable.put(row);<br />Row selectedRow = ntable.get(new Row.Key(“RK1”));<br />System.out.println(selectedRow.getCellList(“col1”).get(0));<br />TableScanner scanner = ntable.openScanner(ntable, new String[]{“col1”});<br />Row scanRow = null;<br />while( (scanRow = scanner.next()) = null) {<br />System.out.println(selectedRow.getCellList(“col1”).get(0));<br />}<br />scanner.close();<br />
  14. 14. Neptune Shell<br />Data Definition<br />CREATE TABLE<br />DROP TABLE<br />SHOW TABLES<br />DESC<br />Data Manipulation<br />SELECT<br />DELETE<br />INSERT<br />TRUNCATE COLUMN<br />TRUNCATE TABLE<br />Cluster Monitoring<br />PING TABLETSERVER <br />REPORT TABLE <br />
  15. 15. Web Console<br />
  16. 16. Performance<br />Number of 1000-byte values read/written per second<br />
  17. 17. HBase/Bigtable Comparison<br />
  18. 18. Powered by Neptune<br />http://searcus.com/nosql<br />twitter search service<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×