Your SlideShare is downloading. ×
0
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Neptune Distributed Data System
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Neptune Distributed Data System

2,113

Published on

Neptune is Distributed Large scale Structured Data Storage, and open source project implementing Google's Bigtable.

Neptune is Distributed Large scale Structured Data Storage, and open source project implementing Google's Bigtable.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,113
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Neptune<br />hjkim<br />http://www.openneptune.com<br />http://dev.naver.com/projects/neptune(korean)<br />babokim@gmail.com<br />
  • 2. Neptune<br />Distributed Data Storage<br />semi-structured data store(not file system)<br />Use Distributed File System for Data file<br />Supports real time and batch processing<br />Google Bigtable clone<br />Data Model, Architecture, Features<br />Open source<br />http://dev.naver.com/projects/neptune(korean)<br />http://www.openneptune.com<br />Goal<br />500 nodes<br />200 GB 이상/per node, Peta bytes<br />
  • 3. Features<br />Schema Management<br />Create, drop, modify table schema<br />Real-time Transaction<br />Single row operation(no join, group by, order by)<br />Multi row operation: like, between<br />Batch Transaction<br />Scanner, Direct Uploader, MapReduce Adapter<br />Scalability<br />Automatic table split & re-assignment<br />Reliability<br />Data file stored in Distributed File System<br />Commit log stored in ChangeLog Cluster<br />Failover<br />Tablet takeover time: max 1 min.<br />Utility<br />Web Console, Shell(simple query), Data Verifier<br />
  • 4. Architecture<br />User Application<br />MapReduce<br />Neptune<br />Master<br />Neptune<br />TabletServer #1<br />TabletServer #2<br />TabletServer #n<br />Table<br />Distributed File System(Hadoop or other)<br />Physical storage<br />
  • 5. Components<br />Master<br />Lock Server<br />Neptune Client<br />Neptune Master<br />ZooKeeper<br />Pleidas<br />Neptune Master<br />NTable<br />Scanner<br />Shell<br />NChubby<br />failover<br />/ event<br />failover<br />/ event<br />Data/Control<br />Control<br />TabletServer #1<br />(Neptune)<br />TabletServer #2<br />(Neptune)<br />TabletServer #n<br />(Neptune)<br />LogServer<br />#1<br />LogServer<br />#2<br />LogServer<br />#n<br />DFS #1<br />(DataNode)<br />Computing #1<br />(Map&Reduce)<br />DFS #2<br />(DataNode)<br />Computing #2<br />(Map&Reduce)<br />DFS #n<br />(DataNode)<br />Computing #n<br />(Map&Reduce)<br />Local disk<br />Local disk<br />Local disk<br />
  • 6. Data Model<br />Table<br />Column#1<br />Column#n<br />Rowkey<br />row #1<br />ck-1 v1, t1<br />TabletA-1<br />rk-1<br /> v2, t2<br />ck-2 v3, t2<br /> v4, t3<br />row #k<br /> v5, t4<br />row #k+1<br />ck-n vn, tn<br />TabletA-2<br />- Sorted by rowKey<br />- Sorted by columnKey<br />row #m<br />Row#1<br />Cell<br />row #m+1<br />Row.Key<br />TabletA-n<br />Column1<br />Column2<br />Column-n<br />Cell.Key<br />Cell1<br />Cell1<br />Cell1<br />Cell.Value(t1)<br />Cell2<br />Cell2<br />Cell2<br />…<br />…<br />row #n<br />Cell.Value(t2)<br />Cell3<br />…<br />…<br />Cell-k<br />Cell-m<br />Cell.Value(tn)<br />Cell-n<br />
  • 7. Index<br />Root Index<br />M.T1.1000:M1<br />…<br />Max:mn<br />M.T1.2000:M2<br />index of Meta Index<br />Meta Tablet<br />m2<br />m1<br />mn<br />T1.100:U1<br />…<br />xx<br />xx<br />…<br />n<br />T1.200:U2<br />T1.1000:UN<br />…<br />T1.2000:UN<br />T1.1100:U1<br />T1.1200:U2<br />…<br />index of User Tablet<br />User defined Tablet<br />U1<br />U2<br />10<br />20<br />…<br />100<br />110<br />120<br />…<br />200<br />…<br />Index of TableMapFile’sblock(max-key, file-offset)<br />Index Record format: . Key - TableName.MaxRowKey<br /> . Value – Tablet Name, assigned host<br />scan<br />64KB<br />TableMapFile(Physical file,sortedby rowkey, columnkey)<br />
  • 8. Data/index file in HDFS<br />TabletA<br />TabletB<br />Column<br />Data/index file<br />
  • 9. TabletServer<br />Minor <br />Compaction<br />MemoryTable<br />ChangeLogServer<br />Data Operation<br />put(key, value)<br />ChangeLog<br />Searcher<br />get(key)<br />Merged<br />MapFile<br />(HDFS)<br />MapFile#2<br />(HDFS)<br />MapFile#1<br />(HDFS)<br />MapFile #n<br />(HDFS)<br />Major Compaction<br />
  • 10. Failover<br />Master fail<br />disabled only Table Schema Management and Tablet Split<br />can execute Multi-Master<br />TabletServer fail<br />assign to other TabletServer by master<br />within 2 minutes<br />
  • 11. MapReduce with Neptune<br />Hadoop<br />TaskTracker<br />TableA<br />Map Task<br />Map Task<br />TabletInputFormat<br />Map Task<br />TabletA-1<br />TaskTracker<br />Partitioned <br />by key<br />TableB<br />Reduce<br />Task<br />TaskTracker<br />Tablet A-2<br />Tablet B-1<br />Map Task<br />Map Task<br />Map Task<br />Tablet A-3<br />TaskTracker<br />Tablet B-2<br />…<br />Reduce<br />Task<br />TaskTracker<br />Map Task<br />Tablet A-N<br />Map Task<br />Map Task<br />DBMS<br />or HDFS<br />META Table<br />
  • 12. Client<br />Client API<br />Single row operation: put/get<br />Multi row operation: like, between<br />Batch operation: scanner/uploader<br />MapReduce: TabletInputFormat<br />Command line Shell<br />NQL(Neptune Query Language)<br />JDBC support<br />Web Console<br />
  • 13. Client API Example<br />TableShematableSchema = <br /> new TableSchema(“T_TEST”, new String[]{“col1”, “col2”});<br />NTable.createTable(tableSchema);<br />NTablentable = Ntable.openTable(“T_TEST”);<br />Row row = new Row(new Row.Key(“RK1”));<br />Row.addCell(“col1”, new Cell(new Cell.Key(“CK1”), “test_value”.getBytes()));<br />ntable.put(row);<br />Row selectedRow = ntable.get(new Row.Key(“RK1”));<br />System.out.println(selectedRow.getCellList(“col1”).get(0));<br />TableScanner scanner = ntable.openScanner(ntable, new String[]{“col1”});<br />Row scanRow = null;<br />while( (scanRow = scanner.next()) = null) {<br />System.out.println(selectedRow.getCellList(“col1”).get(0));<br />}<br />scanner.close();<br />
  • 14. Neptune Shell<br />Data Definition<br />CREATE TABLE<br />DROP TABLE<br />SHOW TABLES<br />DESC<br />Data Manipulation<br />SELECT<br />DELETE<br />INSERT<br />TRUNCATE COLUMN<br />TRUNCATE TABLE<br />Cluster Monitoring<br />PING TABLETSERVER <br />REPORT TABLE <br />
  • 15. Web Console<br />
  • 16. Performance<br />Number of 1000-byte values read/written per second<br />
  • 17. HBase/Bigtable Comparison<br />
  • 18. Powered by Neptune<br />http://searcus.com/nosql<br />twitter search service<br />

×