Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,141
On Slideshare
1,141
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
40
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. dwivedishashwat@gmail.com
  • 2. QUERIESFlow of Hbase read/write
  • 3. QUERYAbout Meter Logs: It best suited for persistent and day by day increasing data, as you data grows you can keep on adding more nodes, and you have lot of facility, process this data such as pig, hive and map-reduce. You can run map-reduce and output the datasets which can be indexed and used for faster search of that huge data.
  • 4. QUERYUsibility HBase isnt suitable for every problem. First, make sure you have enough data. If you have hundreds of millions or billions of rows, then HBase is a good candidate. If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle. Second, make sure you can live without all the extra features that an RDBMS provides (e.g., typed columns, secondary indexes, transactions, advanced query languages, etc.) An application built against an RDBMS cannot be "ported" to HBase by simply changing a JDBC driver, for example. Consider moving from an RDBMS to HBase as a complete redesign as opposed to a port. Third, make sure you have enough hardware. Even HDFS doesnt do well with anything less than 5 DataNodes (due to things such as HDFS block replication which has a default of 3), plus a NameNode. HBase can run quite well stand-alone on a laptop - but this should be considered a development configuration only.
  • 5. QUERYDifference Between NoSQL DB and RDBMS NoSQL is a kind of database that doesnt have a fixed schema like a traditional RDBMS does. With the NoSQL databases the schema is defined by the developer at run time. They dont write normal SQL statements against the database, but instead use an API to get the data that they need. The NoSQL databases can usually scale across different physical servers easily without needing to know which server the data you are looking for is on. Why we should think of using it  Durability  Scalability on fly  Distributed data  Persistence etc.
  • 6. QUERYRow Oriented and Column oriented DBSA column-oriented DBMS is a database management system (DBMS) that stores data tables as sections of columns of data rather than as rows of data, like most relational DBMSs 1,Smith,Joe,40000; 1,2,3; Smith,Jones,Johnson; 2,Jones,Mary,50000; Joe,Mary,Cathy; 3,Johnson,Cathy,44000; 40000,50000,44000;
  • 7. A column-oriented database is different from traditional row-oriented databases because of how they store data. By storing a whole column together instead of a row, you can minimize disk access when selecting a few columns from a row containing many columns. In row-oriented databases theres no difference if you select just one or all fields from a row. NamID Age This would be persisted in a conventional RDBMS as e follows - Bhavi 1,Bhavin,29|2,Roger,301 29 n Roge In a column oriented DBMS this would be persisted as -2 30 1,2|Bhavin,Roger|29,30 r
  • 8. MORE DEPTH OF HBASE CONCEPTS
  • 9. MODES OF HBASE OPERATIONStand Alone :In standalone mode, there is no distributed file system and no Java services/daemons are started. All mappers and reducers run inside a single Java VM.This mode is best suited for testing purpose, and experimentations with HBase.
  • 10. MODES OF HBASE OPERATION…Pseudo-Distributed ModePseudo-distributed mode, Hbase processing is distributed over all of the cores/processors on a single machine. Hbase writes all files to the Hadoop Distributed FileSystem (HDFS), and all services and daemons communicate over local TCP sockets for inter-process communicationA pseudo-distributed mode is simply a distributed mode run on a single host. Use this configuration testing and prototyping on HBase. Do not use this configuration for production nor for evaluating HBase performance
  • 11. MODES OF HBASE OPERATION…Distributed:In distributed mode the daemons are spread across all nodes in the clusterDistributed modes require an instance of the Hadoop Distributed File System (HDFS)
  • 12. BASIC PREREQUISITESJavaSSHDNSNTPulimit and nprocHadoop for Distributed mode
  • 13. BASIC PREREQUISITES IN DETAILJust like Hadoop, HBase requires at least java 6 from Oraclessh must be installed and sshd must be running to use Hadoops scripts to manage remote Hadoop and HBase daemons. You must be able to ssh to all nodes, including your local node, using passwordless loginHBase uses the local hostname to self-report its IP address. Both forward and reverse DNS resolving must work
  • 14. BASIC PREREQUISITES IN DETAIL The clocks on cluster members should be in basic alignments. Some skew is tolerable but wild skew could generate odd behaviors. Run NTP on your cluster, or an equivalent. It uses a lot of files all at the same time. The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is insufficient Any significant amount of loading will lead you to “java.io.IOException...(Too many open files)” Hadoop for Distributed file system and mapreduce processing.
  • 15. HBASE CONFIGURATION FILEShbase-site.xmlhbase-default.xmlhbase-env.shlog4j.propertiesregionservers
  • 16. HBASE-DEFAULT.XMLNot all configuration options make it out to hbase-default.xml. Configuration that it is thought rare anyone would change can exist only in code; the only way to turn up such configurations is via a reading of the source code itself.
  • 17. hbase.rootdir The directory shared by region servers and into which HBase persists.  Default: file:///tmp/hbase-${user.name}/hbase  hdfs://namenode:9000/hbase  For Distributedhbase.master.port The port the HBase Master should bind to. Default is 60000hbase.cluster.distributed The mode the cluster will be in. Possible values are false for standalone mode and true for distributed modehbase.tmp.dir Temporary directory on the local filesystem. Default: /tmp/hbase-${user.name}hbase.regionserver.port The port the HBase RegionServer binds to. Default: 60020There are lot many parameters which need to be change for more customized and optimized Hbase cluster.
  • 18. HBASE-SITE.XMLJust as in Hadoop where you add site- specific HDFS configuration to the hdfs-site.xml file, for HBase, site specific customizations go into the file conf/hbase-site.xml. For the list of configurable properties
  • 19. <?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration> <property> <name>hbase.zookeeper.quorum</name> <value>node1,node2,node3</value> <description>The directory shared by RegionServers. </description> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/export/zookeeper</value> <description>Property from ZooKeepers config zoo.cfg. The directory where the snapshot is stored. </description>
  • 20. </property><property> <name>hbase.rootdir</name> <:value>hdfs//node0:8020/hbase</value> <description>The directory shared by RegionServers. </description></property><property> <name>hbase.cluster.distributed</name> <value>true</value> <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase- env.sh) </description></property>
  • 21. HBASE-ENV.SHSet HBase environment variables in this file. Examples include options to pass the JVM on start of an HBase daemon such as heap size and garbarge collector configs. You can also set configurations for HBase configuration, log directories, niceness, ssh options, where to locate process pid files, etc. Open the file at conf/hbase- env.sh and peruse its content. Each option is fairly well documented. Add your own environment variables here if you want them read by HBase daemons on startup.
  • 22. export JAVA_HOME=/usr/lib//jvm/java-6-sun/export HBASE_CLASSPATH=export HBASE_HEAPSIZE=1000
  • 23. LOG4J.PROPERTIESEdit this file to change rate at which HBase files are rolled and to change the level at which HBase logs messages.Changes here will require a cluster restart for HBase
  • 24. REGIONSERVERSIn this file you list the nodes that will run RegionServers.Eg :regionservernode1regionservernode2regionservernode3
  • 25. CONFIGURATIONSRequired Configurations Java SSH DNS NTP ulimit and nproc Hadoop for Distributed modeRecommended Configurations zookeeper.session.timeout  The default timeout is three minutes (specified in milliseconds). This means that if a server crashes, it will be three minutes before the Master notices the crash and starts recovery Number of ZooKeeper Instances Compression Bigger Regions Balancer  The balancer is a periodic operation which is run on the master to redistribute regions on the cluster. It is configured via hbase.balancer.period and defaults to 300000Still more are there, Just read more to have more optimized cluster.
  • 26. GETTING STARTED WITH HBASEStart HBase./bin/start-hbase.sh starting Master, logging to logs/hbase-user-master-example.org.outConnect to your running HBase via the shell./bin/hbase shell HBase Shell hbase(main):001:0>And on this shell you can type shell command which hbase provides to perform various operations.
  • 27. NEED MORE CLARIFICATION ON QUERIES??Search Read Research and Share to have more better understandability. Right ?http://db.csail.mit.edu/projects/cstore/abadi-sigmod08.pdfhttps://cs.uwaterloo.ca/~aelhelw/papers/dolap11.pdfhttps://ccp.cloudera.com/download/attachments/14549380/CDH2_In stallation_Guide.pdf?version=1&modificationDate=134887134000 0http://hbase.apache.org/https://ccp.cloudera.com/display/CDHDOC/HBase+InstallationLast and the best onewww.google.com :) I am here :dwivedishashwat@gmail.com Twitter : shashwat_2010 Facebook : shriparv@gmail.com Skype: shriparv