• Save
Cloumon enterprise
Upcoming SlideShare
Loading in...5
×
 

Cloumon enterprise

on

  • 4,267 views

monitoring system for opensource cloud computing, ZooKeeper, Hadoop, Cassandra, HBase....

monitoring system for opensource cloud computing, ZooKeeper, Hadoop, Cassandra, HBase....

Statistics

Views

Total Views
4,267
Views on SlideShare
3,976
Embed Views
291

Actions

Likes
3
Downloads
0
Comments
0

8 Embeds 291

http://www.cloumon.org 219
http://cloumon.org 33
http://play.daumcorp.com 30
url_unknown 4
http://www.techgig.com 2
http://localhost:8080 1
http://twitter.com 1
http://a0.twimg.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Cloumon enterprise Cloumon enterprise Presentation Transcript

  • Management/Monitoring for Cloud Computing !   ZooKeeper !   Cassandra !   Hadoop contact@gruter.com 070-8129-2950
  • Cloumon ? (Cloumon®) ZooKeeper, Cassandra, Hadoop , .Copyright © 2011 Gruter Ltd.. All rights reserved 2
  • Cloumon ZooKeeper Cassandra Hadoop •  Znode •  Cassandra •  Hadoop •  Znode •  Cassandra •  Znode •  Hadoop • Copyright © 2011 Gruter Ltd.. All rights reserved 3
  • Why Cloumon? •  –  –  •  –  , –  •  –  –  CloumonCopyright © 2011 Gruter Ltd.. All rights reserved 4
  • Server Storage ERP SCM Mail Virtualization Tech. Automatic Management Tool , , , IT . IT .Copyright © 2011 Gruter Ltd.. All rights reserved 5
  • Traditional vs. Cloud Computing Traditional Cloud(AWS) Request Request Firewall Security Group Load Balancer Elastic AutoScaling (L4) LoadBalancer Physical Server Physical Server EC2 Instance EC2 Instance (Unix/Linux) (Unix/Linux) Cloud Monitoring Watch Database Storage Various DB Cloud Storage (Oracle) (SAN, NAS) (NoSQL, RDBMS) Backup ( Tape, Disk Array) -  -Copyright © 2011 Gruter Ltd.. All rights reserved 6
  • System Architecture for Cloud ComputingCopyright © 2011 Gruter Ltd.. All rights reserved 7
  • Software Components for Cloud Computing Infrastructure Service Runtime Environment Cloud Data Storage Interface Protocol (Cassandra, Hbase) (Thrift, SOAP, REST, …) Management API Lightweight Cont Other Framewor Cloud File Storage ainer(Spring) ks (Hadoop) Data Mapper File System API Server Virtualization Application Server (KVM, Xen, VMWare) (Tomcat, .NET, Thrift Server, …) Development Environment Management/Common Service Management Portal IDL Compiler Common Service (Thrift, WSDL) (Billing, Metering, SSO) Service Repository/Framework plug-in Service Repository Development IDE Distributed Coordinator (Eclipse) (ZooKeeper)Copyright © 2011 Gruter Ltd.. All rights reserved 8
  • ZooKeeper •  –  Distributed Lock Coordinator –  SPOF(Single Point Of Failure) –  : zookeeper.apache.org •  –  •  •  n replication –  Notification •  , , •  –  –  High Availability Master Election –  LockCopyright © 2011 Gruter Ltd.. All rights reserved 9
  • ZooKeeperCopyright © 2011 Gruter Ltd.. All rights reserved 10
  • ZooKeeper Usage 1) create() ZooKeeper create(“10.1.1.1”, EPHEMERAL) subA_server1 getChildren(“/service/subA”, watcher) 10.1.1.1 /service/subA subA 10.1.1.1 client create(“10.1.1.2”, event 10.1.1.2 EPHEMERAL) 3) getChildren cache 4) cache subA subA_server2 6) subA / watcher 10.1.1.3 10.1.1.2 cache event 5) (EPHEMERAL) subA_server3 10.1.1.3Copyright © 2011 Gruter Ltd.. All rights reserved 11
  • ZooKeeper Usage High Availability: Master Election 1) lock master 2) master IP setData subA_master_1 ZooKeeper (10.1.1.1) subA_master /subA/master lock release client (10.1.1.1) 3) getData(“/service/subA_master”) 4) cache -  acquireLock() 5) watcher release event -  register watcher subA_master_2 (10.1.1.2) 6) master lock 7) IP setDataCopyright © 2011 Gruter Ltd.. All rights reserved 12
  • Cloumon ZooKeeper •  –  Znode •  Action •  / / –  CloumonManager Acton •  –  Refresh •  ZooKeeper –  ZooKeeper –  –  Historical Notification Listener CloumonManager ZooKeeper Aciton Alarm Node Manager Manager (Mail, SMS) Znode ZooKeeper Cluster Manager Server #1 Action Manager Host Manager ZooKeeper (Mail, SMS) Server #2 Node Metrics ZooKeeper Data Data Server #nCopyright © 2011 Gruter Ltd.. All rights reserved 13
  • Cloumon ZooKeeper •  Latency min/avg/max: Latency •  Received: Packet •  Sent: Packet •  Outstanding: request •  Zxid: ID •  Mode zookeeper: (leader, follower) •  Node count: znode •  Client Connection ( Connection ) –  remoteAddress: –  Queued: request –  Recved: packet –  Sent: packet –  Sid: session Id –  Lop: connection Operation –  Est: –  to : connection session timeout –  Lcxid: Last cxid of this connection –  Lzxid: connection Id –  Minlat: latency –  Avglat: latency –  Maxlat: latencyCopyright © 2011 Gruter Ltd.. All rights reserved 14
  • Cloumon ZooKeepper Node •  –  Cloumon instance ZooKeeper •  –  Tree View –  / / •  ACL •  Recursive –  •  Validation Check Rule –  ACL •  ACL •  ACL –  Event Notification •  Event Listener •  (Node Deleted, Data Changed, Child Changed) Listener •  ActionCopyright © 2011 Gruter Ltd.. All rights reserved 15
  • Cassandra •  –  Facebook , –  RDBMS NoSQL •  –  Keyspace, ColumnFamily, ColumnKey, Column •  –  N Linux –  Hash(key) –  / •  –  n-replication –  •  –  : Thrift, Avro –  : Java, C/C++, PHP, Python –  3rd : HectorCopyright © 2011 Gruter Ltd.. All rights reserved 16
  • Cassandra Server1 (token=A) Storage RPC Daemon Server5 Service (Thrift, Avro) Server2 (token=V) (token=D) JMX Server4 Server3 (token=O) (token=K) •  (P2P) •  –  token token –  Server2: A → D, Server3: D → K, … Server1: V → A •  Partitioner –  Random Partitioner: Hash(Key) –  OrderPreservingPartitioner: KeyCopyright © 2011 Gruter Ltd.. All rights reserved 17
  • Cloumon Cassandra •  Cassandra Seed –  Seed •  –  –  –  Historical •  Cassandra –  , , Notification Data CloumonManager Cassandra Alarm Cassandra Manager Manager Cassandra (Mail, SMS) / Cassandra Cluster Manager Server #1 Action Manager Host Manager Cassandra (Mail, SMS) Server #2 Metrics Cassandra Data Data Server #nCopyright © 2011 Gruter Ltd.. All rights reserved 18
  • Cloumon Cassandra •  14 , 190 •  Size •  db.Caches •  db.Caches •  Capacity •  db.Commitlog •  Requests •  db.ColumnFailies •  Hits •  db.IndexColumnFamilies •  RecentHitRate •  db.CompactionManager •  db.DynamicEndpointSnitch •  MaxRowSize •  db.StorageProxy •  MemtableDataSize •  db.StorageService •  MemtableSwitchCount •  net.StreamingService •  net.MessagingService •  MemtableThroughputInMB •  net.FailureDetector •  ReadCount •  request.MutationStage •  TotalDiskSpaceUsed •  request.ReadStage •  request.RequestResponseStage •  WriteCount •  request.ReplicateOnWriteStage •  RecentRangeLatencyMicros •  request.ReadRepairStage •  RecentReadLatencyMicros •  internal.AntiEntropyStage •  RecentWriteLatencyMicros •  internal.GossipStage •  internal.MigrationStage •  Joined •  internal.MiscStage •  Keyspaces •  internal.StreamStage •  LeavingNodes •  internal.InternalResponseStage •  internal.FlushSorter •  LiveNodes •  internal.FlushWriter •  Load •  internal.HintedHandoff •  RPCServerRunning •  internal.FILEUTILS-DELETE-POOL •  internal.MemtablePostFlusher •  RemovalStatus •  Token •  StatusCopyright © 2011 Gruter Ltd.. All rights reserved 19
  • Hadoop •  Cloud Computing –  / –  •  Distributed File System –  Very Large Scale Distributed File System •  10K nodes, 100 million files, 10 PB –  Use Commodity Hardware •  self-healing: failover, recovery, backup –  Optimized for batch processing •  Distributed/Parallel Computing framework –  MapReduce , –  HDFS •  / –  •  FIFO, Fair, Capacity •  –  •  Job: •  Task: n Task(Map or Reduce) –  / •  •  , (streaming)Copyright © 2011 Gruter Ltd.. All rights reserved 20
  • Hadoop Secondary NameNode NameNode File Namespace /foo/bar NameNode Request Meta 1 Operation Client Server Status Block Management File Block File Read/Write DataNode #1 DataNode #2 DataNode #3 DataNode #4 DataNode #5 (rack)1 (rack)2Copyright © 2011 Gruter Ltd.. All rights reserved 21
  • Cloumon Hadoop •  Hadoop Metrics –  Hadoop metrics context CloumonContext •  –  Hadoop CloumonManager metrics –  –  –  Historical •  Hadoop –  : , , Notification Data CloumonManager Hadoop Alarm Hadoop Manager Manager (Mail, SMS) Hadoop Cloumon hadoop Cluster Manager Context Server #1 Action Manager Host Manager Cloumon hadoop (Mail, SMS) Context Server #2 Metrics Cloumon hadoop Data Data Context Server #nCopyright © 2011 Gruter Ltd.. All rights reserved 22
  • Cloumon Hadoop •  NameNode –  FSNamesystem •  BlocksTotal, CapacityRemain, CapacityTotal, CapacityUse d, FilesTotal, MissingBlocks, TotalLoad –  Dfs.NameNode •  BlockCorrupted, CreateFileOps, Transactions_avg_time –  Jvm.metrics: •  gcCount, gcTimeMillis, memHeapUsed, logWarn, logError, threadsNew •  DataNode –  Dfs.datanode •  Blocks_operation, bytes_read, bytes_written, avg_time –  Jvm.metrics: •  gcCount, gcTimeMillis, memHeapUsed, logWarn, logError, threadsNewCopyright © 2011 Gruter Ltd.. All rights reserved 23
  • Cloumon ZooKeeperCopyright © 2011 Gruter Ltd.. All rights reserved 24
  • Cloumon ZooKeeperCopyright © 2011 Gruter Ltd.. All rights reserved 25
  • Cloumon CassandraCopyright © 2011 Gruter Ltd.. All rights reserved 26
  • Cloumon HadoopCopyright © 2011 Gruter Ltd.. All rights reserved 27
  • Cloumon HadoopCopyright © 2011 Gruter Ltd.. All rights reserved 28
  • Cloumon •  1 instance –  –  metrics DB •  MySQL : 2000 ~ 3000 Transaction/sec –  •  60 •  ZooKeeper –  ZooKeeper ZooKeeper ( clien t 1 record ) –  ZooKeeper Client /60 –  ZooKeeper •  Cassandra –  190 /60 * –  Cassandra 600 –  190 •  Hadoop –  77 /60 * –  Hadoop 800 –  77 •  –  Historical –  10 12GBCopyright © 2011 Gruter Ltd.. All rights reserved 29
  • Open Source vs. Enterprise Open Source Enterprise X Server O ( ) O O O ZooKeeper ACL X O Listener X O Cassandra X O Hadoop O O User Interface Bad GoodCopyright © 2011 Gruter Ltd.. All rights reserved 30
  • Cloumon release milestone •  Cloumon-0.9: 2011.03 –  –  www.cloumon.org –  –  ZooKeeper , Hadoop •  Cloumon-enterprise-1.0: 2011.05 –  ZooKeeper –  ZooKeeper ACL, Event Listener –  Cassandra –  Hadoop –  UI •  Cloumon-enterprise-2.0: 2011.07 –  Cassandra –  Cassandra Client(Hector) –  (Dashboard) –  Cloumon Manager HA –  UI ( ) –  •  Cloumon-enterprise-3.0: 2011.09 –  Hadoop MapReduce –  HbaseCopyright © 2011 Gruter Ltd.. All rights reserved 31