Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloudera & Zookeeper

512 views

Published on

Cloudera & Zookeeper in DataLAB

Published in: Software
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Cloudera & Zookeeper

  1. 1. Cloudera Manager 클라우데라매니저로구축하는빅데이터에코시스템
  2. 2. Cloudera? 클라우데라는Apache Hadoop에대해가장잘압니다 세계 3대하둡기업: 클라우데라, 호튼웍스, 맵알 하둡의창시자"더그 커팅"도수석아키텍트로합류 빅데이터시스템고도화관련컨설팅 주요제품: DataHub, AnalyticDB, Impala
  3. 3. Cloudera Manager? 빅데이터에코시스템을쉽게 설치하고 관리해주는도구 Cloudera Express (무료) / Enterprise (유료) 클라우데라하둡배포판(CDH)를사용
  4. 4. Apache BigTop 아파치재단의Top‑level 프로젝트 빅데이터에코시스템을패키징, 테스트, 가상화등지원 AWS EMR 4.0부터빌드및패키징프로세스를관리하기 위해Apache Bigtop를사용
  5. 5. 환경 구축 도저히내노트북에가상머신3개를돌릴수없음 Amazon Web Service 클라우드환경에구축 AWS t2.micro / CentOS 7 / 1GiB / 8GiB 먼저인스턴스하나에환경 구축하고, 이미지복제
  6. 6. AWS 관련Tip CentOS는기본으로제공하지않음 Marketplace에서이미지를받아서사용 default 유저아이디는centos (centos@12.214.12.3)
  7. 7. 그러나...
  8. 8. Trouble‑Shooting t2.micro free tier 인스턴스에서절대안돌아간다 메모리문제라고 클라우데라에서말하더라 명시되어있는권장인스턴스m4.large AWS m4.large / CentOS 7 / 8GiB / 8GiB
  9. 9. Trouble‑Shooting 다좋은데, 3개 돌아가는거 생각하면너무비싸다 시간 당$0.123 * 3 = $0.369 AMI를만들어두고 Spot Instance를활용 시간 당$0.0184 * 3 = $0.0552 가격이훨씬저렴하다
  10. 10. Cloudera Manager 설치
  11. 11. Cloudera Manager 설치 공식문서가 너무친절함 http://www.cloudera.com/documentation/manager/5‑1‑ x/Cloudera‑Manager‑Installation‑ Guide/cm5ig_install_on_ec2.html AWS에서설치할때는 https://hadoopabcd.wordpress.com/2015/01/21/installing‑ cloudera‑manager‑and‑cdh‑on‑amazon‑ec2‑part‑1/
  12. 12. Cloudera Manager 설치 가장최신버전을받아서그대로설치진행 $ wget http://archive.cloudera.com/cm5/installer/latest/clouder $ chmod u+x cloudera-manager-installer.bin $ ./cloudera-manager-installer.bin
  13. 13. Trouble‑Shooting AWS Inbound Rules 지정 TCP 7180 ~ 7182 ICMP protocol 각 인스턴스에Elastic IP 할당(고정아이피) CentOS 방화벽해제 service iptables stop
  14. 14. Trouble‑Shooting Fatal Error: SELinux is enabled. It must be disabled to install and use this product.
  15. 15. Trouble‑Shooting $ vi /etc/selinux/config $ SELINUX=disabled # 변경 클라우데라공식문서는친절하다 https://www.cloudera.com/documentation/enterprise/5‑5‑ x/topics/install_cdh_disable_selinux.html
  16. 16. Cloudera Manager 접속
  17. 17. Cloudera Manager 접속 http:// [AWS public IP] :7180/ username : admin password : admin
  18. 18. Cloudera Manager 설정
  19. 19. Trouble‑Shooting
  20. 20. Trouble‑Shooting CDH를설치하기 위해임시로루트계정의ssh 접속허용 설치후에는다시잠금 설정할것 $ sudo vi /etc/ssh/sshd_config $ PermitRootLogin yes $ systemctl restart sshd.service # Log message $ sudo tail -F /var/log/cloudera-scm-agent/cloudera-scm-agent.
  21. 21. Link https://www.cloudera.com/documentation/enterprise/release‑ notes/topics/cm_rn_known_issues.html
  22. 22. ZooKeeper
  23. 23. ZooKeeper 서버가 수십, 수백대인분산처리환경 네트워크장애, 예기치못한중지, 서버확장 분산된시스템간의정보를어떻게 공유할것인지 클러스터에있는서버들의상태를체크 분산된서버들간에동기화를위한락(lock)을처리
  24. 24. ZooKeeper는심플하다 Zookeeper의데이터모델(디렉토리구조) 디렉토리구조의각 노드에데이터를저장(znode) 서버의상태, 락정보, 환경 설정과 같은메타데이터
  25. 25. ZooKeeper 클라이언트들은단일주키퍼서버에연결 클라이언트는TCP 연결로요청을보내거나, 응답을받고, 이벤트를감시 하며하트비트(heartbeats)를보내는것을통해유지 만약주키퍼서버가 사용불가한상태가 되면클라이언트의TCP 연결은 다른주키퍼서버로연결
  26. 26. SPOF (Single Point of Failure) 단일장애지점 문제가 발생하면서비스전체의장애를초래할수도있음 어떻게 해결? 다중화 보통3대의서버를클러스터로묶어서배포(ensemble) 클러스터에포함되는서버는모두같은데이터를저장 클러스터마다리더가 있어서리더가 데이터일관성을보장
  27. 27. SPOF (Single Point of Failure)
  28. 28. 그런데만약리더가 죽으면? 리스트의가장위에있는노드가 리더의역할을대체
  29. 29. Zookeeper 운영시주의사항 데이터의변경이자주발생하는서비스에서ZooKeeper를데이터저장 소로사용하는것은비추 ZooKeeper 서버가 제대로실행되지않을때가 있는데, 대부분서버간의 데이터불일치로인한데이터동기화실패가 그 원인 zoo.cfg라는설정(configuration) 파일에ZooKeeper 서버목록확인

×