The way SoundCloud monitors its services and infrastructure has dramatically changed over the last couple of years. Monitoring, alerting, and being on-call is now deeply ingrained in every engineering team and we have learned a lot on the way.
This talk explains the goals and reasons Prometheus was born, show the early history of the project and how it has helped SoundCloud to fundamentally change its monitoring to support the rapid development of services.
Jenkins is an open source automation server written in Java. Jenkins helps to automate the non-human part of software development process, with continuous integration and facilitating technical aspects of continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
The way SoundCloud monitors its services and infrastructure has dramatically changed over the last couple of years. Monitoring, alerting, and being on-call is now deeply ingrained in every engineering team and we have learned a lot on the way.
This talk explains the goals and reasons Prometheus was born, show the early history of the project and how it has helped SoundCloud to fundamentally change its monitoring to support the rapid development of services.
Jenkins is an open source automation server written in Java. Jenkins helps to automate the non-human part of software development process, with continuous integration and facilitating technical aspects of continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
Virtual machines are generally considered secure. At least, secure enough to power highly multi-tenant, large-scale public clouds, where a single physical machine can host a large number of virtual instances belonging to different customers. Containers have many advantages over virtual machines: they boot faster, have less performance overhead, and use less resources. However, those advantages also stem from the fact that containers share the kernel of their host, instead of abstracting a new independent environment. This sharing has significant security implications, as kernel exploits can now lead to host-wide escalations.
We will show techniques to harden Linux Containers; including kernel capabilities, mandatory access control, hardened kernels, user namespaces, and more, and discuss the remaining attack surface.
Re:invent 2016 Container Scheduling, Execution and AWS Integrationaspyker
Members from over all over the world streamed over forty-two billion hours of Netflix content last year. Various Netflix batch jobs and an increasing number of service applications use containers for their processing. In this session, Netflix presents a deep dive on the motivations and the technology powering container deployment on top of Amazon Web Services. The session covers our approach to resource management and scheduling with the open source Fenzo library, along with details of how we integrate Docker and Netflix container scheduling running on AWS. We cover the approach we have taken to deliver AWS platform features to containers such as IAM roles, VPCs, security groups, metadata proxies, and user data. We want to take advantage of native AWS container resource management using Amazon ECS to reduce operational responsibilities. We are delivering these integrations in collaboration with the Amazon ECS engineering team. The session also shares some of the results so far, and lessons learned throughout our implementation and operations.
Modern Tools for API Testing, Debugging and MonitoringNeil Mansilla
Presented at DocuSign Momentum DevCon 2015 in San Francisco by Neil Mansilla. During the presentation, I discussed a number of useful community tools for API testing/debugging, followed by several live demos of Runscope API testing and monitoring tools. For more information, or to try Runscope, visit https://www.runscope.com
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
PCF Platform Monitoring with Prometheus and GrafanaVMware Tanzu
SpringOne Platform 2017
Alan Strader, Northern Trust; Jamie Christian, Northern Trust
"This presentation will cover Northern Trust's platform monitoring solution which is Grafana, Prometheus and Alertmanager. Specifically:
Enterprise need for monitoring of the platform
Options considered
Rationale for using this particular solution
Architecture of the solution and how it monitors our 5 foundations
A demo or screen captures
What we find valuable and what we look at daily to better manage the platform
Issues encountered as we deployed the solution (bosh/yml/forwarders)
Stories on how it saved us"
Distributed system coordination by zookeeper and introduction to kazoo python...Jimmy Lai
Zookeeper is a coordination tool to let people build distributed systems easier. In this slides, the author summarizes the usage of zookeeper and provides Kazoo Python library as example.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Virtual machines are generally considered secure. At least, secure enough to power highly multi-tenant, large-scale public clouds, where a single physical machine can host a large number of virtual instances belonging to different customers. Containers have many advantages over virtual machines: they boot faster, have less performance overhead, and use less resources. However, those advantages also stem from the fact that containers share the kernel of their host, instead of abstracting a new independent environment. This sharing has significant security implications, as kernel exploits can now lead to host-wide escalations.
We will show techniques to harden Linux Containers; including kernel capabilities, mandatory access control, hardened kernels, user namespaces, and more, and discuss the remaining attack surface.
Re:invent 2016 Container Scheduling, Execution and AWS Integrationaspyker
Members from over all over the world streamed over forty-two billion hours of Netflix content last year. Various Netflix batch jobs and an increasing number of service applications use containers for their processing. In this session, Netflix presents a deep dive on the motivations and the technology powering container deployment on top of Amazon Web Services. The session covers our approach to resource management and scheduling with the open source Fenzo library, along with details of how we integrate Docker and Netflix container scheduling running on AWS. We cover the approach we have taken to deliver AWS platform features to containers such as IAM roles, VPCs, security groups, metadata proxies, and user data. We want to take advantage of native AWS container resource management using Amazon ECS to reduce operational responsibilities. We are delivering these integrations in collaboration with the Amazon ECS engineering team. The session also shares some of the results so far, and lessons learned throughout our implementation and operations.
Modern Tools for API Testing, Debugging and MonitoringNeil Mansilla
Presented at DocuSign Momentum DevCon 2015 in San Francisco by Neil Mansilla. During the presentation, I discussed a number of useful community tools for API testing/debugging, followed by several live demos of Runscope API testing and monitoring tools. For more information, or to try Runscope, visit https://www.runscope.com
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
PCF Platform Monitoring with Prometheus and GrafanaVMware Tanzu
SpringOne Platform 2017
Alan Strader, Northern Trust; Jamie Christian, Northern Trust
"This presentation will cover Northern Trust's platform monitoring solution which is Grafana, Prometheus and Alertmanager. Specifically:
Enterprise need for monitoring of the platform
Options considered
Rationale for using this particular solution
Architecture of the solution and how it monitors our 5 foundations
A demo or screen captures
What we find valuable and what we look at daily to better manage the platform
Issues encountered as we deployed the solution (bosh/yml/forwarders)
Stories on how it saved us"
Distributed system coordination by zookeeper and introduction to kazoo python...Jimmy Lai
Zookeeper is a coordination tool to let people build distributed systems easier. In this slides, the author summarizes the usage of zookeeper and provides Kazoo Python library as example.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
하이퍼레저 페이지 단위 블록 조회(How to retrieve data more than totalQueryLimit)
- 블록 데이터 조회가 한 번의 요청에 10만건으로 제한이 되는 제약이 있는 상황에서 이를 페이징처리하여 조회할 수 있는 방법을 다룹니다.(It deals with how block data queries can be paged and queried in situations where there is a limit of 100,000 requests per time request.)
Kafka monitoring using Prometheus and Grafanawonyong hwang
Kafka Cluster를 모니터링 하기 위한 Prometheus 설정을 가이드하고, 이를 시각화하기 위해 Grafana를 연동하는 방법을 설명합니다.
Guide Prometheus settings for monitoring the Kafka Cluster and explain how to work with Grafana to visualize them.
Web Application Development with Quasar Framework
In this tutorial, You can see a rough development process with Quasar Framework which is known as front-end framework with VueJS components.
- Frontend : Quasar (based on Vue.js)
- Backend : Google firebase
- Result
* Web Page : https://checkin.wonyong.net
* Play Store : https://play.google.com/store/apps/details?id=org.kopochecker.app
- Youtube (Korean) : https://www.youtube.com/watch?v=HEttw-RSXxg&list=PLlWoe5hcgrk4qQVIBxDA3d-5ZRfYuITxb
kubernetes install and practice
* Environment (bare metal installation, not using cloud service)
- VM 1 : Mater node, 30GB, 2 vCPU, 4GB Mem
- VM 2 : Worker node, 30GB, 2 vCPU, 4GB Mem
* Practice
- deploying pod, make a deployment and service
- expose service using ingress(nginx-ingress)
Hyperledger Fabric practice material for Korea Polytechnics students
- Build Your First Network
- Chaincode Development
- Chaincode Devolopment via IBM Blockchain Platform
- Balance Transfer
- Vote system example using 'Balance Transfer' tutorial
Docker Practice for beginner.
- docker install on ubuntu 18.04 LTS
- docker pull/push
- making docker-compose file which serving spring-boot+ mySql application
Blockchain technology is the backbone of the Bitcoin and it can be used in many other disciplines. This presentation will introduce with basics like transaction, hash, crypto scheme.
This presentation is for anyone who wants to know about overall picture of Blockchain technology.
2. Intro
http/https request
80, 443 port
response
Ngrok agent
NAT / Firewall
외부와 양방향 통신 가능한 고정IP이 없는 경우에
Ngrok의 터널링 기능을 활용하면, 운영환경을 개발서버에서 시뮬레이션 할 수 있음
centos7
3. Nginx 설치
• yum install을 위한 nginx.repo 파일 생성 및 편집
# vi /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
# yum install -y nginx
4. Nginx 구동
• 서비스 구동 및 접속 확인
# systemctl start nginx
# systemctl enable nginx
6. Ngrok 구동
• 서비스 구동 및 접속 확인
# ngrok http 80
이 주소로 접속하면 Ngrok 중계서버가 나의 서버를 연결해 준다는 의미
이 주소는 리부팅 등 실행시마다 변동됨 (무료 계정)
또한, 8시간 정도만 주소가 유지됨 (실행 이후 주소 고정을 원할 경우 ‘인증토큰’을 추가해줘야 함 다음 장)
7. Ngrok 인증 토큰 추가
• Ngrok 회원 가입 필요
• 토큰 추가는 무료
• 무료 계정은 분당 요청 : 분당 약 40회까지 허용
유료 계정은 분당 약 120회
회원 가입 후 ‘Your Authtoken’ 메뉴 이동하여 토큰 확인
# ngrok config add-authtoken “인증 토큰”
Ngrok 서비스 중지 후 위 명령어 실행 Ngrok 서비스 실행
8. Ngrok 인증 토큰 추가
인증 토큰 추가 후, 재시작한 Ngrok 서비스에서
변경된 접속 주소 확인 후, 해당 주소로 접속
‘Visit Site’ 버튼 클릭
9. 외부 도메인 연결 + https 적용
준비사항 : 도메인 준비
예> 가비아 이벤트용 도메인 1년에 2,000원
[참고]
10. 도메인 등록 및 CNAME 설정
• 도메인 등록 및 CNAME 설정 후 nslookup으로 등록여부 확인
- 최초 도메인 등록時 DNS전파에 일정 시간 소요 (0분~00분)
12. certbot을 활용한 인증서 발급
• certbot certonly -d [도메인명] --manual --preferred-challenges dns
# certbot certonly -d server.kopo.online --manual --preferred-challenges dns
거의 모든 스텝을 ‘Y’로 입력 진행하다가,
우측의 DNS TXT 필드 입력 문구가 나오면
DNS관리에서 입력 및 저장을 먼저 완료하고
엔터를 입력해야 정상 진행됨
14. nginx에 인증서 설정 및 ssl 서비스 등록
• vi /etc/nginx/conf.d/default.conf
• nginx -s reload
server{
listen 443 ssl;
server_name server.kopo.online;
root /usr/share/nginx/html;
index index.html;
ssl_certificate /etc/letsencrypt/live/server.kopo.online/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/server.kopo.online/privkey.pem;
}
15. 로컬에서 자체 테스트
로컬 개발서버에서 hosts파일을 임시로 편집하여 로컬 브라우저로 접속하였을때 정상여부를 확인
정상이면 hosts파일은 원상복구
16. 인증서 자동 갱신 크론탭 등록
# crontab -e
0 0 1 * * root systemctl stop nginx && certbot renew -q && systemctl start nginx
매달 1일에 인증서 갱신 시도
17. 외부 도메인과 Ngrok 연계
• server.kopo.online 도메인을 Ngrok으로 연계하기 위해서는
유료 계정을 사용해야 한다.
18. 외부 도메인과 Ngrok 연계
• 유료 계정 가입 후 아래 링크로 이동
25$/month, 1년 단위 계약시 20$/month
• https://dashboard.ngrok.com/cloud-edge/domains
27. 외부 도메인과 Ngrok 연계
• 서버에서 /etc/hosts 파일에 server.kopo.online을 추가해야 하는 이유
상기 내용 미반영시,
서버 외부 브라우저 접속시 인증서 불일치 경고 발생(서버내에 있는 브라우저는 정상 접속)
추정 사유> server.kopo.online 접속 -> ngrok 중계, ngrok 인증서 교환 -> 로컬 서버 응답 ->
응답시 server.kopo.online을 참조하는데 위와 같이 설정하지 않으면 다시 ngrok 서버 및 해당
인증서로 연계하여 접속 주소와 인증서의 불일치가 발생하게 됨 (상기 그림)