Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation
Yuto Kawamura
LINE / Z Part Team
At LINE we've been operating Apache Kafka to provide the company-wide shared data pipeline for services using it for storing and distributing data.
Kafka is underlying many of our services in some way, not only the messaging service but also AD, Blockchain, Pay, Timeline, Cryptocurrency trading and more.
Many services feeding many data into our cluster, leading over 250 billion daily messages and 3.5GB incoming bytes in 1 second which is one of the world largest scale.
At the same time, it is required to be stable and performant all the time because many important services uses it as a backend.
In this talk I will introduce the overview of Kafka usage at LINE and how we're operating it.
I'm also going to talk about some engineerings we did for maximizing its performance, solving troubles led particularly by hosting huge data from many services, leveraging advanced techniques like kernel-level dynamic tracing.
UXDev Summit Keynote : A real world story of Angular and Apache Unomi integra...Serge Huber
In this presentation I talk about how we used Angular and Apache Unomi to build our Marketing Factory product. I talk about the process, best practices and lessons learned while selecting and then integration both technologies to deliver our new platform in record time.
Integrating Apache Kafka Into Your Environmentconfluent
Watch this talk here: https://www.confluent.io/online-talks/integrating-apache-kafka-into-your-environment-on-demand
Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams.
This session is part 4 of 4 in our Fundamentals for Apache Kafka series.
Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesLINE Corporation
Yuto Kawamura
LINE / Z Part Team
At LINE we've been operating Apache Kafka to provide the company-wide shared data pipeline for services using it for storing and distributing data.
Kafka is underlying many of our services in some way, not only the messaging service but also AD, Blockchain, Pay, Timeline, Cryptocurrency trading and more.
Many services feeding many data into our cluster, leading over 250 billion daily messages and 3.5GB incoming bytes in 1 second which is one of the world largest scale.
At the same time, it is required to be stable and performant all the time because many important services uses it as a backend.
In this talk I will introduce the overview of Kafka usage at LINE and how we're operating it.
I'm also going to talk about some engineerings we did for maximizing its performance, solving troubles led particularly by hosting huge data from many services, leveraging advanced techniques like kernel-level dynamic tracing.
UXDev Summit Keynote : A real world story of Angular and Apache Unomi integra...Serge Huber
In this presentation I talk about how we used Angular and Apache Unomi to build our Marketing Factory product. I talk about the process, best practices and lessons learned while selecting and then integration both technologies to deliver our new platform in record time.
Integrating Apache Kafka Into Your Environmentconfluent
Watch this talk here: https://www.confluent.io/online-talks/integrating-apache-kafka-into-your-environment-on-demand
Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams.
This session is part 4 of 4 in our Fundamentals for Apache Kafka series.
Secure Your Kubernetes Apps from Attacks with NGINXNGINX, Inc.
On-Demand Recording:
https://www.nginx.com/resources/webinars/secure-your-kubernetes-apps-from-attacks-with-nginx/
With more and more organizations conducting business with their customers online, web applications remain the top attack target for cybercriminals. It’s easy to see why; they are often complex, composed of microservices, and spanning distributed environments, increasing the number of endpoints vulnerable to exploitation. It is no surprise that OWASP Top 10 vulnerabilities such as SQL injection and cross scripting (XSS) remain popular cyberattacks. You need to strengthen security for your containerized apps deployed in Kubernetes by adding a WAF to NGINX Ingress Controller.
In this webinar we look at the cost of a typical application hack and why traditional WAFs don’t work for today’s modern applications. We conclude with a demo that showcases how by combining NGINX Ingress Controller with NGINX App Protect WAF you can protect your apps against common vulnerabilities, create granular policies for app services, and make your Kubernetes clusters a safer place to run your apps.
Integrating Splunk into your Spring ApplicationsDamien Dallimore
How much visibility do you really have into your Spring applications? How effectively are you capturing,harnessing and correlating the logs, metrics, & messages from your Spring applications that can be used to deliver this visibility ? What tools and techniques are you providing your Spring developers with to better create and utilize this mass of machine data ? In this session I'll answer these questions and show how Splunk can be used to not only provide historical and realtime visibility into your Spring applications , but also as a platform that developers can use to become more "devops effective" & easily create custom big data integrations and standalone solutions.I'll discuss and demonstrate many of Splunk's Java apps,frameworks and SDK and also cover the Spring Integration Adaptors for Splunk.
Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022)Lucas Jellema
This session does a quick recap of microservices: why do we want them, what problems do they solve and what are the principles around designing and implementing them? The Dapr.io runtime framework for distributed applications is introduced. Dapr provides a sidecar (almost like a personal assistant to a manager) to an application or microservice, a companion process that handles common tasks such as storing and retrieving state, consuming and publishing messages and events, invoking external services and other microservices as well as handling incoming requests. Participants will do a handson lab with Dapr.io and learn how to quickly implement interactions with various technologies, including Redis and MySQL.
Node(JS) is introduced – a server side JavaScript-based programming language that can be used well for implementing microservices. Some of the main characteristics of NodeJS are discussed (functional programming, asynchronous flows, NPM package manager) as well as common use cases (handle incoming HTTP requests, invoke REST APIs). In the second lab, Node and Dapr are used together to implement microservices that interact with databases and message brokers and each other – in a decoupled fashion.
Slides from the keynote Elie Auvray (CEO and Co-Founder of Jahia) and Serge Huber (CTO and Co-Founder of Jahia) presented at ApacheCon:Core 2015 to introduce the Apache Unomi Project: the Way to Ethical WEM.
It is incumbent on digital enterprises to protect their customer data privacy; the Apache Unomi Project helps manage big data without sacrificing customer data privacy.
About OASIS
OASIS (Organization for the Advancement of Structured Information Standards) is a nonprofit, international consortium whose goal is to promote the adoption of product-independent standards for information formats such as Standard Generalized Markup Language (SGML), Extensible Markup Language (XML), and Hypertext Markup Language (HTML).
About Apache Software Foundation
Established in 1999, the Apache Software Foundation (ASF) is a US charitable organization, funded by individual donations and corporate sponsors. The mission of the ASF is to provide software for the public good by providing services and support for many like-minded software project communities of individuals who choose to join the ASF. (http://www.apache.org/foundation/)
About Apache Unomi Project
Apache Unomi is the reference implementation of the upcoming OASIS Context Server (CXS) standard to help standardize personalization of online experience while promoting ethical web experience management and increased user privacy controls. (http://unomi.incubator.apache.org/)
About Jahia
Jahia Solutions Group (Jahia) is a leading provider of a customizable digital experience management platform which enables enterprises to put their customer at the heart of their business. Our technology breaks down silos, facilitates authentic 1:1 customer relationships powered by agile innovation, trust and a customer-centric digital workforce. This focus ensures digital enterprise transformation for success in today’s competitive business landscape.
Founded in 2002 and headquartered in Geneva, Switzerland, Jahia has its North American headquarters in Washington, D.C. and offices in Toronto, Canada, and throughout Europe. Jahia counts hundreds of global brands and governmental organizations, in more than 20 countries, among its esteemed customers. http://www.jahia.com
Grant Allen, CTO Chief Product Officer at Dow Jones explains how to deploy Flowable at scale in AWS.
It was presented at the Flowfest 2018 in Barcelona, Spain
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유Hyojun Jeon
NDC18에서 발표하였습니다. 현재 보고 계신 슬라이드는 1부 입니다.(총 2부)
- 1부 링크: https://goo.gl/3v4DAa
- 2부 링크: https://goo.gl/wpoZpY
(SlideShare에 슬라이드 300장 제한으로 2부로 나누어 올렸습니다. 불편하시더라도 양해 부탁드립니다.)
Real-time Analytics with Presto and Apache PinotXiang Fu
Presto Con 2021
In this world, most analytics products either focus on ad-hoc analytics, which requires query flexibility without guaranteed latency, or low latency analytics with limited query capability.
In this talk, we will explore how to get the best of both worlds using Apache Pinot and Presto:
1. How people do analytics today to trade-off Latency and Flexibility: Comparison over analytics on raw data vs pre-join/pre-cube dataset.
2. Introduce Apache Pinot as a column store for fast real-time data analytics and Presto Pinot Connector to cover the entire landscape.
3. Deep dive into Presto Pinot Connector to see how the connector does predicate and aggregation push down.
4. Benchmark results for Presto Pinot connector.
Did you know that you can automate Akamai tasks related Luna Control Center without having to log in? Join this session to learn about how you can achieve this using different APIs like PAPI, GTM, and others. We'll showcase in-house tools which will leverage APIs to a) create, edit and push configurations, b) get GTM liveness test details across your account, c) get origin details like hostname, cache keys, certificate info across your accounts, and d) invoke Virtual Prioritization-related functions using the Akamai command-line interface. We will demonstrate how Akamai APIs can help you save time and resources across your Akamai deployment.
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...Riccardo Zamana
Time series Analytics - a deep dive into ADX Azure Data Explorer. Let’s discover with a step-by-step approach the entire ecosystem of features driven by Azure Data eXplorer.
Secure Your Kubernetes Apps from Attacks with NGINXNGINX, Inc.
On-Demand Recording:
https://www.nginx.com/resources/webinars/secure-your-kubernetes-apps-from-attacks-with-nginx/
With more and more organizations conducting business with their customers online, web applications remain the top attack target for cybercriminals. It’s easy to see why; they are often complex, composed of microservices, and spanning distributed environments, increasing the number of endpoints vulnerable to exploitation. It is no surprise that OWASP Top 10 vulnerabilities such as SQL injection and cross scripting (XSS) remain popular cyberattacks. You need to strengthen security for your containerized apps deployed in Kubernetes by adding a WAF to NGINX Ingress Controller.
In this webinar we look at the cost of a typical application hack and why traditional WAFs don’t work for today’s modern applications. We conclude with a demo that showcases how by combining NGINX Ingress Controller with NGINX App Protect WAF you can protect your apps against common vulnerabilities, create granular policies for app services, and make your Kubernetes clusters a safer place to run your apps.
Integrating Splunk into your Spring ApplicationsDamien Dallimore
How much visibility do you really have into your Spring applications? How effectively are you capturing,harnessing and correlating the logs, metrics, & messages from your Spring applications that can be used to deliver this visibility ? What tools and techniques are you providing your Spring developers with to better create and utilize this mass of machine data ? In this session I'll answer these questions and show how Splunk can be used to not only provide historical and realtime visibility into your Spring applications , but also as a platform that developers can use to become more "devops effective" & easily create custom big data integrations and standalone solutions.I'll discuss and demonstrate many of Splunk's Java apps,frameworks and SDK and also cover the Spring Integration Adaptors for Splunk.
Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022)Lucas Jellema
This session does a quick recap of microservices: why do we want them, what problems do they solve and what are the principles around designing and implementing them? The Dapr.io runtime framework for distributed applications is introduced. Dapr provides a sidecar (almost like a personal assistant to a manager) to an application or microservice, a companion process that handles common tasks such as storing and retrieving state, consuming and publishing messages and events, invoking external services and other microservices as well as handling incoming requests. Participants will do a handson lab with Dapr.io and learn how to quickly implement interactions with various technologies, including Redis and MySQL.
Node(JS) is introduced – a server side JavaScript-based programming language that can be used well for implementing microservices. Some of the main characteristics of NodeJS are discussed (functional programming, asynchronous flows, NPM package manager) as well as common use cases (handle incoming HTTP requests, invoke REST APIs). In the second lab, Node and Dapr are used together to implement microservices that interact with databases and message brokers and each other – in a decoupled fashion.
Slides from the keynote Elie Auvray (CEO and Co-Founder of Jahia) and Serge Huber (CTO and Co-Founder of Jahia) presented at ApacheCon:Core 2015 to introduce the Apache Unomi Project: the Way to Ethical WEM.
It is incumbent on digital enterprises to protect their customer data privacy; the Apache Unomi Project helps manage big data without sacrificing customer data privacy.
About OASIS
OASIS (Organization for the Advancement of Structured Information Standards) is a nonprofit, international consortium whose goal is to promote the adoption of product-independent standards for information formats such as Standard Generalized Markup Language (SGML), Extensible Markup Language (XML), and Hypertext Markup Language (HTML).
About Apache Software Foundation
Established in 1999, the Apache Software Foundation (ASF) is a US charitable organization, funded by individual donations and corporate sponsors. The mission of the ASF is to provide software for the public good by providing services and support for many like-minded software project communities of individuals who choose to join the ASF. (http://www.apache.org/foundation/)
About Apache Unomi Project
Apache Unomi is the reference implementation of the upcoming OASIS Context Server (CXS) standard to help standardize personalization of online experience while promoting ethical web experience management and increased user privacy controls. (http://unomi.incubator.apache.org/)
About Jahia
Jahia Solutions Group (Jahia) is a leading provider of a customizable digital experience management platform which enables enterprises to put their customer at the heart of their business. Our technology breaks down silos, facilitates authentic 1:1 customer relationships powered by agile innovation, trust and a customer-centric digital workforce. This focus ensures digital enterprise transformation for success in today’s competitive business landscape.
Founded in 2002 and headquartered in Geneva, Switzerland, Jahia has its North American headquarters in Washington, D.C. and offices in Toronto, Canada, and throughout Europe. Jahia counts hundreds of global brands and governmental organizations, in more than 20 countries, among its esteemed customers. http://www.jahia.com
Grant Allen, CTO Chief Product Officer at Dow Jones explains how to deploy Flowable at scale in AWS.
It was presented at the Flowfest 2018 in Barcelona, Spain
[NDC18] 야생의 땅 듀랑고의 데이터 엔지니어링 이야기: 로그 시스템 구축 경험 공유Hyojun Jeon
NDC18에서 발표하였습니다. 현재 보고 계신 슬라이드는 1부 입니다.(총 2부)
- 1부 링크: https://goo.gl/3v4DAa
- 2부 링크: https://goo.gl/wpoZpY
(SlideShare에 슬라이드 300장 제한으로 2부로 나누어 올렸습니다. 불편하시더라도 양해 부탁드립니다.)
Real-time Analytics with Presto and Apache PinotXiang Fu
Presto Con 2021
In this world, most analytics products either focus on ad-hoc analytics, which requires query flexibility without guaranteed latency, or low latency analytics with limited query capability.
In this talk, we will explore how to get the best of both worlds using Apache Pinot and Presto:
1. How people do analytics today to trade-off Latency and Flexibility: Comparison over analytics on raw data vs pre-join/pre-cube dataset.
2. Introduce Apache Pinot as a column store for fast real-time data analytics and Presto Pinot Connector to cover the entire landscape.
3. Deep dive into Presto Pinot Connector to see how the connector does predicate and aggregation push down.
4. Benchmark results for Presto Pinot connector.
Did you know that you can automate Akamai tasks related Luna Control Center without having to log in? Join this session to learn about how you can achieve this using different APIs like PAPI, GTM, and others. We'll showcase in-house tools which will leverage APIs to a) create, edit and push configurations, b) get GTM liveness test details across your account, c) get origin details like hostname, cache keys, certificate info across your accounts, and d) invoke Virtual Prioritization-related functions using the Akamai command-line interface. We will demonstrate how Akamai APIs can help you save time and resources across your Akamai deployment.
Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...Riccardo Zamana
Time series Analytics - a deep dive into ADX Azure Data Explorer. Let’s discover with a step-by-step approach the entire ecosystem of features driven by Azure Data eXplorer.
Deview 2013 :: Backend PaaS, CloudFoundry 뽀개기Nanha Park
# Part 1
개발자의 주위환경에 대해 살펴보고 Cloud Foundry overview, Cloud Foundry 를 구성하는 components 마지막으로 Deploy 환경에 대해 알아보겠습니다.
# Part 2
설치부터 코드까지, 데모찍은 동영상은 추후 제공예정
부족한 부분은 nanhap@gmail.com 으로 문의메일 주시면 성심성의껏 답변 드리겠습니다. 감사합니다.
Meetup tools for-cloud_native_apps_meetup20180510-vsminseok kim
마이크로서비스로 시스템을 구성하면 서비스간에 연관관계가 줄어들면서 서비스 릴리즈 속도가 높아지고 유연하게 대처할 수 있지만, 관리포인트가 늘어나게 되어 운영상에 많은 어려움을 마주치게 됩니다. 배포 될 때마다 생성되고 소멸되는 마이크로서비스를 다른 마이크로서비스가 쉽게 참조하게 하고 마이크로서비스들의 설정 정보를 일관되게 관리하는 일은 쉬운일이 아닙니다. 이러한 문제를 해결하기 위해 Spring Cloud 프로젝트와 같은 도구를 비롯하여 Pivotal Cloud Foundry와 같은 클라우드 플랫폼등이 있습니다. 이번 밋업에서는 마이크로서비스를 운영할 때의 어려운점과 도움을 주는 다양한 도구들에 대해 알아보도록 하겠습니다.
AWS에서는 다양한 언어에 대한 기계 번역(Translate) 등 AI 기능에 대한 API 서비스를 제공합니다. 본 실습에서는 이들 서비스(Serverless) 환경으로 AWS Amplify를 활용하여 소셜 모바일 앱을 안드로이드 기반으로 만들어 봅니다. 이를 위해 사용자 인증(Cognito), Graphql(Appsync) 등의 기능을 함께 활용합니다. 만들어진 앱은 AWS Device Farm을 통해서 클라우드 상에서 테스트 할 수 있습니다. 추가적으로Amazon Pinpoint를 이용하여 사용자 이벤트를 수집하고 분석하는 기능을 활용합니다.
Similar to 2023 GDG Sondo DevFest - Flutter/ Flavor, PlatformChannel, Environment variable .pptx (20)
35. 왜 Channel 인가?
수많은 네이티브 기능들..
알림, 딥링크, 센서, 카메라, 배터리, 위치, 사운드, 커넥션, 다
른앱과 정보공유, 다른앱 실행,로컬 스토리지 등등…
플랫폼이 시간이 지날수록 추가되는
각종 신규 기능들을 어떻게 다 빠르게
지원할 수 있을까?
48. Pigeon
Pigeon is a code generator tool to make communication between Flutter and the host platform type-
safe, easier, and faster.
Pigeon removes the necessity to manage strings across multiple platforms and languages. It also
improves efficiency over common method channel patterns. Most importantly though, it removes the
need to write custom platform channel code, since pigeon generates it for you.
Supported Platforms
Currently pigeon supports generating:
● Kotlin and Java code for Android
● Swift and Objective-C code for
iOS and macOS
● C++ code for Windows
Flutter Seoul 의 박제창님 비둘
73. iOS In-House Distribution
- TestFlight
- Ad-Hoc
- Firebase App Distribution
- 웹을 이용한 IPA 배포 (feat.Apple Developer Enterprise Program)
74. App Developer Enterprise Development
- Have 100 or more employees.
- Be a legal entity. We do not accept DBAs, fictitious businesses, trade names, or
branches.
- Use the program only to create proprietary, in-house apps for internal use, and to
distribute these apps privately and securely to employees within the organization.
- Have systems in place to ensure only employees can download your internal-use
apps, and to protect membership credentials and assets.
- Participate in and pass Apple’s verification interview and continuous evaluation
process.
75. Unlisted App Distribution
- 2022년 부터 제공한 비공개 AppStore 배포
- 엔터프라이즈 계정이 아닌 일반 개발자 계정으로도 배포가 가능
- 배포방식은 일반 스토어 배포와 동일하게 AppStoreConnect 에 심사
를 맡기지만 추가로 Unlisted App Distribution 이라는 배포 방식을 하
겠다고 description 란에 적어줘야함
- 또 별도의 Unlisted App Distribution 신청서를 작성
