データの民主化のために StackStorm を活用した事例

•

4 likes•3,808 views

1. StackStorm is an open source workflow automation tool that allows users to automate tasks and business processes. 2. It provides actions and workflows that can be triggered via a web UI, API, or rules to interact with various systems like Jenkins, Kubernetes, and AWS. 3. StackStorm also offers high availability, scheduling, and APIs for automation needs.

Technology

StackStorm
/ @laclefyoshi / ysaeki@r.recruit.co.jp

: / @laclefyoshi
• 2011/04
• 2015/09
•
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
• Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017)
• Apache Kafka  
(Apache Kafka Meetup Japan #4; 2018)
3

StackStorm
• https://stackstorm.com/
• OSS  
+ LDAP Auth plugin https://git.io/vxbbJ
•
• Shell Script
• Crontab
5

• JP1:
• Digdag:
• Jenkins: CI
• Airﬂow: GCP Cloud Composer
• Oozie: Hadoop
6

StackStorm
•
• High Availability
• Crontab Scheduler
•
•
•
• Runner Action Workﬂow
•
• REST API
7

$StackStorm API 1. test.test Action       2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test",   “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8$

StackStorm
• Design UI Commercial
• GUI
• OSS
• Commercial Role Based Access Control
• OSS
• Action Workﬂow
• Jinja/YAQL
9

st2XXX, mistral-XXX
replication
cluster
mistral
(PostgreSQL 9.6)
Runner Action
Workﬂow
Rule
10

•
• 1
• Timer
• st2rulesengine Timer HA
• https://docs.stackstorm.com/reference/ha.html#st2rulesengine
• MongoDB Primary
st2rulesengine Timer
• Concurrency policy Zookeeper
Timer
11

GCP?
• BigQuery  
• BigQuery
• SQL
• Tableau
• Pandas R
• StackStorm
BigQuery
12

StackStorm
1. StackStorm
2.  
3. StackStorm
4. StackStorm
5.
6.
17

StackStorm
• Amazon Data Value Chain  
Subscribe Deliver
https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how-
amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017
18

Action
• Action
•
•
•
• LDAP
• Stackdriver
• Action
• SQL BigQuery
• BigQuery CSV
•
19

• StackStorm Action
• Action Workﬂow  
• YAML
•  
• YAML  
• JP1 GUI
20

API
• YAML
• GUI API
• Action Workﬂow
• StackStorm Action Workﬂow API
• API
•
22

1/2
23
2. StackStorm Web UI  
StackStorm API
1. Action Workﬂow
API

2/2
24
JP1
2. JP1 1
StackStorm API
1. Action Workﬂow
API

• StackStorm
• API
• StackStorm
•
• Rule
• JP1
• Workﬂow
•
25

26
• StackStorm
• Action Workﬂow YAML
•
•
Action Workﬂow

My talk at Strata Data Conference, London, May 2017. https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57619 Abstract: Modern businesses have data at their core, but this data is changing continuously. How can you harness this torrent of information in real time? The answer: stream processing. The core platform for streaming data is Apache Kafka, and thousands of companies are using Kafka to transform and reshape their industries, including Netflix, Uber, PayPal, Airbnb, Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: to succeed, many technologies need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we engineers would like to work and how we actually end up working in practice. Michael Noll explains how Apache Kafka helps you radically simplify your data processing architectures by building normal applications to serve your real-time processing needs rather than building clusters or similar special-purpose infrastructure—while still benefiting from properties typically associated exclusively with cluster technologies, like high scalability, distributed computing, and fault tolerance. Michael also covers Kafka’s Streams API, its abstractions for streams and tables, and its recently introduced interactive queries functionality. Along the way, Michael shares common use cases that demonstrate that stream processing in practice often requires database-like functionality and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (for example, in the form of event-driven, containerized microservices). As you’ll see, Kafka makes such architectures equally viable for small-, medium-, and large-scale use cases.

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

HostedbyConfluent

Interacting with Apache Kafka seems straightforward at first, you “just” push and pull messages. Yet it can quickly become a source of frustration as the user encounters timeouts, vague error descriptions and disappearing messages. Experience helps a lot and I’m here to share what I know. In this talk you will learn the tips & tricks I wish I had known at the beginning of my Apache Kafka journey. We’ll discuss topics like producer acknowledgments, server and consumer parameters (auto_offset_reset anyone?) that are commonly overlooked causing lots of developer’s pain. I’ll share with you how to generate code that works as expected on the first run, making your first integration painless. These tips will kickstart your Apache Kafka experience in Python and save you hours of debugging.

大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会Takayuki Kyowa

Real time stock processing with apache nifi, apache flink and apache kafka

Timothy Spann

Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...

confluent

The field of astronomy is rapidly changing away from the traditional notion of a lone astronomer pointing a telescope at a single object in a static sky. Initiatives such as the Sloan Digital Sky Survey have ushered in a collaborative big data era of wide-field sky surveys, in which telescopes collect observations continuously while sweeping across the visible night sky. This method of data collection enables not only very deep imaging of far and faint objects but is also optimal for searching for objects that might be changing or moving. By analyzing the differences in astronomical image data from one night to the next, astronomers can detect "transient" objects, such as variable stars, supernova, and near Earth asteroids. New sky surveys provide a wealth of scientific value for astronomers but not without technical challenges. Survey data need to be automatically processed and the results immediately distributed to the scientific community in order to enable rapid follow-up observations as transient astronomy can be highly time sensitive. Detection alert data distribution mechanisms need to be robust and reliable to maintain scientific integrity without data loss. Additionally, alerting systems need to be scalable to support a data volume unprecedented in astronomy, as transient detection rates have increased to exceed all historical data in a single night. A streaming architecture is an ideal architecture for automated distribution and processing of transient data in real time as it is being collected. In this talk, we will discuss how Kafka and Avro are being used in wide-field astronomical sky survey pipelines to serialize and distribute transient data, the design choices behind this system, and how this alert stream system has been successfully deployed in production to distribute transient detection alerts to the scientific research community in excess of 1 million events per night.

Apache Kafka lessons learned @PAYBACK

Maxim Shelest

Apache Pulsar Community-Jennifer

StreamNative

Spark Compute as a Service at Paypal with Prabhu Kasinathan

Databricks

Apache Spark is a gift to the big data community, which adds tons of new features on every release. However, it’s difficult to manage petabyte-scale Hadoop clusters with hundreds of edge nodes, multiple Spark releases and demonstrate operational efficiencies and standardization. In order to address these challenges, Paypal has developed and deployed a REST0based Spark platform: Spark Compute as a Service (SCaaS),which provides improved application development, execution, logging, security, workload management and tuning. This session will walk through the top challenges faced by PayPal administrators, developers and operations and describe how Paypal’s SCaaS platform overcomes them by leveraging open source tools and technologies, like Livy, Jupyter, SparkMagic, Zeppelin, SQL Tools, Kafka and Elastic. You’ll also hear about the improvements PayPal has added, which enable it to run greater than 10,000 Spark applications in production effectively.

Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server

BizTalk360

ELK-Stack is world’s most popular log management platform. These open-source products are most commonly used in log analysis in IT environments. Logstash collects and parses logs, Elasticsearch indexes and stores the information. Kibana then presents the data in visualizations that provide actionable insights into one’s environment/software. Ashwin is going to brief about ELK-stack and show how this popular log management platform can be used with BizTalk servers. Including installing ELK stack in Windows and demo on how BizTalk data can be logged and analyzed in ELK-Stack. And he is going to discuss about some of the uses cases you can use ELK-stack with BizTalk and Azure.

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Michael Noll

Log analysis using Logstash,ElasticSearch and Kibana

Avinash Ramineni

E2E Data Pipeline - Apache Spark/Airflow/Livy

Rikin Tanna

Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...

HostedbyConfluent

Since Pac-Man was originally released in the '80s, it has been a beacon of fun and joy for people of all ages. What few people know is that this game can also be used to inspire developers on how to build event streaming applications. In this near-zero-slides talk, attendees will get to play the game to generate events. As they play, the presenter will write from scratch a scoreboard using ksqlDB -- an open-source event streaming database built for Apache Kafka. After building the scoreboard, it will be discussed the different strategies to make the data available elsewhere so any interested service could leverage it with ease. Examples of these services will be provided to monitor in near real-time the scoreboard, revealing whoever is the most proficient Pac-Man player in the room.

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps 5. Playing with Kafka using Wirbelsturm Audience: developers, operations, architects Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/ Verisign is a global leader in domain names and internet security. Tools mentioned: - Wirbelsturm (https://github.com/miguno/wirbelsturm) - kafka-storm-starter (https://github.com/miguno/kafka-storm-starter) Blog post at: http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/ Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

confluent

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Matt Stubbs

Speaker: Cedrick Lunven, Developer Advocate, DataStax Speaker Bio: Cedrick is a Developer Advocate at DataStax where he finds opportunities to share his passions by speaking about developing distributed architectures and implementing reference applications for developers. In 2013, he created FF4j, an open source framework for Feature Toggle which he still actively maintains. He is now contributor in JHipster team. Talk Synopsis: We have all introduced more or less functional programming and asynchronous operations into our applications in order to speed up and distribute treatments (e.g., multi-threading, future, completableFuture, etc.). To build truly non-blocking components, optimize resource usage, and avoid "callback hell" you have to think reactive—everything is an event. From the frontend UI to database communications, it’s now possible to develop Java applications as fully reactive with frameworks like Spring WebFlux and Reactor. With high throughput and tunable consistency, applications built on top of Apache Cassandra™ fit perfectly within this pattern. DataStax has been developing Apache Cassandra drivers for years, and in the latest version of the enterprise driver we introduced reactive programming. During this session we will migrate, step by step, a vanilla CRUD Java service (SpringBoot / SpringMVC) into reactive with both code review and live coding. Bring home a working project! Filmed at Skills Matter/Code Node London on 9th May 2019 as part of the Big Data LDN Meetup Blueprint Series. Meetup sponsored by DataStax.

Apache Pulsar: A borderless community

StreamNative

Scala at foursquare

jorgeortiz85

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...

gethue

Livy is a new open source Spark REST Server for submitting and interacting with your Spark jobs from anywhere. Livy is conceptually based on the incredibly popular IPython/Jupyter, but implemented to better integrate into the Hadoop ecosystem with multi users. Spark can now be offered as a service to anyone in a simple way: Spark shells in Python or Scala can be ran by Livy in the cluster while the end user is manipulating them at his own convenience through a REST api. Regular non-interactive applications can also be submitted. The output of the jobs can be introspected and returned in a tabular format, which makes it visualizable in charts. Livy can point to a unique Spark cluster and create several contexts by users. With YARN impersonation, jobs will be executed with the actual permissions of the users submitting them. Livy also enables the development of Spark Notebook applications. Those are ideal for quickly doing interactive Spark visualizations and collaboration from a Web browser! This talk is technical and details the architecture and design decisions taken for developing this server, as well as its internals. It also describes the alternatives we tried and the challenges that were faced. The capabilities of Livy will then be lived demo in Hue’s Notebook Application through a real life scenario. https://spark-summit.org/eu-2015/events/building-a-rest-job-server-for-interactive-spark-as-a-service/

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

confluent

Robin is a Developer Advocate at Confluent, the company founded by the creators of Apache Kafka, as well as an Oracle Groundbreaker Ambassador. His career has always involved data, from the old worlds of COBOL and DB2, through the worlds of Oracle and Hadoop, and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Databricks

Since mid-2016, Spark-as-a-Service has been available to researchers in Sweden from the Rise SICS ICE Data Center at www.hops.site. In this session, Dowling will discuss the challenges in building multi-tenant Spark structured streaming applications on YARN that are metered and easy-to-debug. The platform, called Hopsworks, is in an entirely UI-driven environment built with only open-source software. Learn how they use the ELK stack (Elasticsearch, Logstash and Kibana) for logging and debugging running Spark streaming applications; how they use Grafana and InfluxDB for monitoring Spark streaming applications; and, finally, how Apache Zeppelin can provide interactive visualizations and charts to end-users. This session will also show how Spark applications are run within a ‘project’ on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In addition, hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.afka topics are protected from access by users that are not members of the project. We will also discuss the experiences of our users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications.

What's hot

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

HostedbyConfluent

大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会Takayuki Kyowa

Real time stock processing with apache nifi, apache flink and apache kafka

Timothy Spann

Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...

confluent

Apache Kafka lessons learned @PAYBACK

Maxim Shelest

Apache Pulsar Community-Jennifer

StreamNative

Spark Compute as a Service at Paypal with Prabhu Kasinathan

Databricks

Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server

BizTalk360

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Michael Noll

Log analysis using Logstash,ElasticSearch and Kibana

Avinash Ramineni

E2E Data Pipeline - Apache Spark/Airflow/Livy

Rikin Tanna

Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...

HostedbyConfluent

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

confluent

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Matt Stubbs

Apache Pulsar: A borderless community

StreamNative

Scala at foursquare

jorgeortiz85

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...

gethue

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

confluent

What's hot (20)

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会

Real time stock processing with apache nifi, apache flink and apache kafka

Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...

Apache Kafka lessons learned @PAYBACK

Apache Pulsar Community-Jennifer

Spark Compute as a Service at Paypal with Prabhu Kasinathan

Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Log analysis using Logstash,ElasticSearch and Kibana

E2E Data Pipeline - Apache Spark/Airflow/Livy

Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...

Apache Kafka 0.8 basic training - Verisign

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Apache Pulsar: A borderless community

Scala at foursquare

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

Similar to データの民主化のために StackStorm を活用した事例

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Databricks

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

Chris Fregly

ストリーム処理を支えるキューイングシステムの選び方

Yoshiyasu SAEKI

Apereo CAS: State of the Project 2018

Misagh Moayyed

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Lightbend

In a recent survey, 90% of over 2400 developers reported having at least some real-time functionality in their systems. Enterprises are realizing that the ability to extract value from streaming data in near real-time is the new competitive advantage. Two technologies–Akka Streams and Kafka Streams–have emerged as popular tools to use with Apache Kafka for addressing the shared requirements of availability, scalability, and resilience for both streaming microservices and Fast Data. So which one should you use for specific use cases?

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

Chris Fregly

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

Jacek Laskowski

DataMass Summit 2019 Edition --> http://summit.datamass.io There is quite a bit to learn about any stream processing engine. But at a reasonably high level they actually are very similar and have lots in common. Not only do all have to offer a high-level stream processing API to describe distributed streaming dataflows, but also a low-level API for more sophisticated streaming topologies. The engines translate the dataflow description into their internal runtime representation. That’s where the differences are and where we’ll be looking at. This talk compares two modern stream processing engines — Kafka Streams and Spark Structured Streaming. We’ll be talking about their internals and how the engines manage stateless and stateful streams. You will learn about their similarities and differences that should shed more light on the question when to use which engine.

ElasticBeanstalk で新規事業を爆速ローンチする

Ryo Shibayama

Minami Aoyama Night #1 で発表したスライドです。ザッパラスで行っているリーンな開発におけるインフラ周りの話をしました。また、タイミングよく @i2key さんのブログポスト「新規事業のグロース期を支えるエンジニアリングについて」が投稿されましたが、その文脈においての一部が ElasticBeanstalk を使ったインフラストラクチャなのではないかと感じています。

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Sarah Guido

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Java-Jersey 到 Python-Flask 服務不中斷重構之旅

Max Lai

Perl ウェブ開発の中世〜CGI と Plack の間〜

鉄次尾形

Flink September 2015 Community Update

Robert Metzger

Jug - ecosystem

Florent Ramiere

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Helena Edelson

Spark Uber Development Kit

DataWorks Summit/Hadoop Summit

Queryable State for Kafka Streamsを使ってみた

Yoshiyasu SAEKI

Chti jug - 2018-06-26

Florent Ramiere

Apache spark

Sameer Mahajan

What's new with Apache Camel 3? | DevNation Tech Talk

Red Hat Developers

Apache Camel is a leading open source integration framework that has been around for more than a decade. With the release of Apache Camel 3, the Camel family has been extended to include a full range of projects that are tailored to popular platforms including Spring Boot, Quarkus, Kafka, Kubernetes, and others - creating an ecosystem. Join this webinar to learn what’s new in Camel 3 and about Camel projects: - Latest features in Camel 3 - Quick demos of Camel 3, Camel Quarkus, Camel K, and Camel Kafka Connector - Present insights into what's coming next Speakers: Andrea Cosentino, Claus Ibsen

Similar to データの民主化のために StackStorm を活用した事例 (20)

KafkaとAWS Kinesisの比較

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

ストリーム処理を支えるキューイングシステムの選び方

Apereo CAS: State of the Project 2018

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

ElasticBeanstalk で新規事業を爆速ローンチする

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Java-Jersey 到 Python-Flask 服務不中斷重構之旅

Perl ウェブ開発の中世〜CGI と Plack の間〜

Flink September 2015 Community Update

Jug - ecosystem

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Spark Uber Development Kit

Queryable State for Kafka Streamsを使ってみた

Chti jug - 2018-06-26

Apache spark

What's new with Apache Camel 3? | DevNation Tech Talk

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Microsoft - Power Platform_G.Aspiotis.pdf

Uni Systems S.M.S.A.

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Neo4j

Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

GridMate - End to end testing is a critical piece to ensure quality and avoid...

ThomasParaiso2

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 4

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Video Streaming: Then, Now, and in the Future

Microsoft - Power Platform_G.Aspiotis.pdf

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

PHP Frameworks: I want to break free (IPC Berlin 2024)

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

FIDO Alliance Osaka Seminar: Overview.pdf

GridMate - End to end testing is a critical piece to ensure quality and avoid...

Elizabeth Buie - Older adults: Are we really designing for our future selves?

20240607 QFM018 Elixir Reading List May 2024

Epistemic Interaction - tuning interfaces to provide information for AI support

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Climate Impact of Software Testing at Nordic Testing Days

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

データの民主化のために StackStorm を活用した事例

1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp

2. • StackStorm • • • 2

3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka   (Apache Kafka Meetup Japan #4; 2018) 3

4. StackStorm

5. StackStorm • https://stackstorm.com/ • OSS   + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5

6. • JP1: • Digdag: • Jenkins: CI • Airﬂow: GCP Cloud Composer • Oozie: Hadoop 6

7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workﬂow • • REST API 7

8. StackStorm API 1. test.test Action       2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test",   “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8

9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workﬂow • Jinja/YAQL 9

10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workﬂow Rule 10

11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11

12. GCP? • BigQuery   • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12

13.

14. • • DB • • • →   14

15. 1/2 • 15

16. 2/2 • StackStorm 16

17. StackStorm 1. StackStorm 2.   3. StackStorm 4. StackStorm 5. 6. 17

18. StackStorm • Amazon Data Value Chain   Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18

19. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19

20. • StackStorm Action • Action Workﬂow   • YAML •   • YAML   • JP1 GUI 20

21. 21

22. API • YAML • GUI API • Action Workﬂow • StackStorm Action Workﬂow API • API • 22

23. 1/2 23 2. StackStorm Web UI   StackStorm API 1. Action Workﬂow API

24. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workﬂow API

25. • StackStorm • API • StackStorm • • Rule • JP1 • Workﬂow • 25

26. 26 • StackStorm • Action Workﬂow YAML • • Action Workﬂow

データの民主化のために StackStorm を活用した事例

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to データの民主化のために StackStorm を活用した事例

Similar to データの民主化のために StackStorm を活用した事例 (20)

Recently uploaded

Recently uploaded (20)

データの民主化のために StackStorm を活用した事例