AtlasCamp 2015: Get your add-on in shape for Data Center

•

0 likes•752 views

Michael Heemskerk The launch of JIRA, Confluence and Stash Data Center opens up both new opportunities and new challenges for add-on developers. Data Center installations consist of multiple clustered nodes that serve thousands of users. Data Center add-ons need to cope with distributed state, high load, large numbers of users, issues, pages or repositories. Join Michael Heemskerk, Stash architect, to learn all about the challenges, best practices and frameworks that are available for add-on developers that want to make their add-on ready for Data Center.

Technology

Get your add-on in shape for
Data Center
MICHAEL HEEMSKERK • ARCHITECT • ATLASSIAN • @MICHAELHEEMSKRK

Data Center
Database
Shared home
Local home
Local home
Local home

Caching
• Avoid it if you can
• Request-level caching
• atlassian-cache

ATLASSIAN-CACHE
API
• CacheFactory
• Cache<K, V>
• CachedReference<V>
• CacheLoader<K, V>
• CacheSettings
• CacheListener

ATLASSIAN-CACHE
cache
Node 1
cache
Node 2
Local caches

ATLASSIAN-CACHE
cache
Node 1
cache
Node 2
Remote caches
Replication

ATLASSIAN-CACHE
cache
Node 1
cache
Node 2
Hybrid caches
Invalidation

Synchronisation
• Avoid it if you can
• Optimistic locking in the database
• atlassian-beehive: cluster-aware lock service

Event handling
• Events are node-local only
• Options for node-node coordination
• atlassian-cache (+ listener)
• product-speciﬁc APIs

Scheduling
Node 1
Node 2
JobRunner
JobRunner
SchedulerService
JobRunner
JobConﬁg
scheduled jobs
JobRunner
JobConﬁg
SchedulerService
JobRunner
JobRunner
scheduled jobs
replicated *

ClusterMessagingService
ClusterEvent
TopicService
BucketedExecutor

Documentation
• Data Center add-on development: http://bit.ly/1IguwLq
• Atlassian Cache 2 Overview: http://bit.ly/1Il8WZg
• Stash - Making cluster-safe plugins: http://bit.ly/1G9aeE6

Thank you!
MICHAEL HEEMSKERK • ARCHITECT • ATLASSIAN • @MICHAELHEEMSKRK

Get your add-on in shape for Data Center
Submit your feedback:
go.atlassian.com/acaddondc

What's hot

Kafka On YARN (KOYA): An Open Source Initiative to integrate Kafka & YARN

DataWorks Summit

Oracle SOA suite and Coherence dehydration

Michel Schildmeijer

Data science lifecycle with Apache Zeppelin

DataWorks Summit/Hadoop Summit

CloudStack currently provides a variety bespoke high availability mechanisms for resources such as virtual machines, hosts, and virtual routers. Each of these implementations duplicates the HA check/recovery cycle, as well as, concurrency, persistence, and clustering required manage high available for any CloudStack resource. The High Availability Resource Management Service has been developed to consolidate these concerns -- providing a robust, extensible HA mechanism. Using this service, plugins only need to define health check, activity check, and fence operations.

When the Cloud is a Rockin: High Availability in Apache CloudStack

John Burwell

Apache Zeppelin is interactive data analytics environment for large scale data processing systems. It deeply integrates to Apache spark and many other frameworks, provides beautiful interactive web-based interface, data visualization, collaborative work environment and many other nice features to make your data science lifecycle more fun and enjoyable. Helium is a framework that manages pluggable components like Visualization, Spell inside of Zeppelin. Pluggable component extends Zeppelin's capability and particularly useful when Zeppelin is being used as a collaborative data science environment. Moon will demonstrate create custom visualization, publish to Helium online registry and use them in the notebook. Also talk about how Helium framework and Helium online registry works behind the scene and future roadmap as well. You'll see not only how easy creating and publishing Helium package is but also what possibility these pluggable modules gives to Zeppelin as a data science tool and business intelligence tool.

Helium makes Zeppelin fly!

DataWorks Summit

Organizations like Starbucks, HPE, and PayPal (see our customers) have selected the Akka toolkit for their enterprise scale distributed applications; and when it comes to squeezing out the best possible performance, the secret is using two particular modules in tandem: Akka Cluster and Akka Streams. In this webinar by Nolan Grace, Senior Solution Architect at Lightbend, we look at these two Akka modules and discuss the features that will push your application architecture to the next tier of performance. For the full blog post, including the video, visit: https://www.lightbend.com/blog/akka-at-enterprise-scale-performance-tuning-distributed-applications

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Lightbend

Twelve Factor apps are built for agility and rapid deployment. They enable continuous delivery and reduce the time and cost for new developers to join a project. At the same time, they are architected to exploit the principles of modern cloud platforms while permitting maximum portability between them. Finally, they can scale up without significant changes to tooling, architecture or development practices. In this talk, you’ll learn the principles and best practices espoused by the Twelve Factor app. We’ll discuss how to structure your code, manage dependencies, store configuration, run admin tasks, capture log files, and more. You’ll learn how modern Java deployments can benefit

12 Factor App: Best Practices for JVM Deployment

Joe Kutner

As enterprises move to cloud-based analytics, the risk of cloud security breaches poses a serious threat. Encrypting data at rest and in transit is a major first step. However, data must still be decrypted in memory for processing, exposing it to an attacker who has compromised the operating system or hypervisor. Trusted hardware such as Intel SGX has recently become available in latest-generation processors. Such hardware enables arbitrary computation on encrypted data while shielding it from a malicious OS or hypervisor. However, it still suffers from a significant side channel: access pattern leakage. We present Opaque, a package for Apache Spark SQL that enables very strong security for SQL queries: data encryption, computation verification, and access pattern leakage protection (a.k.a. obliviousness). Opaque achieves these guarantees by introducing new oblivious distributed relational operators that provide 2000x performance gain over state of the art oblivious systems, as well as novel query planning techniques for these operators implemented using Catalyst.

Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...

Spark Summit

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

William Hill is one of the UK’s largest, most well-established gaming companies with a global presence across 9 countries with over 16,000 employees. In recent years the gaming industry and in particular sports betting, has been revolutionised by technology. Customers now demand a wide range of events and markets to bet on both pre-game and in-play 24/7. This has driven out a business need to process more data, provide more updates and offer more markets and prices in real time. At William Hill, we have invested in a completely new trading platform using Apache Kafka. We process vast quantities of data from a variety of feeds, this data is fed through a variety of odds compilation models, before being piped out to UI apps for use by our trading teams to provide events, markets and pricing data out to various end points across the whole of William Hill. We deal with thousands of sporting events, each with sometimes hundreds of betting markets, each market receiving hundreds of updates. This scales up to vast numbers of messages flowing through our system. We have to process, transform and route that data in real time. Using Apache Kafka, we have built a high throughput, low latency pipeline, based on Cloud hosted Microservices. When we started, we were on a steep learning curve with Kafka, Microservices and associated technologies. This led to fast learnings and fast failings. In this session, we will tell the story of what we built, what went well, what didn’t go so well and what we learnt. This is a story of how a team of developers learnt (and are still learning) how to use Kafka. We hope that you will be able to take away lessons and learnings of how to build a data processing pipeline with Apache Kafka.

Building High-Throughput, Low-Latency Pipelines in Kafka

confluent

There are many tools and frameworks for monitoring. Usually when you think of an Open Source solution, you don’t think to implement it in a COTS product. Nevertheless, this session will tell you how you can implement tools such as Prometheus, Grafana and ELK into such an Enterprise application platform. Monitoring performance, throughput and error rate is important to be in control of your transactions. If you use a Service Bus or SOA/BPM suite product there are a lot out of the box diagnostics waiting for you. The puzzle here is how to get it out in a useful way. Besides of the many commercial solutions also Open Source tools can help you out with it. You can export runtime diagnostics out of the Diagnostics framework, monitor your SOA Composites and trace down Service Bus statistics using Prometheus and Grafana. The session will elaborate how to set up a proper monitoring using these tools, also in a proactive way where automated monitoring is a must for every application environment.

OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform

NETWAYS

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Mac...

Helena Edelson

Spark Summit EU talk by William Benton

Spark Summit

The business systems of an organization are a continuous source of events. Each system also needs to know about events happening in the other systems. Exchanging these events through direct API calls creates a web of inter-dependencies, is fragile and fails to scale. We examine how this problem can be solved through the use of right integration patterns implemented as a light-weight event hub that leverages the power of Kafka and Confluent to operate at enterprise scale. We demonstrate how JavaScript with its event-driven programming model can be a good fit for implementing an event hub that ensures guaranteed message delivery in the face of failures within the individual subscriber systems. Many organizations having large engineering teams skilled in NodeJS and a multitude of NodeJs applications. We show how these teams can easily leverage the power of Kafka and scale their applications with the right architectural building blocks. We also offer insights from our own experience of building NodeJS based Kafka applications.

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

HostedbyConfluent

Agenda: • Brief overview of Spark provided spark-shell, spark-submit • Overview of Spark ContextOverview of Zeppelin and Jupyter notebooks for Spark • Introduction to IBM Spark Kernel • Introduction to Cloudera Livy and Spark JobServer Github Link: Previous meetups:- 1) Introduction to Resilient Distributed Dataset and deep dive Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-and-resilient-distributed-dataset-basics-and-deep-dive Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/225159947/ Video: https://www.youtube.com/watch?v=MkeRWyF1y_0 Github: https://github.com/SatyaNarayan1/spark_meetup 2) Introduction to Spark DataFrames/SQL and Deep dive Slides: http://www.slideshare.net/sachinparmarss/deep-dive-spark-data-frames-sql-and-catalyst-optimizer Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/226419828/ Video: https://www.youtube.com/watch?v=h71MNWRv99M Github: https://github.com/parmarsachin/spark-dataframe-demo 3) Apache Spark - Introduction to Spark Streaming and Deep dive Slides: http://www.slideshare.net/differentsachin/apache-spark-introduction-to-spark-streaming-and-deep-dive-57671774 Meetup: http://www.meetup.com/Big-Data-Developers-in-Bangalore/events/227008581/ Video: Github: https://github.com/agsachin/spark-meetup Looking forward to have a great interactive session. Do provide feedback.

Interactive Analytics using Apache Spark

Sachin Aggarwal

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

PivotalOpenSourceHub

Toon is a leading brand in the European smart energy market, currently expanding internationally, providing energy usage insights, eco-friendly energy management and smart thermostat use for the connected home. As value added services become ever more relevant in this market, we have the need to ensure that we can easily and safely on-board new tenants into our data platform. In this talk we’re going to guide you across a less discussed side of using Spark in production – devops. We will speak about our journey from an on-premise cluster to a managed solution in the cloud. A lot of moving parts were involved: ETL flows, data sharing with 3rd parties and data migration to the new environment. Add to this the need to have a multi-tenant environment, revamp our toolset and deploy a live public facing service. It’s possible to find a lot of great examples of how Spark is used for data-science purposes. On the data engineering side, we need to deploy production services, ensure data is cleaned, secured and available, and keep the data-science teams happy. We’d like to share some of the options we took and some of the lessons learned from this (ongoing) transition.

Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira

Spark Summit

Today, there are several compliance use cases ‒ archiving, e-discovery, supervision and surveillance, to name a few ‒ that appear naturally suited as Hadoop workloads, but haven’t seen wide adoption. In this session, you’ll learn about common limitations, how Apache Spark helps and some new blueprints for modernizing this architecture and disrupt existing solutions. Additionally, we’ll review the rising role of Apache Spark in this ecosystem, leveraging machine learning and advanced analytics in a space that has traditionally been restricted to fairly rote reporting.

Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan Volz

Databricks

CEP Integration for Apache Stratos 4.0.0

Manula Thantriwatte

Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop

Ayon Sinha

What's hot (20)

Kafka On YARN (KOYA): An Open Source Initiative to integrate Kafka & YARN

Oracle SOA suite and Coherence dehydration

Data science lifecycle with Apache Zeppelin

When the Cloud is a Rockin: High Availability in Apache CloudStack

Helium makes Zeppelin fly!

Akka at Enterprise Scale: Performance Tuning Distributed Applications

12 Factor App: Best Practices for JVM Deployment

Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

Building High-Throughput, Low-Latency Pipelines in Kafka

OSMC 2021 | Use OpenSource monitoring for an Enterprise Grade Platform

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Mac...

Spark Summit EU talk by William Benton

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

Interactive Analytics using Apache Spark

#GeodeSummit: Easy Ways to Become a Contributor to Apache Geode

Using Apache Spark in the Cloud—A Devops Perspective with Telmo Oliveira

Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan Volz

CEP Integration for Apache Stratos 4.0.0

Eventual Consistency @WalmartLabs with Kafka, Avro, SolrCloud and Hadoop

Similar to AtlasCamp 2015: Get your add-on in shape for Data Center

Hazelcast 101

Emrah Kocaman

BigData Developers MeetUp

Christian Johannsen

Caching 101: Caching on the JVM (and beyond)

Louis Jacomet

You definitely have heard about the SMACK architecture, which stands for Spark, Mesos, Akka, Cassandra, and Kafka. It’s especially suitable for building a lambda architecture system. But what is SDACK? Apparently it’s very much similar to SMACK except the “D" stands for Docker. While SMACK is an enterprise scale, multi-tanent supported solution, the SDACK architecture is particularly suitable for building a data product. In this talk, I’ll talk about the advantages of the SDACK architecture, and how TrendMicro uses the SDACK architecture to build an anomaly detection data product. The talk will cover: 1) The architecture we designed based on SDACK to support both batch and streaming workload. 2) The data pipeline built based on Akka Stream which is flexible, scalable, and able to do self-healing. 3) The Cassandra data model designed to support time series data writes and reads.

Using the SDACK Architecture to Build a Big Data Product

Evans Ye

JIRA and Confluence are introducing new product editions with support for high availability and clustering. Depending on your add-on, supporting enterprise-grade deployments of JIRA and Confluence could require updates to your plugin. Get a sneak preview of the application architecture for HA and clustering, and learn how you can ensure your plugin is ready to run in these mission critical installations.

AtlasCamp 2014: Preparing Your Plugin for JIRA Data Center

AtlasCamp 2015: Get your add-on in shape for Data Center

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AtlasCamp 2015: Get your add-on in shape for Data Center

Similar to AtlasCamp 2015: Get your add-on in shape for Data Center (20)

More from Atlassian

More from Atlassian (20)

Recently uploaded

Recently uploaded (20)

AtlasCamp 2015: Get your add-on in shape for Data Center