Downloaded 115 times
Uploaded byMuleSoft
Object Store V2 Workshop
The document outlines Mulesoft's Object Store V2, a managed service for sharing application states in multi-clustered environments while preventing state loss. It highlights key features such as unlimited key storage, fast access, and integration with the Anypoint platform, while also detailing improvements over Object Store V1. Confidentiality and caution are emphasized regarding forward-looking statements and the changing nature of the presented information.
More Related Content
![[Final] best practices for access management (mule soft meetups riyadh) - j...](https://cdn.slidesharecdn.com/ss_thumbnails/finalbestpracticesforaccessmanagementmulesoftmeetups-riyadh-july2021-210724163604-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Object Store V2 Workshop
Object Store V2 Workshop
- 1. Vivin Nath Senior ProductManager Object Store V2
- 2. All contents ©MuleSoft Inc. Safe harbor statement The information in this presentation is confidential and proprietary to MuleSoft and may not be disclosed without the permission of MuleSoft. This presentation is not subject to your license agreement or any other service or subscription agreement with MuleSoft. MuleSoft has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation and MuleSoft's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by MuleSoft at any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This document is for informational purposes and may not be incorporated into a contract. MuleSoft assumes no responsibility for errors or omissions in this document, except if such damages were caused by MuleSoft intentionally or grossly negligent. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
- 3. All contents ©MuleSoft Inc. Object Store V2 3 External Apps Object Store is a fully managed service that enables sharing of states in multi- clustered environments on CloudHub without any risk of state loss during disaster events or application restarts Object Store V2 On-premises Cloud
- 4. All contents ©MuleSoft Inc. Why Object Store? 4 • Share states across multi-clustered environments • Leverage fast caching layer for I/O intensive applications • Make applications run faster by removing bottleneck caused by state management • Enable distributed batch processing capabilities • Remove friction in development of applications through fully managed service integrated with Anypoint platform • Lowers Total Cost of Ownership
- 5. All contents ©MuleSoft Inc. Object Store Use Cases Mule components Batch Processes TokensIdempotency Filter Until Successful Resequencer Distributed Cache Aggregator
- 6. All contents ©MuleSoft Inc. Object Store Use Cases User defined components API Caching API Rate Limiting Distributed Synchronization
- 7. All contents ©MuleSoft Inc. Object Store V1 Object Store V2 What’s new with Object Store V2 7 • 1 MB value limit • Limited to 100K Keys/App • Key-Value pairs • No shared application store • HA unavailable for in-memory stores • Not ideal for distributed batch processes • No alerting and notification service • No rollbacks and compensations • Values size up to 10 MB • Unlimited Keys/App (requires subscription) • Key-lists (string, numbers, binary, counter) • Share states across applications • Full HA across multiple availability zones • Suitable for distributed batch processes • Fast access and horizontally scalable • Rest interface for external applications • Usage statistics • Alerting and notification
- 8. All contents ©MuleSoft Inc. Workshop Demo # 1– Resilient OSV2 8 Application: osv2emprecords Object Store API
- 9. All contents ©MuleSoft Inc. Workshop Demo # 2 (API Caching with Global Store) 9 API App1 : Leagues API CACHE OBJECT-STORE-1 Global Store API App 2: Leagues API CACHE OBJECT-STORE-2
- 10. All contents ©MuleSoft Inc. API caching policy Workshop Demo # 2 10
- 11. All contents ©MuleSoft Inc. Workshop Demo – Resilient OSV2 11 Application: osv2emprecords Object Store API
- 12. All contents ©MuleSoft Inc. Object Store V2 Highlights 12 Global Store Secure Unlimited keys/app Faster access and horizontally scalable Values up to 10 MB Key-List support Real time persistence Fully Integrated with Anypoint Platform
- 13. All contents ©MuleSoft Inc. Participating in OSV2 Beta 13 • Beta Participation opened in Jan 2017 • Contact your CSM/AE if you want to enroll into Beta • OS V2 available to use at end of Q2
- 14.
Editor's Notes
- #4 So, let’s start by understanding what Object Store in the context of Anypoint platform really is. Put simply, it is a service that allows you to share and save application states on the cloud reliably without any risk of loss of data. So, even in the event of a worker or application restart, the application states stored on the object store will be preserved. That’s the main purpose of object store.
- #5 App integration, Data integration, ETL, messaging Today, the demand for application and data connectivity within and outside the enterprise is growing rapidly. Many IT organizations have tried to address these rising connectivity demands by installing a disparate set of application integration solutions, ETL tools and messaging platforms. As many of these heterogeneous platforms weren’t originally designed for talking to each other, they often need to share application states and data to work seamlessly. With addition of object store as a service, Anypoint platform is making it very easy to share application states in multi-clustered environments. Another use case is to enable sharing of states across distributed batch applications, which could be leveraged for batch and ETL jobs. For example, batch jobs can store states and records on the object store and retrieve them as it processes each record. Developers can use object store as a fast caching layer for I/O intensive applications, API calls and DB reads. It can be used to remove bottlenecks caused by shared states so that you can make your applications run faster. It is ideal for critical applications with Disaster recovery requirements as Object Store V2 is highly available service with replication across multiple availability zones. Finally, object store V2 removes friction in development and deployment of applications by offering a fully managed service as part of the Anypoint platform; as a result, users no longer have to build, buy or setup custom or third party services. Share states in multi-clustered environments (used for mule reliability components, batch processing, API caching) Enable distributed batch processing capabilities Remove friction in development of applications through fully managed service integrated with Anypoint platform Synchronizing your code with locks provided by Mule enables you to synchronize access to shared resources in a clustered environment Enables distributed batch processing capabilities (serves as our foray into advanced ETL use cases) Enables distributed locks to synchronize access to shared resources in clustered environment (API rate limiting)
- #6 Object store enables many use cases. Firstly, it finds use inside mule components. Mule Reliability components: Inside Anypoint platform, Mule provides several components for architecting highly reliable and distributed applications. Until-successful scope, idempotent filtering, resequencing are some of the most frequently used tools inside mule applications to enable reliability and resiliency. These internally use object store. With the launch of object store V2, now you can rest assured that all your internal states and data stored by mule components are much more resilient and reliable on the cloud. By default, on cloudhub in-memory storage is used by Mule components and as a result, if the worker running the mule application on the cloud restarts, the states stored in-memory would be lost. This would no longer be a problem when using Object Store V2 as it offers a persistence layer for storing of the mule states. In addition, to finding use inside mule components, object store can also be leveraged by external user defined components and applications.
- #7 In addition, to finding use inside mule components, object store can also be leveraged by external user defined components and applications. API first approach -> many devices -> single API -> rate limiting The API-first approach has gained momentum. We now live in a multi-screen world where businesses do not have the option of building a single application for a single device. Typically, the efficient approach is for companies to build API layers that can serve all their applications so that the same application can serve multiple devices such as desktop, tablets and mobiles. However, this API-first approach needs to be tempered with some policies around limiting the number of API calls that can be made in a given time window. API rate limiting: In a situation where the enterprise is running its own datacenter, such rate limits can most easily be applied at the web server layer, but on CloudHub, since the customers running applications in cloud do not have access to that infrastructure, the rate limiting policies will need to be applied at the application level. In such cases where the rate limits apply across applications, applications will need to communicate among themselves to ensure the limits are not exceeded. This requires a sophisticated object store that can be accessed by different applications. And, Object Store V2 is ideally suited to enable such use cases. API Caching: A high performance object store can function as a high speed cache, which minimizes calls to external public cloud services that are sometimes used for API caching. Use of Object Store V2 integrated within CloudHub will enable faster response times and will lower the total cost of ownership for such applications. Distributed memory cache: Lastly, you can use OS for distributed memory cache use cases. Public cloud infrastructures are ideal for massive application scale out to support high availability, but scaling out reliably requires some form of distributed memory cache. This is especially true for for distributed applications requiring collaboration and synchronization. Here is where the Object store V2 can serve as a caching layer to enable distributed memory caching and synchronization. In addition to all these, Object store V2 could also be used to build applications that require distributed batch processing capabilities such as some ETL, reporting and big data processing use cases.
- #8 Currently in cloudhub, when you use mule components such as idempotency filter components in your application and deploy them on the cloud, they use the in-memory storage option by default. This in-memory version is not truly distributed or HA, so if you restart your application or your worker on the cloud restarts for some reason, these in-memory entries will be lost. The OS V2 overcomes this limitation by storing the states into a distributed persistent storage layer, thus ensuring that you can truly leverage powerful capabilities of mule in a highly reliable and resilient manner. Furthermore, by default on the cloud in-memory storage is used by Mule components. And since volume of memory is constrained on the machine hosting the process, the value size is limited to 32 KB. Now, 32 KB is limiting for many use cases such as batch applications. With OSV2, application can store and retrieve states that are of sizes of up to 10MB. The current object store support key-value pairs, with OSV2 we are offering key-list support allowing applications to store multiple values for each key entry. Furthermore, OSV2 will support for strings, numbers, binary, counters, and locks. Locks are not supported in the beta release but will be released subsequently. Key-list also helps your applications scale better (for e.g. if you are using the aggregator component in mule, you can keep appending values instead of retrieving and storing values for every value that you need to aggregate on) OSV2 is well suited for applications with disaster recovery and high availability requirements OSV2 also provides a rest interface for node.js, and external applications. CLI interface would be available in future. OSV2 also introduces the concept of Confirmable timeout which can be used to implement transactions. Confirmable timeout is the Time period within which an ACK signal should be received by the the object store service for the confirmable value to be committed to object store. For example, to process a transaction a client application can write a value along with an attribute that tells the object store service how long it should wait before the application confirms or revokes the value. After issuing these attributes for a value that needs to be confirmed, if the client application sends an ACK within the timeout period (TTL), then value would be associated with the key and stored in object store, otherwise if a negative acknowledgement is received or timeout expires, the value would not be confirmed. We also support shared application store, so the same object store could be shared by multiple applications(how???) In addition to all this, in future we will be rolling out usage statistics, alerting and notification based on the usage. We are also thinking about global application store and on-premises support later on for OSV2. Since it is built on a shared nothing architecture, you can scale out your applications to your needs. Lastly, the entire transaction from the transport to storage of the key-values are authenticated and secured to meet enterprise security requirements. Persistent (user object store) and In-memory (Mule components) Key-lists (for connector during GA ) Enable transactions
- #9 I’ll cover 2 simple use cases. In the first use case, we will use a simple application using a user defined object store to store employee records and retrieve them through http end point. In the second example, we will demonstrate a API caching use case to show the performance gains with the OSV2 and illustrate shared application store capabilities. Show Object Store View Key-Values and Delete Key capabilities; default TTL (14 days); Max TTL 30 days Talk about supported runtimes.
- #10 Now, let’s proceed to our next example demonstrating shared API caching use case where 2 applications share the same API cache. We have created 2 different applications receiving and processing requests about Victor’s favorite La Liga teams. We have deployed a custom global API caching policy that uses the Object Store V2 to store and retrieve entries for both these applications. Now, I’m going to place a request to get the team info for a city say Madrid. As you can see, the team info is retrieved. This API response is now stored in the shared cache in OSV2. Now, I’m going to place the same request but on a different application. When I send the same request again, you will notice that the response was retrieved from the cache even though it was the first time this API request was sent to the second App. Now, let’s try doing the same but placing the request first to second app.
- #11 Now, I’m going to place a request to get the team info for a city say Madrid. As you can see, the team info is retrieved. This API response is now stored in the shared cache in OSV2. Now, I’m going to place the same request but on a different application. When I send the same request again, you will notice that the response was retrieved from the cache even though it was the first time this API request was sent to the second App. Now, let’s try doing the same but placing the request first to second app.
- #13 Fully integrated with Anypoint Platform, Compatible with old OS, so you do not have to change your application, just redeploy them. Persistent and in-memory options Real time persistence (snapshots persisted in real time) Support for key-lists (strings, numbers, binary, counters, locks ) Values up to 10 MB Flexible TTL DR across multiple zones Scalable to address your needs Full HA across zones offers resilient solution End to end secure transport (https) and storage (encrypted with keys) Rest and CLI interface for node.js, and other external applications Same API as the old object store Global stores (you can use it today but the UI for each application will show same object store) Confirmable timeout to enable rollbacks