Don't call me cache - Dotan Horovits

1. Powering Your Application with XAP
(Using payment processing as an example)
“Don’t call me cache”
Dotan Horovits
Director, Customer Services
Senior solutions architect

2. About me
Dotan Horovits
Director, Customer Services
Senior system and solutions architect
technology evangelist with interest in many things
• distributed high-performance systems
• big data solutions
• cloud technologies
• And of course: agile software methodologies
• And much more…
dotan@gigaspaces.com
horovits.wordpress.com
@horovits

3. Short Intro to Caching Evolution
Cache
In process caching
of Key->Value data
structure
Distribute Cache
Partitioned cache
nodes
IMDG
Partitioned system
of record
In Memory
Application Platform
Collocated IMDG and
Processing
Cache
Cache is good for repetitive data reads
But it is limited in capacity
It also doesn’t handle write-heavy scenarios

4. Short Intro to Caching Evolution
Cache
In process caching
of Key->Value data
structure
Distribute Cache
Partitioned cache
nodes
IMDG
Partitioned system
of record
In Memory
Application Platform
Collocated IMDG and
Processing
Distribute Cache
Allows you to distribute your cache over numerous machines so you get
Increased Capacity
But it doesn’t support write heavy scenarios
It’s also Limited to query by Id
What about the rest of your app? - Business logic & messaging??

5. Short Intro to Caching Evolution
Cache
In process caching
of Key->Value data
structure
Distribute Cache
Partitioned cache
nodes
IMDG
Partitioned system
of record
In Memory
Application Platform
Collocated IMDG and
Processing
IMDG solves these problems!
You get increased capacity
IMDG is also a System of Record with:
Query APIs
Optimized data access
Data integrity
It solves your write scalability problem
.

6. Short Intro to Caching Evolution
Cache
In process caching
of Key->Value data
structure
Distribute Cache
Partitioned cache
nodes
IMDG
Partitioned system
of record
In Memory
Application Platform
Collocated IMDG and
Processing
In Memory Application Platform
XAP for end to end scaling
Its an IMDG that hosts your Business
logic & has messaging services!
It Provides Parallel processing of data
You get linear scalability
You get high availability
How does XAP work?

7. Here’s What a Payment Authorization Process Looks Like
Payment
Authorization
Request
Basic Validation User Profile Check
Merchant
Profile Check
Payment
Authorization
Approved

8. Write the Payment Object to XAP
Cash Register
Application Payment
User payment Cluster
@SpaceId()
public Long getId() {
return id;
}
@SpaceRouting
public Long getUserId() {
return userId;
}
}
@SpaceIndex
public Long getCardId() {
return cardId;
}
}
The primary key of this object in the grid
The grid will use this attribute to route
the object to a particular partition
Payment object
A secondary index for query optimization

9. Write the Payment Object to XAP
Cash Register
Application
User payment Cluster
Payment
@SpaceId
public Long getId() {
return id;
}
@SpaceRouting
public Long getUserId() {
return userId;
}
@SpaceIndex
public Long getCardId() {
return cardId;
}
The primary key of this object in the grid
The grid will use this attribute to route
the object to a particular partition
Payment object
A Secondary index for query optimization

10. Write the Payment Object to XAP
Cash Register
Application
User payment Cluster
Payment
Index
@SpaceId
public Long getId() {
return id;
}
@SpaceRouting
public Long getUserId() {
return userId;
}
@SpaceIndex
public Long getCardId() {
return cardId;
}
The primary key of this object in the grid
The grid will use this attribute to route
the object to a particular partition
Payment object
A Secondary index for query optimization

11. Let's Talk About Data Model for a Second

12. User Id (Routing)| User Name|…
Card Id| Card Data | UserID (Routing)…
Transaction ID | CardId | User ID (Routing)…
1  *
1  *
Data Model

13. Let's Get Back to the Process

14. Payment Validation
@EventTemplate
public SQLQuery<Payment> getNewPayment()
{
SQLQuery<Payment> query = new SQLQuery<Payment>(Payment.class," paymentStatus = ? ");
query.setParameter(1, Payment.PaymentStatus.New);
return query;
}
Queue
Payment
Validator

15. Payment Validation
@SpaceDataEvent
public Payment validatePayment(Payment payment)
{
…
}
private boolean basicPaymentValidation(Payment payment, User user, Card card)
{
…
}
Queue
Payment
Class PaymentAuthorization{
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
Validator
Payment Authorization

16. Basic Validation
if (basicPaymentValidation(payment,user,card)) {
gigaSpace.write(userPaymentMsg);
gigaSpace.write(vendorPaymentMsg);
gigaSpace.write(paymentAuthorization);
payment.setPaymentStatus(Payment.PaymentStatus.Processing);
} else {
payment.setPaymentStatus(Payment.PaymentStatus.SuspectedFraud);
}
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
User
Validation
Vendor
Validation
Payment
Authorization
Queue
Queue
Queue Validator

17. User Validation
Class PaymentAuthorization{
Enum Status
Boolean userCheck
Boolean vendorCheck
}
Read
Message
Process
Queue
UserCreditCardPayment
Querying
@SpaceDataEvent
public void validateUser(UserPaymentMsg event) {
…
space.readMultiple(new Card(userId))
space.readMultiple(new Payment(userId))
if (valid()) {
IdQuery<PaymentAutjorzation> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthoirzation.class, paymentId);
space.change(idQuery, new ChangeSet().set(“userCheck", true));
} else { ... }
}

18. User Validation
Read
Message
Process
Queue
UserCreditCardPayment
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
Querying
@SpaceDataEvent
public void validateUser(UserPaymentMsg event) {
…
space.readMultiple(new Card(userId))
space.readMultiple(new Payment(userId))
if (valid()) {
IdQuery<PaymentAutjorzation> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthoirzation.class, paymentId);
space.change(idQuery, new ChangeSet().set(“userCheck", true));
} else { ... }
}

19. User Validation
Read
Message
Process
Queue
UserCreditCardPayment
@SpaceDataEvent
public void validateUser(UserPaymentMsg event) {
…
space.readMultiple(new Card(userId))
space.readMultiple(new Payment(userId))
if (valid()) {
IdQuery<PaymentAutjorzation> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class, paymentId);
space.change(idQuery, new ChangeSet().set(“userCheck", true));
} else { ... }
}
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
Querying

20. Vendor Validation
@SpaceDataEvent
public void validateVendor(VendorPaymentMsg vendorPaymentMsg) {
AsyncFuture<Boolean> future = vendorGigaSpace.execute(
new VendorValidationTask(vendorPaymentMsg),
vendorPaymentMsg.getMerchant());
IdQuery<PaymentAuthorization> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class,
vendorPaymentMsg.getPaymentId());
if (future.get()) gigaSpace.change(idQuery, new ChangeSet().
set("vendorCheck", true));
else gigaSpace.change(idQuery, new ChangeSet().
set("vendorCheck”, false));
}
Read
Message
Process
Queue
Task
class PaymentAuthoirzation {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}

21. Vendor Validation
@SpaceDataEvent
public void processVendorPaymentValidation() {
AsyncFuture<Boolean> result= space.execute(
new DistributedTask(new VendorValidationTask(Payment)))
if (result.get()) {
IdQuery<PaymentAutjorzation> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class,
paymentId);
space.change(idQuery, new ChangeSet().set(“vendorCheck”, true));
} else { ... }
}
Payment Cluster
Task
Vendor Cluster
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}

22. Vendor Validation
public void validateVendor(VendorPaymentMsg vendorPaymentMsg,
GigaSpace gigaSpace) {
AsyncFuture<Boolean> future = vendorGigaSpace.execute(new
VendorValidationTask(vendorPaymentMsg),
vendorPaymentMsg.getMerchant());
IdQuery<PaymentAuthorization> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class,
vendorPaymentMsg.getPaymentId());
if (future.get()) gigaSpace.change(idQuery, new
ChangeSet().set("vendorCheck", true));
else gigaSpace.change(idQuery, new ChangeSet().set("vendorCheck", false));
}
Payment Cluster Vendor Cluster
Task
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}

23. Vendor Validation
Payment Cluster Vendor Cluster
Task
public void validateVendor(VendorPaymentMsg vendorPaymentMsg,
GigaSpace gigaSpace) {
AsyncFuture<Boolean> future = vendorGigaSpace.execute(new
VendorValidationTask(vendorPaymentMsg),
vendorPaymentMsg.getMerchant());
IdQuery<PaymentAuthorization> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class,
vendorPaymentMsg.getPaymentId());
if (future.get()) gigaSpace.change(idQuery, new
ChangeSet().set("vendorCheck", true));
else gigaSpace.change(idQuery, new ChangeSet().set("vendorCheck", false));
}
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}

24. Vendor Validation Check
@Override
public Boolean execute() throws Exception {
return validatePayment(vendorPaymentMsg);
}
Task
DealsMerchant
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
Querying

25. Vendor Validation Check
Task
DealsMerchant
@Override
public Boolean execute() throws Exception {
return validatePayment(vendorPaymentMsg);
}
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}
Querying

26. Vendor Validation
Payment Cluster Vendor Cluster
Callback Task
public void validateVendor(VendorPaymentMsg vendorPaymentMsg,
GigaSpace gigaSpace) {
AsyncFuture<Boolean> future = vendorGigaSpace.execute(new
VendorValidationTask(vendorPaymentMsg),
vendorPaymentMsg.getMerchant());
IdQuery<PaymentAuthorization> idQuery = new
IdQuery<PaymentAuthorization>(PaymentAuthorization.class,
vendorPaymentMsg.getPaymentId());
if (future.get()) gigaSpace.change(idQuery, new
ChangeSet().set("vendorCheck", true));
else gigaSpace.change(idQuery, new ChangeSet().set("vendorCheck", false));
}
Class PaymentAuthorization {
Status status;
Boolean userCheck;
Boolean vendorCheck;
}

27. Yey! Payment Has Been Authorized
@EventTemplate
public PaymentAuthorization getNewPayment() {
return new PaymentAuthorization(null, true, true, PaymentAuthorizationStatus.New);
}
Queue
Payment
Authorization

28. Yey! Payment Has Been Authorized
@SpaceDataEvent
public void completePaymentValidation(PaymentAuthorization paymentAuthorization) {
Payment payment = gigaSpace.readById(Payment.class,paymentAuthorization.getPaymentId());
payment.setPaymentStatus(Payment.PaymentStatus.Closed);
gigaSpace.write(payment);
paymentAuthorization.setPaymentAuthorizationStatus(
PaymentAuthorization.PaymentAuthorizationStatus.Done);
}
Queue
Payment
Authorization

29. What’s next?
• Multi-site deployment?
• Persistence to DB for auditing?
• Using Scala? .NET? scripts?
• Schema evolution?
• Running our clusters on the cloud?
• Etc.

30. Q&A
Source code (Java) publically available on GitHub:
https://github.com/Gigaspaces/fraud-detection-example

31. Thank you!
Dotan Horovits
dotan@gigaspaces.com
horovits.wordpress.com
@horovits