LA Cassandra Day 2015 - Testing Cassandra

©2013 DataStax Conﬁdential. Do not distribute without consent.
@chbatey
Christopher Batey 
Technical Evangelist for Apache Cassandra
LA 2015: Testing Cassandra applications

@chbatey
Who am I?
• Based in London
• Technical Evangelist for Apache Cassandra
•Work on Stubbed Cassandra
•Help out Apache Cassandra users
• Previous: Cassandra backed apps at BSkyB

@chbatey
Agenda
• Cassandra failure scenarios
• Developer tools
• Stubbed Cassandra

@chbatey
Production?
Application

@chbatey
Production?
ApplicationApplicationApplicationApplicationApplication
replication factor: 3
consistency: ONE

@chbatey
Production?
DC1
DC2

@chbatey
Read timeout
Application C
R1
R2
R3
C=QUROUM
Replication factor: 3
timeout
timeout
Read timeout

@chbatey
Read timeout
• Received acknowledgements
• Required acknowledgements
• Consistency level
• wasDataReceived

@chbatey
Write timeout
Application C
R1
R2
R3
C=QUROUM
timeout
timeout
Write timeout

@chbatey
Retrying writes
• Cassandra does not roll back!

@chbatey
Write timeout
• Received acknowledgements
• Required acknowledgements
• Consistency level
• CAS and Batches are more complicated:
- WriteType: SIMPLE, BATCH, UNLOGGED_BATCH,
BATCH_LOG, CAS

@chbatey
Batches
• BATCH_LOG
- Timed out waiting for batch log replicas
- Retry if you like
• BATCH
- Written to batch log but timed out waiting for actual replica
- Will eventually be committed (serial read)
• UNLOGGED_BATCH
- Unknown - retry if you like

@chbatey
Idempotent writes
• All writes are idempotent with the following exceptions:
- Counters
- lists

@chbatey
Unavailable
Application C
R1
R2
R3
C=QUROUM
Unavailable

@chbatey
Unavailable
• Alive Replicas
• Required Replicas
• Consistency

@chbatey
Coordinator issue
Application C
R1
R2
R3
C=QUROUM
???
Set a socket read timeout!

@chbatey
Unit & Integration testing
• Requirements: Quick to run, deterministic, not brittle!!
• Integration tests against an embedded Cassandra?
• cassandra-unit
• Integration tests against an external Cassandra running on developer / CI
machines
• CCM
• Regular install
• Docker
• Vagrant
• Mocking the driver

@chbatey
Acceptance testing
Application
Acceptance
test
prepare data
verification

@chbatey
How do this if it was a HTTP service?
• Release it - great book
• Wiremock - mocking HTTP
services
• Saboteur - adding network
latency
If you want to build a fault tolerant application
you better test faults!

@chbatey
Stubbed Cassandra
Application
Acceptance
test
prime on admin port (REST)
verification on admin port
Admin endpoints
Native protocol

@chbatey
Two sides of Scassandra
• Java
-Acts as a unit/integration testing library
-90% of effort on this side
• Standalone
-Runs as a separate process

@chbatey
Java Client
• Embeds Stubbed Cassandra in a Java library
- Start / stop the server
- Prime the server to return rows and/or errors
- Verify exactly the queries your application has executed

@chbatey
Starting / Stopping
@ClassRule 
public static final ScassandraServerRule SCASSANDRA = new
ScassandraServerRule();
 
Scassandra scassandra = ScassandraFactory.createServer();
scassandra.start();
PrimingClient primingClient = scassandra.primingClient();
ActivityClient activityClient = scassandra.activityClient();

@chbatey
Activity Client
• Query
‣ Query text
‣ Consistency
• PrepreparedStatementExecution
‣ Prepared statement text
‣ Bound variables
 
public List<Query> retrieveQueries();
public List<PreparedStatementExecution> retrievePreparedStatementExecutions()
 
public List<Connection> retrieveConnections();
public void clearAllRecordedActivity()
 
public void clearConnections(); 
public void clearQueries(); 
public void clearPreparedStatementExecutions();

@chbatey
Priming Client
• PrimingRequest
‣ Either a Query or PreparedStatement
‣ Query text or QueryPattern (regex)
‣ Consistency (default all)
‣ Result (success, read timeout, unavailable etc)
‣ Rows for successful response
‣ Column types for rows
‣ Variable types for prepared statements
public void prime(PrimingRequest prime)
 
public List<PrimingRequest> retrievePreparedPrimes() 
public List<PrimingRequest> retrieveQueryPrimes() 
public void clearAllPrimes() 
public void clearQueryPrimes() 
public void clearPreparedPrimes()

@chbatey
Example time
public interface PersonDao { 
void connect(); 
 
void disconnect(); 
 
List<Person> retrievePeople(); 
 
List<Person> retrievePeopleByName(String firstName); 
 
void storePerson(Person person); 
}

@chbatey
Testing the connect method
@Test 
public void shouldConnectToCassandraWhenConnectCalled() { 
//given 
activityClient.clearConnections(); 
//when 
underTest.connect(); 
//then 
assertTrue("Expected at least one connection", 
activityClient.retrieveConnections().size() > 0); 
}

@chbatey
Testing retrieve
public List<Person> retrieveNames() { 
ResultSet result; 
try { 
Statement statement = new SimpleStatement("select * from person"); 
statement.setConsistencyLevel(ConsistencyLevel.QUORUM); 
result = session.execute(statement); 
} catch (ReadTimeoutException e) { 
throw new UnableToRetrievePeopleException(); 
} 
 
List<Person> people = new ArrayList<>(); 
for (Row row : result) { 
people.add(new Person(row.getString("first_name"), row.getInt("age"))); 
} 
return people; 
}

@chbatey
Test the query + consistency
@Test 
public void testQueryIssuedWithCorrectConsistencyUsingMatcher() { 
//given 
Query expectedQuery = Query.builder() 
.withQuery("select * from person") 
.withConsistency("QUORUM").build(); 
 
//when 
underTest.retrievePeople(); 
 
//then 
assertThat(activityClient.retrieveQueries(), containsQuery(expectedQuery)); 
}
Hamcrest matcher

@chbatey
Testing behaviour
@Test 
public void testRetrievingOfNames() throws Exception { 
// given 
Map<String, ?> row = ImmutableMap.of( 
"first_name", "Chris", 
"last_name", "Batey", 
"age", 29); 
primingClient.prime(PrimingRequest.queryBuilder() 
.withColumnTypes(column("age", PrimitiveType.INT)) 
.withRows(row) 
.build()); 
 
//when 
List<Person> names = underTest.retrievePeople(); 
 
//then 
assertEquals(1, names.size()); 
assertEquals(“Chris Batey", names.get(0).getName()); 
}
Each Cassandra Row == Java map
Tell Scassandra about your schem

@chbatey
Testing errors
@Test(expected = UnableToRetrievePeopleException.class) 
public void testHandlingOfReadRequestTimeout() throws Exception { 
// given 
PrimingRequest primeReadRequestTimeout = PrimingRequest.queryBuilder() 
.withResult(Result.read_request_timeout) 
.build(); 
primingClient.prime(primeReadRequestTimeout); 
 
//when 
 
//then 
} Expecting custom exception

@chbatey
Testing retries
 
@Override 
public RetryDecision onReadTimeout(Statement statement, ConsistencyLevel cl, int requiredResponses,
int receivedResponses, boolean dataRetrieved, int nbRetry) { 
if (nbRetry < configuredRetries) { 
return RetryDecision.retry(cl); 
} else { 
return RetryDecision.rethrow(); 
} 
} 
 
@Override 
public RetryDecision onWriteTimeout(Statement s, ConsistencyLevel cl, WriteType wt, int requiredAcks,
int receivedAcks, int nbRetry) { 
return DefaultRetryPolicy.INSTANCE.onWriteTimeout(s, cl, wt, receivedAcks, receivedAcks, nbRetry); 
} 
 
@Override 
public RetryDecision onUnavailable(Statement statement, ConsistencyLevel cl, int requiredReplica,
int aliveReplica, int nbRetry) { 
return DefaultRetryPolicy.INSTANCE.onUnavailable(statement, cl, requiredReplica, aliveReplica, nbRetry); 
}

@chbatey
Testing retries
@Test 
public void testRetriesConfiguredNumberOfTimes() throws Exception { 
PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() 
.withResult(Result.read_request_timeout) 
.build(); 
primingClient.prime(readTimeoutPrime); 
 
try { 
} catch (UnableToRetrievePeopleException e) { 
} 
 
assertEquals(CONFIGURED_RETRIES + 1, activityClient.retrieveQueries().size()); 
}

@chbatey
Coordinator issue
Application C
R1
R2
R3
C=QUROUM
???

@chbatey
Testing slow connection
@Test(expected = UnableToSavePersonException.class) 
public void testThatSlowQueriesTimeout() throws Exception { 
// given 
PrimingRequest preparedStatementPrime = PrimingRequest.preparedStatementBuilder() 
.withQueryPattern("insert into person.*") 
.withVariableTypes(VARCHAR, INT, list(TIMESTAMP)) 
.withFixedDelay(1000) 
.build(); 
primingClient.prime(preparedStatementPrime); 
underTest.connect(); 
 
underTest.storePerson(new Person("Christopher", 29, Collections.emptyList())); 
}
Delays the response by 1000ms
Expect a custom exception

@chbatey
Testing slow connection
SocketOptions socketOptions = new SocketOptions(); 
socketOptions.setReadTimeoutMillis(500); 
cluster = Cluster.builder() 
.addContactPoint("localhost") 
.withPort(port) 
.withRetryPolicy(new RetryReads()) 
.withSocketOptions(socketOptions) 
.build();
Set a read timeout
@Override 
public void storePerson(Person person) { 
try { 
BoundStatement bind = storeStatement.bind(person.getName(), person.getAge()); 
session.execute(bind); 
} catch (NoHostAvailableException e) { 
throw new UnableToSavePersonException(); 
} 
}

@chbatey
Summary of features
• Start as many instances as you like in the same JVM
• Prime all primitive types + collections
• Prime prepared statements
• Verify number of connections your application makes
• Test slow queries

@chbatey
Future features
• Loading of schema for priming prepared statements
• Pretending to be multiple nodes

@chbatey
How do I get it?
•It is on maven central
•Or go to www.scassandra.org
•Executable jar + REST API
<dependency> 
<groupId>org.scassandra</groupId> 
<artifactId>java-client</artifactId> 
<version>0.6.0</version> 
<scope>test</scope> 
</dependency>

@chbatey
I want to help
• Server: https://github.com/scassandra/scassandra-
server
• Client: https://github.com/scassandra/scassandra-java-
client
• Tests: https://github.com/scassandra/scassandra-it-java-
driver-2

@chbatey
Summary
• Testing is good - I want more tools to help me
• Existing tools for Cassandra are great at happy path
scenarios
• Stubbed Cassandra allows testing of edge cases

@chbatey
The end - Questions?
www.scassandra.org
http://christopher-batey.blogspot.co.uk/
Ping me on twitter @chbatey

@chbatey
How does this work?
• Server implemented in Scala
• Binary port for your Cassandra application to connect to
• Admin port for the REST API
• Thin Java wrapper to make it easy to be used in JUnit
tests

LA Cassandra Day 2015 - Testing Cassandra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to LA Cassandra Day 2015 - Testing Cassandra

Similar to LA Cassandra Day 2015 - Testing Cassandra (20)

More from Christopher Batey

More from Christopher Batey (20)

Recently uploaded

Recently uploaded (20)

LA Cassandra Day 2015 - Testing Cassandra