The document discusses Google App Engine task queues and cron jobs. It provides examples of creating and using default and named push queues, as well as pull queues. It explains how to add tasks, lease tasks from pull queues, and delete tasks. It also covers using task queues within transactions. Additionally, it demonstrates configuring and using cron jobs to schedule recurring tasks.

1. GAE Developer - Day3
Simon @ MiCloud
2014Q1

2. Last Class
● Datastore basic
● Datastore operation
● Restrictions
● Transaction

3. Create an Entity with Key
Entity job = new Entity("User", "simonsu@mitac.com.tw");
job.setProperty("name", "Simon Su");
job.setProperty("start", "20140103");
job.setProperty("create", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}

4. Create an Entity with Ancestor
Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
Entity job = new Entity("Jobs", userKey);
job.setProperty("name", "engineer");
job.setProperty("start", new Date());
try {
Key k = ds.put(job);
out.println("Done..." + k.getId());
} catch (Exception e) {
e.printStackTrace();
}

5. Get data with Key
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
try {
Entity ent = ds.get(key);
out.print(ent);
Map m = ent.getProperties();
out.println(m.toString());
} catch (EntityNotFoundException e) {
e.printStackTrace();
}

6. Get data with Ancestor
Query q = new Query("Jobs");
Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
q.setAncestor(ancestor);
PreparedQuery results = ds.prepare(q);
Iterator iter = results.asIterable().iterator();
while(iter.hasNext()) {
Entity ent = (Entity) iter.next();
out.println(ent);
}

7. Today
● Memcache basic
● Task Queue concept
● Push Queue
● Pull Queue

8. Memcache

9. Why memcache
● Improve Application Performance
● Reduce Application Cost
● Caching for Read heavy operations
● Caching In Front of Datastore
● Semi-durable Shared state Across App
Instances

10. Memcache - operation
MemcacheService cache = MemcacheServiceFactory.getMemcacheService();
cache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
value = (byte[]) cache.get(key);
// save to cache
if (value == null) {
// ........
cache.put(key, value);
}

11. Async memcache - get
// Using the asynchronous cache
AsyncMemcacheService asyncCache =
MemcacheServiceFactory.getAsyncMemcacheService();
asyncCache.setErrorHandler(
ErrorHandlers.getConsistentLogAndContinue(Level.INFO));
// read from cache
Future<Object> futureValue = asyncCache.get(key);
// ... do other work in parallel to cache retrieval
value = (byte[]) futureValue.get();

12. Async memcache - put
if (value == null) {
// get value from other source
// ........
// asynchronously populate the cache
// Returns a Future<Void> which can be used to block until completion
asyncCache.put(key, value);
}

13. Using jcache - init
import java.util.Collections;
import net.sf.jsr107cache.*;
Cache cache;
try {
CacheFactory cacheFactory =
CacheManager.getInstance().getCacheFactory();
cache = cacheFactory.createCache(Collections.emptyMap());
} catch (CacheException e) {
// ...
}

14. Using jcache - get & put
String key; // ...
byte[] value; // ...
// Put the value into the cache.
cache.put(key, value);
// Get the value from the cache.
value = (byte[]) cache.get(key);

15. Testing with ab benchmark
ab -c 50 -n 50 http://[app-address]/[servlet_path]
or
ab -c 50 -t 50 http://[app-address]/[servlet_path]

16. Other Operation
● getAll(), putAll(), deleteAll()
A single read or write operation for multiple memcache
entries
● increment(key, delta), incrementAll(...)
Provide atomic increment of numeric value(s)
● getIdentifiable(), putIfUntouched()
A mechanism to update a value consistently by
concurrent requests

17. Memcache Caveat
● Memcache Is Volatile
● Performance suggest: Batch size < 32 MB
● Dedicated memcache (pay): 10K OPS/s/GB,
range 1-20 GB

18. Task Queue

19. Why Task Queue
● Frontend restriction of 60 second limit
● Need longer-running tasks
● Asynchronously outside front end request
● Best practice:
○ Email
○ Write datastore
○ Web crawler
○ Backend jobs

20. Task Queue Types
● By name
○ Default Queue (Push Queue)
○ Named Queue
● By type
○ Push queues
■ tasks execute automatically
■ only for App Engine app
○ Pull queues
■ tasks wait to be leased
■ available to workers outside the app
■ tasks can be batched

21. queue.xml
<queue-entries>
<queue>
<name>Queue-Name</name>
<rate>1/s</rate>
...
</queue>
<queue>
<name>Queue-Name2</name>
<mode>pull</mode>
...
</queue>
</queue-entries>
Push Queue
Pull Queue

22. Task Queue Manage Console

23. Task Queue - Default Push Queue
Queue queue = QueueFactory.getDefaultQueue();
//named queue:
//QueueFactory.getQueue("Qname");
TaskOptions task =
TaskOptions.Builder
.withUrl("/path-to-my-worker")
.param(key, value);
queue.add(task);
Reachable Servlet

24. Detail Configure
● Task queues use token bucket algorithm [ref]

25. ● rate - usual rate
● bucket-size - cap on peak demand
● max concurrent requests
● Retry options
○ task-retry-limit - min retries
○ task-age-limit - min elapsed time to keep retrying
(Both task-retry-limit and task-age-limit are reached, task is deleted)
○ min-backoff-seconds -- min delay between retries
○ max-backoff-seconds -- max delay between retries
○ max-doublings -- max times to double delay
● more parameters see DOC
queue.xml parameters

26. Task Queue - With Transaction
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Queue queue = QueueFactory.getDefaultQueue();
try {
Transaction txn = ds.beginTransaction();
// … other operations
queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker"));
// … other operations
txn.commit();
} catch (DatastoreFailureException e) {
// … exception handle
}

27. Pull Queue - Create Queue
Queue queue =
QueueFactory.getQueue("queue_name");
TaskOptions task =
TaskOptions.Builder
.withMethod(TaskOptions.Method.PULL)
.payload(payload);
queue.add(task);

28. Pull Queue - Lease
Queue queue =
QueueFactory.getQueue("queue_name");
tasks = q.leaseTasks(
duration,
TimeUnit.SECONDS,
how-many);

29. Pull Queue - Delete Task
//delete named task
queue.deleteTask("task_ name");
//purge all tasks from queue
QueueFactory.getQueue("queue_name").purge();

30. Task Queue ACL - queue.xml
<queue>
<name>pull-queue1</name>
<mode>pull</mode>
<acl>
<user-email>me@gmail.com</user-email>
<writer-email>you@gmail.com</writer-email>
<writer-email>she@gmail.com</writer-email>
</acl>
</queue>

31. Cron

32. Cron
<cronentries>
<cron>
<url>/recache</url>
<description>
Put your description here
</description>
<schedule>every 5 minutes</schedule>
</cron>
</cronentries>

33. ● every 12 hours
● every 5 minutes from 10:00 to 14:00
● 2nd,third mon,wed,thu of march 17:00
● every monday 09:00
● 1st monday of sep,oct,nov 17:00
● every day 00:00
● every N (hours|mins|minutes) ["from" (time) "to" (time)]
● every 2 hours synchronized
Cron Schedule Format

34. Cron Manage Console

35. Restrictions
● 20 cron jobs for free apps
● 100 for paid apps

36. Task Queue / Cron Reference
● Task Queues Developer Guide (Java) (Python)
● Task Queue Configuration (Java) (Python)
● Task Queue API Ref (JavaDoc) (Python)
● Task Queue REST API Ref
● Scheduled Tasks with Cron (Java) (Python)
● Task queue quota and limits
● Get started with Task API article (Python)
● Google I/O 2011: Putting Task Queues to Work (video)