GAE Developer - Day3
Simon @ MiCloud
2014Q1
Last Class
● Datastore basic
● Datastore operation
● Restrictions
● Transaction
Create an Entity with Key
Entity job = new Entity("User", "simonsu@mitac.com.tw");
job.setProperty("name", "Simon Su");
jo...
Create an Entity with Ancestor
Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
Entity job = new Entity...
Get data with Key
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Key key = KeyFactory.createKey("Use...
Get data with Ancestor
Query q = new Query("Jobs");
Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw");
q...
Today
● Memcache basic
● Task Queue concept
● Push Queue
● Pull Queue
Memcache
Why memcache
● Improve Application Performance
● Reduce Application Cost
● Caching for Read heavy operations
● Caching In ...
Memcache - operation
MemcacheService cache = MemcacheServiceFactory.getMemcacheService();
cache.setErrorHandler(
ErrorHand...
Async memcache - get
// Using the asynchronous cache
AsyncMemcacheService asyncCache =
MemcacheServiceFactory.getAsyncMemc...
Async memcache - put
if (value == null) {
// get value from other source
// ........
// asynchronously populate the cache
...
Using jcache - init
import java.util.Collections;
import net.sf.jsr107cache.*;
Cache cache;
try {
CacheFactory cacheFactor...
Using jcache - get & put
String key; // ...
byte[] value; // ...
// Put the value into the cache.
cache.put(key, value);
/...
Testing with ab benchmark
ab -c 50 -n 50 http://[app-address]/[servlet_path]
or
ab -c 50 -t 50 http://[app-address]/[servl...
Other Operation
● getAll(), putAll(), deleteAll()
A single read or write operation for multiple memcache
entries
● increme...
Memcache Caveat
● Memcache Is Volatile
● Performance suggest: Batch size < 32 MB
● Dedicated memcache (pay): 10K OPS/s/GB,...
Task Queue
Why Task Queue
● Frontend restriction of 60 second limit
● Need longer-running tasks
● Asynchronously outside front end re...
Task Queue Types
● By name
○ Default Queue (Push Queue)
○ Named Queue
● By type
○ Push queues
■ tasks execute automaticall...
queue.xml
<queue-entries>
<queue>
<name>Queue-Name</name>
<rate>1/s</rate>
...
</queue>
<queue>
<name>Queue-Name2</name>
<...
Task Queue Manage Console
Task Queue - Default Push Queue
Queue queue = QueueFactory.getDefaultQueue();
//named queue:
//QueueFactory.getQueue("Qnam...
Detail Configure
● Task queues use token bucket algorithm [ref]
● rate - usual rate
● bucket-size - cap on peak demand
● max concurrent requests
● Retry options
○ task-retry-limit - min ...
Task Queue - With Transaction
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Queue queue = QueueFact...
Pull Queue - Create Queue
Queue queue =
QueueFactory.getQueue("queue_name");
TaskOptions task =
TaskOptions.Builder
.withM...
Pull Queue - Lease
Queue queue =
QueueFactory.getQueue("queue_name");
tasks = q.leaseTasks(
duration,
TimeUnit.SECONDS,
ho...
Pull Queue - Delete Task
//delete named task
queue.deleteTask("task_ name");
//purge all tasks from queue
QueueFactory.get...
Task Queue ACL - queue.xml
<queue>
<name>pull-queue1</name>
<mode>pull</mode>
<acl>
<user-email>me@gmail.com</user-email>
...
Cron
Cron
<cronentries>
<cron>
<url>/recache</url>
<description>
Put your description here
</description>
<schedule>every 5 min...
● every 12 hours
● every 5 minutes from 10:00 to 14:00
● 2nd,third mon,wed,thu of march 17:00
● every monday 09:00
● 1st m...
Cron Manage Console
Restrictions
● 20 cron jobs for free apps
● 100 for paid apps
Task Queue / Cron Reference
● Task Queues Developer Guide (Java) (Python)
● Task Queue Configuration (Java) (Python)
● Tas...
Upcoming SlideShare
Loading in …5
×

Google App Engine Developer - Day3

1,309 views

Published on

GAE Java - Day3
About memcache, task queue concept and operation

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,309
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Google App Engine Developer - Day3

  1. 1. GAE Developer - Day3 Simon @ MiCloud 2014Q1
  2. 2. Last Class ● Datastore basic ● Datastore operation ● Restrictions ● Transaction
  3. 3. Create an Entity with Key Entity job = new Entity("User", "simonsu@mitac.com.tw"); job.setProperty("name", "Simon Su"); job.setProperty("start", "20140103"); job.setProperty("create", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  4. 4. Create an Entity with Ancestor Key userKey = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); Entity job = new Entity("Jobs", userKey); job.setProperty("name", "engineer"); job.setProperty("start", new Date()); try { Key k = ds.put(job); out.println("Done..." + k.getId()); } catch (Exception e) { e.printStackTrace(); }
  5. 5. Get data with Key DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Key key = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); try { Entity ent = ds.get(key); out.print(ent); Map m = ent.getProperties(); out.println(m.toString()); } catch (EntityNotFoundException e) { e.printStackTrace(); }
  6. 6. Get data with Ancestor Query q = new Query("Jobs"); Key ancestor = KeyFactory.createKey("User", "simonsu@mitac.com.tw"); q.setAncestor(ancestor); PreparedQuery results = ds.prepare(q); Iterator iter = results.asIterable().iterator(); while(iter.hasNext()) { Entity ent = (Entity) iter.next(); out.println(ent); }
  7. 7. Today ● Memcache basic ● Task Queue concept ● Push Queue ● Pull Queue
  8. 8. Memcache
  9. 9. Why memcache ● Improve Application Performance ● Reduce Application Cost ● Caching for Read heavy operations ● Caching In Front of Datastore ● Semi-durable Shared state Across App Instances
  10. 10. Memcache - operation MemcacheService cache = MemcacheServiceFactory.getMemcacheService(); cache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache value = (byte[]) cache.get(key); // save to cache if (value == null) { // ........ cache.put(key, value); }
  11. 11. Async memcache - get // Using the asynchronous cache AsyncMemcacheService asyncCache = MemcacheServiceFactory.getAsyncMemcacheService(); asyncCache.setErrorHandler( ErrorHandlers.getConsistentLogAndContinue(Level.INFO)); // read from cache Future<Object> futureValue = asyncCache.get(key); // ... do other work in parallel to cache retrieval value = (byte[]) futureValue.get();
  12. 12. Async memcache - put if (value == null) { // get value from other source // ........ // asynchronously populate the cache // Returns a Future<Void> which can be used to block until completion asyncCache.put(key, value); }
  13. 13. Using jcache - init import java.util.Collections; import net.sf.jsr107cache.*; Cache cache; try { CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory(); cache = cacheFactory.createCache(Collections.emptyMap()); } catch (CacheException e) { // ... }
  14. 14. Using jcache - get & put String key; // ... byte[] value; // ... // Put the value into the cache. cache.put(key, value); // Get the value from the cache. value = (byte[]) cache.get(key);
  15. 15. Testing with ab benchmark ab -c 50 -n 50 http://[app-address]/[servlet_path] or ab -c 50 -t 50 http://[app-address]/[servlet_path]
  16. 16. Other Operation ● getAll(), putAll(), deleteAll() A single read or write operation for multiple memcache entries ● increment(key, delta), incrementAll(...) Provide atomic increment of numeric value(s) ● getIdentifiable(), putIfUntouched() A mechanism to update a value consistently by concurrent requests
  17. 17. Memcache Caveat ● Memcache Is Volatile ● Performance suggest: Batch size < 32 MB ● Dedicated memcache (pay): 10K OPS/s/GB, range 1-20 GB
  18. 18. Task Queue
  19. 19. Why Task Queue ● Frontend restriction of 60 second limit ● Need longer-running tasks ● Asynchronously outside front end request ● Best practice: ○ Email ○ Write datastore ○ Web crawler ○ Backend jobs
  20. 20. Task Queue Types ● By name ○ Default Queue (Push Queue) ○ Named Queue ● By type ○ Push queues ■ tasks execute automatically ■ only for App Engine app ○ Pull queues ■ tasks wait to be leased ■ available to workers outside the app ■ tasks can be batched
  21. 21. queue.xml <queue-entries> <queue> <name>Queue-Name</name> <rate>1/s</rate> ... </queue> <queue> <name>Queue-Name2</name> <mode>pull</mode> ... </queue> </queue-entries> Push Queue Pull Queue
  22. 22. Task Queue Manage Console
  23. 23. Task Queue - Default Push Queue Queue queue = QueueFactory.getDefaultQueue(); //named queue: //QueueFactory.getQueue("Qname"); TaskOptions task = TaskOptions.Builder .withUrl("/path-to-my-worker") .param(key, value); queue.add(task); Reachable Servlet
  24. 24. Detail Configure ● Task queues use token bucket algorithm [ref]
  25. 25. ● rate - usual rate ● bucket-size - cap on peak demand ● max concurrent requests ● Retry options ○ task-retry-limit - min retries ○ task-age-limit - min elapsed time to keep retrying (Both task-retry-limit and task-age-limit are reached, task is deleted) ○ min-backoff-seconds -- min delay between retries ○ max-backoff-seconds -- max delay between retries ○ max-doublings -- max times to double delay ● more parameters see DOC queue.xml parameters
  26. 26. Task Queue - With Transaction DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Queue queue = QueueFactory.getDefaultQueue(); try { Transaction txn = ds.beginTransaction(); // … other operations queue.add(TaskOptions.Builder.withUrl("/path-to-my-worker")); // … other operations txn.commit(); } catch (DatastoreFailureException e) { // … exception handle }
  27. 27. Pull Queue - Create Queue Queue queue = QueueFactory.getQueue("queue_name"); TaskOptions task = TaskOptions.Builder .withMethod(TaskOptions.Method.PULL) .payload(payload); queue.add(task);
  28. 28. Pull Queue - Lease Queue queue = QueueFactory.getQueue("queue_name"); tasks = q.leaseTasks( duration, TimeUnit.SECONDS, how-many);
  29. 29. Pull Queue - Delete Task //delete named task queue.deleteTask("task_ name"); //purge all tasks from queue QueueFactory.getQueue("queue_name").purge();
  30. 30. Task Queue ACL - queue.xml <queue> <name>pull-queue1</name> <mode>pull</mode> <acl> <user-email>me@gmail.com</user-email> <writer-email>you@gmail.com</writer-email> <writer-email>she@gmail.com</writer-email> </acl> </queue>
  31. 31. Cron
  32. 32. Cron <cronentries> <cron> <url>/recache</url> <description> Put your description here </description> <schedule>every 5 minutes</schedule> </cron> </cronentries>
  33. 33. ● every 12 hours ● every 5 minutes from 10:00 to 14:00 ● 2nd,third mon,wed,thu of march 17:00 ● every monday 09:00 ● 1st monday of sep,oct,nov 17:00 ● every day 00:00 ● every N (hours|mins|minutes) ["from" (time) "to" (time)] ● every 2 hours synchronized Cron Schedule Format
  34. 34. Cron Manage Console
  35. 35. Restrictions ● 20 cron jobs for free apps ● 100 for paid apps
  36. 36. Task Queue / Cron Reference ● Task Queues Developer Guide (Java) (Python) ● Task Queue Configuration (Java) (Python) ● Task Queue API Ref (JavaDoc) (Python) ● Task Queue REST API Ref ● Scheduled Tasks with Cron (Java) (Python) ● Task queue quota and limits ● Get started with Task API article (Python) ● Google I/O 2011: Putting Task Queues to Work (video)

×