This talk is about the "Connector" [1] addon, but focused on the Jobs Queue, which can be used independently of any "connection". We will show, with examples, how we can use this module to delay intensive tasks which would block the users if they were executed synchronously. We'll also see how we can optimize the workload of the jobs using different Job Channels. [1] http://odoo-connector.com

1. A Jobs Queue for processing tasks
asynchronously
Guewen Baconnier & Leonardo Pistone
Camptocamp
1 / 32

2. Guewen Baconnier
Developer @ Camptocamp
OCA committer, OCA Delegate
Connector author
@guewenb
@guewen
Leonardo Pistone
Developer @ Camptocamp
OCA committer, OCA Delegate
@lepistone
About us
2 / 32

3. Computers are slow!
3 / 32

4. Computers are slow!
Humans want them to be fast!
3 / 32

5. The problem
4 / 32

6. User Odoo Server
Waiting
Heavytask!
5 / 32

7. Loading...
6 / 32

8. Still loading...
7 / 32

9. Still loading... Please be patient.
8 / 32

10. Don't leave yet, it's still loading
9 / 32

11. You may not believe it, but the application is
actually loading...
10 / 32

12. Take a minute to get a coffee, because it's
loading...
11 / 32

13. Come on...
12 / 32

14. We can try to save a few seconds
13 / 32

15. We can try to save a few seconds
But we have more radical solutions
13 / 32

16. User Odoo Server
Heavytask!
Connector
Runner
14 / 32

17. Connector
odoo-connector.com
15 / 32

18. Queue it!
Dependency on connector
16 / 32

19. Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(session,model_name,record_id):
#doanheavytaskonrecord_idofmodel_name
16 / 32

20. Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(session,model_name,record_id):
#doanheavytaskonrecord_idofmodel_name
Delay a job:
session=ConnectorSession.from_env(self.env)
a_heavy_task.delay(session,'res.partner',1)
16 / 32

21. Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
17 / 32

22. Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3
./openerp-server--load=web,connector--workers=4
17 / 32

23. Channels
18 / 32

24. HTTPWorkers
csv
magento
root
Running jobs
Capacity
Channels
19 / 32

25. Properties
20 / 32

26. Priority
10 50 999
Priority
import_order.delay(session,1) #defaultis10
import_order.delay(session,2,priority=50)
import_order.delay(session,10,priority=999)
21 / 32

27. B C
Now + 6:00 + 12:00 +18:00 + 24:00
A
ETA
import_order.delay(session,1) #A
import_order.delay(session,1,eta=6*60*60) #B
import_order.delay(session,2,eta=datetime.now()+timedelta(days=1)) #C
22 / 32

28. Retries
import_order.delay(session,1,max_retries=3)
23 / 32

29. Retries
import_order.delay(session,1,max_retries=3)
Invoke a retry
@job
defimport_order(session,args):
try:
do_operation()
except(socket.gaierror,socket.error,socket.timeout)aserr:
raiseRetryableError(
'Anetworkerrorcausedthefailureofthejob:'
'%s'%err)
23 / 32

30. Best Practices
24 / 32

31. Outdating Data in jobs can become outdated.
No:
@job
defexample(session,record_id,vals):
export(record_id,vals)
Yes:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
25 / 32

32. Outdating
Existence
A job can refer to a record which has been deleted. Always
check if it still exists.
No:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
Yes:
@job
defexample(session,record_id):
record=session.env['model'].browse(record_id)
ifrecord.exists():
export(record)
26 / 32

33. Outdating
Existence
Idempotence
A job should, when possible, produce the same result
when executed several times.
No:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
Yes:
@job
defexample(session,record_id):
record=session.env['model'].browse(record_id)
ifrecord.exists():
ifnotrecord.exported:
export(session.env['model'].browse(record_id))
27 / 32

34. Useful Patterns
28 / 32

35. Fanout Job
A job generating other jobs.
@job
defimport_file(session,filepath):
withopen(filepath)asf:
forlineinf:
import_line.delay(session,line)
29 / 32

36. Try or delay
If an operation failed, try it later.
@job
defdo_operation(session,args):
#work
try:
do_operation(session,args)
exceptTimeoutError:
do_operation.delay(session,args,eta=10*60)
30 / 32

37. Extract highly concurrent tasks
And put them in a one-by-one channel.
31 / 32

38. Thanks!
32 / 32

39. Thanks!
OCA Sponsors
32 / 32