This document discusses using Akeneo's batch framework to synchronize applications. It describes: 1) The akeneo/batch framework can be used to import, export, and perform mass actions through jobs made up of steps. 2) Jobs contain steps that can be triggers to perform tasks quickly or item processors to handle data line by line through readers, processors, and writers. 3) Complex mappings can be declared and compiled into optimized PHP code to transform data between sources and destinations very quickly for large datasets.

1. Synchronize
applications with
akeneo/batch
Grégory Planchat
@gplanchat
Kiboko SAS, France

2. Grégory Planchat
@gplanchat
Kiboko SAS, France

3. Grégory Planchat
@gplanchat
Kiboko SAS, France
You may know me from
akeneopim-ug.slack.com

4. Data integration
It is all about importing and
exporting data

5. akeneo/batch
• Exists since Akeneo’s inception in the
AkeneoBatchBundle
• framework-agnostic

6. Used to…
• Import data
• Export
• Mass actions
• …

7. $batchJob = new Job(
‘inventory_job',
$eventDispatcher,
$jobRepository,
[
new ItemStep(
‘inventory_step',
$eventDispatcher,
$jobRepository,
new InventoryFileReader(),
new InventoryImportProcessor(),
new InventoryDatabaseWriter()
),
]
);
What does a job look like?

8. $execution = new JobExecution();
$execution->setJobParameters(
new JobParameters(
[
'cachePath' => $filename,
]
)
);
$batchJob->execute($execution);
How to run a job ?

9. Under the hood
Run Tier Job Tier Step Tier Data Tier

10. $batchJob = new Job(
‘foo_job',
$eventDispatcher,
new JobRepository(),
[
new ItemStep(
...
),
new ItemStep(
...
),
new MyAwesomeNotificationStep(
...
),
new RandomStep(
...
),
]
);
Unlimited number of Steps

11. Principles
• Every Job has at least one Step
• Akeneo provides 2 types of steps:
• Triggers
• Item processing
• The Job provides a JobParameters object for
execution-specific configuration

12. Trigger
• Runs a unique action, should be fast, on an
external element of the batch (eg. A command line)
• Only provides a status (success / failed)
• Examples : download files, clear a cache, send an
e-mail, run a command line, etc.

13. class ApiCacheFetching extends AbstractStep
{
public function doExecute(StepExecution $stepExecution)
{
$file = fopen(
$stepExecution->getJobParameters()->get(‘cachePath'),
‘w'
);
$api = fopen(‘http://api.example.com/foo?bar=3', 'r');
stream_copy_to_stream($api, $file);
fclose($api);
fclose($file);
}
}
A trigger Step: download a
file and put it in local storage

14. class ApiCacheCleanup extends AbstractStep
{
public function doExecute(StepExecution $stepExecution)
{
$file = $stepExecution->getJobParameters()
->get('cachePath');
unlink($file);
}
}
A trigger Step: remove a file
from local storage

15. public function doExecute(StepExecution $stepExecution)
{
$builder = new ProcessBuilder(
[
'/usr/bin/env',
'php',
$this->indexerCommandPath,
'--reindex',
$this->index,
]
);
$process = $builder->getProcess();
$process->setTimeout(3600);
$process->run();
if (!$process->isSuccessful()) {
$stepExecution->addFailureException(
new ProcessFailedException($process));
}
$stepExecution->addSummaryInfo('index', $process->getOutput());
}
A trigger step: run Magento
reindex

16. $batchJob = new Job(
‘inventory_job',
$eventDispatcher,
$jobRepository,
[
new ApiCacheFetching(...),
new ItemStep(
‘inventory_step',
$eventDispatcher,
$jobRepository,
new InventoryFileReader(),
new InventoryDefaultProcessor(),
new InventoryDatabaseWriter()
),
new ApiCacheCleanup(...),
]
);
How to integrate it with the Job
We have added
our 2 Steps
into the Job

17. Item processing
• Handles a data source one line at a time
• Input and output can be two different storage types
• Can reject or ignore certain items
• Provides a status (success / failure)

18. Item processing

19. Item processing, Read

20. Item processing, Writing

21. Using Iterators
• Eases the data reading
• Lots of classes in the SPL and 3rd party libraries
provides iteration feature

22. class IteratorReader implements
ItemReaderInterface,
StepExecutionAwareInterface,
InitializableInterface
{
public function setStepExecution(StepExecution $stepExecution)
{
$this->stepExecution = $stepExecution;
}
public function initialize()
{
$this->iterator = new FooIterator(...);
$this->iterator->rewind();
}
public function read()
{
...
}
}
Using an Iterator

23. public function initialize()
{
$filePath = $this->stepExecution
->getJobParameters()->get('cachePath')
$xml = simplexml_load_file($filePath);
$this->iterator = new ArrayIterator(
$xml->xpath(‘//inventory/stock/virtual')
);
$this->iterator->rewind();
}
public function read()
{
if (!$this->iterator->valid()) {
return null;
}
$this->stepExecution->incrementReadCount();
$item = this->iterator->current();
$this->iterator->next();
return $item;
}
Using SimpleXML

24. Limits
• No conditional execution
• No parallel execution
• Processors are unpleasant and complex to write

25. How to go further?
• Create a new type of Step
• Use the Extract-Transform-Load pattern
• Ease data mapping

26. Use case #1
• 145.000 product SKU
• 70 websites
• Inventories and prices are coming from 46 ERP

27. The pipeline
• Extract data from the sources
• Transform the data (type and contents)
• Load the data into the destination
• Open-sourced as php-etl/pipeline

28. $pipeline = (new PipelinePipeline(new PipelineRunner()))
->extract(new ProductExtractor($productEndpoint, $start, $limit))
->transform(new ProductTransformer($mapper, $logger))
->transform(new ProductMappingToDoctrineEntityBySKUTransformer($om, $mapper))
->load(new DoctrineBatchLoader($batchSize, $om))
;
Declare a pipeline

29. $batchJob = new Job(
‘product_job',
$eventDispatcher,
$jobRepository,
[
new PipelineStep(
‘product_step',
$eventDispatcher,
$jobRepository,
$pipeline
),
]
);
How to integrate it with the Job

30. Use case #1

31. Use case #2
• 8.000 product SKU per tenant
• Big concurrency
• A batch of 25 products imported in 20 seconds

32. How do we go fast?
• Optimize code for speed makes it hard to read
• Stop caring about code readability, makes
maintenance hard
• Code compilers are great, let’s use:
• nikic/php-parser
• symfony/expression-language

33. $mapper = (new ArrayBuilder(null, $interpreter))
->children()
->constant('[type]', 'SIMPLE')
->copy('[ak_applications.default.value]', '[applications]')
->copy('[ak_tree.default.value][last_name]', '[tree]')
->list('[products]', 'merge( input["units"], input["shippings"] )')
->children()
->copy('[code]', '[ean]')
->expression(
'[name]',
'filter( input["label"], locale("fr_FR"), scope("web") )')
->end()
->end()
->end()
->getMapper();
Declare your mapping

34. $compiler = new CompilerCompiler(new CompilerStrategySpaghetti());
$mapper = $compiler->compile(
CompilerStandardCompilationContext::build(
new EmptyPropertyPath(), __DIR__, 'FooArraySpaghettiMapper'
),
$mapper
);
Compile your mapper

35. $output['ak_applications.default.value'] = array_values((function (array $input) {
return array_slice($input, 0, 1, true);
})((function (array $input) : array {
$output = array_filter($input, function (array $item) {
return in_array($item['locale'], ["fr_FR", null]);
});
return $output;
})($input["values"]["applications"] ?? [])))[0]["data"] ?? null;
$output['ak_arbre.name'] = implode(",", (array) (array_values((function (array $input) {
return array_slice($input, 0, 1, true);
})((function (array $input) : array {
$output = array_filter($input, function (array $item) {
return in_array($item['locale'], ["fr_FR", null]);
});
return $output;
})($input["values"]["arbre"] ?? [])))[0]["data"] ?? null));
Spaghetti-generated code

36. Use case #2
• Now imports 6000 products in 20 sec

37. Is it a problem ?
• Code is auto-generated by the library
• Code is optimized to be very fast
• You do not need to maintain it
• Xdebug and Blackfire will be fully functional

38. Merci
Grégory Planchat
@gplanchat
Kiboko SAS, France