Symfony Camp UA 2018

2. Filesystem abstraction
layer and message
queue for Symfony

3. About me
SergeevYuriy
Technical lead
Digital department 1+1 media
7 years with PHP
3 years with Symfony 😍
https://www.facebook.com/sergeevyv

4. Agenda
● Image processing packages (VichUploaderBundle,
LiipImagiuneBundle)
● Filesystem abstraction layer (Gaufrette)
● Message queue package (Enqueue)
● Symfony Messenger Component

5. Start from scratch
$form = $this->createFormBuilder()
->add('attachment', FileType::class)
->getForm();
// ...
if ($form->isSubmitted() && $form->isValid()) {
/** @var SymfonyComponentHttpFoundationFileUploadedFile $file */
$file = $form['attachment']->getData();
$file->move(
$this->getParameter('upload_directory'),
'image.jpg'
);
}

6. Real life problems
Folders structure
File names
Files & entities binding
Storage
Scaling
Thumbnails
Performance
Background handlers

7. Symfony Filesystem
Component
Basic utilities for the filesystem.
Methods: mkdir, exists, copy, touch, chown, chgrp, chmod, remove, rename, symlink, readlink,
makePathRelative, mirror, isAbsolutePath, dumpFile, appendToFile
$fileSystem->exists('/tmp/photos');
$fileSystem->dumpFile('file.txt', 'Hello World');
$fileSystem->remove(array('symlink', '/path/to/directory', 'activity.log'));

8. What about images?

9. VichUploaderBundle
- File and folder names
- Inject the file into the entity
- Delete the file upon removal of the entity
- Templating helpers
https://github.com/dustin10/VichUploaderBundle

10. Mapping
/**
* @VichUploadableField(mapping="product_image", fileNameProperty="imageName")
*/
private $imageFile;
/**
* @ORMColumn(type="string", length=255)
*/
private $imageName;

11. Manual upload
/**
* @ORMColumn(type="datetime")
*/
private $updatedAt;
//...
public function setImageFile(?File $image = null): void
{
$this->imageFile = $image;
if (null !== $image) {
$this->updatedAt = new DateTimeImmutable();
}
}

12. Configuration (events)
vich_uploader:
# ...
mappings:
product_image:
# ...
inject_on_load: false
delete_on_update: true
delete_on_remove: true

13. vich_uploader:
storage: file_system # one of ['gaufrette', 'flysystem',
'file_system']
mappings:
product_image:
# ...
inject_on_load: false
delete_on_update: true
delete_on_remove: true
Configuration (storage)

14. Gaufrette
http://knplabs.github.io/Gaufrette
Filesystem abstraction layer.
Data Abstraction layer Storage
Supported adapters:
Local, AWS S3, Google Cloud Storage, OpenCloud, FTP,
SFTP etc.

15. Basic usage
use GaufretteFilesystem;
use GaufretteAdapterLocal as LocalAdapter;
// First, you need a filesystem adapter
$adapter = new LocalAdapter('/var/media');
// Then, you can access your filesystem directly
var_dump($filesystem->read('myFile')); // bool(false)
$filesystem->write('myFile', 'Hello world!');
//Then, create filesystem with adapter
$filesystem = new Filesystem($adapter);

16. Extras: Resolvable filesystem
https://github.com/Gaufrette/extras
- AwsS3PublicUrlResolver: Create a URL for an object stored
on S3 with public ACL.
- AwsS3PresignedUrlResolver: Create a temporary URL, valid
for a given amount of time.
- StaticUrlResolver: Resolves the object into an URL by
concatenating a prefix with object path.

17. Usage example
$client = // AwsS3 client instantiation
$expDate = new DateTime('+ 1 hour');
$decorated = new Filesystem(
new AwsS3($client, 'my_bucket', ['directory' => 'root/dir'])
);
$filesystem = new ResolvableFilesystem(
$decorated,
new AwsS3PresignedUrlResolver($client, 'my_bucket', 'root/dir', $expDate)
);
$url = $filesystem->resolve('/foo/bar.png');
// https://eu-west-1.blabla.aws.com/my_bucket/root/dir/foo/bar.png?token

18. Symfony Integration -
KnpGaufretteBundle
https://github.com/KnpLabs/KnpGaufretteBundle
Filesystem map service: 'knp_gaufrette.filesystem_map'
Filesystem services: 'gaufrette.%filesystem_alias%'
Stream wrapper: protocol and filesystems config and ability to use code
like 'gaufrette://domain/file.txt'

19. Adapters and
Filesystems configuration
knp_gaufrette:
adapters:
foo:
local:
directory: /path/to/my/filesystem
create: true
knp_gaufrette:
filesystems:
bar:
adapter: foo
alias: foo_filesystem
aws:
adapter: profile_photos
profile_photos:
aws_s3:
service_id: 'acme.aws_s3.client'
bucket_name: 'images'
detect_content_type: true
custom_adapter:
service:
id: 'my.adapter.service'

20. Basic usage
// get from filesystem map service
$container->get('knp_gaufrette.filesystem_map')->get('bar');
//get by filesystem alias “foo”
$container->get('gaufrette.foo_filesystem');
//stream_wrapper usage example
$fileStream = sprintf('gaufrette://your_defined_fs/%s', 'path/to/file.pdf');
$response = new BinaryFileResponse($fileStream);

21. Back to VichUploader
vich_uploader:
storage: gaufrette
mappings:
product_image:
uri_prefix: /images/products
upload_destination: aws # gaufrette filesystem name

22. What about
thumbnails?

23. LiipImagineBundle
https://github.com/liip/LiipImagineBundle
- Filters, filter sets (size, orientation etc.)
- Post processors (JPEG Optim, Opti PNG etc.)
- Runtime filter configuration
- Background processing
- Data Loaders
- Cache Resolvers
<img src="{{ asset('/relative/path/to/image.jpg') | imagine_filter('my_thumb') }}" />

24. Data Loaders
liip_imagine:
loaders:
stream.profile_photos:
stream:
wrapper: gaufrette://profile_photos
Custom loader
interface LoaderInterface
{
public function find($path);
}

25. Cache Resolvers
interface ResolverInterface
{
public function isStored($path, $filter);
public function resolve($path, $filter);
public function store(BinaryInterface $binary, $path, $filter);
public function remove(array $paths, array $filters);
}

26. When to do it?

27. kernel.terminate
Do stuff

28. kernel.terminate
How it works: fastcgi_finish_request()
When to use:
- short time processes
- small number of requests
Possible problems:
- exceptions breaks profiler
- execution control complexity
- pm.max_children limitation

29. Enqueue
https://github.com/php-enqueue/enqueue-dev
- Remote Procedure Calls
- Job queue
- Async commands
- frameworks integration: Symfony, Magento, Laravel,
Yii2
Supported transports: AMQP, Amazon SQS, Kafka, Redis,
Gearman, Mongodb etc.

30. How it works
Producer Consumer
Queue
Messages

31. Message
use EnqueueClientMessage;
use EnqueueClientMessagePriority;
$message = new Message();
$message->setBody('Hello %username%');
$message->setPriority(MessagePriority::NORMAL);

32. Message Producer
//send to all "fire and forget"
$producer->sendEvent('a_topic', 'Hello there!');
use EnqueueClientProducerInterface;
$producer = $container->get(ProducerInterface::class);
// send to one consumer
$producer->sendCommand('a_processor_name', 'Hello there!');

33. Spool Producer
use EnqueueClientSpoolProducer;
$spoolProducer = $container->get(SpoolProducer::class);
// messages is being sent on console.terminate or kernel.terminate event
$spoolProducer->sendEvent('a_topic', 'Hello there!');
$spoolProducer->sendCommand('a_processor_name', 'Hello there!');
// you could send queued messages manually by calling flush method
$spoolProducer->flush();

34. Message Processor
use InteropQueuePsrProcessor;
use InteropQueuePsrMessage;
use InteropQueuePsrContext;
class SendMailProcessor implements PsrProcessor
{
public function process(PsrMessage $message, PsrContext $context)
{
$this->mailer->send('foo@example.com', $message->getBody());
return self::ACK;
}
}

35. Consumer extensions
use InteropQueuePsrContext;
use EnqueueConsumptionChainExtension;
use EnqueueConsumptionQueueConsumer;
use EnqueueConsumptionExtensionReplyExtension;
/** @var InteropQueuePsrContext $psrContext */
$queueConsumer = new QueueConsumer(
$psrContext,
new ChainExtension([
new ReplyExtension()
])
);

36. Reply extension
use EnqueueConsumptionResult;
use InteropQueuePsrProcessor;
use InteropQueuePsrMessage;
use InteropQueuePsrContext;
class SendMailProcessor implements PsrProcessor
{
public function process(PsrMessage $message, PsrContext $context)
{
$this->mailer->send('foo@example.com', $message->getBody());
$replyMessage = $context->createMessage('Message has been sent');
return Result::reply($replyMessage);
}
}

37. Async commands
(enqueue/async-command)
use EnqueueAsyncCommand{CommandResult, Commands, RunCommand};
$promise = $producer->sendCommand(
Commands::RUN_COMMAND,
new RunCommand('debug:container'),
true
);
// do other stuff.
if ($replyMessage = $promise->receive(5000)) {
$result = CommandResult::jsonUnserialize($replyMessage->getBody());
echo $result->getOutput();
}

38. Testing & Debugging
enqueue:
transport: 'null:'
client:
traceable_producer: true
use EnqueueClientTraceableProducer;
use SymfonyBundleFrameworkBundleClient;
// ...
$service = $client->getContainer()->get('a_service');
// the method calls inside $producer->send('fooTopic', 'messageBody');
$service->do();
$producer = $client->getContainer()->get(TraceableProducer::class);
$traces = $producer->getTopicTraces('fooTopic');
//use inside SomeServiceTest
$this->assertCount(1, $traces);
$this->assertEquals('messageBody', $traces[0]['message']);

39. Integration with
LiipImagineBundle
enqueue:
transport: "amqp:"
client: ~
liip_imagine:
enqueue: true
./bin/console enqueue:consume --setup-broker -vvv
Run consumer

40. Long running commands
● Clear all Doctrine ORM entity managers (to prevent outdated entities from being
updated)
● Reset all closed Doctrine ORM entity managers (after a failed transaction)
● Close all database connections (to prevent database timeout errors)
● Clear all Monolog "fingers crossed" handlers (clears messages and resets the handler
when there was no failure during the execution of a task)
● Close all Monolog buffer handlers (clears log messages that were buffered during the
execution of a task)
● Flush all Swift Mailer "in memory" spools (i.e. send spooled e-mails)
● Flush all unsent Sentry errors (in case they are handled async)
https://github.com/LongRunning/LongRunning
''long_running.delegating_cleaner'' service

41. Alternative - Messenger
Component
https://symfony.com/doc/current/components/messenger.html

42. Message Bus
use AppMessageMyMessage;
use SymfonyComponentMessengerMessageBus;
use SymfonyComponentMessengerHandlerLocatorHandlerLocator;
use SymfonyComponentMessengerMiddlewareHandleMessageMiddleware;
$bus = new MessageBus([
new HandleMessageMiddleware(new HandlerLocator([
MyMessage::class => $handler,
])),
]);
//Default middleware: LoggingMiddleware, SendMessageMiddleware, HandleMessageMiddleware
$result = $bus->dispatch(new MyMessage(/* ... */));

43. Message Handler
use AppMessageMyMessage;
class MyMessageHandler
{
public function __invoke(MyMessage $message)
{
// Message processing...
}
}

44. Message Envelope
use SymfonyComponentMessengerEnvelope;
use SymfonyComponentMessengerTransportSerializationSerializerConfiguration;
$bus->dispatch(
(new Envelope($message))->with(new SerializerConfiguration([
'groups' => ['my_serialization_groups'],
]))
);
//Built-in envelopes: SerializerConfiguration, ValidationConfiguration, ReceivedMessage

45. Conclusions

46. Links
VichUploaderBundle - https://github.com/dustin10/VichUploaderBundle
LiipImagineBundle - https://github.com/liip/LiipImagineBundle
Gaufrette - http://knplabs.github.io/Gaufrette/
KnpGaufretteBundle - https://github.com/KnpLabs/KnpGaufretteBundle
Enqueue - https://github.com/php-enqueue/enqueue-dev
Enqueue Bundle - https://github.com/php-enqueue/enqueue-bundle
LongRunning - https://github.com/LongRunning/LongRunning
Symfony Messenger - https://symfony.com/doc/current/components/messenger.html

47. Thanks!