SlideShare a Scribd company logo
Iterators & Generators
Practical Uses in Memory Management
Reach out
● Software Engineer
● Systems Administrator
● All around nerd
@aramonc
Memory Management in PHP
● Garbage collected
○ Reference counts
○ Happens after variable is unset, leave function
scope, or script ends, GC buffer is full
● All values are stored in hash tables of ZVAL structs
● Arrays are dynamically allocated by doubling the
size
● Much better in PHP 7 than 5.6
What are Iterators and Generators
Iterators are:
● An object that can be iterated over using a loop
● PHP has several built in the SPL
● Can create additional ones
● Special ones are the recursive & filter iterators
Generators are:
● A special kind of custom iterator
● Keywords & interfaces that allow you to define an
Iterator without having to create a class that
implements the interface
https://github.com/aramonc/ig_talk
Generating many values: The object
<?php
namespace ArcIgTalk;
class User
{
public $_id;
public $firstName;
public $lastName;
public $email;
public $password;
public $phoneNumber;
}
Generating many values: Generate some values
<?php
...
for ($i = 0; $i < 1000000; $i++) {
$user = new User();
$user->_id = new ObjectID();
$user->email = $faker->email;
$user->password = $faker->password(12, 32);
$user->firstName = $faker->firstName();
$user->lastName = $faker->lastName;
$user->phoneNumber = $faker->phoneNumber;
$users[] = $user;
if ($i % 5000 === 0) {
$collection->insertMany($users);
$users = [];
}
}
Generating many values: Generate some values
Processing values: Paginated Provider
<?php
namespace ArcIgTalk;
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): array
{
$cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]);
$users = [];
foreach ($cursor as $data) {
$users[] = $this->createUserEntity($data->getArrayCopy());
}
return $users;
}
}
Processing values: Paginated Provider
<?php
namespace ArcIgTalk;
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): array
{
$cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]);
$users = [];
foreach ($cursor as $data) {
$users[] = $this->createUserEntity($data->getArrayCopy());
}
return $users;
}
}
Processing values: Simple approach
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 10000; $up++) {
foreach ($provider->getPagedList($up) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Simple approach
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 10000; $up++) {
foreach ($provider->getPagedList($up) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Simple approach
Processing values: Why does it take so long?
● 3 nested loops
○ for ($up = 0; $up < 10000; $up++)
○ foreach ($provider->getPagedList($up) as $user)
○ foreach ($cursor as $data)
● O(n^3) or n*m*p
Processing values: What is going on?
<?php
namespace ArcIgTalk;
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): array
{
$cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]);
$users = [];
foreach ($cursor as $data) {
$users[] = $this->createUserEntity($data->getArrayCopy());
}
return $users;
}
}
Processing values: What is going on?
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 10000; $up++) {
foreach ($provider->getPagedList($up) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Make it faster
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 1000; $up++) {
foreach ($provider->getPagedList($up, 1000) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Make it faster
Processing values: What is going on?
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 1000; $up++) {
foreach ($provider->getPagedList($up, 1000) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Make it faster
<?php
$provider = new Provider(new Client("mongodb://datastore:27017"));
$emailLogPath = __DIR__ . '/../data/user.txt';
$emailLog = fopen($emailLogPath, 'w+');
for ($up = 0; $up < 100; $up++) {
foreach ($provider->getPagedList($up, 10000) as $user) {
fwrite($emailLog, $user->email . "n");
}
}
fclose($emailLog);
Processing values: Make it faster
Processing values: Make it better
<?php
namespace ArcIgTalk;
class Provider
{
public function getPagedList(int $page = 0, int $limit = 100): Generator
{
$cursor = $this->collection->find([],[ 'limit' => $limit,'skip' => $page * $limit]);
foreach ($cursor as $data) {
/** @var $data BSONDocument */
yield $this->createUserEntity($data->getArrayCopy());
}
}
}
Processing values: Make it better
Processing values: It’s a bit slower
● 3 nested loops
○ for ($up = 0; $up < 10000; $up++)
○ foreach ($provider->getPagedList($up) as $user)
○ foreach ($cursor as $data)
● Effectively only 2
● O(n^2) or n*m
● Extra time for YIELD to go back & forth
We can do more! Let’s filter the list!
Filter values: Abstract the pagination
<?php
namespace ArcIgTalk;
class Service
{
const LIMIT = 100000;
public function getList(int $limit): Generator
{
$pages = ceil($limit / self::LIMIT);
$innerLimit = $pages > 1 ? self::LIMIT : $limit;
for ($page = 0; $page < $pages; $page++) {
foreach ($this->provider->getPagedList($page, $innerLimit) as $user) {
yield $user;
}
}
}
}
Filter values: The filter
<?php
...
function filterGmailUsers(User $user): bool
{
$emailParts = explode('@', $user->email);
return $emailParts[1] === 'gmail.com';
}
$filtered = new CallbackFilterIterator(
$service->getList(1000000),
'filterGmailUsers'
);
foreach ($filtered as $user) {
fwrite($emailLog, $user->email . "n");
}
Filter values: The filter
http://bit.ly/2tgw6z7
Questions?

More Related Content

What's hot

Working with web_services
Working with web_servicesWorking with web_services
Working with web_services
Lorna Mitchell
 
Hacking hhvm
Hacking hhvmHacking hhvm
Hacking hhvm
wajrcs
 
Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
 Connecting Content Silos: One CMS, Many Sites With The WordPress REST API Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
Caldera Labs
 
Javascript
JavascriptJavascript
Javascript
orestJump
 
Models and Service Layers, Hemoglobin and Hobgoblins
Models and Service Layers, Hemoglobin and HobgoblinsModels and Service Layers, Hemoglobin and Hobgoblins
Models and Service Layers, Hemoglobin and Hobgoblins
Ross Tuck
 
Web Apps in Perl - HTTP 101
Web Apps in Perl - HTTP 101Web Apps in Perl - HTTP 101
Web Apps in Perl - HTTP 101
hendrikvb
 
J querypractice
J querypracticeJ querypractice
J querypractice
Inbal Geffen
 
jQuery in 15 minutes
jQuery in 15 minutesjQuery in 15 minutes
jQuery in 15 minutes
Simon Willison
 
Jqeury ajax plugins
Jqeury ajax pluginsJqeury ajax plugins
Jqeury ajax plugins
Inbal Geffen
 
The promise of asynchronous php
The promise of asynchronous phpThe promise of asynchronous php
The promise of asynchronous php
Wim Godden
 
PHP 5.3 and Lithium: the most rad php framework
PHP 5.3 and Lithium: the most rad php frameworkPHP 5.3 and Lithium: the most rad php framework
PHP 5.3 and Lithium: the most rad php framework
G Woo
 
jQuery
jQueryjQuery
jQuery
Jay Poojara
 
Rails GUI Development with Ext JS
Rails GUI Development with Ext JSRails GUI Development with Ext JS
Rails GUI Development with Ext JS
Martin Rehfeld
 
Developing applications for performance
Developing applications for performanceDeveloping applications for performance
Developing applications for performance
Leon Fayer
 
Learning jQuery in 30 minutes
Learning jQuery in 30 minutesLearning jQuery in 30 minutes
Learning jQuery in 30 minutes
Simon Willison
 
Eve - REST API for Humans™
Eve - REST API for Humans™Eve - REST API for Humans™
Eve - REST API for Humans™
Nicola Iarocci
 
Php summary
Php summaryPhp summary
Php summary
Michelle Darling
 
Play á la Rails
Play á la RailsPlay á la Rails
Play á la Rails
Sebastian Nozzi
 
JQuery introduction
JQuery introductionJQuery introduction
JQuery introduction
NexThoughts Technologies
 
jQuery from the very beginning
jQuery from the very beginningjQuery from the very beginning
jQuery from the very beginning
Anis Ahmad
 

What's hot (20)

Working with web_services
Working with web_servicesWorking with web_services
Working with web_services
 
Hacking hhvm
Hacking hhvmHacking hhvm
Hacking hhvm
 
Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
 Connecting Content Silos: One CMS, Many Sites With The WordPress REST API Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
Connecting Content Silos: One CMS, Many Sites With The WordPress REST API
 
Javascript
JavascriptJavascript
Javascript
 
Models and Service Layers, Hemoglobin and Hobgoblins
Models and Service Layers, Hemoglobin and HobgoblinsModels and Service Layers, Hemoglobin and Hobgoblins
Models and Service Layers, Hemoglobin and Hobgoblins
 
Web Apps in Perl - HTTP 101
Web Apps in Perl - HTTP 101Web Apps in Perl - HTTP 101
Web Apps in Perl - HTTP 101
 
J querypractice
J querypracticeJ querypractice
J querypractice
 
jQuery in 15 minutes
jQuery in 15 minutesjQuery in 15 minutes
jQuery in 15 minutes
 
Jqeury ajax plugins
Jqeury ajax pluginsJqeury ajax plugins
Jqeury ajax plugins
 
The promise of asynchronous php
The promise of asynchronous phpThe promise of asynchronous php
The promise of asynchronous php
 
PHP 5.3 and Lithium: the most rad php framework
PHP 5.3 and Lithium: the most rad php frameworkPHP 5.3 and Lithium: the most rad php framework
PHP 5.3 and Lithium: the most rad php framework
 
jQuery
jQueryjQuery
jQuery
 
Rails GUI Development with Ext JS
Rails GUI Development with Ext JSRails GUI Development with Ext JS
Rails GUI Development with Ext JS
 
Developing applications for performance
Developing applications for performanceDeveloping applications for performance
Developing applications for performance
 
Learning jQuery in 30 minutes
Learning jQuery in 30 minutesLearning jQuery in 30 minutes
Learning jQuery in 30 minutes
 
Eve - REST API for Humans™
Eve - REST API for Humans™Eve - REST API for Humans™
Eve - REST API for Humans™
 
Php summary
Php summaryPhp summary
Php summary
 
Play á la Rails
Play á la RailsPlay á la Rails
Play á la Rails
 
JQuery introduction
JQuery introductionJQuery introduction
JQuery introduction
 
jQuery from the very beginning
jQuery from the very beginningjQuery from the very beginning
jQuery from the very beginning
 

Similar to Iterators & generators: practical uses in memory management

Doctrine For Beginners
Doctrine For BeginnersDoctrine For Beginners
Doctrine For Beginners
Jonathan Wage
 
関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい
Hisateru Tanaka
 
Electrify your code with PHP Generators
Electrify your code with PHP GeneratorsElectrify your code with PHP Generators
Electrify your code with PHP Generators
Mark Baker
 
PHP POWERPOINT SLIDES
PHP POWERPOINT SLIDESPHP POWERPOINT SLIDES
PHP POWERPOINT SLIDES
Ismail Mukiibi
 
Mojolicious
MojoliciousMojolicious
Mojolicious
Marcos Rebelo
 
How to Create Login and Registration API in PHP.pdf
How to Create Login and Registration API in PHP.pdfHow to Create Login and Registration API in PHP.pdf
How to Create Login and Registration API in PHP.pdf
Appweb Coders
 
RESTful web services
RESTful web servicesRESTful web services
RESTful web services
Tudor Constantin
 
Ruby/Rails
Ruby/RailsRuby/Rails
Ruby/Rails
rstankov
 
Php update and delet operation
Php update and delet operationPhp update and delet operation
Php update and delet operation
syeda zoya mehdi
 
Zend Framework Study@Tokyo #2
Zend Framework Study@Tokyo #2Zend Framework Study@Tokyo #2
Zend Framework Study@Tokyo #2
Shinya Ohyanagi
 
PHP 5.3 Overview
PHP 5.3 OverviewPHP 5.3 Overview
PHP 5.3 Overview
jsmith92
 
Framework
FrameworkFramework
Framework
Nguyen Linh
 
Drupal 8 migrate!
Drupal 8 migrate!Drupal 8 migrate!
Drupal 8 migrate!
Pavel Makhrinsky
 
Quality assurance for php projects with PHPStorm
Quality assurance for php projects with PHPStormQuality assurance for php projects with PHPStorm
Quality assurance for php projects with PHPStorm
Michelangelo van Dam
 
4. Php MongoDB view_data
4. Php MongoDB view_data4. Php MongoDB view_data
4. Php MongoDB view_data
Razvan Raducanu, PhD
 
Php
PhpPhp
The Zen of Lithium
The Zen of LithiumThe Zen of Lithium
The Zen of Lithium
Nate Abele
 
Mojolicious
MojoliciousMojolicious
Mojolicious
Lenz Gschwendtner
 
Web Scraping with PHP
Web Scraping with PHPWeb Scraping with PHP
Web Scraping with PHP
Matthew Turland
 
Advanced Php - Macq Electronique 2010
Advanced Php - Macq Electronique 2010Advanced Php - Macq Electronique 2010
Advanced Php - Macq Electronique 2010
Michelangelo van Dam
 

Similar to Iterators & generators: practical uses in memory management (20)

Doctrine For Beginners
Doctrine For BeginnersDoctrine For Beginners
Doctrine For Beginners
 
関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい
 
Electrify your code with PHP Generators
Electrify your code with PHP GeneratorsElectrify your code with PHP Generators
Electrify your code with PHP Generators
 
PHP POWERPOINT SLIDES
PHP POWERPOINT SLIDESPHP POWERPOINT SLIDES
PHP POWERPOINT SLIDES
 
Mojolicious
MojoliciousMojolicious
Mojolicious
 
How to Create Login and Registration API in PHP.pdf
How to Create Login and Registration API in PHP.pdfHow to Create Login and Registration API in PHP.pdf
How to Create Login and Registration API in PHP.pdf
 
RESTful web services
RESTful web servicesRESTful web services
RESTful web services
 
Ruby/Rails
Ruby/RailsRuby/Rails
Ruby/Rails
 
Php update and delet operation
Php update and delet operationPhp update and delet operation
Php update and delet operation
 
Zend Framework Study@Tokyo #2
Zend Framework Study@Tokyo #2Zend Framework Study@Tokyo #2
Zend Framework Study@Tokyo #2
 
PHP 5.3 Overview
PHP 5.3 OverviewPHP 5.3 Overview
PHP 5.3 Overview
 
Framework
FrameworkFramework
Framework
 
Drupal 8 migrate!
Drupal 8 migrate!Drupal 8 migrate!
Drupal 8 migrate!
 
Quality assurance for php projects with PHPStorm
Quality assurance for php projects with PHPStormQuality assurance for php projects with PHPStorm
Quality assurance for php projects with PHPStorm
 
4. Php MongoDB view_data
4. Php MongoDB view_data4. Php MongoDB view_data
4. Php MongoDB view_data
 
Php
PhpPhp
Php
 
The Zen of Lithium
The Zen of LithiumThe Zen of Lithium
The Zen of Lithium
 
Mojolicious
MojoliciousMojolicious
Mojolicious
 
Web Scraping with PHP
Web Scraping with PHPWeb Scraping with PHP
Web Scraping with PHP
 
Advanced Php - Macq Electronique 2010
Advanced Php - Macq Electronique 2010Advanced Php - Macq Electronique 2010
Advanced Php - Macq Electronique 2010
 

More from Adrian Cardenas

Http/2 lightning
Http/2   lightningHttp/2   lightning
Http/2 lightning
Adrian Cardenas
 
Http/2
Http/2Http/2
Creating a phar
Creating a pharCreating a phar
Creating a phar
Adrian Cardenas
 
First there was the command line
First there was the command lineFirst there was the command line
First there was the command line
Adrian Cardenas
 
Learning the command line
Learning the command lineLearning the command line
Learning the command line
Adrian Cardenas
 
Conquering the Command Line
Conquering the Command LineConquering the Command Line
Conquering the Command Line
Adrian Cardenas
 
Communicating on the web
Communicating on the webCommunicating on the web
Communicating on the web
Adrian Cardenas
 

More from Adrian Cardenas (7)

Http/2 lightning
Http/2   lightningHttp/2   lightning
Http/2 lightning
 
Http/2
Http/2Http/2
Http/2
 
Creating a phar
Creating a pharCreating a phar
Creating a phar
 
First there was the command line
First there was the command lineFirst there was the command line
First there was the command line
 
Learning the command line
Learning the command lineLearning the command line
Learning the command line
 
Conquering the Command Line
Conquering the Command LineConquering the Command Line
Conquering the Command Line
 
Communicating on the web
Communicating on the webCommunicating on the web
Communicating on the web
 

Recently uploaded

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
What is Master Data Management by PiLog Group
What is Master Data Management by PiLog GroupWhat is Master Data Management by PiLog Group
What is Master Data Management by PiLog Group
aymanquadri279
 
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
kalichargn70th171
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
TaghreedAltamimi
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Green Software Development
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
Shane Coughlan
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 

Recently uploaded (20)

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
What is Master Data Management by PiLog Group
What is Master Data Management by PiLog GroupWhat is Master Data Management by PiLog Group
What is Master Data Management by PiLog Group
 
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 

Iterators & generators: practical uses in memory management

  • 1. Iterators & Generators Practical Uses in Memory Management
  • 2. Reach out ● Software Engineer ● Systems Administrator ● All around nerd @aramonc
  • 3. Memory Management in PHP ● Garbage collected ○ Reference counts ○ Happens after variable is unset, leave function scope, or script ends, GC buffer is full ● All values are stored in hash tables of ZVAL structs ● Arrays are dynamically allocated by doubling the size ● Much better in PHP 7 than 5.6
  • 4. What are Iterators and Generators Iterators are: ● An object that can be iterated over using a loop ● PHP has several built in the SPL ● Can create additional ones ● Special ones are the recursive & filter iterators Generators are: ● A special kind of custom iterator ● Keywords & interfaces that allow you to define an Iterator without having to create a class that implements the interface
  • 6. Generating many values: The object <?php namespace ArcIgTalk; class User { public $_id; public $firstName; public $lastName; public $email; public $password; public $phoneNumber; }
  • 7. Generating many values: Generate some values <?php ... for ($i = 0; $i < 1000000; $i++) { $user = new User(); $user->_id = new ObjectID(); $user->email = $faker->email; $user->password = $faker->password(12, 32); $user->firstName = $faker->firstName(); $user->lastName = $faker->lastName; $user->phoneNumber = $faker->phoneNumber; $users[] = $user; if ($i % 5000 === 0) { $collection->insertMany($users); $users = []; } }
  • 8. Generating many values: Generate some values
  • 9. Processing values: Paginated Provider <?php namespace ArcIgTalk; class Provider { public function getPagedList(int $page = 0, int $limit = 100): array { $cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]); $users = []; foreach ($cursor as $data) { $users[] = $this->createUserEntity($data->getArrayCopy()); } return $users; } }
  • 10. Processing values: Paginated Provider <?php namespace ArcIgTalk; class Provider { public function getPagedList(int $page = 0, int $limit = 100): array { $cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]); $users = []; foreach ($cursor as $data) { $users[] = $this->createUserEntity($data->getArrayCopy()); } return $users; } }
  • 11. Processing values: Simple approach <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 10000; $up++) { foreach ($provider->getPagedList($up) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 12. Processing values: Simple approach <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 10000; $up++) { foreach ($provider->getPagedList($up) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 14. Processing values: Why does it take so long? ● 3 nested loops ○ for ($up = 0; $up < 10000; $up++) ○ foreach ($provider->getPagedList($up) as $user) ○ foreach ($cursor as $data) ● O(n^3) or n*m*p
  • 15. Processing values: What is going on? <?php namespace ArcIgTalk; class Provider { public function getPagedList(int $page = 0, int $limit = 100): array { $cursor = $this->collection->find([],['limit' => $limit,'skip' => $page * $limit]); $users = []; foreach ($cursor as $data) { $users[] = $this->createUserEntity($data->getArrayCopy()); } return $users; } }
  • 16. Processing values: What is going on? <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 10000; $up++) { foreach ($provider->getPagedList($up) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 17. Processing values: Make it faster <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 1000; $up++) { foreach ($provider->getPagedList($up, 1000) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 19. Processing values: What is going on? <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 1000; $up++) { foreach ($provider->getPagedList($up, 1000) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 20. Processing values: Make it faster <?php $provider = new Provider(new Client("mongodb://datastore:27017")); $emailLogPath = __DIR__ . '/../data/user.txt'; $emailLog = fopen($emailLogPath, 'w+'); for ($up = 0; $up < 100; $up++) { foreach ($provider->getPagedList($up, 10000) as $user) { fwrite($emailLog, $user->email . "n"); } } fclose($emailLog);
  • 22. Processing values: Make it better <?php namespace ArcIgTalk; class Provider { public function getPagedList(int $page = 0, int $limit = 100): Generator { $cursor = $this->collection->find([],[ 'limit' => $limit,'skip' => $page * $limit]); foreach ($cursor as $data) { /** @var $data BSONDocument */ yield $this->createUserEntity($data->getArrayCopy()); } } }
  • 24. Processing values: It’s a bit slower ● 3 nested loops ○ for ($up = 0; $up < 10000; $up++) ○ foreach ($provider->getPagedList($up) as $user) ○ foreach ($cursor as $data) ● Effectively only 2 ● O(n^2) or n*m ● Extra time for YIELD to go back & forth
  • 25. We can do more! Let’s filter the list!
  • 26. Filter values: Abstract the pagination <?php namespace ArcIgTalk; class Service { const LIMIT = 100000; public function getList(int $limit): Generator { $pages = ceil($limit / self::LIMIT); $innerLimit = $pages > 1 ? self::LIMIT : $limit; for ($page = 0; $page < $pages; $page++) { foreach ($this->provider->getPagedList($page, $innerLimit) as $user) { yield $user; } } } }
  • 27. Filter values: The filter <?php ... function filterGmailUsers(User $user): bool { $emailParts = explode('@', $user->email); return $emailParts[1] === 'gmail.com'; } $filtered = new CallbackFilterIterator( $service->getList(1000000), 'filterGmailUsers' ); foreach ($filtered as $user) { fwrite($emailLog, $user->email . "n"); }