Process any amount of data.Any timeJuozas Kaziukėnas // juokaz.com // @juokaz
Juozas Kaziukėnas, LithuanianYou can call me JoeMore info http://juokaz.com
The problem
Developers are lazy
Code should be lazy
file_get_contens
Buffering
Wait
Memory usage
SAW GRAPH
Memory vsperformance
Reading files
LINE-BY-LINE READING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);echo $lin...
LINE-BY-LINE PROCESSING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);proces...
XML
Read XML files
$z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->...
$z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->...
Running processes
Pipes
LINUXwget http://example.com/lol.xml.gzgunzip lol.xml.gz
LINUXwget -O- http://example.com/lol.xml.gz | gunzip
$command = "wget -O- $url | gunzip";$process = proc_open($command,array(array("pipe","r"),array("pipe","w"),array("pipe","...
READING FROM MYSQL
MYSQL_UNBUFFERED_QUERY$lh = mysql_connect(server, uname, pword);$qry = "SELECT * FROM my_bigass_table";$rh = mysql_unbuffe...
Outputting data
PHP$result = ;foreach ($data as $item) {$line = Name  . $item[name] . PHP_EOL;$result .= $line;}echo $result;
PHP$result = ;ob_end_clean();foreach ($data as $item) {echo Name  . $item[name] . PHP_EOL;flush();}
PHP 5.5
Generators
Yield
LINE-BY-LINE PROCESSING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);proces...
LINE-BY-LINE PROCESSINGfunction read() {$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($fi...
Making HTTP requests
Open connections limit
Connection Sharing with CURL inPHP: How to re-use HTTPconnections to knock 70% off RESTnetwork time.http://technosophos.co...
Next steps
Gearman
Hadoop
Why PHP?
What’s possible
Using it for good
THANKS!Juozas Kaziukėnas@juokaz
Process any amount of data any time - Web5 conference
Process any amount of data any time - Web5 conference
Process any amount of data any time - Web5 conference
Upcoming SlideShare
Loading in …5
×

Process any amount of data any time - Web5 conference

836 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
836
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Process any amount of data any time - Web5 conference

  1. 1. Process any amount of data.Any timeJuozas Kaziukėnas // juokaz.com // @juokaz
  2. 2. Juozas Kaziukėnas, LithuanianYou can call me JoeMore info http://juokaz.com
  3. 3. The problem
  4. 4. Developers are lazy
  5. 5. Code should be lazy
  6. 6. file_get_contens
  7. 7. Buffering
  8. 8. Wait
  9. 9. Memory usage
  10. 10. SAW GRAPH
  11. 11. Memory vsperformance
  12. 12. Reading files
  13. 13. LINE-BY-LINE READING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);echo $line;}fclose($file_handle);
  14. 14. LINE-BY-LINE PROCESSING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);process($line);}fclose($file_handle);
  15. 15. XML
  16. 16. Read XML files
  17. 17. $z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->name !== product);// now that were at the right depth, hop to the next <product/> until the end of the treewhile ($z->name === product){// either one should work//$node = new SimpleXMLElement($z->readOuterXML());$node = simplexml_import_dom($doc->importNode($z->expand(), true));// now you can use $node without going insane about parsingvar_dump($node->element_1);// go to next <product />$z->next(product);}
  18. 18. $z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->name !== product);// now that were at the right depth, hop to the next <product/> until the end of the treewhile ($z->name === product){// either one should work//$node = new SimpleXMLElement($z->readOuterXML());$node = simplexml_import_dom($doc->importNode($z->expand(), true));// now you can use $node without going insane about parsingvar_dump($node->element_1);// go to next <product />$z->next(product);}
  19. 19. Running processes
  20. 20. Pipes
  21. 21. LINUXwget http://example.com/lol.xml.gzgunzip lol.xml.gz
  22. 22. LINUXwget -O- http://example.com/lol.xml.gz | gunzip
  23. 23. $command = "wget -O- $url | gunzip";$process = proc_open($command,array(array("pipe","r"),array("pipe","w"),array("pipe","w")),$pipes);while(!feof($pipes[1])) {$buffer .= fgets($pipes[1], 128);if (strpos($buffer, <Item>)) {// detected start of the item}if (strpos($buffer, </Item>)) {// detected end of the item}}
  24. 24. READING FROM MYSQL
  25. 25. MYSQL_UNBUFFERED_QUERY$lh = mysql_connect(server, uname, pword);$qry = "SELECT * FROM my_bigass_table";$rh = mysql_unbuffered_query($qry, $lh);while ($res = mysql_fetch_row($rh)){process($res);}
  26. 26. Outputting data
  27. 27. PHP$result = ;foreach ($data as $item) {$line = Name . $item[name] . PHP_EOL;$result .= $line;}echo $result;
  28. 28. PHP$result = ;ob_end_clean();foreach ($data as $item) {echo Name . $item[name] . PHP_EOL;flush();}
  29. 29. PHP 5.5
  30. 30. Generators
  31. 31. Yield
  32. 32. LINE-BY-LINE PROCESSING$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);process($line);}fclose($file_handle);
  33. 33. LINE-BY-LINE PROCESSINGfunction read() {$file_handle = fopen("myfile", "r");while (!feof($file_handle)) {$line = fgets($file_handle);yield $line;}fclose($file_handle);}$data = read();
  34. 34. Making HTTP requests
  35. 35. Open connections limit
  36. 36. Connection Sharing with CURL inPHP: How to re-use HTTPconnections to knock 70% off RESTnetwork time.http://technosophos.com/content/connection-sharing-curl-php-how-re-use-http-connections-knock-70-rest-network-time
  37. 37. Next steps
  38. 38. Gearman
  39. 39. Hadoop
  40. 40. Why PHP?
  41. 41. What’s possible
  42. 42. Using it for good
  43. 43. THANKS!Juozas Kaziukėnas@juokaz

×