Process any amounts of data. Any time - PHP Benelux 2013

7,036 views
6,425 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,036
On SlideShare
0
From Embeds
0
Number of Embeds
124
Actions
Shares
0
Downloads
19
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Process any amounts of data. Any time - PHP Benelux 2013

  1. Process any amounts of data. Any time Juozas Kaziukėnas // juokaz.com // @juokaz
  2. Juozas Kaziukėnas, LithuanianYou can call me JoeMore info http://juokaz.com
  3. Last year at PHPBNL12
  4. Planking competition
  5. The problem
  6. CRON
  7. PHP developers are lazy
  8. Code should be lazy
  9. file_get_contens
  10. Buffering
  11. Wait
  12. Memory usage
  13. Saw graph
  14. Memory vsperformance
  15. Reading files
  16. Line-by-line reading$file_handle = fopen("myfile", "r");while (!feof($file_handle)) { $line = fgets($file_handle); echo $line;}fclose($file_handle);
  17. Line-by-line processing$file_handle = fopen("myfile", "r");while (!feof($file_handle)) { $line = fgets($file_handle); process($line);}fclose($file_handle);
  18. XML
  19. Read XML files
  20. $z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->name !== product);// now that were at the right depth, hop to the next <product/> until the end of the treewhile ($z->name === product){ // either one should work //$node = new SimpleXMLElement($z->readOuterXML()); $node = simplexml_import_dom($doc->importNode($z->expand(), true)); // now you can use $node without going insane about parsing var_dump($node->element_1); // go to next <product /> $z->next(product);}
  21. $z = new XMLReader;$z->open(data.xml);$doc = new DOMDocument;// move to the first <product /> nodewhile ($z->read() && $z->name !== product);// now that were at the right depth, hop to the next <product/> until the end of the treewhile ($z->name === product){ // either one should work //$node = new SimpleXMLElement($z->readOuterXML()); $node = simplexml_import_dom($doc->importNode($z->expand(), true)); // now you can use $node without going insane about parsing var_dump($node->element_1); // go to next <product /> $z->next(product);}
  22. Running processes
  23. Pipes
  24. Linuxwget http://example.com/lol.xml.gzgunzip lol.xml.gz
  25. Linuxwget -O- http://example.com/lol.xml.gz | gunzip
  26. $command = "wget -O- $url | gunzip";$process = proc_open($command, array( array("pipe","r"), array("pipe","w"), array("pipe","w") ),$pipes);while(!feof($pipes[1])) { $buffer .= fgets($pipes[1], 128); if (strpos($buffer, <Item>)) { // detected start of the item } if (strpos($buffer, </Item>)) { // detected end of the item }}fclose($pipes[1]);proc_close($process);
  27. Reading from MySQL
  28. mysql_unbuffered_query$lh = mysql_connect( server, uname, pword );$qry = "SELECT * FROM my_bigass_table";$rh = mysql_unbuffered_query( $qry, $lh );while ( $res = mysql_fetch_row( $rh ) ){ process($res);}
  29. Outputting data
  30. PHP$result = ;foreach ($data as $item) { $line = Name . $item[name] . PHP_EOL; $result .= $line;}echo $result;
  31. PHP$result = ;ob_end_clean();foreach ($data as $item) { echo Name . $item[name] . PHP_EOL; flush();}
  32. PHP 5.5
  33. Generators
  34. Yield
  35. Line-by-line processing$file_handle = fopen("myfile", "r");while (!feof($file_handle)) { $line = fgets($file_handle); process($line);}fclose($file_handle);
  36. Line-by-line processingfunction read() { $file_handle = fopen("myfile", "r"); while (!feof($file_handle)) { $line = fgets($file_handle); yield $line; } fclose($file_handle);}$data = read();foreach ($data as $line) {}
  37. Making HTTP requests
  38. Open connections limit
  39. Connection Sharing with CURL inPHP: How to re-use HTTPconnections to knock 70% off RESTnetwork time.http://technosophos.com/content/connection-sharing-curl-php-how-re-use-http-connections-knock-70-
  40. Next steps
  41. Gearman
  42. Hadoop
  43. Why PHP?
  44. What’s possible
  45. Using it for good
  46. Thanks

×