SlideShare a Scribd company logo
smoke
Friday, 1 May 2009

Hi!
smoke
                Ben Schwarz
                @benschwarz
                http://github.com/benschwarz



Friday, 1 May 2009

Hi!
Lunch time
Friday, 1 May 2009

Starting with a story... It was lunch time.
2pm
Friday, 1 May 2009

It was well and truly past eating time
1 Hour
Friday, 1 May 2009

I had one hour. What do you do?
Code lunch

Friday, 1 May 2009

No need for eating. What will I do?
Sinatra
                       http://sinatrarb.com




Friday, 1 May 2009

We use Sinatra!
Yahoo Pipes
                             http://pipes.yahoo.com




Friday, 1 May 2009

Annd data from other peoples websites. No RDBMS for me!
Yahoo Pipes
                     •   Get data from multiple
                         web based sources

                     •   Mash it up a bit

                     •   Provide a single output
                         in either JSON or XML




Friday, 1 May 2009
YQL
Friday, 1 May 2009
YQL?
Friday, 1 May 2009

Yeah, its kinda SQL for the web. Most people don’t even really like writing SQL
@lachlanhardy


Friday, 1 May 2009

Lachlan convinced me to have a second look at YQL after slagging it o
Time to start

Friday, 1 May 2009
Friday, 1 May 2009

The code isn’t important, but basically it just talks to a yahoo pipe that I made up.
Friday, 1 May 2009

I made “munch”. It pulls from various recipe and cooking related web sites like taste.com.au
and friends. It reads RSS and scrapes sites. Pretty simple.
2:46pm
Friday, 1 May 2009

46 six minutes later. I’d been eected.
“I need a new
                        web site”
                        - Me, probably avoiding real work




Friday, 1 May 2009

I wanted to blog about this, sadly my blog is cached files sitting on a slicehost account.
Mephisto isn’t even running any longer.
Friday, 1 May 2009

I quickly threw a grid together.
HTML5
Friday, 1 May 2009

I wanted to experiment.
Sinatra
Friday, 1 May 2009

a bit more sinatra.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
?



Friday, 1 May 2009

I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.
Friday, 1 May 2009

of course, I could’ve just got a tumblr account. That didn’t let me use sinatra or mess around
with HTML5.
Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
Delicious




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                     Twitter




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                                 When I wasn’t talking to @someone
                     Twitter




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                                 When I wasn’t talking to @someone
                     Twitter

                      Flickr




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                                 When I wasn’t talking to @someone
                     Twitter

                                 Selected (special tag)
                      Flickr




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                                 When I wasn’t talking to @someone
                     Twitter

                                 Selected (special tag)
                      Flickr

                         ?




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
All
                     Delicious

                                 When I wasn’t talking to @someone
                     Twitter

                                 Selected (special tag)
                      Flickr

                         ?

                         ?




Friday, 1 May 2009

It was getting easier, I had constraints around the use of my data
Friday, 1 May 2009

The yahoo pipe looked like this. A bit of a mental leap.
Friday, 1 May 2009

Some things stood out. like union, I could join multiple sources of data together and output
them as one
Friday, 1 May 2009

or sort them by the publication date
Friday, 1 May 2009

and filter out the tweets to people. Sure a regex would’ve been smarter, but this was 90%
good enough
Friday, 1 May 2009

I started thinking about my process, the data coming from all over the place.
“What if I could
                     write something
                       like pipes?”

Friday, 1 May 2009

I wanted to do something myself (again)
“What if I could
                 write a ruby block
                   based DSL?”


Friday, 1 May 2009

I’d never written one, totally a good time to jump right in head first.
“What if you got
                      plain old Ruby
                         objects?”

Friday, 1 May 2009

If the data returned could be ruby, irrespective of what format it started as... it would be
awesome
Or JSON
Friday, 1 May 2009
Friday, 1 May 2009

Enter, “Smoke.”
Smoke
Friday, 1 May 2009

Enter, “Smoke.”
Friday, 1 May 2009

A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”
Friday, 1 May 2009

A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”
Friday, 1 May 2009

A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”
Friday, 1 May 2009

A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better
Friday, 1 May 2009

The whole point of this was to bring these multiple feeds together, re interpret them as one
common feed.
Friday, 1 May 2009

This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required
Creates




Friday, 1 May 2009

This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required
Creates




Friday, 1 May 2009

This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That can be extended and the items from the two sources can have some transformations as
well.
Friday, 1 May 2009

That accessor
Friday, 1 May 2009

Get the first item output (output is an array of the items from the source)
Friday, 1 May 2009

It gives us this
What just happened?




Friday, 1 May 2009

Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.
What just happened?
                     • Queried the web




Friday, 1 May 2009

Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.
What just happened?
                     • Queried the web
                     • Auto-magically parsed the data




Friday, 1 May 2009

Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.
What just happened?
                     • Queried the web
                     • Auto-magically parsed the data
                     • Returned a clean object



Friday, 1 May 2009

Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.
Friday, 1 May 2009

We can also output JSON
Friday, 1 May 2009

Its string escaped and not worth reading, but you could pass this through a web based app
straight down to javascript
Method
                     chaining
Friday, 1 May 2009

Methods can also be chained without invoking a new source or joining sources together
Friday, 1 May 2009

Here we can get a predefined source, shorten the result set and re sort it.
Friday, 1 May 2009

Here we can get a predefined source, shorten the result set and re sort it.
Friday, 1 May 2009

Here we can get a predefined source, shorten the result set and re sort it.
Transformations
                                     Amount of items
                     Truncate

                                     By a single key
                       Sort

                                     One key to another
                     Rename

                                     Items that match regex
                      Keep

                                     Items that match regex
                     Discard

                                     Ruby (default) or JSON
                     Output



Friday, 1 May 2009

Right now, you can do these things
YQL
                     Definitions

Friday, 1 May 2009

I’m looking at adding support for YQL definitions
Friday, 1 May 2009

They look like this, I’ve never written one, but it doesn’t look too hard. You can add search to
your own preexisting site / rss etc
http://github.com/spullara/yql-tables




Friday, 1 May 2009

There are a collection of community provided yql definitions hosted on github
All
                        these
                      fucking
                     web sites


Friday, 1 May 2009

Need I say more?
My message

Friday, 1 May 2009

Forgetting all the stu about pipes, yql and smoke, I want you to try something
“Re-interpret
                       the web”

Friday, 1 May 2009

Reconsider data, re-represent it to your own needs.
Thanks


Friday, 1 May 2009
Thanks
                     github.com/benschwarz/smoke
                     http://github.com/benschwarz/munch
                     http://sinatrarb.com
                     http://pipes.yahoo.com/pipes/pipe.edit?_id=fHK7LuUW3hGZHgSfbbsjiw
                     http://twitter.com/lachlanhardy
                     http://pipes.yahoo.com/pipes
                     http://developer.yahoo.com/yql
                     http://github.com/spullara/yql-tables
                     http://www.flickr.com/photos/95579828@N00/71240905
                     http://www.flickr.com/photos/hockadilly/2910737768
                     http://www.flickr.com/photos/maxblack/2655837607




Friday, 1 May 2009

More Related Content

Similar to Smoke

iTunes for Scholarly Publishing?
iTunes for Scholarly Publishing?iTunes for Scholarly Publishing?
iTunes for Scholarly Publishing?
Crossref
 
Tab Digi Development #1: listening & talking
Tab Digi Development #1: listening & talkingTab Digi Development #1: listening & talking
Tab Digi Development #1: listening & talkingHannah Rudman
 
Tumblr XHack2012
Tumblr XHack2012Tumblr XHack2012
Tumblr XHack2012xhack
 
Using Social Media in Professional Development
Using Social Media in Professional DevelopmentUsing Social Media in Professional Development
Using Social Media in Professional Development
Stratepedia Presentations
 
Write That Down! Van Wilder's Guide to Social Media
Write That Down!  Van Wilder's Guide to Social MediaWrite That Down!  Van Wilder's Guide to Social Media
Write That Down! Van Wilder's Guide to Social Media
Erica OGrady
 
Escape The Lab Tri Upa Slides
Escape The Lab Tri Upa SlidesEscape The Lab Tri Upa Slides
Escape The Lab Tri Upa Slides
bolt peters
 

Similar to Smoke (9)

iTunes for Scholarly Publishing?
iTunes for Scholarly Publishing?iTunes for Scholarly Publishing?
iTunes for Scholarly Publishing?
 
Session6
Session6Session6
Session6
 
Tab Digi Development #1: listening & talking
Tab Digi Development #1: listening & talkingTab Digi Development #1: listening & talking
Tab Digi Development #1: listening & talking
 
Tumblr XHack2012
Tumblr XHack2012Tumblr XHack2012
Tumblr XHack2012
 
Session6
Session6Session6
Session6
 
Using Social Media in Professional Development
Using Social Media in Professional DevelopmentUsing Social Media in Professional Development
Using Social Media in Professional Development
 
Session 7
Session 7 Session 7
Session 7
 
Write That Down! Van Wilder's Guide to Social Media
Write That Down!  Van Wilder's Guide to Social MediaWrite That Down!  Van Wilder's Guide to Social Media
Write That Down! Van Wilder's Guide to Social Media
 
Escape The Lab Tri Upa Slides
Escape The Lab Tri Upa SlidesEscape The Lab Tri Upa Slides
Escape The Lab Tri Upa Slides
 

Recently uploaded

BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdfBENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
DR SETH JOTHAM
 
Superficial & Deep Fascia of the NECK.pptx
Superficial & Deep Fascia of the NECK.pptxSuperficial & Deep Fascia of the NECK.pptx
Superficial & Deep Fascia of the NECK.pptx
Dr. Rabia Inam Gandapore
 
micro teaching on communication m.sc nursing.pdf
micro teaching on communication m.sc nursing.pdfmicro teaching on communication m.sc nursing.pdf
micro teaching on communication m.sc nursing.pdf
Anurag Sharma
 
Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...
Sujoy Dasgupta
 
ARTIFICIAL INTELLIGENCE IN HEALTHCARE.pdf
ARTIFICIAL INTELLIGENCE IN  HEALTHCARE.pdfARTIFICIAL INTELLIGENCE IN  HEALTHCARE.pdf
ARTIFICIAL INTELLIGENCE IN HEALTHCARE.pdf
Anujkumaranit
 
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIONDACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
DR SETH JOTHAM
 
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
VarunMahajani
 
The Normal Electrocardiogram - Part I of II
The Normal Electrocardiogram - Part I of IIThe Normal Electrocardiogram - Part I of II
The Normal Electrocardiogram - Part I of II
MedicoseAcademics
 
New Drug Discovery and Development .....
New Drug Discovery and Development .....New Drug Discovery and Development .....
New Drug Discovery and Development .....
NEHA GUPTA
 
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness JourneyTom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
greendigital
 
Physiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdfPhysiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdf
MedicoseAcademics
 
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
i3 Health
 
Are There Any Natural Remedies To Treat Syphilis.pdf
Are There Any Natural Remedies To Treat Syphilis.pdfAre There Any Natural Remedies To Treat Syphilis.pdf
Are There Any Natural Remedies To Treat Syphilis.pdf
Little Cross Family Clinic
 
Flu Vaccine Alert in Bangalore Karnataka
Flu Vaccine Alert in Bangalore KarnatakaFlu Vaccine Alert in Bangalore Karnataka
Flu Vaccine Alert in Bangalore Karnataka
addon Scans
 
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptxMaxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Dr. Rabia Inam Gandapore
 
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists  Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
Saeid Safari
 
heat stroke and heat exhaustion in children
heat stroke and heat exhaustion in childrenheat stroke and heat exhaustion in children
heat stroke and heat exhaustion in children
SumeraAhmad5
 
Knee anatomy and clinical tests 2024.pdf
Knee anatomy and clinical tests 2024.pdfKnee anatomy and clinical tests 2024.pdf
Knee anatomy and clinical tests 2024.pdf
vimalpl1234
 
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
GL Anaacs
 
Prix Galien International 2024 Forum Program
Prix Galien International 2024 Forum ProgramPrix Galien International 2024 Forum Program
Prix Galien International 2024 Forum Program
Levi Shapiro
 

Recently uploaded (20)

BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdfBENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
BENIGN PROSTATIC HYPERPLASIA.BPH. BPHpdf
 
Superficial & Deep Fascia of the NECK.pptx
Superficial & Deep Fascia of the NECK.pptxSuperficial & Deep Fascia of the NECK.pptx
Superficial & Deep Fascia of the NECK.pptx
 
micro teaching on communication m.sc nursing.pdf
micro teaching on communication m.sc nursing.pdfmicro teaching on communication m.sc nursing.pdf
micro teaching on communication m.sc nursing.pdf
 
Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...Couples presenting to the infertility clinic- Do they really have infertility...
Couples presenting to the infertility clinic- Do they really have infertility...
 
ARTIFICIAL INTELLIGENCE IN HEALTHCARE.pdf
ARTIFICIAL INTELLIGENCE IN  HEALTHCARE.pdfARTIFICIAL INTELLIGENCE IN  HEALTHCARE.pdf
ARTIFICIAL INTELLIGENCE IN HEALTHCARE.pdf
 
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIONDACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
ACUTE SCROTUM.....pdf. ACUTE SCROTAL CONDITIOND
 
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
Pulmonary Thromboembolism - etilogy, types, medical- Surgical and nursing man...
 
The Normal Electrocardiogram - Part I of II
The Normal Electrocardiogram - Part I of IIThe Normal Electrocardiogram - Part I of II
The Normal Electrocardiogram - Part I of II
 
New Drug Discovery and Development .....
New Drug Discovery and Development .....New Drug Discovery and Development .....
New Drug Discovery and Development .....
 
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness JourneyTom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
Tom Selleck Health: A Comprehensive Look at the Iconic Actor’s Wellness Journey
 
Physiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdfPhysiology of Chemical Sensation of smell.pdf
Physiology of Chemical Sensation of smell.pdf
 
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
New Directions in Targeted Therapeutic Approaches for Older Adults With Mantl...
 
Are There Any Natural Remedies To Treat Syphilis.pdf
Are There Any Natural Remedies To Treat Syphilis.pdfAre There Any Natural Remedies To Treat Syphilis.pdf
Are There Any Natural Remedies To Treat Syphilis.pdf
 
Flu Vaccine Alert in Bangalore Karnataka
Flu Vaccine Alert in Bangalore KarnatakaFlu Vaccine Alert in Bangalore Karnataka
Flu Vaccine Alert in Bangalore Karnataka
 
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptxMaxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
Maxilla, Mandible & Hyoid Bone & Clinical Correlations by Dr. RIG.pptx
 
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists  Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
Ozempic: Preoperative Management of Patients on GLP-1 Receptor Agonists
 
heat stroke and heat exhaustion in children
heat stroke and heat exhaustion in childrenheat stroke and heat exhaustion in children
heat stroke and heat exhaustion in children
 
Knee anatomy and clinical tests 2024.pdf
Knee anatomy and clinical tests 2024.pdfKnee anatomy and clinical tests 2024.pdf
Knee anatomy and clinical tests 2024.pdf
 
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
HOT NEW PRODUCT! BIG SALES FAST SHIPPING NOW FROM CHINA!! EU KU DB BK substit...
 
Prix Galien International 2024 Forum Program
Prix Galien International 2024 Forum ProgramPrix Galien International 2024 Forum Program
Prix Galien International 2024 Forum Program
 

Smoke

  • 2. smoke Ben Schwarz @benschwarz http://github.com/benschwarz Friday, 1 May 2009 Hi!
  • 3. Lunch time Friday, 1 May 2009 Starting with a story... It was lunch time.
  • 4. 2pm Friday, 1 May 2009 It was well and truly past eating time
  • 5. 1 Hour Friday, 1 May 2009 I had one hour. What do you do?
  • 6. Code lunch Friday, 1 May 2009 No need for eating. What will I do?
  • 7. Sinatra http://sinatrarb.com Friday, 1 May 2009 We use Sinatra!
  • 8. Yahoo Pipes http://pipes.yahoo.com Friday, 1 May 2009 Annd data from other peoples websites. No RDBMS for me!
  • 9. Yahoo Pipes • Get data from multiple web based sources • Mash it up a bit • Provide a single output in either JSON or XML Friday, 1 May 2009
  • 11. YQL? Friday, 1 May 2009 Yeah, its kinda SQL for the web. Most people don’t even really like writing SQL
  • 12. @lachlanhardy Friday, 1 May 2009 Lachlan convinced me to have a second look at YQL after slagging it o
  • 13. Time to start Friday, 1 May 2009
  • 14. Friday, 1 May 2009 The code isn’t important, but basically it just talks to a yahoo pipe that I made up.
  • 15. Friday, 1 May 2009 I made “munch”. It pulls from various recipe and cooking related web sites like taste.com.au and friends. It reads RSS and scrapes sites. Pretty simple.
  • 16. 2:46pm Friday, 1 May 2009 46 six minutes later. I’d been eected.
  • 17. “I need a new web site” - Me, probably avoiding real work Friday, 1 May 2009 I wanted to blog about this, sadly my blog is cached files sitting on a slicehost account. Mephisto isn’t even running any longer.
  • 18. Friday, 1 May 2009 I quickly threw a grid together.
  • 19. HTML5 Friday, 1 May 2009 I wanted to experiment.
  • 20. Sinatra Friday, 1 May 2009 a bit more sinatra.
  • 21. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 22. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 23. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 24. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 25. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 26. Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 27. ? Friday, 1 May 2009 I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it once and be able to keep it updated.
  • 28. Friday, 1 May 2009 of course, I could’ve just got a tumblr account. That didn’t let me use sinatra or mess around with HTML5.
  • 29. Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 30. Delicious Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 31. All Delicious Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 32. All Delicious Twitter Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 33. All Delicious When I wasn’t talking to @someone Twitter Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 34. All Delicious When I wasn’t talking to @someone Twitter Flickr Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 35. All Delicious When I wasn’t talking to @someone Twitter Selected (special tag) Flickr Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 36. All Delicious When I wasn’t talking to @someone Twitter Selected (special tag) Flickr ? Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 37. All Delicious When I wasn’t talking to @someone Twitter Selected (special tag) Flickr ? ? Friday, 1 May 2009 It was getting easier, I had constraints around the use of my data
  • 38. Friday, 1 May 2009 The yahoo pipe looked like this. A bit of a mental leap.
  • 39. Friday, 1 May 2009 Some things stood out. like union, I could join multiple sources of data together and output them as one
  • 40. Friday, 1 May 2009 or sort them by the publication date
  • 41. Friday, 1 May 2009 and filter out the tweets to people. Sure a regex would’ve been smarter, but this was 90% good enough
  • 42. Friday, 1 May 2009 I started thinking about my process, the data coming from all over the place.
  • 43. “What if I could write something like pipes?” Friday, 1 May 2009 I wanted to do something myself (again)
  • 44. “What if I could write a ruby block based DSL?” Friday, 1 May 2009 I’d never written one, totally a good time to jump right in head first.
  • 45. “What if you got plain old Ruby objects?” Friday, 1 May 2009 If the data returned could be ruby, irrespective of what format it started as... it would be awesome
  • 46. Or JSON Friday, 1 May 2009
  • 47. Friday, 1 May 2009 Enter, “Smoke.”
  • 48. Smoke Friday, 1 May 2009 Enter, “Smoke.”
  • 49. Friday, 1 May 2009 A search, which I named “ruby” uses yql and does a web based (search engine) search for the keyword of “ruby”
  • 50. Friday, 1 May 2009 A search, which I named “ruby” uses yql and does a web based (search engine) search for the keyword of “ruby”
  • 51. Friday, 1 May 2009 A search, which I named “ruby” uses yql and does a web based (search engine) search for the keyword of “ruby”
  • 52. Friday, 1 May 2009 A search, which I named “ruby” uses yql and does a web based (search engine) search for the keyword of “ruby”
  • 53. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 54. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 55. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 56. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 57. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 58. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 59. Friday, 1 May 2009 and hey, I’m gonna grab the RSS feed from slashdot. “emit” will transform the objects that my feed returns. I’ll rename any keys of link to url, this will make my data mind-map better
  • 60. Friday, 1 May 2009 The whole point of this was to bring these multiple feeds together, re interpret them as one common feed.
  • 61. Friday, 1 May 2009 This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if required
  • 62. Creates Friday, 1 May 2009 This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if required
  • 63. Creates Friday, 1 May 2009 This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if required
  • 64. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 65. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 66. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 67. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 68. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 69. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 70. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 71. Friday, 1 May 2009 That can be extended and the items from the two sources can have some transformations as well.
  • 72. Friday, 1 May 2009 That accessor
  • 73. Friday, 1 May 2009 Get the first item output (output is an array of the items from the source)
  • 74. Friday, 1 May 2009 It gives us this
  • 75. What just happened? Friday, 1 May 2009 Thinking a little more about what actually just happened. It parsed the data to a ruby object, irrespective of the format it came in as.
  • 76. What just happened? • Queried the web Friday, 1 May 2009 Thinking a little more about what actually just happened. It parsed the data to a ruby object, irrespective of the format it came in as.
  • 77. What just happened? • Queried the web • Auto-magically parsed the data Friday, 1 May 2009 Thinking a little more about what actually just happened. It parsed the data to a ruby object, irrespective of the format it came in as.
  • 78. What just happened? • Queried the web • Auto-magically parsed the data • Returned a clean object Friday, 1 May 2009 Thinking a little more about what actually just happened. It parsed the data to a ruby object, irrespective of the format it came in as.
  • 79. Friday, 1 May 2009 We can also output JSON
  • 80. Friday, 1 May 2009 Its string escaped and not worth reading, but you could pass this through a web based app straight down to javascript
  • 81. Method chaining Friday, 1 May 2009 Methods can also be chained without invoking a new source or joining sources together
  • 82. Friday, 1 May 2009 Here we can get a predefined source, shorten the result set and re sort it.
  • 83. Friday, 1 May 2009 Here we can get a predefined source, shorten the result set and re sort it.
  • 84. Friday, 1 May 2009 Here we can get a predefined source, shorten the result set and re sort it.
  • 85. Transformations Amount of items Truncate By a single key Sort One key to another Rename Items that match regex Keep Items that match regex Discard Ruby (default) or JSON Output Friday, 1 May 2009 Right now, you can do these things
  • 86. YQL Definitions Friday, 1 May 2009 I’m looking at adding support for YQL definitions
  • 87. Friday, 1 May 2009 They look like this, I’ve never written one, but it doesn’t look too hard. You can add search to your own preexisting site / rss etc
  • 88. http://github.com/spullara/yql-tables Friday, 1 May 2009 There are a collection of community provided yql definitions hosted on github
  • 89. All these fucking web sites Friday, 1 May 2009 Need I say more?
  • 90. My message Friday, 1 May 2009 Forgetting all the stu about pipes, yql and smoke, I want you to try something
  • 91. “Re-interpret the web” Friday, 1 May 2009 Reconsider data, re-represent it to your own needs.
  • 93. Thanks github.com/benschwarz/smoke http://github.com/benschwarz/munch http://sinatrarb.com http://pipes.yahoo.com/pipes/pipe.edit?_id=fHK7LuUW3hGZHgSfbbsjiw http://twitter.com/lachlanhardy http://pipes.yahoo.com/pipes http://developer.yahoo.com/yql http://github.com/spullara/yql-tables http://www.flickr.com/photos/95579828@N00/71240905 http://www.flickr.com/photos/hockadilly/2910737768 http://www.flickr.com/photos/maxblack/2655837607 Friday, 1 May 2009