Making Email a Useful Web App Web 2.0 Conference April 23, 2008 San Francisco Andy Denmark TripIt, Co-Founder and VP Engineering

Web 2.0 Conference

April 23, 2008

San Francisco

Andy Denmark

TripIt, Co-Founder and VP Engineering

Email Bankruptcy “ I routinely declare email bankruptcy and simply delete my entire inbox.” - Michael Arrington, TechCrunch “ Wondering if I can get a Bear Sterns bailout for my email bankruptcy.” - willpate on Twitter “ (He) just declared email bankruptcy. I've read those declarations enough to know it means our email tools aren't usable.” - garrickvanburen on Twitter So, why is email still around despite all the challenges and criticism… ?

Email Is Still The “Killer App” Email is a natural extension of a metaphor for communication that humans have used forever Everyone has email and the overwhelming majority of all person to person Internet communication takes place there No matter what else people use to communicate, they all use email too Email contains an amazing amount of context and data about our lives Our relationships Our purchases Our thoughts and the thoughts of people we know

Email is a natural extension of a metaphor for communication that humans have used forever

Everyone has email and the overwhelming majority of all person to person Internet communication takes place there

No matter what else people use to communicate, they all use email too

Email contains an amazing amount of context and data about our lives

Our relationships

Our purchases

Our thoughts and the thoughts of people we know

What More Can We Do With Email? Spam and mass email marketing may be contributing to the problems outlined earlier But think of all the things we can do to make our applications better by making better use of email: Broadcasting data Query interfaces Data transfer Intelligent parsing of data Mining of social connections (“Inbox 2.0”)

Spam and mass email marketing may be contributing to the problems outlined earlier

But think of all the things we can do to make our applications better by making better use of email:

Broadcasting data

Query interfaces

Data transfer

Intelligent parsing of data

Mining of social connections (“Inbox 2.0”)

Does An Email App Make Sense? PROS Ubiquity – everyone has email Exposure – email is on all day and captures a lot of attention Simplicity – easy to send email, everyone knows how to do it Embedded identity CONS May just not fit your service May be too complex for your users to understand It’s challenging to properly support all email clients, versions and formats Building and supporting a system that sends, receives, and reads email is a pain

PROS

Ubiquity – everyone has email

Exposure – email is on all day and captures a lot of attention

Simplicity – easy to send email, everyone knows how to do it

Embedded identity

CONS

May just not fit your service

May be too complex for your users to understand

It’s challenging to properly support all email clients, versions and formats

Building and supporting a system that sends, receives, and reads email is a pain

Prehistoric Email Apps

Domain Registration Email Sign-Up Email to: [email_address] From address authentication for new sign-ups Choose to receive your bill via email or USPS Authentication schemes for modify/delete were CRYPT-PW and PGP

Email to: [email_address]

From address authentication for new sign-ups

Choose to receive your bill via email or USPS

Authentication schemes for modify/delete were CRYPT-PW and PGP

Archie File Query Email Interface Available Interfaces Text-based client Graphical client Telnet client Email

Available Interfaces

Text-based client

Graphical client

Telnet client

Email

Browsing The Web Via Email “ For personal reasons, I do not browse the web from my computer. (I also have no net connection much of the time.) To look at page I send mail to a demon which runs wget and mails the page back to me. It is very efficient use of my time, but it is slow in real time.” -- Richard Stallman (2007-12-15)

About TripIt ?

***CONFIDENTIAL***

Forward Any Confirmation Email Automated Online Master Itinerary Automated Weather, Maps, Guides… Trip Sharing & Social Alerts iCal & Atom Support Email & Mobile Access Add Meetings, Activities, Restaurants, etc.

Why Email Made Sense For TripIt The native format of travel data is varied and unfriendly The travel industry isn’t moving to expose that data Partnering with big travel companies and building APIs would take too long Users already have all their travel data in email, but that data is “dead” Auto-account creation using the embedded identity of an email message It has become an “instant delight” factor for users

The native format of travel data is varied and unfriendly

The travel industry isn’t moving to expose that data

Partnering with big travel companies and building APIs would take too long

Users already have all their travel data in email, but that data is “dead”

Auto-account creation using the embedded identity of an email message

It has become an “instant delight” factor for users

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Broadcast ?

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Broadcast One Way User requests email information to be sent to them Typically triggered by an action or an alert Examples Mailing Lists Google Alerts

One Way

User requests email information to be sent to them

Typically triggered by an action or an alert

Examples

Mailing Lists

Google Alerts

Example: Mailing Lists Searchable knowledge base built as a by-product of natural human communication

Example: Google Alerts “ Real time” alerts delivered right to my desktop Useful way to asynchronously broadcast data I’ve requested as it’s available With linkage to the web, alerts are actionable and controllable

“ Real time” alerts delivered right to my desktop

Useful way to asynchronously broadcast data I’ve requested as it’s available

With linkage to the web, alerts are actionable and controllable

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Broadcast CLI Users emails a command and Email App replies with a response Typically a standard list of commands and responses Examples I Want Sandy TripIt Twitter

CLI

Users emails a command and Email App replies with a response

Typically a standard list of commands and responses

Examples

I Want Sandy

TripIt

Twitter

Example: I Want Sandy http://iwantsandy.com/

Example: TripIt

Example: Twitter Messaging platform with multiple UIs Web (http://twitter.com/) SMS (40404) IM (Jabber: twitter@twitter.com) Facebook (http://www.facebook.com/) FriendFeed ( http://friendfeed.com/ ) Desktop (e.g. Twhirl) Email I can tweet by email?!? Yes, it’s just another way to use the messaging platform’s CLI Very natural, ubiquitous and free (no SMS charges)! See: http://emailtwitter.com/ and http://twittermail.com/

Messaging platform with multiple UIs

Web (http://twitter.com/)

SMS (40404)

IM (Jabber: twitter@twitter.com)

Facebook (http://www.facebook.com/)

FriendFeed ( http://friendfeed.com/ )

Desktop (e.g. Twhirl)

Email

I can tweet by email?!?

Yes, it’s just another way to use the messaging platform’s CLI

Very natural, ubiquitous and free (no SMS charges)!

See: http://emailtwitter.com/ and http://twittermail.com/

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Email Forwarding Users forward emails and Email App recognizes and parses them Typically converts “unstructured” data into “structured” data making it more usable to the app and user Examples Flickr Blogger Broadcast

Email Forwarding

Users forward emails and Email App recognizes and parses them

Typically converts “unstructured” data into “structured” data making it more usable to the app and user

Examples

Flickr

Blogger

Example: Flickr

Example: Blogger

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Broadcast Intelligent Agent Structured data from email is enriched by the Email App by retrieving related information from other sites Typically leverages “keys” such as geo-coding to capture relevant data Examples TrackMyShipments TripIt

Intelligent Agent

Structured data from email is enriched by the Email App by retrieving related information from other sites

Typically leverages “keys” such as geo-coding to capture relevant data

Examples

TrackMyShipments

TripIt

Example: TrackMyShipments

Example: TripIt Timezone Online Checkin Flight Status Seat Advice Flight Details Weather

Types Of Email App Services Command Line Interface Data Transfer Semantic Parsing Email Forwarding Users forward emails and Email App recognizes and parses them Typically converts “unstructured” data into “structured” data making it more usable to the app and user Examples Flickr Blogger Broadcast CLI Users emails a command and Email App replies with a response Typically a standard list of commands and responses Examples I Want Sandy TripIt Twitter One Way User requests email information to be sent to them Typically triggered by an action or an alert Examples Listserv Google Alerts Intelligent Agent Structured data from email is enriched by the Email App by retrieving related information from other sites Typically leverages “keys” such as geo-coding to capture relevant data Examples TrackMyShipments TripIt

Email Forwarding

Users forward emails and Email App recognizes and parses them

Typically converts “unstructured” data into “structured” data making it more usable to the app and user

Examples

Flickr

Blogger

CLI

Users emails a command and Email App replies with a response

Typically a standard list of commands and responses

Examples

I Want Sandy

TripIt

Twitter

One Way

User requests email information to be sent to them

Typically triggered by an action or an alert

Examples

Listserv

Google Alerts

Intelligent Agent

Structured data from email is enriched by the Email App by retrieving related information from other sites

Typically leverages “keys” such as geo-coding to capture relevant data

Examples

TrackMyShipments

TripIt

Caution: Email Is Full Of Problems CAUTION STANDARDS CAUTION CAUTION SPAM CAUTION CAUTION PRIVACY CAUTION CAUTION DELIVERABILITY CAUTION

Deliverability Issues Your reputation is attached to your IPs Monitor your logs, it’s a dynamic system Work with the ESPs, they are as much invested in this as you are Even if you're a good citizen, there are no guarantees email will get through Tips for getting email delivered… No Open Relays Proper DNS Configuration (forward + reverse resolution) SPF/DKIM Privacy Policy RFC Compliance ESP “Whitelists” Bounce Handling CAUTION DELIVERABILITY CAUTION

Your reputation is attached to your IPs

Monitor your logs, it’s a dynamic system

Work with the ESPs, they are as much invested in this as you are

Even if you're a good citizen, there are no guarantees email will get through

Tips for getting email delivered…

No Open Relays

Proper DNS Configuration (forward + reverse resolution)

SPF/DKIM

Privacy Policy

RFC Compliance

ESP “Whitelists”

Bounce Handling

Spam Issues No silver bullet solutions, but a suite of counter-measures… Spam assessment tools Spam Firewalls (Hardware/Software) ‏ Improper/forged headers are key factor Non-RFC Compliant Mailers (don't retry, don't try all your MXs, etc.) ‏ DHCP/home IPs w/ no reverse DNS Blacklists (be very careful) ‏ But all of these cause you to run the risk of catching legitimate email in the net Spammers like to go fast, so slowing things down in an RFC compliant way can help CAUTION SPAM CAUTION

No silver bullet solutions, but a suite of counter-measures…

Spam assessment tools

Spam Firewalls (Hardware/Software) ‏

Improper/forged headers are key factor

Non-RFC Compliant Mailers (don't retry, don't try all your MXs, etc.) ‏

DHCP/home IPs w/ no reverse DNS

Blacklists (be very careful) ‏

But all of these cause you to run the risk of catching legitimate email in the net

Spammers like to go fast, so slowing things down in an RFC compliant way can help

Privacy Issues People are very protective of their Inbox, it’s their private space Laws vary wildly but there’s a baseline in common sense Make sure people can opt-out/in easily Build mechanisms that prevent you from contacting people who don’t wish to be contacted Listen to your communities, you’ll get some of this wrong, but you can fix it Be transparent and responsive CAUTION PRIVACY CAUTION

People are very protective of their Inbox, it’s their private space

Laws vary wildly but there’s a baseline in common sense

Make sure people can opt-out/in easily

Build mechanisms that prevent you from contacting people who don’t wish to be contacted

Listen to your communities, you’ll get some of this wrong, but you can fix it

Be transparent and responsive

Standards Issues There are no standards that are strictly followed across mail clients Pretty much every mail-related RFC has been outright violated or creatively interpreted by at least one major mail client White space, HTML markup, attachments are all mangled and re-interpreted depending on which mail client is working on the message If you build a mail application that reads email you will have to deal with this, it’s expensive and painful CAUTION STANDARDS CAUTION

There are no standards that are strictly followed across mail clients

Pretty much every mail-related RFC has been outright violated or creatively interpreted by at least one major mail client

White space, HTML markup, attachments are all mangled and re-interpreted depending on which mail client is working on the message

If you build a mail application that reads email you will have to deal with this, it’s expensive and painful

Key Take-aways Email is not dead! Treasure-trove of data waiting to be mined At least four different types of apps Broadcast, CLI, Data transfer, Semantic parsing Remember the gotchas: Deliverability, Spam, Privacy, Standards If you get it right, huge payoff for you and your users

Email is not dead!

Treasure-trove of data waiting to be mined

At least four different types of apps

Broadcast, CLI, Data transfer, Semantic parsing

Remember the gotchas:

Deliverability, Spam, Privacy, Standards

If you get it right, huge payoff for you and your users

Questions Andy Denmark TripIt, Co-Founder and VP Engineering

Andy Denmark

TripIt, Co-Founder and VP Engineering

Appendix ESP Whitelists/Resources Microsoft Hotmail/MSN: http://postmaster.msn.com/Services.aspx https://postmaster.live.com/snds/ http://postmaster.live.com/Troubleshooting.aspx http://download.microsoft.com/download/e/3/3/e3397e7c-17a6-497d-9693-78f80be272fb/enhance_deliver.pdf http://www.senderscorecertified.com/ by ReturnPath Yahoo http://help.yahoo.com/l/us/yahoo/mail/postmaster/ Google http://www.google.com/mail/help/bulk_mail.html AOL http://postmaster.aol.com/

ESP Whitelists/Resources

Microsoft Hotmail/MSN:

http://postmaster.msn.com/Services.aspx

https://postmaster.live.com/snds/

http://postmaster.live.com/Troubleshooting.aspx

http://download.microsoft.com/download/e/3/3/e3397e7c-17a6-497d-9693-78f80be272fb/enhance_deliver.pdf

http://www.senderscorecertified.com/ by ReturnPath

Yahoo

http://help.yahoo.com/l/us/yahoo/mail/postmaster/

Google

http://www.google.com/mail/help/bulk_mail.html

AOL

http://postmaster.aol.com/

Appendix cont’d Mail Server Setup http://domainkeys.sourceforge.net/ http://www.openspf.org/ http://www.maawg.org/about/MAAWG_Sender_BCP/MAAWG_Senders_BCP_Combine.pdf CLI Examples http://microformats.org/wiki/picoformats Relevant RFCs http://rfc.net/rfc2821.html (SMTP) http://rfc.net/rfc2822.html (Mail Headers) http://rfc.net/rfc2045.html (MIME part 1) http://rfc.net/rfc2046.html (MIME part 2)

Mail Server Setup

http://domainkeys.sourceforge.net/

http://www.openspf.org/

http://www.maawg.org/about/MAAWG_Sender_BCP/MAAWG_Senders_BCP_Combine.pdf

CLI Examples

http://microformats.org/wiki/picoformats

Relevant RFCs

http://rfc.net/rfc2821.html (SMTP)

http://rfc.net/rfc2822.html (Mail Headers)

http://rfc.net/rfc2045.html (MIME part 1)

http://rfc.net/rfc2046.html (MIME part 2)