1.
Online storage services for the masses and the case of Pithos Nectarios Koziris CSLAB-NTUA / Vice Chair GRNET GridNets 2009, AIT-Athens, 9 /9/2009 www.pithos.gr

2.
Why Online Storage? <ul><li>Hard Disk costs are decreasing per TB (Sept 09: 80$/TB) </li></ul><ul><li>Speed ….? 10s+MB/sec? 100+MB/sec with SSDs! </li></ul><ul><li>MTTF: 1.000.000-1.500.000h –> Annual replacement ~1% </li></ul><ul><li>But.. true annual replacement up to 10%! </li></ul><ul><li>(FAST07: ‘Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?’) </li></ul><ul><li>Many redundancy options </li></ul><ul><li>Portability? </li></ul><ul><ul><li>USB sticks 64GB common today (~120$) </li></ul></ul>

3.
Pros <ul><li>Access anyplace, anytime </li></ul><ul><li>Reliability </li></ul><ul><li>Multiple workplaces, multiple PCs (sync between) </li></ul><ul><li>Sharing/collaboration </li></ul><ul><li>Security ? </li></ul><ul><li>Backup to the net/restore anywhere </li></ul><ul><li>Access from online applications </li></ul><ul><li>Access from different devices (mobiles…) </li></ul>

4.
Open issues <ul><li>Ease of use </li></ul><ul><li>Security </li></ul><ul><li>Privacy </li></ul><ul><li>Provider reliability (xdrive..) </li></ul><ul><li>Speed (uploading takes time….) </li></ul><ul><li>Integration with local OS/applications/websites </li></ul><ul><li>Filesize limits </li></ul><ul><li>Multiplatform access (mobiles/netbooks etc) </li></ul><ul><li>No clear business model! </li></ul>Still a long way to go….

5.
Online Storage for users <ul><li>ADrive: www.adrive.com </li></ul><ul><ul><li>50GB free/2GB per file </li></ul></ul><ul><li>Microsoft Skydrive: SkyDrive.live.com </li></ul><ul><ul><li>25GB free/50MB per file </li></ul></ul><ul><li>Mozy (Home Free 2GB) & 3.75 €/ month unlimited space </li></ul><ul><li>Box.net </li></ul><ul><ul><li>1GB free </li></ul></ul><ul><li>DropBox (uses Amazon S3): www.dropbox.com </li></ul><ul><ul><li>2GB free, 50GB at 9,9$/month, 100GB at 19,9$/month </li></ul></ul><ul><li>Livedrive.com </li></ul><ul><ul><li>100GB /£39.95 per year </li></ul></ul><ul><li>Esnips 5GB free </li></ul><ul><li>Humyo, Infinit, YouSendIt, Wuala, etc.. </li></ul><ul><li>Remember Xdrive.com? </li></ul><ul><ul><li>2005: 5GB free, 50GB at $9.95 /month </li></ul></ul><ul><ul><li>AOL acquired it 2005 / 5M$ selling/ closed it on January 12, 2009. </li></ul></ul>

6.
Online Storage as a Service: Amazon S3 <ul><li>unlimited storage through a simple web services interface </li></ul><ul><li>scalability , high availability, and low latency at commodity costs. </li></ul><ul><li>REST-style HTTP interface // SOAP interface </li></ul><ul><li>5GB file size per bucket in S3 </li></ul><ul><li>DropBox, SmugMug, Sli deshare, Twitter, Woot.com and many other sites use Amazon S3 to host images /videos/files . </li></ul><ul><li>AWS: +Elastic Block Store (EBS) for EC2 VMs </li></ul>

7.
SmugMug example <ul><li>2006: SmugMug spent $6,000 per TB of space </li></ul><ul><li>S3 since April 2006 - Savings of US$500,000 per year by using S3 </li></ul><ul><li>Despite outages/slowdowns of S3 in 2006 continued. </li></ul><ul><li>saved almost $1 million in storage costs in 12 months… </li></ul>

8.
GRNET <ul><li>GRNET is the greek academic, education and research network (NREN) </li></ul><ul><li>Offers services since 1995 </li></ul><ul><li>1998 GRNET company formed (fully publicly owned) </li></ul><ul><li>2002 GRNET 2 upgrade , 2004 GRNET 2 B’ phase ( λ based) </li></ul><ul><li>200 8 GRNET 3 upgrade (10G, dark fiber) </li></ul><ul><li>2009 GRNET4 </li></ul><ul><li>Virtualization at all levels: </li></ul><ul><ul><li>Networking, Storage, Computing (VMs) </li></ul></ul>

9.
GRNET3 DF Network <ul><li>Basic characteristics </li></ul><ul><ul><li>35 PoPs </li></ul></ul><ul><ul><li>Includes : </li></ul></ul><ul><ul><ul><li>8410 km </li></ul></ul></ul><ul><ul><ul><li>MANs Attica & Thessaloniki </li></ul></ul></ul><ul><ul><ul><li>DF local loops at 33 cities </li></ul></ul></ul>

10.
Pithos <ul><li>From Wikipedia : </li></ul><ul><li>Pithos (plural pithoi) is the ancient Greek word (πίθος, πίθοι) for a large storage jar of a characteristic shape . </li></ul><ul><li>Both Storage for users and Storage as a Service from GRNET </li></ul><ul><ul><li>At the current stage, 22 universities have Shibbolethised their AA process (total number of possible users > 300,000) </li></ul></ul><ul><li>1. Personal Online Storage: </li></ul><ul><ul><li>50GB/user </li></ul></ul><ul><ul><li>Versioning (per file) </li></ul></ul><ul><ul><li>Backup </li></ul></ul><ul><ul><li>Groups </li></ul></ul><ul><ul><li>Sharing </li></ul></ul><ul><ul><li>Full text Search and tagging </li></ul></ul><ul><ul><li>No filesize limit/no file expiration </li></ul></ul><ul><ul><li>no traffic-bandwidth limit </li></ul></ul>www.pithos.gr and more…

11.
Pithos <ul><ul><li>(more:) </li></ul></ul><ul><ul><li>rich web GUI client (access via the REST API) based on GWT </li></ul></ul><ul><ul><li>desktop client (access via the REST API) </li></ul></ul><ul><ul><li>Firefox addon (via REST) </li></ul></ul><ul><li>2. Storage as a Service: </li></ul><ul><ul><li>REST API </li></ul></ul><ul><ul><li>WebDAV interface </li></ul></ul>A truly scalable online file storage infrastructure

12.
GSS software for Pithos <ul><li>Stands for GRNET Storage Service </li></ul><ul><li>It’s the software behind PITHOS </li></ul><ul><ul><li>Provides a virtual disk (storage) accessed over the network </li></ul></ul><ul><ul><ul><li>Users are able to upload, share, and index their files </li></ul></ul></ul><ul><ul><ul><li>Inspired by Amazon S3, but going beyond… </li></ul></ul></ul><ul><ul><li>Targeting research & academic community </li></ul></ul><ul><ul><ul><li>> 300,000 users </li></ul></ul></ul><ul><ul><li>Funded by the national project HellasGrid ( www.hellasgrid.gr ) </li></ul></ul><ul><li>Open Source Code http:// code.google.com/p/gss / </li></ul><ul><li>(designed and code developed by www.ebs.gr ) </li></ul>

13.
User Functionality <ul><li>offers users a file system abstraction, complete with file/folder hierarchical structures </li></ul><ul><li>Basic functions </li></ul><ul><ul><li>Usual file system operations, e.g. copy, move, etc. </li></ul></ul><ul><ul><li>Delete plus “trash-bin” functions </li></ul></ul><ul><ul><li>50GB free space per user, easily extendible </li></ul></ul><ul><li>Advanced functions </li></ul><ul><ul><li>File sharing with selected users, user groups or public </li></ul></ul><ul><ul><li>Automatic file versioning, e.g. keep history of the file </li></ul></ul><ul><ul><li>Text search and tagging capabilities for file names or folders </li></ul></ul><ul><ul><li>Shibbolized authentication, aka no need for extra credentials </li></ul></ul><ul><ul><li>Access over IPv6 </li></ul></ul>

14.
GUI Secure Web Access File Sharing User Credentials Available Resources Versioning IPv6 Support Public access (if enabled) Search Capabilities Properties Flexible File Naming

16.
Technical Details <ul><li>All GSS functionality provided on the application layer via application server farms </li></ul><ul><ul><li>Implemented in Java, dual stack (IPv6) </li></ul></ul><ul><li>Multiple access interfaces </li></ul><ul><ul><li>Direct API (REST), web-based client (AJAX), desktop client (Tcl/Tk), WebDAV. </li></ul></ul><ul><ul><li>Use Shibboleth ( shibboleth.internet2.edu ) for AA </li></ul></ul><ul><ul><li>Use student credentials from university catalogues </li></ul></ul><ul><li>Infrastructure </li></ul><ul><ul><li>12 servers ( IBM x3550, 2.0 GHz 16GB RAM ), SAN ( 140TB, SATA disks ), 10Gbps connection ( to GRNET backbone ) </li></ul></ul>

17.
webdav interface (XP/Vista/MacOS/Linux)

18.
FireGSS plugin for Firefox using REST

19.
Another Pithos Firefox plugin…

20.
Pithos Architecture

21.
Logical Topology - Components <ul><li>Cluster of GSS servers (handling main functionality and APIs) </li></ul><ul><li>Solr server(s) ( for indexing and search functionality) </li></ul><ul><li>DB server (for storing meta-data) </li></ul><ul><li>GUI web clients </li></ul>

23.
Shibboleth-based Access

24.
Current Status <ul><li>Beta testing Completed (over 1500 users) </li></ul><ul><li>Performance testing </li></ul><ul><ul><li>On going process </li></ul></ul><ul><li>Enrich functionality </li></ul><ul><ul><li>Open source effort http://code.google.com/p/gss/ </li></ul></ul><ul><ul><li>Take advantage of REST API for third-party applications – Firefox Plugins </li></ul></ul>

25.
Issues to discuss further <ul><li>Cost per TB? </li></ul><ul><ul><li>Commodity Disks… </li></ul></ul><ul><li>Scalability </li></ul><ul><ul><li>Scalable metadata structures </li></ul></ul><ul><ul><li>Fast, symmetric access-NGNs (FTTH) </li></ul></ul><ul><li>Reliability </li></ul><ul><ul><li>software based </li></ul></ul><ul><li>Storage for VMs (Block Store etc) </li></ul>

26.
<ul><li>For more info: </li></ul><ul><li>http://pithos.grnet.gr </li></ul><ul><li>http://www.pithos.gr </li></ul><ul><li>http://code.google.com/p/gss/ </li></ul><ul><li>contact: [email_address] </li></ul><ul><li>www.cslab.ece.ntua.gr/~nkoziris </li></ul>www.pithos.gr