Online storage services for the masses and the case of Pithos Nectarios Koziris CSLAB-NTUA / Vice Chair GRNET GridNets 200...
Why Online Storage? <ul><li>Hard Disk costs are decreasing per TB  (Sept 09: 80$/TB) </li></ul><ul><li>Speed ….?  10s+MB/s...
Pros <ul><li>Access anyplace, anytime </li></ul><ul><li>Reliability </li></ul><ul><li>Multiple workplaces, multiple PCs (s...
Open issues <ul><li>Ease of use </li></ul><ul><li>Security </li></ul><ul><li>Privacy </li></ul><ul><li>Provider reliabilit...
Online Storage for users <ul><li>ADrive:  www.adrive.com </li></ul><ul><ul><li>50GB free/2GB per file </li></ul></ul><ul><...
Online Storage as a Service: Amazon S3 <ul><li>unlimited storage through a simple web services interface  </li></ul><ul><l...
SmugMug example  <ul><li>2006:  SmugMug  spent  $6,000 per  TB  of space </li></ul><ul><li>S3 since April 2006  - Savings ...
GRNET <ul><li>GRNET is the greek academic, education and research network (NREN) </li></ul><ul><li>Offers services since  ...
GRNET3 DF Network <ul><li>Basic characteristics </li></ul><ul><ul><li>35  PoPs </li></ul></ul><ul><ul><li>Includes : </li>...
Pithos <ul><li>From Wikipedia :  </li></ul><ul><li>Pithos (plural pithoi) is the ancient Greek word (πίθος, πίθοι) for a  ...
Pithos <ul><ul><li>(more:) </li></ul></ul><ul><ul><li>rich web GUI client (access via the REST API)  based on GWT </li></u...
GSS software for Pithos <ul><li>Stands for GRNET Storage Service </li></ul><ul><li>It’s the software behind PITHOS </li></...
User Functionality <ul><li>offers users a file system abstraction, complete with file/folder hierarchical structures </li>...
GUI Secure Web Access File Sharing User Credentials Available Resources Versioning IPv6 Support Public access  (if enabled...
 
Technical Details <ul><li>All GSS functionality provided on the application layer via application server farms </li></ul><...
webdav   interface (XP/Vista/MacOS/Linux)
FireGSS plugin for Firefox using REST
Another Pithos Firefox plugin…
Pithos Architecture
Logical Topology - Components <ul><li>Cluster of GSS servers (handling main functionality and APIs) </li></ul><ul><li>Solr...
 
Shibboleth-based Access
Current Status <ul><li>Beta testing Completed (over 1500 users) </li></ul><ul><li>Performance testing </li></ul><ul><ul><l...
Issues   to discuss further <ul><li>Cost per TB?  </li></ul><ul><ul><li>Commodity Disks… </li></ul></ul><ul><li>Scalabilit...
<ul><li>For more info: </li></ul><ul><li>http://pithos.grnet.gr   </li></ul><ul><li>http://www.pithos.gr   </li></ul><ul><...
Upcoming SlideShare
Loading in …5
×

Online storage for the masses and the case of pithos

1,571 views
1,483 views

Published on

Invited talk at GRIDNETS 2009 (Pithos online storage for Greek Researchers-students, 50GB free space)

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,571
On SlideShare
0
From Embeds
0
Number of Embeds
32
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Online storage for the masses and the case of pithos

  1. 1. Online storage services for the masses and the case of Pithos Nectarios Koziris CSLAB-NTUA / Vice Chair GRNET GridNets 2009, AIT-Athens, 9 /9/2009 www.pithos.gr
  2. 2. Why Online Storage? <ul><li>Hard Disk costs are decreasing per TB (Sept 09: 80$/TB) </li></ul><ul><li>Speed ….? 10s+MB/sec? 100+MB/sec with SSDs! </li></ul><ul><li>MTTF: 1.000.000-1.500.000h –> Annual replacement ~1% </li></ul><ul><li>But.. true annual replacement up to 10%! </li></ul><ul><li>(FAST07: ‘Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?’) </li></ul><ul><li>Many redundancy options </li></ul><ul><li>Portability? </li></ul><ul><ul><li>USB sticks 64GB common today (~120$) </li></ul></ul>
  3. 3. Pros <ul><li>Access anyplace, anytime </li></ul><ul><li>Reliability </li></ul><ul><li>Multiple workplaces, multiple PCs (sync between) </li></ul><ul><li>Sharing/collaboration </li></ul><ul><li>Security ? </li></ul><ul><li>Backup to the net/restore anywhere </li></ul><ul><li>Access from online applications </li></ul><ul><li>Access from different devices (mobiles…) </li></ul>
  4. 4. Open issues <ul><li>Ease of use </li></ul><ul><li>Security </li></ul><ul><li>Privacy </li></ul><ul><li>Provider reliability (xdrive..) </li></ul><ul><li>Speed (uploading takes time….) </li></ul><ul><li>Integration with local OS/applications/websites </li></ul><ul><li>Filesize limits </li></ul><ul><li>Multiplatform access (mobiles/netbooks etc) </li></ul><ul><li>No clear business model! </li></ul>Still a long way to go….
  5. 5. Online Storage for users <ul><li>ADrive: www.adrive.com </li></ul><ul><ul><li>50GB free/2GB per file </li></ul></ul><ul><li>Microsoft Skydrive: SkyDrive.live.com </li></ul><ul><ul><li>25GB free/50MB per file </li></ul></ul><ul><li>Mozy (Home Free 2GB) & 3.75 €/ month unlimited space </li></ul><ul><li>Box.net </li></ul><ul><ul><li>1GB free </li></ul></ul><ul><li>DropBox (uses Amazon S3): www.dropbox.com </li></ul><ul><ul><li>2GB free, 50GB at 9,9$/month, 100GB at 19,9$/month </li></ul></ul><ul><li>Livedrive.com </li></ul><ul><ul><li>100GB /£39.95 per year </li></ul></ul><ul><li>Esnips 5GB free </li></ul><ul><li>Humyo, Infinit, YouSendIt, Wuala, etc.. </li></ul><ul><li>Remember Xdrive.com? </li></ul><ul><ul><li>2005: 5GB free, 50GB at $9.95 /month </li></ul></ul><ul><ul><li>AOL acquired it 2005 / 5M$ selling/ closed it on January 12, 2009. </li></ul></ul>
  6. 6. Online Storage as a Service: Amazon S3 <ul><li>unlimited storage through a simple web services interface </li></ul><ul><li>scalability , high availability, and low latency at commodity costs. </li></ul><ul><li>REST-style HTTP interface // SOAP interface </li></ul><ul><li>5GB file size per bucket in S3 </li></ul><ul><li>DropBox, SmugMug, Sli deshare, Twitter, Woot.com and many other sites use Amazon S3 to host images /videos/files . </li></ul><ul><li>AWS: +Elastic Block Store (EBS) for EC2 VMs </li></ul>
  7. 7. SmugMug example <ul><li>2006: SmugMug spent $6,000 per TB of space </li></ul><ul><li>S3 since April 2006 - Savings of US$500,000 per year by using S3 </li></ul><ul><li>Despite outages/slowdowns of S3 in 2006 continued. </li></ul><ul><li>saved almost $1 million in storage costs in 12 months… </li></ul>
  8. 8. GRNET <ul><li>GRNET is the greek academic, education and research network (NREN) </li></ul><ul><li>Offers services since 1995 </li></ul><ul><li>1998 GRNET company formed (fully publicly owned) </li></ul><ul><li>2002 GRNET 2 upgrade , 2004 GRNET 2 B’ phase ( λ based) </li></ul><ul><li>200 8 GRNET 3 upgrade (10G, dark fiber) </li></ul><ul><li>2009 GRNET4 </li></ul><ul><li>Virtualization at all levels: </li></ul><ul><ul><li>Networking, Storage, Computing (VMs) </li></ul></ul>
  9. 9. GRNET3 DF Network <ul><li>Basic characteristics </li></ul><ul><ul><li>35 PoPs </li></ul></ul><ul><ul><li>Includes : </li></ul></ul><ul><ul><ul><li>8410 km </li></ul></ul></ul><ul><ul><ul><li>MANs Attica & Thessaloniki </li></ul></ul></ul><ul><ul><ul><li>DF local loops at 33 cities </li></ul></ul></ul>
  10. 10. Pithos <ul><li>From Wikipedia : </li></ul><ul><li>Pithos (plural pithoi) is the ancient Greek word (πίθος, πίθοι) for a large storage jar of a characteristic shape . </li></ul><ul><li>Both Storage for users and Storage as a Service from GRNET </li></ul><ul><ul><li>At the current stage, 22 universities have Shibbolethised their AA process (total number of possible users > 300,000) </li></ul></ul><ul><li>1. Personal Online Storage: </li></ul><ul><ul><li>50GB/user </li></ul></ul><ul><ul><li>Versioning (per file) </li></ul></ul><ul><ul><li>Backup </li></ul></ul><ul><ul><li>Groups </li></ul></ul><ul><ul><li>Sharing </li></ul></ul><ul><ul><li>Full text Search and tagging </li></ul></ul><ul><ul><li>No filesize limit/no file expiration </li></ul></ul><ul><ul><li>no traffic-bandwidth limit </li></ul></ul>www.pithos.gr and more…
  11. 11. Pithos <ul><ul><li>(more:) </li></ul></ul><ul><ul><li>rich web GUI client (access via the REST API) based on GWT </li></ul></ul><ul><ul><li>desktop client (access via the REST API) </li></ul></ul><ul><ul><li>Firefox addon (via REST) </li></ul></ul><ul><li>2. Storage as a Service: </li></ul><ul><ul><li>REST API </li></ul></ul><ul><ul><li>WebDAV interface </li></ul></ul>A truly scalable online file storage infrastructure
  12. 12. GSS software for Pithos <ul><li>Stands for GRNET Storage Service </li></ul><ul><li>It’s the software behind PITHOS </li></ul><ul><ul><li>Provides a virtual disk (storage) accessed over the network </li></ul></ul><ul><ul><ul><li>Users are able to upload, share, and index their files </li></ul></ul></ul><ul><ul><ul><li>Inspired by Amazon S3, but going beyond… </li></ul></ul></ul><ul><ul><li>Targeting research & academic community </li></ul></ul><ul><ul><ul><li>> 300,000 users </li></ul></ul></ul><ul><ul><li>Funded by the national project HellasGrid ( www.hellasgrid.gr ) </li></ul></ul><ul><li>Open Source Code http:// code.google.com/p/gss / </li></ul><ul><li>(designed and code developed by www.ebs.gr ) </li></ul>
  13. 13. User Functionality <ul><li>offers users a file system abstraction, complete with file/folder hierarchical structures </li></ul><ul><li>Basic functions </li></ul><ul><ul><li>Usual file system operations, e.g. copy, move, etc. </li></ul></ul><ul><ul><li>Delete plus “trash-bin” functions </li></ul></ul><ul><ul><li>50GB free space per user, easily extendible </li></ul></ul><ul><li>Advanced functions </li></ul><ul><ul><li>File sharing with selected users, user groups or public </li></ul></ul><ul><ul><li>Automatic file versioning, e.g. keep history of the file </li></ul></ul><ul><ul><li>Text search and tagging capabilities for file names or folders </li></ul></ul><ul><ul><li>Shibbolized authentication, aka no need for extra credentials </li></ul></ul><ul><ul><li>Access over IPv6 </li></ul></ul>
  14. 14. GUI Secure Web Access File Sharing User Credentials Available Resources Versioning IPv6 Support Public access (if enabled) Search Capabilities Properties Flexible File Naming
  15. 16. Technical Details <ul><li>All GSS functionality provided on the application layer via application server farms </li></ul><ul><ul><li>Implemented in Java, dual stack (IPv6) </li></ul></ul><ul><li>Multiple access interfaces </li></ul><ul><ul><li>Direct API (REST), web-based client (AJAX), desktop client (Tcl/Tk), WebDAV. </li></ul></ul><ul><ul><li>Use Shibboleth ( shibboleth.internet2.edu ) for AA </li></ul></ul><ul><ul><li>Use student credentials from university catalogues </li></ul></ul><ul><li>Infrastructure </li></ul><ul><ul><li>12 servers ( IBM x3550, 2.0 GHz 16GB RAM ), SAN ( 140TB, SATA disks ), 10Gbps connection ( to GRNET backbone ) </li></ul></ul>
  16. 17. webdav interface (XP/Vista/MacOS/Linux)
  17. 18. FireGSS plugin for Firefox using REST
  18. 19. Another Pithos Firefox plugin…
  19. 20. Pithos Architecture
  20. 21. Logical Topology - Components <ul><li>Cluster of GSS servers (handling main functionality and APIs) </li></ul><ul><li>Solr server(s) ( for indexing and search functionality) </li></ul><ul><li>DB server (for storing meta-data) </li></ul><ul><li>GUI web clients </li></ul>
  21. 23. Shibboleth-based Access
  22. 24. Current Status <ul><li>Beta testing Completed (over 1500 users) </li></ul><ul><li>Performance testing </li></ul><ul><ul><li>On going process </li></ul></ul><ul><li>Enrich functionality </li></ul><ul><ul><li>Open source effort http://code.google.com/p/gss/ </li></ul></ul><ul><ul><li>Take advantage of REST API for third-party applications – Firefox Plugins </li></ul></ul>
  23. 25. Issues to discuss further <ul><li>Cost per TB? </li></ul><ul><ul><li>Commodity Disks… </li></ul></ul><ul><li>Scalability </li></ul><ul><ul><li>Scalable metadata structures </li></ul></ul><ul><ul><li>Fast, symmetric access-NGNs (FTTH) </li></ul></ul><ul><li>Reliability </li></ul><ul><ul><li>software based </li></ul></ul><ul><li>Storage for VMs (Block Store etc) </li></ul>
  24. 26. <ul><li>For more info: </li></ul><ul><li>http://pithos.grnet.gr </li></ul><ul><li>http://www.pithos.gr </li></ul><ul><li>http://code.google.com/p/gss/ </li></ul><ul><li>contact: [email_address] </li></ul><ul><li>www.cslab.ece.ntua.gr/~nkoziris </li></ul>www.pithos.gr

×