AFS case study


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AFS case study

  1. 1. AFS Case Study Manfred at zeropiu.itEuroBsdCon November 2006
  2. 2. Agenda•  Overview •  Basic Concepts •  AFS Servers Type •  Arla Overview•  Best practice •  Planning •  AFS Convention•  Case Study •  Solution •  Architecture •  Result Pagina 2
  3. 3. OverviewAndrew File System is a distributed file system designed to :•  handle terabytes of data•  handle thousands of users•  working in WAN environment Pagina 3
  4. 4. Brief history of a AFS•  1983 Andrew Project started at Carnegie Mellon University (CMU)•  1987 Coda research work begun (based on AFS)•  1988 First use of AFS version 3 First use of AFS outside Carnegie Mellon University•  1988 Institutional File System project at University of Michigan –•  1989 Transarc Corporation founded to commercialize AFS,•  1993 Arla project started at Kungliga Tekniska Högskolan•  1998 Transarc Corporation becomes wholly owned subsidiary of IBM•  2000 IBM releases OpenAFS as OpenSource (IBM License),•  2000 OpenAFS release version 1.0 based on Transarc 3.6•  2001 OpenAFS release version 1.2 first release with better support of new operating system and fix several memory leak•  2005 OpenAFS release version 1.4 with a lot of new feature•  2005 AFS was discontinued from IBM Pagina 4
  5. 5. Basic Concepts•  Transparent Access and Uniform Namespace •  Cell •  Partitions and Volumes •  Mount Points•  Scalability •  Client Caching •  Replication•  Security •  Authentication and secure communication •  Authorization and flexible access control•  System Management •  Single system interface •  Delegation •  Backup Pagina 5
  6. 6. Transparent Access and Uniform Namespace•  Cell •  Cell is collection of file servers and workstation •  The directories under /afs are cells , unique tree •  Fileserver contains volumes•  Volumes •  Volumes are "containers" or sets of related files and directories •  Have size limit •  3 type rw,ro,backup•  Mount Point •  Access to a volume is provided through a mount point •  A mount point looks and just like a static directory Pagina 6
  7. 7. Scalability•  Cache Manager (Client Side) •  Maintain information about identities users •  Retrieve data from fileserver •  Keeps chunks of retrieved files on local disk (cache)•  Replication •  Frequently accessed data can be replicated (read-only) on several server •  Cache Manager make use of replicate volumes first Pagina 7
  8. 8. Security•  Authentication •  Kerberos IV native (kaserver) •  External Kerberos V •  Unique identity •  Encryption communication on data transfer (crypt option)•  Authorization •  Access control list with 7 types permissions •  Groups definition by user Pagina 8
  9. 9. System Management•  Single system interface •  Configuration changes can made from any client •  Move volume in transparent way •  On-line upgrade and extend system•  Delegation •  Group delegation •  Admin delegation•  Backup •  Backup volume and file •  Built in backup function •  User direct access (backup mounted) Pagina 9
  10. 10. Example write operation client side1 create file rpc2 write chunks into cache (interrupted by store_data RPC)3 read from cache4 transfer over network5 write to /vicepXX Pagina 10
  11. 11. Example write operation server side1 Create file2 Check metadata, permission, quota and return file path3 write file into /vicepXX4 Update meta data on server5 Update db Pagina 11
  12. 12. AFS Servers Type•  Fileserver machine •  file storage•  Database server machine •  File and Volume localization •  ACL and groups administration •  Authentication provider•  Binary distribution •  Master server for afs binary (specific architecture)•  System control machine •  Time server •  AFS configuration master Pagina 12
  13. 13. AFS Server Process•  Bosserver, system monitor•  Fileserver, serves file•  Volserver, serves volume data•  Vlserver, volume location server•  Kaserver, kerberos IV server•  Ptserver, protection server (group,acl)•  Buserver, backup server•  Upserver •  Update conf •  Update binary Pagina 13
  14. 14. Weakness•  File restriction •  Pipes •  Devices files •  Sockets •  Unicode name•  AFS Lock •  Only advisory locks (byte-range locking underway)•  ACL •  Only on directory•  Volume •  Read only •  Manual sync•  Write on close •  Date time operation Pagina 14
  15. 15. Arla•  AFS client alternative•  *BSD support•  Disconnected operation Pagina 15
  16. 16. Where used ?•  University •  Cmu, Stanford,MIT,KTH(Sweden), Chemitz(Germany), Roma3(italy),…•  Research Labs •  SLAC, DESY,CERN(EUROPE),INFN(ITALY),…•  Companies: •  Intel,Morgan Stanley,Pictage,.. Pagina 16
  17. 17. Conventions and Best Practices•  AFS file space layout •  Server planning •  Volume naming and schemas •  Volume replication•  Username schemas•  Partition Filesystem•  Backup planning•  Security consideration•  Client Cache tuning•  AFS limitations Pagina 17
  18. 18. Cell Name• Convention •  Company Domain name •  Company Kerberos Realm• Cell name •  Short name (Max size cell is 64 characters) •  Cell name can contain only lowercase characters •  Suitable for different operating system. (Do not include command shell metacharacters). Pagina 18
  19. 19. Server planning• Fileserver •  Ratio 200:1 client server (many site today have 1000:1) •  Replica server location •  Big machine vs small machine• Database server •  3 machine for election algorithm (ubik) •  Separate from Fileserver• Update server •  One system• Binary distribution •  One system per architecture Pagina 19
  20. 20. Volume naming and schemas• Volume name limit •  Read/write volume names can be up to 22 characters in length •  The .readonly and .backup extensions are reserved word •  root.afs and root.cell name are used for default• Volume naming •  Mount point prefix name (user.manfred) •  Function suffix name Pagina 20
  21. 21. Volume layout and replication• Volume •  User have its own volume for simplify load balance operations (move,backup) •  Volume for group of file (binary, documents ..)• Replication is not appropriate for volumes that changefrequently• Replicate the root.afs and root.cell most as possible• Backup volume use the same partition (it is a copy of thesource volumes vnode index) Pagina 21
  22. 22. Username• Username • Characters, which have special meanings to the command shell • The colon ( : ), because AFS reserves it as a field separator in protection group names; • The period ( . ); it is conventional used to identify special username that have administrator capability (ex. manfred.admin ) • AFS UID, 32766, is reserved for the user anonymous.• UID maching, unix uid / AFS uid •  Unix ldap •  NIS •  kerberos ldap backend •  smb Pagina 22
  23. 23. Partition Filesystem (inode vs iname)Inode faster •  Dedicated partition •  Special fsck for the system partition •  No journaling file system •  Restore on same filesystem layout (same inode structure)Iname slower •  OS fsck •  Filesystem independent, with advantage of journaling •  The aren’t special requirement for /VicepXX, it could be a mounted filesystem •  Simply restore operation Pagina 23
  24. 24. Backup• Native backup system and recovery AFS can be configured to create a full or incremental backup• Volume dump This operation permit to create a binary file with all information of backup volume• Backup system with AFS support • Amanda • Bacula • Other commercial product Pagina 24
  25. 25. Security consideration•  User Accounts: •  Kerberos integration with modified login utility •  replace kaserver with Unix Kerberos solution or Windows AD (OpenAFS support basic Kerberos 5 2b protocol) •  including the unlog command in every users .logout file or equivalent•  Server Machines •  Change the AFS server encryption key on a frequent and regular schedule. •  Particularly limit access to the local superuser root account on a server machine.•  System Administrators •  Create an administrative account for each administrator separate from the personal account •  assign AFS privileges only to the administrative account. •  Set the token lifetime for administrative accounts to a fairly short amount of time. Pagina 25
  26. 26. Client Cache•  Cache Size •  single user machine 128MB •  Multi-user machine 1GB/4GB•  Cache partition •  Directory, the partition must grantee enough space •  Disk partition, better performance (Terminal Server)•  Login Integration Pagina 26
  27. 27. AFS limitationsGeneral Limit• OpenAFS can support a maximum of 104.000 clients per server• tmpfs no work as AFS Cache, (ramdisk work)• Max 255 partition per server (/vicepa-/vicepiv), no limits in partition size• Max 4,294,967,295 volumes per partition (this a limit of VLDB),• Max Volume size is 2TB• Max file limit per directory is 64,000 files (less than 16 characters).Windows Limit• Write-on-close, the changes are synchronized only on close operation• No integration on Microsoft DFS• No support for files greater than 2GB on windows platform ( work inprogress). Pagina 27
  28. 28. Case StudyItalsempione is nowadays the biggest Italian fully indipendent forwarding company covering any service related to transports and logistics with a worldwide agency network.Company:•  Head Quarter in Italy•  16 Branch Office in Italy•  7 branch outside Italy•  400 PC , Windows XX•  150 PC , Linux•  8 Windows NT Domain•  Wide Area Network•  No IT stuff on the branch office Pagina 28
  29. 29. Solution•  Primary goals •  Reduce cost of Software License •  Simplify System Administration task.•  Solution •  Thin client replacement, terminal server •  Server Virtualization , VMware •  Storage Virtualization, OpenAFS Pagina 29
  30. 30. Architecture Pagina 30
  31. 31. ArchitectureHead Quarter3 Fileserver Machines • User:Server rate 200:1. • The read-write information volumes are replicated with circular schema • The volumes of binary and programs are replicated on all fileserver. • The fileserver are based on OpenBSD 3.9.3 Database Servers • installed on the same machine of fileserver2 Authentication Servers •  Heimdal Kerberos V •  ldap backend (samba, heimdal, unix, profile info)8 VMmachine • windows terminal server image • Linux terminal server image • OpenBSD network service image Pagina 31
  32. 32. Architecture •  Cell name= domain name •  Main Directory tree = country/city/function •  User Directory tree = usr/m/manfred •  User volume •  Volume name Directory usage Volume name User home user.username prefix= mount point User home backup user.username.backup Suffix= function Application apps.applicationname OS Software software.soname •  Volume replication Groups groups.groupname Binary data VMware image image.osname Root volume(afs,cell) Pagina 32
  33. 33. Architecture•  Partition •  inode base •  Small partition for quick check •  Odd vicepX for rw volume even for ro volume•  Backup •  Bacula for incremental / total dump •  User backup volume mounted in home dir•  Monitoring •  Zabbix •  AFS monitor and performance Pagina 33
  34. 34. HardwareFileserver /DbServer: • 1GB of RAM, • 3GHz Xeon single processor • 2x36Gb SCSI RAID 1 for operating system partition • 4x 143GB SCSI RAID5 storage (/vicepXX)Authentication server: •  1GB of RAM •  3GHz Xeon single processor •  2x36Gb SCSI RAID 1 for operating system and db backendVMmachine: • 4GB of RAM • 3GHz Xeon dual processor. • 2x36Gb SCSI RAID 1 for operating system and local vmware image. Pagina 34
  35. 35. Why OpenBSD•  OpenAFS support •  Porting Server side and client side•  Security level•  Heimdal integration •  AFS emulation •  LDAP backend •  2ab protocol (large kerberos ticket)•  Small and fast•  Stable Pagina 35
  36. 36. Consideration• Iron server vs Small server •  small number of inexpensive fileservers (provides equivalent performance) •  inexpensive incremental increase in capacity •  better manageability and redundancy.NFS file sharing vs AFS •  AFS resulted in a 60% decrease in network traffic. •  The servers load decreased by 80% •  task execution time was reduced by 30%. Pagina 36
  37. 37. Benefit • Reduced cost • Reduced software costs for 150.000 Euro • Increase performance (Server and Desktop) • Reduced down time • Reduced helpdesk load • Simplify System Administration task •  Improved manageability •  Full disaster recovery protection •  Data accessible from Spain to Singapore with a •  High security level •  Single sign-on Pagina 37
  38. 38. NextOpenAFS •  Lock subsystem •  Windows support •  Kerberos V supportExternal project ( •  Ptserver with ldap backend •  Web interface Pagina 38
  39. 39. ReferenceInstall •  bsdcan • afs-bpw-2005-iowa.htmlOpenafs • • • • Pagina 39
  40. 40. The End Pagina 40