ATA Over Ethernet
7 March 2008
Lawrence Public Library, Lawrence, KS
What is it?
• Encapsulation of ATA commands inside
• Non-routeable: transmitted directly via
Ethernet, rather than (e.g.) TCP/IP
• Hosts can use AoE-attached disks as if they
were directly (e.g. via SATA) attached.
• Cheaper lower-level alternative to iSCSI
How to get it?
• Purchase EtherDrive appliance from Coraid.
This is a commodity x86(64) server, generally
SuperMicro, with a number of hot-swap SATA
trays and Coraid OS.
• Purchase a device based on LayerWalker
• Roll your own
Roll your own?
• Build commodity Linux server.
• Put physical disks for export (or RAID arrays)
into an LVM VG.
• Create as many separate LVM LV’s as you want
• Export LV’s using vblade(d), kvblade, or
• Physical volumes need not be (S)ATA!
Wasn’t this ATA over Ethernet?
• LVM appears to take care of all of it.
• I have successfully created and exported LV’s
via AoE on top of SATA, SCSI, and Adaptec SCSI
• This means you can throw a bunch of old SCSI
disks into a Linux box, export them via AoE,
and use them for stuff.
Gigabit Ethernet switch
OS on SSD LAN
AOE shelf AOE shelf
AOE shelf AOE shelf
Example configuration explained
• Linux server has OS mounted on SSD
• Connected to AOE “shelves” via dedicated
• AOE shelves configured with appropriate RAID
• Each shelf exports a single logical device
• Logical devices configured on server as PV’s in
• LV’s created on server in VG
• In the example configuration, it is trivially easy
to add storage to the VG on the server:
• Attach another AOE shelf (e.g. #5)
• Extend VG with device exported by shelf (e.g.
• vgextend vg00 /dev/etherd/e5.0
• Extend appropriate LV(s), filesystems
• AoE is non-routeable. SAN must be on same logical
Ethernet LAN as server
• Set up AoE devices on server as PV’s and put them into
an LVM VG. Otherwise not so easy to extend later.
• Set up syslog on related AoE shelves to point to single
host for ease of management.
• 1500 byte MTU only allows for two 512b sectors.
Increase MTU to 9000 for better performance.
• AoE volumes potentially accessible via multiple
machines. If mounting the same volume from two
machines, use an FS that supports this, like ClusterFS.
Required Software for Linux
• AoE modules have been in the kernel since
• aoetools – tools to detect and display
available AoE volumes, as well as tools to
export LV’s/partitions via AoE –
(my personal experiments with AoE)
• http://www.lbserver.org/aoe/ (AoE bootable CD, can
both mount and export volumes)
I can help!
• Available to consult on AoE
• (843) 819-9065