The Quick Migration of File Servers

© Hitachi, Ltd. 2018. All rights reserved.
06/05/2018
Keiichi Matsuzawa*†
Mitsuo Hayasaka*
Takahiro Shinagawa†
The Quick Migration of File Servers
* Hitachi, Ltd.
† University of Tokyo

Background
• Most cases of installing new file servers require server
and data migration across file servers
Source: Storage Magazine, Vol.16, No.2, 2017
20%
24%
32%
39%
56%
1
2
3
4
5
What are the top drivers
for your new NAS system?
Improve performance for existing apps
Replace old hardware
Deploy new application
Support virtual server better
Use as a backup target
Migrate from
existing old server
1 / 23

Challenges for migration scheme
1. Independent of the source server
– Heterogenous file server migration is needed to:
• Change to a more attractive vendor and product
• Dispose of End-of-Life products
• Aggregate small file servers to a enterprise file server
– Should not modify the source server
2. No long downtime for clients (<30 seconds)
– Common OSs define network file access timeout about tens
of seconds
3. No performance overhead after migration
– Improving performance is one of top drivers
2 / 23

• Pre-copy data per file [rsync][robocopy]
– Tracking the last updates takes long time to traverse tree
• Copy disk volume [SnapMirror]
– Source and target must share the same disk layout
Existing solutions
Source Target
Cannot mount
volume
>1day downtime
to track the last
update
rsync
Source Target
3 / 23

Our approach
• Post-Copy per file
– Clients change connection before copying data
– Target server copies metadata and data on demand
• Transfer data only via common interface (ex. NFS)
X
5: return
3: read(“/A/X”)
from target
Source Target
A
root
A
X
root
B
Y
4: copy(“/A/X”)
Client
1: read(“/A/X”)
from source
2: remount
4 / 23

Data migration method
• Migrate whole file namespace by:
1. On-demand migration
– Copies objects accessed by clients on-demand
2. Background migration
– Copies un-accessed objects in background
3. Stub-based file management:
– Enables intermediate migration state and supports above
migrations
5 / 23

On-demand migration
• Copy directories and files recursively when accessed
• After preparing root directory, any files are accessible
– Accessing directory only generates intermediate state
objects to reduce time to copy
root
A B C
root root
A B C
X Y Z
root
A B C
X Y Z
lookup “/” lookup “/A/” lookup “/A/Y”
read( “/A/Y” )
Create objects without
data
6 / 23

Background migration (1)
• Copy un-accessed files by clients
• Crawling program traverses file namespace and
induces on-demand migration
– Should run when access load is low (ex. midnight or
weekend)
root
A B C
X Y Z
root
A B C
X Y Z
root
A B C
X Y Z
Run crawling
program
7 / 23

Background migration (2)
• Crawling program may run intermittently
– Copy huge file share across several nights
– Need to handle updated files in the daytime
• Restart from the root directory each time
– Count intermediate objects until reaching zero
root
A B C
X Y Z
root
A B C
X Y Z
root
A B C
X Y Z
count: 4 count: 2 count: 0
8 / 23

• Should consider characteristics of file system:
• Data structure
– Hierarchical structure
– Protocol dependent metadata (ex. ACL, uid)
• Operations
– Rename/move objects
– Hardlinks
– Truncate
• Client behaviors (ex. Return code handling, thumbnail
preview)
• Product dependent features (ex. Compression, quota)
Issues of post-copy per file (1)
Scope of this
presentation
9 / 23

• Cause timeout to create inodes under huge directory
– Retrieving metadata requires many packet exchanges and
takes long time
A
X
read(“/A/Y”)
Source Target
Y Z …
A
Target
A
X Y Z …
Create all at once?
10 / 23

• Keep consistency of source and target objects even
after move operation
root
X
move(“/B”,”/A/C”)
Source Target
A B
Y
root
X
A B
Target
root
X
A
C
Different path for
source and target
Y
C
11 / 23

Stub management
• Concept used in Hierarchical Storage Management
– Stub is a virtual object does not have data contents but
stores actual data location
– File system intercepts stub accesses and inserts additional
operation
• Apply the concept for file server migration
– Allow intermediate states storing partial data and metadata
– Retrieve un-copied data when stub is accessed
12 / 23

Dummy Partial Normal
File and directory status
• Define intermediate states storing partial (meta)data
– Defer (meta)data copy until the file is accessed
– Dummy object can be created by only parent directory entry
– Partial object has full metadata and parts of data blocks
• Regards hole blocks as un-copied blocks
– Normal object has full metadata and data
• Migration is completed and not produce any source server accesses
Stub info.
Metadata Metadata
Data
inodeAccess
whole
data
Access
metadata
Access
parent
directory
Data
Intermediate states
inode
Stub info.
inode Internal
data
structure
Migration completedMost attributes are empty
13 / 23

• Additional metadata to manage intermediate states
• Stores full path name of corresponding objects in
source server
• Maintains the relationship even after move operations
Stub info.
Stub information
root
X
move(“/B”,”/A/C”)
Source Target
A B
Y
root
X
A B
/B
root
X
A
C
Y
/B/Y
/B
read(“/A/C/Y”)
status: dummy
source path: /A/X
Generate path name
from parent stub info.
14 / 23

Software stack of target server
Implementation
• Implemented on Linux-based file server.
– Stub management is implemented in kernel VFS layer to
intercept file accesses effectively
– Retriever and crawling programs run in user space
– Used in our production server
Linux Kernel File System
Stub management
Retriever program
libsmbclient Crawling
program
Retrieve data/metadata
kernel
nfsd
Samba
kernel
NFS client
Command queue on
unix domain socket
File accesses
File accesses
Clients
Source
15 / 23

Protocol dependent issues
• NFS hardlinks
– Reproduce hardlink
using representative
file whose name is
based on inode number
• NFS JUKEBOX return code
– Return JUKEBOX if NFS server needs long time to respond to
suppress excessive client retry
– NFS client waits a while and retries later
– Copy process keeps going in background
Our implementation
X
hidden
10Y
Representative
file
NG
X Y X Y
Copied
separately
Hard-linked
(inode# = 10)
16 / 23

Performance evaluation
• Access files under two situations:
A) Pre-copied files
B) Un-copied files that induces on-demand migration
• Evaluation items:
1. Access downtime for client changing connection
2. File access performance overhead after migration
B) on-demand migrationA) pre-copied file
ReadRead
Copy
17 / 23

Test Environment
• Same machine for file servers and client:
– CPU: Xeon E5-2603×2
– DRAM: 64GB
– NIC: 10GbE Copper × 1
– Disk: NVMe SSD
• OS for client and source server:
– NFSv3: Ubuntu Server 16.04LTS
– SMB3: Windows Server 2016 10GbE
SW
Source Target
Client
18 / 23

Downtime (1)
• Read 1GiB file halfway, change connection, and
resume reading
– Total amount of time of remount, open, and reading first byte
is downtime for client
• Change directory depth of test file and width of each
directory
• Remount does not depend on situations
– Less than 0.3 seconds
10MB Read Close Unmount Mount Open 10MB Read 10MB Read
Connect to source Connect to target
10MB Read
remount open read
19 / 23

Downtime (2)
• Open depends on directory structure due to migration
of directories on the pathname
– NFS results shorter for small directory due to less metadata
than SMB (ex. ACL)
– NFS takes much longer for large directory because Linux NFS
client insert 5 seconds wait for NFS JUKEBOX return code
• Client does not cause timeout error due to the code
0.001
0.005
0.001
0.005
0.023
0.023
0.023
0.023
0.052
0.277
5.684
90.075
0.11
0.623
1.62
26.217
0.001
0.01
0.1
1
10
100
1 2 3 4
Timetoopen(seconds)
系列1 系列2 系列3 系列4
Directory parameters (depth, width)
20 / 23

Downtime (3)
• Reading first byte increase response time a little due
to on-demand data migration
– Short time even for 1GB file because target server copies
only accesses blocks before response
– SMB results longer due to more packet exchanges to open
• In total, the downtime is less than 30s in most cases.
– Other cases are supported by NFS JUKEBOX return code.
0.017 0.022
0.058
0.217
1 2
Time to read first byte (seconds)
系列1 系列2
21 / 23

Overhead after migration
• Using filebench for practical case
– Client reads 10,000 files randomly so files are copied little
by little
– As migration progresses, the overhead decreases
0
500
1000
1500
2000
2500
1 21 41 61 81 101
系列1 系列2
(Readperseconds)
(sec)
0
500
1000
1500
2000
1 21 41 61 81 101
系列1 系列2
(Readperseconds)
(sec)
22 / 23

6. Conclusion
• Quick migration scheme of heterogeneous file servers
– Post-copy of file data and metadata
– Combine on-demand and background migration built on stub
management
• Achieve requirements
– Independent and no modification of source server
– Short downtime (<30 seconds in usual case)
– No performance overhead remained after migration
23 / 23

The Quick Migration of File Servers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Quick Migration of File Servers

Similar to The Quick Migration of File Servers (20)

More from Shinagawa Laboratory, The University of Tokyo

More from Shinagawa Laboratory, The University of Tokyo (8)

Recently uploaded

Recently uploaded (20)

The Quick Migration of File Servers