IBM Spectrum Scale with Active File Management (AFM) allows storing data safely across geographically distributed sites using a clustered file system cache. AFM moves data between the home cluster where data is primarily stored and cache clusters where data is made available on demand or periodically to increase availability. Modes like read-only, single-writer, and independent-writer define how data is cached, modified, and synchronized between sites.

1. IBM Systems
Technical Symposium
Store your data
safely at a
geographically
distributed site using
Spectrum Scale with
AFM
Trishali Nayar
(Spectrum Scale
Development)

2. Please note
• IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as
the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed. Therefore, no
assurance can be given that an individual user will achieve results similar to those
stated here.

3. Introduction
• Spectrum Scale is a fast, scalable and complete storage
solution for today’s data-intensive enterprise.
• Integrated tools designed to help organizations manage
petabytes of data and millions of files.
• Active File Management is a clustered file system cache,
using the underlying file system.
• Moves data on demand, periodically and continuously
which makes it extremely flexible.
• Helps increase global collaboration and immensely
increases data availability.

4. Definitions
Home Cluster/Site
The cluster or main site where data is stored.
Cache Cluster/Site
The cluster where data is cached.
Note:
The home and cache sites are created independent of each other in
terms of storage and network configuration. The number of nodes in
each of these sites can vary based on workload.

5. Fitment
Block
iSCSI
Global Namespace
Analytics
Transparent
HDFS
Spark
OpenStack
Cinder
Glance
Manilla
Object
Swift S3
Transparent Cloud
Powered by
IBM Spectrum Scale
Automated data placement and data migration
Disk Tape Shared Nothing
Cluster
FlashWorldwide Data
Distribution(AFM)
Site B
Site A
Site C
SMBNFS
POSIX
File
Encryption DR Site
AFM-DR
JBOD/JBOF
Spectrum Scale
RAID
Compression

6. Node Definitions
Gateway (GW) Node
On the cache site, a few nodes in the cluster are assigned special
responsibility of acting as gateway nodes. These gateway nodes are
used to send and receive data from the home cluster.
Multiple nodes can be configured as gateway nodes for load balancing,
workload distribution and better performance. The master GW node
manages the entire data transfer for the fileset.
Application Node
An application node is any node in the cache cluster that gets I/O
requests from applications.
A node can be both an application node and a GW node.

7. File system Operations
Synchronous Operations
Operations done at the cache like reads, lookups or stats which need to get a
response from the home site, before the application can be served.
The first time one gets a cache “miss” performance, but future times it becomes a
cache “hit”.
Configuring revalidation is possible for some modes.
Asynchronous Operations
Operations done at the cache like creating directories/files, writes, renames,
removes, truncates or setting permissions/attributes etc.
Once the operation is completed on the local filesystem at the cache and queued at
the GW node, the response is returned to the application.
The GW node maintains a queue of all these asynchronous operations that need to
be performed at the home cluster. These will happen at the home cluster after some
delay and this process is asynchronous, but continuous.

8. Data Flow
Pull Data
This is used to refer to the direction of data flow, when data is pulled
into the AFM cache from the home. Eg- on demand
Push Data
This is used to refer to the direction of data flow, when data is pushed
from the AFM cache to home. Or from primary site to secondary site, in
case of Disaster Recovery scenarios.
Revalidation
The process of comparing the metadata at cache and home to
determine if the data has changed at home. And if it has, then fetch the
latest contents.

9. Modes Available
Read-only (RO)
This is a mode used for pulling data from home. The data can be pulled on-demand i.e. on access or it
can be prefetched as well. The data is modified only at home and any changes get pulled into the cache
after the revalidation duration. The cache behaves like a read-only file system and creating and
modifying files is not allowed.
Single-writer (SW)
When a cache is configured in this mode, the cache site can exclusively write data. All asynchronous
operations at the cache get pushed to the home site asynchronously, hiding WAN latencies. This also
helps provide better performance to any applications which are run at the cache, as write-back caching
is done. When any asynchronous operation happens, an application can proceed as soon as the
operation happens locally on its filesystem at the cache. This same operation also gets queued on the
AFM gateway node.
There is a 1:1 relationship between the AFM single-writer cache fileset and the home fileset. This
implies that all the data is to be written at the single cache site and the home is used only for reading.
AFM cannot detect or prevent home site modification of data, the administrators need to ensure that the
data is not modified or accidently corrupted.
Local-update (LU)
This is used to pull data from home, but any changes made at the cache are not pushed to the home.
When a cache is configured in this mode, the cached data is available for both reading and writing. But
the data modified at the cache site is not sent back to the home site. So, this mode serves as a scratch-
cache. After the data is modified at cache, new updates made at home for that particular data object are
not pulled into the cache.

10. Modes Available
Independent-writer (IW)
This mode allows multiple cache filesets, located in different cache clusters to be associated with a
single home fileset, hence this is an example of N:1 mapping. But the important point to be noted is that
each cache site should perform asynchronous operations (includes writes) on different files. There is no
inter-cluster locking for a file getting modified, at multiple cache clusters. Each cache makes its updates
independently and these changes in the IW caches are pushed to the home. In case multiple sites
modify the same file and cause conflicts then the last writer will win. It is administrator’s responsibility to
control who has write access to files, to avoid such conflicts.
Once data is updated at home, all connected IW caches can fetch those changes on-demand based on
the revalidation intervals set. So on next data access all the IW caches will get synchronized with the
home. Data can also be pre-fetched into the cache.
Note:
As seen in the above modes, depending on where the data is created/modified sometimes the home
site can be referred to as the local site and the cache site can be referred to as the remote or edge or
geographically disperse site. Eg. In RO mode, the home cluster can be called the local site and cache
cluster can be considered as the remote site.
The vice versa is also true Eg- in the SW/IW mode the cache site is where data is generated and can be
considered as the local site and the home site can be considered as the remote site. So these terms
local or remote site can be applied to both the cache and home sites, based on location of data creation
and direction of data flow.

11. Capabilities
Eviction
When the cache needs to be smaller than home, you can save
storage costs.
Eviction means that data blocks of files residing in the cache are
removed from the local file system, but the metadata of these files
is retained at the cache.
Automatic Eviction: The automatic eviction is based on fileset
quotas.
Manual Eviction: can be done for specific files selected by an
Information Lifecycle Management (ILM) policy. This adds more
flexibility in terms of specifying which particular files shouldn’t be
eating up your disk space.

12. Capabilities
Prefetch Data
This refers to pre-populating the cache or pulling in the data from home
in advance. This can be done for the entire data at home. Or it can be
done for selective files, based on Information Lifecycle Management
(ILM) policy where you can specify which file names or files based on
modification time etc., need to be prefetched.
Parallel I/O
If the files written at the cache or read into the cache, are of large size
and above a configurable threshold limit, then the parallel I/O feature of
AFM can be used. This feature helps to break this write/read of a large
file into various chunks and distributes these chunks across multiple
gateway (GW) nodes in the cluster. Hence multiple channels of
communication with the home cluster can be used to quickly move the
data to and from the home site.

13. Data Distribution
• Media and Entertainment
• Software/Binary Distribution

14. Backup at Home
• Healthcare
and Life
Sciences
• Data
Archives/
Libraries
• Govt.
Institutions

15. Global Namespace
• Central and
Branch Offices
• Data can be made
available at all
sites.

16. Ingest and Disseminate Data

17. Capabilities
• Relation is always at a fileset level. Only supports independent
filesets.
• The ‘mmafmctl’ command has options like getstate,
flushpending, resumeRequeued
• Can create a new home for a SW/IW fileset.
• Can create a new cache from a home.
• Peer-snapshots.

18. Disconnection
• Cache can continue despite no connectivity with home or periods
when home is inaccessible.
• Updates to home are queued.
• Data is served from local cache, there is no revalidation with home
• Data not available in cache return as not existing error (ENOENT)

19. Disaster Recovery
Primary and Secondary
AFM can be used for disaster recovery (DR) solutions and there are 2
sites/clusters in this case, the primary and secondary.
Both these sites have the entire data on them.
The AFM modes useful for DR are also called primary and secondary
filesets.
When an AFM fileset is configured in this mode, the primary(RW,
Active) site exclusively creates or writes data. The secondary site(RO,
Passive) cannot modify the data. AFM has a mandatory 1:1 mapping
between primary and secondary filesets. This ensures that only a
particular primary can talk to a secondary.
Note: AFM DR feature is disabled by default and customers need
to review the deployment with the Spectrum Scale development
for approval

20. Disaster Recovery
NAS client
AFM
(configured as primary)
AFM
(configured as
secondary)
Push all updates
asynchronously
Client switches to
secondary on failure

21. Disaster Recovery
Failover
An operational mode in which the functions of a system component (such as a
server/network) are assumed by secondary system components when the
primary component becomes unavailable through either failure or scheduled
down time.
Failback
The process of restoring operations and applications to the primary facility after
they had been moved to a secondary machine or facility during failover.
Recovery Point Objective (RPO)
The interval indicating the amount of data loss which can be tolerated in the
event of failures or disasters.
Recovery time objective (RTO)
The amount of time it takes for an application to fail over when a disaster
occurs.

22. Useful Links
https://www.ibm.com/support/knowledgecenter/en/STXKQY/ibmspe
ctrumscale_welcome.html
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com
.ibm.spectrum.scale.v4r23.doc/b1lins_quickreference_afm.htm
https://developer.ibm.com/storage/2017/07/12/edge-caching-
across-multiple-sites-premium-access-performance-saving-storage-
costs/

23. Notice and disclaimers
• Copyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted
in any form without written permission from IBM.
• U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM.
• Information in these presentations (including information relating to products that have not yet been announced by IBM) has been
reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall
have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or
implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to,
loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to
the terms and conditions of the agreements under which they are provided.
• IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have
been previously installed. Regardless, our warranty terms apply.”
• Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
• Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented
as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost,
savings or other results in other operating environments may vary.
• References in this document to IBM products, programs, or services does not imply that IBM intends to make such products,
programs or services available in all countries in which IBM operates or does business.
• Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily
reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor
shall constitute legal or other guidance or advice to any individual participant or their specific situation.
• It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s
business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or
warrant that its services or products will ensure that the customer is in compliance with any law.

24. Notice and disclaimers continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or
other publicly available sources. IBM has not tested those
products in connection with this publication and cannot confirm
the accuracy of performance, compatibility or any other claims
related to non-IBM products. Questions on the capabilities of
non-IBM products should be addressed to the suppliers of those
products. IBM does not warrant the quality of any third-party
products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM expressly disclaims all
warranties, expressed or implied, including but not limited
to, the implied warranties of merchantability and fitness for
a particular, purpose.
The provision of the information contained herein is not intended
to, and does not, grant any right or license under any IBM
patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, AIX, BigInsights, Bluemix, CICS,
Easy Tier, FlashCopy, FlashSystem, GDPS, GPFS,
Guardium, HyperSwap, IBM Cloud Managed Services, IBM
Elastic Storage, IBM FlashCore, IBM FlashSystem, IBM
MobileFirst, IBM Power Systems, IBM PureSystems, IBM
Spectrum, IBM Spectrum Accelerate, IBM Spectrum Archive,
IBM Spectrum Control, IBM Spectrum Protect, IBM Spectrum
Scale, IBM Spectrum Storage, IBM Spectrum Virtualize, IBM
Watson, IBM z Systems, IBM z13, IMS, InfoSphere, Linear
Tape File System, OMEGAMON, OpenPower, Parallel
Sysplex, Power, POWER, POWER4, POWER7, POWER8,
Power Series, Power Systems, Power Systems Software,
PowerHA, PowerLinux, PowerVM, PureApplica- tion, RACF,
Real-time Compression, Redbooks, RMF, SPSS, Storwize,
Symphony, SystemMirror, System Storage, Tivoli,
WebSphere, XIV, z Systems, z/OS, z/VM, z/VSE, zEnterprise
and zSecure are trademarks of International Business
Machines Corporation, registered in many jurisdictions
worldwide. Other product and service names might
be trademarks of IBM or other companies. A current list of
IBM trademarks is available on the Web at "Copyright and
trademark information" at:
www.ibm.com/legal/copytrade.shtml.
Linux is a registered trademark of Linus Torvalds in the United
States, other countries, or both. Java and all Java-based
trademarks and logos are trademarks or registered
trademarks of Oracle and/or its affiliates.