With the recent release of Data ONTAP 8.1, NetApp now supports the ability to perform an in-place expansion (or “in-place upgrade”) of 32-bit aggregates to a 64-bit format. This non-disruptive process is triggered when adding disks to a 32-bit aggregate that would cause the aggregate size to exceed 16TB. While it’s possible to monitor, it’s largely invisible to most users. So let’s take a closer look to see what actually happens “behind the scenes” during an in-place expansion.But first, some background.
In WAFL, each file is described by an inode (containing metadata and pointers). Small files consist of inodes pointing directly to data blocks, while larger files have additional layers of indirection called “indirect blocks”.
With the introduction of 64-bit aggregates, WAFL now supports both 32-bit and 64-bit indirect block formats (pointer sizes). However, with larger pointer sizes comes a reduction in the number of “addresses” that an indirect block can hold. With 32-bit block pointers, that number is 510 at the FlexVol layer, while a 64-bit indirect block only contains 255 -- half the number of block pointers.
So replacing these 32-bit indirects with 64-bit indirects must include some sort of overarching structure…right?
"Buffer trees" (or "buftrees") allow WAFL to leverage a hierarchical metadata structure to keep track of the locations of data blocks of files, with the top level of each file being the inode of the file. For more details on FlexVol architecture, read the USENIX 2008 paper entitled, "FlexVol: Flexible, Efficient File Volume Virtualization in WAFL".
Once the expansion process is initiated, a scanner walks all inodes and volumes within an aggregate to perform the conversion. It is during this in-place expansion period that an aggregate or volume may contain both 32-bit and 64-bit buffer trees in a sort of “mixed” format. All new write traffic will always write out blocks in the 64-bit format. Upon completion, all buftrees in the active file system (excluding Snapshots) become 64-bit buftrees.
Nice, but there’s "no free lunch" here: Given the increase in block pointer size with 64-bit aggregates, the expansion process must also check for space and allocate additional blocks. But how?
Prior to the administrator triggering an in-place expansion, Data ONTAP 8.1 has already done some "homework" of its own: during an 8.1 upgrade, Data ONTAP runs a prequalification scanner to check existing volumes and estimates if additional blocks are needed to expand volumes, maintain space guarantees, etc.
Then, prior to the administrator actually triggering the expansion, WAFL estimates the amount of additional space consumed by the expansion process. If a volume does not have enough free space, the expansion does not start.
In my testing with the ONTAP virtual simulator, it was also possible to still run out of space even after increasing the volume size (as I was intentionally writing large files to a volume to intentionally consume space). Luckily, the expansion process includes checkpoints and restarts automatically when space becomes available. Of course most real world deployments will not encounter this scenario, as the addition of actual disks will result in more usable capacity.
Bottom-line: Data ONTAP 8.1 (both 7-Mode and Cluster-Mode) nondisruptively expand existing 32-bit aggregates to 64-bit aggregates, making Data ONTAP 8.1 a compelling upgrade for most organizations.
