VMDK growth patterns (VMFS3 / 5)

I got curious about block sizes and how thin provisioned .vmdks and snapshot .vmdks grow, especially now that VMFS5 uses a 1MB “unified” block size.

I came up with the following table:

VMDK File growth patterns
VMFS3 ESXi 4.1………Thin Provision…..Snapshots
1MB block size………………..2MB……………………16MB
8MB block size……………….16MB……………………16MB

VMFS5 ESXi 5
Native 1MB block…………….1MB…………………….16MB
Upgraded VMF3 – 8MB block……16MB…………………….16MB

So with VMFS3, thin provisioned VMDKs grow at twice the default block size. ie as data is added, a VMDK on a datastore created with a 1MB block size will grow in 2MB chunks. A VMDK on an 8MB VMFS3 datastore will grow in 16MB chunks.

For VMFS5 thin provisioned VMDKs will grow in 1MB chunks.

When a snapshot is taken (VMFS3 or VMFS5, thin or thick VMDK) that snapshot VMDK file will grow in 16MB chunks.

Note that biggest problem I had was finding the file size quickly enough – the vCenter GUI using Browse Datastore or the Storage Views tab took forever to come up-to-date.

Backing data:

Obtained using ls *.vmdk -s -h -l from the ESXi (either 4.1 or 5) local shell
(-s shows used size in 1k blocks and translates to the size shown in the “Browse Datastore” window)

VMFS3 8MB block size datastore
After formatting the VMDK:
131072 -rw——- 1 root root 40.0G Aug 14 02:08 TestVM_1-flat.vmdk
After adding three 1MB files
139264 -rw——- 1 root root 40.0G Aug 14 02:45 TestVM_1-flat.vmdk
After adding 9 1MB files
147456 -rw——- 1 root root 40.0G Aug 14 02:51 TestVM_1-flat.vmdk
After adding 9 1MB files
155648 -rw——- 1 root root 40.0G Aug 14 02:56 TestVM_1-flat.vmdk

Snapshots:
initial
8192 -rw——- 1 root root 84.0k Aug 14 02:58 TestVM_1-000001-delta.vmdk
Add one 1MB file
24576 -rw——- 1 root root 16.1M Aug 14 02:58 TestVM_1-000001-delta.vmdk
Add 15 1MB files
40960 -rw——- 1 root root 32.1M Aug 14 03:03 TestVM_1-000001-delta.vmdk
Add 16 1MB files
57344 -rw——- 1 root root 48.1M Aug 14 03:05 TestVM_1-000001-delta.vmdk

1MB block size datastore
After formatting the VMDK:
126976 -rw——- 1 root root 40.0G Aug 14 03:09 TestVM_1-flat.vmdk
add 1MB file
128000 -rw——- 1 root root 40.0G Aug 14 03:18 TestVM_1-flat.vmdk
add 1MB file
144384 -rw——- 1 root root 40.0G Aug 14 03:19 TestVM_1-flat.vmdk
add 1MB file
146432 -rw——- 1 root root 40.0G Aug 14 03:20 TestVM_1-flat.vmdk

Snapshots
initial snapshot
1024 -rw——- 1 root root 84.0k Aug 14 03:24 TestVM_1-000001-delta.vmdk
Add 15 1MB files
17408 -rw——- 1 root root 16.1M Aug 14 03:24 TestVM_1-000001-delta.vmdk
Add 16 1MB files
33792 -rw——- 1 root root 32.1M Aug 14 03:28 TestVM_1-000001-delta.vmdk

VMFS5 (1MB native)
After format
95232 -rw——- 1 root root 40.0G Aug 14 03:37 TestVM5_1-flat.vmdk
Add 1MB file
96256 -rw——- 1 root root 40.0G Aug 14 03:37 TestVM5_1-flat.vmdk
Add 1MB file
97280 -rw——- 1 root root 40.0G Aug 14 03:41 TestVM5_1-flat.vmdk
Add 1MB file
98304 -rw——- 1 root root 40.0G Aug 14 03:42 TestVM5_1-flat.vmdk

Snapshot
1024 -rw——- 1 root root 84.0k Aug 14 03:45 TestVM5_1-000001-delta.vmdk
Add four 1MB files
17408 -rw——- 1 root root 16.1M Aug 14 03:45 TestVM5_1-000001-delta.vmdk
Add sixteen 1MB file
33792 -rw——- 1 root root 32.1M Aug 14 03:46 TestVM5_1-000001-delta.vmdk
Add sixteen 1MB files
50176 -rw——- 1 root root 48.1M Aug 14 04:32 TestVM5_1-000001-delta.vmdk

Upgraded VMFS5 – 8MB block
/vmfs/volumes/4e476be1-075130d1-51da-00219b9529d7/TestVM5 # ls *-delta.vmdk -s -h -l
57344 -rw——- 1 root root 48.1M Aug 14 07:38 TestVM5_1-000001-delta.vmdk
sixteen 1MB files added
/vmfs/volumes/4e476be1-075130d1-51da-00219b9529d7/TestVM5 # ls *-delta.vmdk -s -h -l
73728 -rw——- 1 root root 64.1M Aug 14 07:38 TestVM5_1-000001-delta.vmdk

This entry was posted in Computing, Storage, Virtualization, VMware and tagged , , , , , , . Bookmark the permalink.

5 Responses to VMDK growth patterns (VMFS3 / 5)

  1. We always find it best to handle snapshots on the storage tier depending on you SAN vendor of course.

    Vmware Snapshots plus Vmware thin provisioning is not a great idea and puts a performance overhead on the host.

    • JAndrews says:

      VMware snapshots are best done in moderation, note tho that a proper VMware-aware array snapshot will include VMware snapshots first to quiesce the guests.

      Array snaps are the fastest short-term backup you can get, but also the most expensive.

  2. NiTRo says:

    Hi joshua, did you try on nfs datastore ?
    BTW, do you know if CBT is independent from vmfs blocksize ?
    Thanks for this interesting post 🙂

    • JAndrews says:

      I have not tried it on NFS, look for an update soon (assuming I remember ;))

      CBT is a separate file that tracks changed blocks. It will add to the list when your VMDK grows but it’s only a list of blocks.

  3. JAndrews says:

    On an NFS datastore it will be defined by the NAS – I tested on a NetApp NFS share and the file grew in 4k blocks (the NetApp blocksize) but in fits and starts, 4k, 8k, 12k, or 16k at a time while making 512b file changes. It’s more erratic if the partition is not aligned.

Leave a Reply