How to backup MongoDB database using lvm snapshots – Part 1

In this series of blog posts for MongoDB replica sets, I will show you how to properly run full backups using lvm snapshot followed by incremental backups using the oplog. I will also cover restores with point in time recovery using the previous backup. Snapshots work by creating pointers between the live data and a special snapshot volume. These pointers are theoretically equivalent to “hard links.” As the working data diverges from the snapshot, the snapshot process uses a copy-on-write strategy. As a result, the snapshot only grows as data is modified. 

What is LVM

Logical volume manager – lvm is a program that abstracts disk images from physical devices. It provides a number of raw disk manipulation and snapshot capabilities, useful for system management. A logical volume snapshot is a copy-on-write technology that monitors changes to an existing volume’s data blocks. When a write is made to one of the blocks, the block’s value at the snapshot time is copied to a snapshot volume. 

To be able to utilize lvm snapshot, your server must be using logical volume management, especially the partition where you mount your MongoDB data directory. Let’s see this with actual example:

I’m not going into details on how to configure lvm and how to create your volumes. It’s something that is already very well covered in different blog posts. I’ll quickly share sample output of pvdisplay, vgdisplay, lvdisplay from my test system. We will use the output in our backup commands later. Additionally try to separate your MongoDB directories from the rest of your file system so the backup will only have MongoDB related data.

 

root@mongodbhc:~# pvdisplay

  --- Physical volume ---

  PV Name               /dev/sdc1

  VG Name               vgdiskdata

  PV Size               <50.00 GiB / not usable 3.00 MiB

  Allocatable           yes

  PE Size               4.00 MiB

  Total PE              12799

  Free PE               7679

  Allocated PE          5120

  PV UUID               d1xGEN-mwW0-edRe-l1dF-SAjp-qmMA-1RdE9a

The above output from pvdisplay shows that there is a physical volume /dev/sdc1 and it’s used in a volume group vgdiskdata. Please note in this case we only have a single physical volume, but there could be more in your case.

 

root@mongodbhc:~# vgdisplay

  --- Volume group ---

  VG Name               vgdiskdata

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  341

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                1

  Open LV               1

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size               <50.00 GiB

  PE Size               4.00 MiB

  Total PE              12799

  Alloc PE / Size       5120 / 20.00 GiB

  Free  PE / Size       7679 / <30.00 GiB

  VG UUID               d99fRZ-Naup-ixHj-GRYl-XIBp-tJgv-T958AO

The output of vgdisplay as shown above, has the volume group name, its format, size and other parameters like Free. From a total 50GiB we have <30GiB free. We can extend this volume group size if we add more physical volumes to it.

 

root@mongodbhc:~# lvdisplay

  --- Logical volume ---

  LV Path                /dev/vgdiskdata/lvmongo

  LV Name                lvmongo

  VG Name                vgdiskdata

  LV UUID                kJ2e5W-DVWM-rXwC-xEwn-H5kP-Ws0w-TtuI2L

  LV Write Access        read/write

  LV Creation host, time mongodbhc, 2022-09-27 10:59:30 +0000

  LV Status              available

  # open                 1

  LV Size                20.00 GiB

  Current LE             5120

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:1




root@mongodbhc:~#

Finally, lvdisplay output shows the logical volume path, name and to which volume group it belongs to. See the name matches with our volume group vgdiskdata as that’s the volume group in which it’s created. Each time we create a snapshot from this logical volume lvmongo, it will be created in its volume group vgdiskdata. This is where the size of the volume group is important.

Now, if we want to see where this logical volume lvmongo is mounted to, we can run the command lsblk or use df. In my case, the MongoDB data directory is set as /mnt/mongo, so that is the file system path that I want to take backups on.

 

root@mongodbhc:~# lsblk /dev/sdc1

NAME                 MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sdc1                   8:33   0  50G  0 part

??vgdiskdata-lvmongo 253:1    0  20G  0 lvm  /mnt/mongo

root@mongodbhc:~# df -hTP /mnt/mongo/

Filesystem                     Type  Size  Used Avail Use% Mounted on

/dev/mapper/vgdiskdata-lvmongo xfs    20G  2.2G   18G  11% /mnt/mongo

Taking the snapshot

From the above output we can see the logical volume is mounted to /mnt/mongo mountpoint and its type lvm. So this is the partition on the file system that belongs to the logical volume /dev/vgdiskdata/lvmongo

Now that we have all of the information for file system partitions and logical volumes on our system, let’s create a lvm snapshot backup. We can restore our database in case of a failure scenario using this backup. 

Before we create the snapshot, let’s use some best practices from MongoDB and lock the database for writes. Even though this is not required for lvm snapshot, taking this step ensures success of the backup and eventual restore by guaranteeing that writes do not happen during the snapshot process. As soon as we create the snapshot and mount as read-only on the file system, we will unlock the database. This will be also useful for us to get the oplog position as we want to take incremental backups after each full backup.

Typically we run backups on MongoDB Secondary nodes, so this should have zero impact. In particular, you could have a hidden node or delayed Secondary as a dedicated backup node in your replica set. 

If you are running the command fsyncLock() in a mongo shell, it will look like this:

mongohc:SECONDARY> db.fsyncLock()

{

"info" : "now locked against writes, use db.fsyncUnlock() to unlock",

"lockCount" : NumberLong(1),

"seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1666351737, 1),

"signature" : {

"hash" : BinData(0,"5cqLjc3Rr7I9/Y+8D+dqeGrUBCY="),

"keyId" : NumberLong("7148394471967686661")

}

},

"operationTime" : Timestamp(1666351737, 1)

}

mongohc:SECONDARY>

 

You will most likely be using a script to perform this sequence of steps. This could be scripted as fsynclock.js file and then called with mongo client like this:

mongo -u<username> -p<password> --port <port> --quiet fsynclock.js

 

One additional note I’d like to highlight here for incremental backups using the oplog. If you want to take oplog backups between each full daily backup, you should take the oplog position now, when the database is locked and before you create the snapshot. This is something that you can perform using a script like below:

root@mongodbhc:~# cat backup_oplog_ts.js

var local = db.getSiblingDB('local');

var last = local['oplog.rs'].find().sort({'$natural': -1}).limit(1)[0];

var result = {};

if(last != null) {

    result = {position : last['ts']};

}

print(JSON.stringify(result));

root@mongodbhc:~# mongo -u<username> -p<password> --port <port> --quiet backup_oplog_ts.js > oplog_position

 

The oplog_position file will have information like below:

{"position":{"$timestamp":{"t":1666355398,"i":1}}}

 

We have full backup up to this point of time, for our incremental backups we will use this timestamp as a starting point. How to run the incremental backups will be part two of this blog series.

Our next command will create a snapshot using lvmongo as a source volume.

root@mongodbhc:~# lvcreate -L500M -s -n mongosnap_21oct2022 /dev/vgdiskdata/lvmongo

  Logical volume "mongosnap_21oct2022" created.

root@mongodbhc:~#

Few notes on the above command:

-L  specifies the size of the snapshot. We need to ensure there is enough space in the volume group where the original volume is created for the extra space we specify here. Even if we go above the available size, it will only use what is remaining in the volume group. This is important if you plan to keep your snapshot for a longer time on a system with heavy writes.

root@mongodbhc:~# lvcreate -L40G -s -n largesnap /dev/vgdiskdata/lvmongo

  Reducing COW size 40.00 GiB down to maximum usable size 20.08 GiB.

  Logical volume "largesnap" created.

root@mongodbhc:~#

-n specifies the snapshot name, please note the name like snapshot is reserved and cannot be used.

-s specifies it’s a snapshot

 

Let’s check how the lvdisplay output looks now

root@mongodbhc:~# lvdisplay

  --- Logical volume ---

  LV Path                /dev/vgdiskdata/lvmongo

  LV Name                lvmongo

  VG Name                vgdiskdata

  LV UUID                kJ2e5W-DVWM-rXwC-xEwn-H5kP-Ws0w-TtuI2L

  LV Write Access        read/write

  LV Creation host, time mongodbhc, 2022-09-27 10:59:30 +0000

  LV snapshot status     source of

                         mongosnap_21oct2022 [active]

  LV Status              available

  # open                 1

  LV Size                20.00 GiB

  Current LE             5120

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:1




  --- Logical volume ---

  LV Path                /dev/vgdiskdata/mongosnap_21oct2022

  LV Name                mongosnap_21oct2022

  VG Name                vgdiskdata

  LV UUID                fv7BHZ-JClM-axTQ-AJUU-XTkP-DezD-c8e8Q2

  LV Write Access        read/write

  LV Creation host, time mongodbhc, 2022-10-21 10:57:57 +0000

  LV snapshot status     active destination for lvmongo

  LV Status              available

  # open                 0

  LV Size                20.00 GiB

  Current LE             5120

  COW-table size         500.00 MiB

  COW-table LE           125

  Allocated to snapshot  0.31%

  Snapshot chunk size    4.00 KiB

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:3




root@mongodbhc:~#

As we can see, there is another logical volume with the name: mongosnap_21oct2022 and part of the same volume group vgdiskdata. For lvmongo we can see details about “Source of mongosnap_21oct2022, while for mongosnap_21oct2022, we can see “Active destination for lvmongo”

Additionally we can use the command lvs that has the output below. There are two logical volumes, belonging to the same volume group, but for the snapshot we can see its origin is lvmongo.

root@mongodbhc:~# lvs

  LV                  VG         Attr       LSize   Pool Origin  Data%  Meta%  Move Log Cpy%Sync Convert

  lvmongo             vgdiskdata owi-aos---  20.00g

  mongosnap_21oct2022 vgdiskdata swi-a-s--- 500.00m      lvmongo 3.25

 

The new snapshot can be used as logical volume and mounted on the file system. From there we can copy the files to offsite locations. Using this approach, we can restore the exact same copy of the database at the time we took the backup on any system. 

The steps will include:

Mount the snapshot to tmp mount as a read only file system. This is important because with lvm snapshot, the data is changing. The snapshot is capturing the changes and that will be reflecting on the file system. If there are writes while we backup and copy the archive elsewhere, the database will be in a corrupted state when we try to restore.

mkdir /tmp/mongosnap

mount -t xfs -o nouuid,ro /dev/vgdiskdata/mongosnap_21oct2022 /tmp/mongosnap/

 

Before we go further, the snapshot is taken, mounted as read only on the file system. Let’s unlock the database for writes and let the replication resume. Between the lock and unlock it should be a few seconds time, so this should not be any impactful on replication lag.

mongohc:SECONDARY> db.fsyncUnlock()

{

"info" : "fsyncUnlock completed",

"lockCount" : NumberLong(0),

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1666351827, 1),

"signature" : {

"hash" : BinData(0,"/ZbNlG1binKXSO9f4trXCc2LdsE="),

"keyId" : NumberLong("7148394471967686661")

}

},

"operationTime" : Timestamp(1666351737, 1)

}

mongohc:SECONDARY>

 

Or if you want to call this from mongo client, place the command in a fsyncunlock.js file and execute 

mongo -u<username> -p<password> --port <port> --quiet fsyncunlock.js

 

Tar the directory with the designated name and move the tar backup file to backups location. This can be an NFS mount point, separate disk on the system, or even copied to a cloud bucket. 

tar -czf mongodb_backup_$(date '+%Y%m%d%H%M').tar.gz -C /tmp/mongosnap/ .

mv mongodb_backup_$(date '+%Y%m%d%H%M').tar.gz /backups/

Or

tar -czf /backups/mongodb_backup_$(date '+%Y%m%d%H%M').tar.gz --absolute-names /tmp/mongosnap

 

Finally, unmount the partition and remove the snapshot. 

umount /tmp/mongosnap

lvremove /dev/vgdiskdata/mongosnap_21oct2022

If you repeat this daily, you can have daily full backup and between each full backup take incremental backups using the oplog. 

NEXT:

INCREMENTAL BACKUPS USING THE OPLOG is covered in  Part 2 of this series.

RESTORES WITH POINT IN TIME RECOVERY is covered in Part 3 of this series.

 

Conclusion

Running LVM provides additional flexibility and enables the possibility of using snapshots to back up MongoDB. Using daily full backup followed by incremental oplog backup will allow you to do PITR. You can restore your database just before you execute an erroneous operation and recover your data with confidence.