Manually cleaning NetWorker backups on an Avamar system
Having recently got involved on a site, it came to light very quickly that some housekeeping was required on the Avamar system, as such I spent 2 days reading a large number of EMC documentation to come up with a plan.
The scope of this document is to provide a how to like procedure aimed at cleaning up rouge and outdated NetWorker backups held on an Avamar system.
Steps involved:
PRE-REQUISITE TASKS
Prior to running the manual cleanup we need perform the following checks:
- Open a putty session and login the Avamar server as 'admin' user.
- Load the Admin keys:
# ssh-agent bash # ssh-add .ssh/dpnid
NOTE: We need to load the admin keys to allow us to authenticate correctly with the Avamar system.Check to make sure no backup sessions are currently running:
# avmaint sessions | grep path
- Check to see if we any running/hung sessions:
# avmaint sessions | grep sessionid
The above will list all running and hung sessions (including replication sessions) - If you need to manually kill a hung session, perform:
# avmaint kill <sessionid>
replace <sessionid> with the value found in the previous step
MANUAL CLEANUP
Once the pre-requisite checks have been carried out, we can start with the cleanup operation.
- Stop the scheduler so no new backups from NetWorker to Avamar start:
# dpnctl stop sched
- Stop the maintenance scheduler so no checkpoint, healthcheck or garbage collection tasks start:
# dpnctl stop maint
- Run the following command to identify NetWorker savesets older than a given date:
# delete-snapups --before=mm/dd/yyyy --domain=/NetWorker > cleanup_candidates.sh
This will generate a file named "cleanup_candidates.sh" that is auto formatted like a script to delete entries. So, when you execute the file, it starts deleting. - To determine the number of entries in the script that the cleanup will perform, we can simply grep and count the lines:
# grep "^m" cleanup_candidates.sh | wc -l
- Execute the 'cleanup_candidates.sh' script, to purge the entries:
# chmod +x ./cleanup_candidates.sh # ./cleanup_candidates.sh
AVAMAR HOUSEKEEPING
Once the manual cleanup has completed, we need to run a series of Avamar housekeeping tasks (checkpoint, healthcheck and garbage collection)
- Run a manual checkpoint task and monitor until it completes:
# avmaint checkpoint --ava # watch avmaint cpstatus
the status field will change to 'completed' once the checkpoint has finished (press 'ctrl-c' to exit the watch process) - Run a manual healthcheck task and monitor until it completes:
# avmaint hfscheck --rolling --ava # watch avmaint hfscheckstatus
the 'percent-completed' field will give an indication of how far the healthcheck has got, likewise the status field will change to 'completed' once the healthcheck task has finished (press 'ctrl-c' to exit the watch process) - Run a manual garbage collection task:
# avmaint garbagecollect --ava --maxtime=3600
(setting maxtime=3600 allows the garbage collection to run for only 1 hour) - Monitor the garbage collection task until it completes:
# watch avmaint gcstatus
the status field will change to 'idle' once the garbage collection task has finished (press 'ctrl-c' to exit the watch process)
POST MAINTENANCE
Once the manual cleanup and housekeeping tasks have completed, we can restarted the schedulers.
- Restart the maintenance window scheduler:
# dpnctl start maint
- Restart the backup scheduler:
# dpnctl start sched
- Check the status of both schedulers:
# dpnctl status
we need to confirm that the 'maintenance window scheduler' is 'enabled' and the 'backup scheduler' is 'up'