Shutdown and Startup an Avamar system (step-by-step)

The scope of this post is to share a how to like procedure aimed at the shutdown and startup of Avamar system.

Steps involved:

  1. Pre-Requisite Tasks
  2. Generate a valid Checkpoint
  3. Shutdown Avamar Processes
  4. Shutdown the Linux OS
  5. Power-off all components
  6. Startup Linux Systems
  7. Start the Avamar Software

PRE-REQUISITE TASKS

  1. ssh onto the Avamar utility node and login 'admin' user.
  2. Load the ssh keys:
    # ssh-agent bash
    # ssh-add .ssh/dpnid
  3. Stop running backup jobs (via command-line)
    # avmain sessions | grep sessionid
    # avmail kill <sessionid>
  4. Suspend future cron job activity
    # suspend_crons
    NOTE: You will receive a warning about currently running cron jobs. CAUTION: Any running “cp_cron” or “gc_cron” processes must be allowed to complete. NOTE: If “hfscheck_cron” is running, this process can be stopped but stopping it also stops the validation of the associated checkpoint (making that a temporarily unreliable checkpoint for Avamar system rollbacks).
  5. To kill hfscheck_cron job:
    # hfscheck_kill
  6. To kill replication processes:
    # ps -efm | grep _cron
    # kill <process_id>
    # ps -efm | grep avtar
    # kill <process_id>
  7. Check for any running cron jobs or if previous cron jobs have completed:
    # ps -efww | grep _cron

GENERATE A VALID CHECKPOINT

Prior to shutting down the Avamar system, we must have a recent validated (HFScheck) checkpoint. If there are complications with the shutdown, we may have to rollback the system to the last checkpoint.

  1. Shows the CheckPoint and HFS Check:
    # cplist
    NOTE:
      If we have no valid checkpoint or recent checkpoint we must manually create them before we can continue:
  2. Run a manual checkpoint task:
    # avmaint checkpoint --ava
    # watch avmaint cpstatus
    the status field will change to 'completed' once the checkpoint has finished (press 'ctrl-c' to exit the watch process)
  3. Run a manual healthcheck task:
    # avmaint hfscheck --rolling --ava
  4. Monitor the healthcheck task until it completes:
    # watch avmaint hfscheckstatus
    the 'percent-completed' field will give an indication of how far the healthcheck has got, likewise the status field will change to 'completed' once the healthcheck task has finished (press 'ctrl-c' to exit the watch process)

STOPPING AVAMAR PROCESSES

Once the pre-requisite checks have been carried out, and we can verify that no cron processes are running, then it safe to continue.

  1. Check current software status:
    # dpnctl status
  2. Stop Avamar
    # dpnctl stop
    NOTE: You will be asked if you are sure you want to do this. Type ‘Y' to continue with the shutdown. NOTE: The gsan process may take a while to stop so be patient.
  3. Check current software status:
    # dpnctl status
    verify that the 'gsan', 'mcs' and 'ems' processes are stopped/down

SHUTDOWN LINUX OS

  1. Switch to the root user.
    # su -
  2. Load the dpn OpenSSH key:
    # ssh-agent bash
    # ssh-add ~dpn/.ssh/dpnid
    # export SYSPROBEUSER=root
  3. Run the Linux shutdown:
    # mapall --parallel --bg 'poweroff'
    this will shut down Linux on the Avamar host system “nodes” in parallel except for the spare node; KVM output on the systems would show "power down" when the host system is ready to power down.
  4. Shutdown the spare node:
    # ssh —x root@SPARENODE_IP ‘poweroff'
  5. Shutdown the utility node:
    # poweroff
    NOTE: Datastore nodes will not actually power off but will remain at a halted state with “power down” on the screen.

POWER-OFF ALL COMPONENTS

Once the “power off” process completes, the Avamar systems and any other components in the configuration can be powered off. (i.e. cabinet power, or any network switches used with the Avamar host systems)

STARTUP LINUX SYSTEMS

To startup Avamar host systems, power up any dependant equipment (cabinet power systems, network switches, etc) before powering up the Avamar nodes.

Power on all the Avamar host systems and wait a couple minutes for the systems to boot up.

Ping the respective node IP addresses from another host to verify the Avamar systems are running.Once the Avamar host systems are up...

  1. Login the Avamar server as 'admin' user.
  2. Load the admin ssh keys:
    # ssh-agent bash
    # ssh-add ~admin/.ssh/admin_key
  3. Verify the storage nodes are up and can be reached by the utility node:
    # mapall uptime
  4. Verify the spare node has properly started up:
    # ssh —x admin@SPARENODE_IP ‘uptime'

START THE AVAMAR SOFTWARE

  1. Check software status to verify processes are stopped/down:
    # dpnctl status
  2. Start software:
    # dpnctl start
    you will be prompted to confirm you wish to do this, type ‘Y' to continue with the startup. NOTE: The 'gsan' process may take a while to start so be patient.
  3. Check current software status:
    # dpnctl status
    confirm that the 'gsan', 'mcs' and 'ems' services are started.

The Avamar system should now be up and running.