Shutdown and Startup an Avamar system (step-by-step)

Published 22 Mar 2014 Read time 3 min(s) 52 sec(s) (10450 views). Storage

The scope of this post is to share a how to like procedure aimed at the shutdown and startup of Avamar system.

Steps involved:

Pre-Requisite Tasks
Generate a valid Checkpoint
Shutdown Avamar Processes
Shutdown the Linux OS
Power-off all components
Startup Linux Systems
Start the Avamar Software

PRE-REQUISITE TASKS

ssh onto the Avamar utility node and login 'admin' user.
Load the ssh keys:
```
# ssh-agent bash
# ssh-add .ssh/dpnid
```

Stop running backup jobs (via command-line)

# avmaint sessions | grep sessionid
# avmail kill <sessionid>

Suspend future cron job activity
```
# suspend_crons
```
NOTE: You will receive a warning about currently running cron jobs. CAUTION: Any running “cp_cron” or “gc_cron” processes must be allowed to complete. NOTE: If “hfscheck_cron” is running, this process can be stopped but stopping it also stops the validation of the associated checkpoint (making that a temporarily unreliable checkpoint for Avamar system rollbacks).
To kill hfscheck_cron job:
```
# hfscheck_kill
```

To kill replication processes:

# ps -efm | grep _cron
# kill <process_id>
# ps -efm | grep avtar
# kill <process_id>

Check for any running cron jobs or if previous cron jobs have completed:
```
# ps -efww | grep _cron
```

GENERATE A VALID CHECKPOINT

Prior to shutting down the Avamar system, we must have a recent validated (HFScheck) checkpoint. If there are complications with the shutdown, we may have to rollback the system to the last checkpoint.

Shows the CheckPoint and HFS Check:
```
# cplist
```
NOTE:
Run a manual checkpoint task:
```
# avmaint checkpoint --ava
# watch avmaint cpstatus
```
the status field will change to 'completed' once the checkpoint has finished (press 'ctrl-c' to exit the watch process)
Run a manual healthcheck task:
```
# avmaint hfscheck --rolling --ava
```
Monitor the healthcheck task until it completes:
```
# watch avmaint hfscheckstatus
```
the 'percent-completed' field will give an indication of how far the healthcheck has got, likewise the status field will change to 'completed' once the healthcheck task has finished (press 'ctrl-c' to exit the watch process)

STOPPING AVAMAR PROCESSES

Once the pre-requisite checks have been carried out, and we can verify that no cron processes are running, then it safe to continue.

Check current software status:
```
# dpnctl status
```
Stop Avamar
```
# dpnctl stop
```
NOTE: You will be asked if you are sure you want to do this. Type ‘Y' to continue with the shutdown. NOTE: The gsan process may take a while to stop so be patient.
Check current software status:
```
# dpnctl status
```
verify that the 'gsan', 'mcs' and 'ems' processes are stopped/down

SHUTDOWN LINUX OS

Switch to the root user.
```
# su -
```

Load the dpn OpenSSH key:

# ssh-agent bash
# ssh-add ~dpn/.ssh/dpnid
# export SYSPROBEUSER=root

Run the Linux shutdown:
```
# mapall --parallel --bg 'poweroff'
```
this will shut down Linux on the Avamar host system “nodes” in parallel except for the spare node; KVM output on the systems would show "power down" when the host system is ready to power down.

Shutdown the spare node:

# ssh —x root@SPARENODE_IP ‘poweroff'

Shutdown the utility node:
```
# poweroff
```
NOTE: Datastore nodes will not actually power off but will remain at a halted state with “power down” on the screen.

POWER-OFF ALL COMPONENTS

Once the “power off” process completes, the Avamar systems and any other components in the configuration can be powered off. (i.e. cabinet power, or any network switches used with the Avamar host systems)

STARTUP LINUX SYSTEMS

To startup Avamar host systems, power up any dependant equipment (cabinet power systems, network switches, etc) before powering up the Avamar nodes.

Power on all the Avamar host systems and wait a couple minutes for the systems to boot up.

Ping the respective node IP addresses from another host to verify the Avamar systems are running.Once the Avamar host systems are up...

Load the admin ssh keys:

# ssh-agent bash
# ssh-add ~admin/.ssh/admin_key

Verify the storage nodes are up and can be reached by the utility node:
```
# mapall uptime
```
Verify the spare node has properly started up:
```
# ssh —x admin@SPARENODE_IP ‘uptime'
```

START THE AVAMAR SOFTWARE

Check software status to verify processes are stopped/down:
```
# dpnctl status
```
Start software:
```
# dpnctl start
```
you will be prompted to confirm you wish to do this, type ‘Y' to continue with the startup. NOTE: The 'gsan' process may take a while to start so be patient.
Check current software status:
```
# dpnctl status
```
confirm that the 'gsan', 'mcs' and 'ems' services are started.

The Avamar system should now be up and running.