Troubleshooting the SAM-FS Archiver

Published   Read time 5 min(s) 10 sec(s)   (2648 views).  SAM-FS / QFS  

When you experience problems with archiving, we firstly need to run the archiver -lv to check if there are any errors in the archiver.cmd file. But we also need to check the following:

  • Are there errors in the archiver.cmd file?
  • Run the command "archiver -lv" to check the configuration syntax in the archiver.cmd file.
  • Is media available?
  • Is the archiver running?
  • Does the site have a valid license?
  • Is the archiver working, but no files are being archived?
  • Are there files in the file system that need to be archived?
  • Do files have to be staged before archive copies can be made? Is the file system full?
  • Are any of the drives reporting hardware problems?

Checking Archiver Log files for possible errors reported

  • Log files related to the archiver process that may be helpful in problem determination are:
    • /var/adm/sam-log
    • /var/adm/messages
    • /var/opt/SUNWsamfs/trace/sam-archiverd
    • /var/opt/SUNWsamfs/devlog
    • archiver log file (configurable)
  • The samu cli display may also provide some assistance. 
    # samu
:a filesystem

Supporting Documentation and Additional data collection

The sam-archiverd may not be performing the tasks as the system administrator has intended. If this is the case, the following things need to be checked.

  • Run the sam-archiverd manually with all list options (-A or -lv) and examine the output to assure that the desired action has been specified properly.
  • Run archiver -lv can be used to check the syntax of commands in the /etc/opt/SUNWsamfs/archiver.cmd file.
  • When you run the archiver -lv, you may see a message such as: 
    3 errors in "/etc/opt/SUNWsamfs/archiver.cmd"
 *** No archiving will be done. ***
  • If you see this message, make sure you fix all errors in the archiver.cmd file.
  • The most common symptom an archiver problem is a full file system. Often this is a result of an archive set running out of media. The following error will be evident in the archiver -lv output.: 
    2 archive sets have no VSNs available.
  • Make sure that VSNs are assigned to all archive sets, including the archive set for the file system (such as samfs1, samfs2, etc). The file system archive sets must be defined in the VSNS section of the archiver.cmd or no archiving will occur. The following error indicates that the file system archive sets have not been defined: 
    # /opt/SUNWsamfs/sbin/archiver -lv
    samfs1.1 has no volumes defined
    samfs2.1 has no volumes defined
    2 archive sets have no volumes defined
    Reading '/etc/opt/SUNWsamfs/archiver.cmd'.
  • Assure that all volumes required for archiving are available and have sufficient space on them for archiving.

Enabling archiver Tracing

If the sam-archiverd appears to cause excessive, unexplainable media activity or appears to be doing nothing, turn on the trace facility and examine the trace file.

  • To do this, edit the /etc/opt/SUNWsamfs/defaults.conf file and insert a line that says: 
    trace
sam-archiverd = on
endtrace
  • When enabling tracing where sam-archiverd is the name of the daemon which we want to trace and "all" says to trace all events. You will need to perform a samd stop and samd config to start the tracing. The trace file is located /var/opt/SUNWsamfs/trace/sam-archiverd by default.
  • To enabling daemon tracing of the archiver by command line issue the following: samset debug all

Archiver Diagnostics

Reasons your SAM-FS environment might not be archiving files include the following:

  • The archiver.cmd file has a syntax error. Run the archiver -lv command to identify the error, then correct the flagged lines.
  • The archiver.cmd file has a wait directive in it. Either remove the wait directive or override it by using the samu utility's :arrun command.
  • No volumes are available. You can view this from archiver -lv command output. Add more volumes as needed. You might have to export existing cartridges to free up slots in the automated library.
  • The volumes for an archive set are full. You can export cartridges and replace them with new cartridges (make sure that the new cartridges are labeled), or you can recycle the cartridges. For more information on recycling, see the Sun StorageTek Storage Archive Manager Archive Configuration and Administration Guide.
  • The VSN section of the archiver.cmd file does not list correct media. Check your regular expressions and VSN pools to ensure that they are correctly defined.
  • There is not enough space to archive any file on the available volumes. If you have larger files and it appears that the volumes are nearly full, the cartridges might be as full as the SAM environment allows. If this is the case, add cartridges or recycle.
  • The archiver.cmd file has the no_archive directive set for directories or file systems that contain large files.
  • The archive -n (archive never) command has been used to specify too many directories, and the files are never archived.
  • Large files are busy. Thus, they never reach their archive age and are not archived.
  • Hardware or configuration problems exist with the automated library.
  • Network connection problems exist between client and server. Ensure that the client and the server have established communications.

Additional Archiver Diagnostics

In addition to examining the items on the previous list, you should check the following when troubleshooting the archiver:

  • The syslog file (by default, /var/adm/sam-log). This file can contain archiver messages that indicate the source of a problem.
  • Volume capacity. Ensure that all required volumes are available and have sufficient space on them for archiving.
  • The trace files. If the archiver appears to cause excessive, unexplainable cartridge activity or appears to be doing nothing, turn on the trace facility and examine the trace file. For information on trace files, see the defaults.conf man page.
  • The truss -p pid command. You can use this command on the archiver process (sam-archiverd) to identify the system call that is not responding. For more information on the truss command, see the truss man page.
  • The showqueue command. This command displays the content of the archiver queue files and displays the progress of archiving. You can use it to observe the state of archiver requests that are being scheduled or archived. Any archive request that cannot be scheduled generates a message indicating the reason.