A brief history of the NetWorker database (updated)

Published   Read time 4 min(s) 23 sec(s)   (2682 views).  NetWorker  

This post provides a brief history of how the NetWorker databases have evolved over time.

NetWorker 4.x

In NetWorker 4.x there were only 2 hosts within a NetWorker environment, the server and it's clients.

  • The Media Index was using WISS (Wisconsin Storage System)
  • The File Index was also using WISS
  • The Configuration DB was using text files
Media Index /nsr/mm ss*.* vol*.*
Client File Index /nsr/index/cl_name ss*.* vol*.*
Configuration DB /nsr/res nsr.res nsrjb.res nsrla.res servers

NetWorker 5.0

Storage Nodes got introduced. There was no real change with respect to the databases, just that the configuration file nsrjb.res now belonged to the Storage Node's package:

Media Index /nsr/mm ss*.* vol*.*
Client File Index /nsr/index/cl_name ss*.* vol*.*
Configuration DB /nsr/res nsr.res nsrjb.res nsrla.res servers

NetWorker 6.0

Two major changes occurred:

  1. In the Media Index the new tablespace clients, representing the ClintID, was added
  2. The File Index was converted into BLISS (Better Legato Information Storage System)
    • More resilient
    • Improved managability (no more compression required)
    • No recovery of the CFI to be able to run new backups
Media Index /nsr/mm/mmvolume6 clients*.* ss*.* vol*.*
Client File Index /nsr/index/cl_name/db6 ssid#.k0 ssid#.k1 ssid#.rec
Configuration DB /nsr/res nsr.res nsrjb.res nsrla.res servers

NetWorker 7.0

Hashing of the major resources was introduced allowing their direct and fast addressing:

Media Index /nsr/mm/mmvolume6 clients*.* ss*.* vol*.*
Client File Index /nsr/index/cl_index/db6 ssid#.k0 ssid#.k1 ssid#.rec
Configuration DB /nsr/res nsrdb/00 : nsrdb/09 nsrla.res servers

In fact this change was executed in two phases:

  1. For the NetWorker Windows server this was NetWorker 6.2
  2. For all other NetWorker server platforms, this was NetWorker 7.0

NetWorker 7.3

There were a lot of databases changes

  • The Media index was interanlly improved with more resilient due to journaling and checksums
  • The Configuration DD clent-side resources were now also hashed to store the SSL keys
  • The Jornaling DB was introduced to store NetWorker jobs' status information
Media Index /nsr/mm/mmvolume6 clients*.* ss*.* vol*.*
Client File Index /nsr/index/cl_name/db6 ssid#.k0 ssid#.k1 ssid#.rec
Configuration DB /nsr/res nsrdb/00 : nsrdb/09 nsrladb/00 : nsrladb09 servers
Jobs Database /nsr/res/jobsdb general/00 : general/00 indication/00 : indication/09 savejobs/00 : savejobs/09 ssninfo/00 : ssinfo/09

NetWorker 7.4

To support the Client Push functionality, two new databases were added:

  1. The Client Push DB to store the client's software version information
  2. The Software Repository to maintain the various software versions
Media Index /nsr/mm/mmvolume6 clients*.* ss*.* vol*.*
Client File Index /nsr/index/cl_name/db6 ssid#.k0 ssid#.k1 ssid#.rec
Configuration DB /nsr/res nsrdb/00 : nsrdb/09 nsrladb/00 : nsrladb/09 servers
Jobs Database /nsr/res/jobsdb general/00 : general/09 indication/00 : indication/09 savejobs/00 : savejobs/09 ssninfo/00 : ssinfo/09
Client Push DB /nsr/res cpdb/00 : cpdb/09
SW Repository /nsr/repository

NetWorker 9.0

In NetWorker 9.0, SQLlite replaces WiSS as the media database.

The new database features the following advantages:

  • Object caching — a targeted cache facility that operates independently of file system or database caching to maintain recently used objects in memory for subsequent requests.
  • Parallel request processing (or multi-threading) — the database in previous versions of NetWorker was single-threaded, which means it could handle only one database request at a time. As a result, long transactions could delay the performance of any other transactions (for example, an operational request would have to wait until the bootstrap was completed). With multi-threading, the database can handle requests in parallel, so that the operational request would be picked up and handled immediately without requiring completion of the bootstrap.
  • Request handling — any request that takes multiple seconds gets logged. You can choose to log all requests by setting dbgcommand to level 1 or higher in debug mode.
  • Bootstrap compatibility — the mechanism being used for SQLlite is the same as for the WiSS database. If a problem occurs after upgrading, you can use a bootstrap taken from the previous system and recover that information directly into the SQL database. If you must temporarily downgrade NetWorker but already performed backups using NetWorker 9.0 that you want to maintain, you can perform a bootstrap and recover the media database into NetWorker 8.2, and that data will be recovered into the WiSS database.

The nsrmmdbd process handles the migration automatically during NetWorker startup after the upgrade. The database migration does not require any user intervention, and occurs in three stages (clients, volumes, and then save set records), with the daemon log indicating the beginning of each stage and logging any errors that occur. Once all data has been moved to the new database, NetWorker performs a cross check to verify the success of the migration. The migration process typically takes between 1-3 minutes but can take longer for very large databases (1 GB or greater).

On successful migration, the /nsr/mm/mmvolume6 directory gets appended with a timestamp to indicate that the directory is no longer active. You can keep this directory if you do not have a current bootstrap backup, otherwise it can be removed. The mmvolrel directory gets used as the new directory for the SQLlite database.

If the database migration does not complete, nsrmmdbd notifies you that migration has failed and the legacy WiSS database continues to run to process jobs until the migration is successful.