Chapter 13. In-service Data Model Upgrade

When we want to change the data model used by ConfD, the simplest method is to stop and restart ConfD with the new .fxs files in place. CDB will then detect the change and perform an upgrade, automatically or assisted by external programs, as described in Chapter 5, CDB - The ConfD XML Database.

If it is necessary that ConfD keeps running throughout the data model change, we can instead control the upgrade from an external program using a set of MAAPI functions, as described in this chapter. The CDB upgrade will be performed in this case too of course, and all the techniques described in the CDB chapter are applicable here too. But in addition, this procedure requires careful synchronization between different ConfD components, e.g. transactions may not span the data model change, and all components must update any related data, while still being able to revert to the original data model in case problems are detected.

The following four sections describe the phases and corresponding MAAPI function calls that comprise this upgrade procedure, and the steps that must be taken by the program controlling the procedure. A complete example showing the procedure can be found in examples.confd/in_service_upgrade/simple in the bundled examples collection. The code excerpts and description refer to this example. See also the confd_lib_maapi(3) manual page for the definitions of the MAAPI functions.

All the new .fxs files, clispecs, MIBs, etc, as well as the docroot tree for the Web UI (if used), that are to be used after the upgrade, must be installed separately from the current ones, and the current ones may not be removed until the upgrade has completed successfully. A good way to organize this is to have the references to the installation directories in confd.conf use a symbolic link. This way we can switch the on-disk data to the new version by simply changing the link, without the need for complex modification of confd.conf. Files that are unchanged between the two versions should be duplicated, or possibly (hard-)linked if disk space is limited.

In the example, we use two directories pkg/v1 and pkg/v2 for the old and new versions, respectively, and a symlink pkg/current that points to the currently used version. In confd.conf both /confdConfig/loadPath/dir and /confdConfig/webui/docroot are then given with the use of the symlink. Multiple loadPath directories are of course also possible, by having subdirectories below v1 and/or v2.

For MIB (.bin) files other than the ones built-in to ConfD, confd.conf offers two possibilities: we can either specify the actual file names with /confdConfig/snmpAgent/mibs/file elements, or use /confdConfig/snmpAgent/mibs/fromLoadPath to tell ConfD to load these files from the directories given via /confdConfig/loadPath. To have the symlink scheme work for the MIB files on upgrade, we need to use the latter alternative, and only specify built-in MIBs via /confdConfig/snmpAgent/mibs/file.

The upgrade.c program in the example controls the upgrade procedure. It can be run either standalone or via a osCommand specification in the clispec. In both cases it must connect a MAAPI socket and associate it with a user session. When running from the CLI, it must use the user session id provided by the environment variable CONFD_MAAPI_USID for this, otherwise it can start a new user session:

static int set_usess(int sock)
{
    char *user = "admin";
    const char *groups[] = {"admin"};
    char *context = "system";
    struct confd_ip ip;

    /* must use usid from CLI to be allowed to run across upgrade */
    if ((usid_env = getenv("CONFD_MAAPI_USID")) != NULL) {
        return maapi_set_user_session(sock, atoi(usid_env));
    } else {
        ip.af = AF_INET;
        inet_pton(AF_INET, "127.0.0.1", &ip.ip.v4);
        return maapi_start_user_session(sock, user, context, groups, 1,
                                        &ip, CONFD_PROTO_TCP);
    }
}

Applications connected to ConfD, e.g. data providers and CDB subscribers, are not directly affected by the upgrade procedure. If they need to take some action due to the upgrade, they should be subscribed to CONFD_NOTIF_UPGRADE_EVENT event notifications (see Chapter 12, Notifications), and will then be notified of the different phases of the upgrade. If nothing else, most applications should call confd_load_schemas() (see confd_lib_lib(3)) when an upgrade has completed, in order to update the in-memory representation of the data model. The cdb_subscriber.c program in the example shows how this can be done.

After having set up the MAAPI socket, the first step in the actual upgrade procedure is to call the maapi_init_upgrade() function. Its purpose is to bring ConfD into "upgrade mode", where no transactions are running, and the northbound agents have entered a state that does not allow new transactions to be started.

    progress("Initializing upgrade...\n");
    phase = "Init";
    /* run notifier in separate process
       - maapi_init_upgrade() blocks */
    notifier = run_notifier(timeout, force);
    OK(maapi_init_upgrade(maapisock, timeout,
                          force ? MAAPI_UPGRADE_KILL_ON_TIMEOUT : 0));
    if (notifier != -1)
        kill(notifier, SIGTERM);
    notifier = -1;
    progress("Init OK\n");
    maapi_prio_message(maapisock, "all",
                       "\n>>> System upgrade in progress...\n");

If users have sessions in configure mode when this function is called, they are given the opportunity to exit from configure mode voluntarily. The function call will block until all transactions have been terminated (although not longer than specified by the timeout). For this reason, we fork() a process that periodically sends out messages to all users, informing them of the imminent upgrade.

If any transactions remain when the timeout expires, maapi_init_upgrade() will fail with confd_errno CONFD_ERR_TIMEOUT, unless the MAAPI_UPGRADE_KILL_ON_TIMEOUT flag was used. If upgrade.c was given the -f (force) option, it will pass this flag to maapi_init_upgrade(), and any remaining transactions will be forcibly terminated instead.

When maapi_init_upgrade() is called, a CONFD_UPGRADE_INIT_STARTED event notification is sent, and when it completes successfully, a CONFD_UPGRADE_INIT_SUCCEEDED event notification is sent.

When maapi_init_upgrade() has completed successfully, the next step is to call maapi_perform_upgrade(). This tells ConfD to load the new .fxs files etc, and we must pass it a list of directories to load these files from. These are the directories that will become the new loadPath directories once the upgrade is complete. These directories will also be searched for CDB "init files" (see Section 5.8, “Loading initial data into CDB”), corresponding to the /confdConfig/cdb/initPath directories that can be specified in confd.conf.

    progress("Performing upgrade...\n");
    phase = "Perform";
    /* set up new loadpath directory */
    snprintf(buf, sizeof(buf), PKG_DIR "/%s", version);
    load_dir[0] = &buf[0];
    OK(maapi_perform_upgrade(maapisock, &load_dir[0], ndirs));
    progress("Perform OK\n");

At this point confd.conf and hence the current symlink must still point to the current version, and thus we pass the new directory "explicitly" to the function as "./pkg/v2". In this example ConfD was also started with the --addloadpath option specifying an additional loadPath directory. The contents of this directory ($CONFD_DIR/etc/confd) does not change in the upgrade, but we must pass the same directory to maapi_perform_upgrade() too - the files found in the given directories will completely replace what ConfD is currently using.

A number of different problems may be detected during the loading of the new files, e.g. .fxs files may have a version that is incompatible with the ConfD version, or they may reference namespaces that can not be found. These problems will make maapi_perform_upgrade() fail with confd_errno CONFD_ERR_BAD_CONFIG, and confd_lasterr() giving information about the details of the problem. If the loading is successful, CDB will start its special upgrade transaction, and perform any automatic upgrade operations that are needed, before maapi_perform_upgrade() returns.

When maapi_perform_upgrade() completes successfully, a CONFD_UPGRADE_PERFORMED event notification is sent.

When maapi_perform_upgrade() has completed successfully, we must call maapi_commit_upgrade() to tell ConfD to make the upgrade permanent. This will also tell CDB to commit its upgrade transaction, and we may need to take some actions for this before the call:

In the example, all the changes can be handled by the automatic CDB upgrade, and we just proceed with the call:

    progress("Committing upgrade...\n");
    phase = "Commit";
    OK(maapi_commit_upgrade(maapisock));
    relink(version);
    progress("Commit OK\n");

    maapi_prio_message(maapisock, "all",
                       ">>> System upgrade has completed successfully.\n");

maapi_commit_upgrade() may fail if the upgraded data does not pass validation, and the errors returned in this case are the same as for e.g. maapi_apply_trans(). Since this will also make ConfD automatically revert to the pre-upgrade state, we must not change the on-disk data to reflect the upgrade until maapi_commit_upgrade() has succeeded. In the code above, the relink() call changes the symlink to point to the new version in an atomic manner.

When maapi_commit_upgrade() completes successfully, a CONFD_UPGRADE_COMMITED event notification is sent.

We can abort the upgrade at any point before the maapi_commit_upgrade() call by calling maapi_abort_upgrade(). However as noted above, this should not be done when one of the other functions fails, since ConfD aborts the upgrade automatically in those cases.

When maapi_abort_upgrade() aborts an upgrade, a CONFD_UPGRADE_ABORTED event notification is sent.

When we use the ConfD High Availability functionality, it is critical that all nodes in the HA cluster agree on the data model used. For this reason we can not do in-service upgrade on a ConfD instance that is part of a HA cluster. A ConfD node in HA state SLAVE or MASTER executing maapi_init_upgrade() will fail with confd_errno CONFD_ERR_HA_WITH_UPGRADE. Conversely, when an in-service upgrade is in progress, calling confd_ha_beslave() will also result in this error, and connections from slaves will be rejected.

Thus in-service upgrade is only possible while in HA-state NONE.

To do the in-service upgrade on a HA cluster, we must thus use "rolling upgrade":

Alternatively, since the HA configuration should be able to handle that a node is stopped and restarted without service interruption, we may simply use the upgrade method described in the CDB chapter for the "rolling upgrade".