Chapter 28. Advanced Topics

The ConfD management backplane keeps a lock for each datastore: running, startup and candidate. These locks are usually referred to as the global locks and they provide a mechanism to grant exclusive access to the datastore the lock guards.

The global locks are the only locks that can explicitly be taken through a northbound agent, for example by the NETCONF <lock> operation, or by calling maapi_lock().

A global lock can be taken for the whole datastore, or it can be a partial lock (for a subset of the datamodel). Partial locks are exposed through NETCONF and MAAPI.

An agent can request a global lock to ensure that it has exclusive write-access to a datastore. When a global lock is held by an agent it is not possible for anyone else to write to the datastore the lock guards - this is enforced by the transaction engine. A global lock on a datastore is granted to an agent if there are no other holders of it (including partial locks), and if all dataproviders approve the lock request. Each dataprovider (CDB and/or external dataproviders) will have its lock() callback invoked to get a chance to refuse or accept the lock. The output of confd --status includes locking status. For each user session locks (if any) per datastore is listed.

A northbound agent starts a user session towards ConfD's management backplane. Each user session can then start multiple transactions. A transaction is either read/write or read-only and is always started against a specific datastore.

The transaction engine has its internal locks, one for every datastore. These transaction locks exists to serialize configuration updates towards the datastore and are separate from the global locks.

As a northbound agent wants to update a datastore with a new configuration it will implicitly grab and release the transactional lock corresponding to the datastore it is trying to modify. The transaction engine takes care of managing the locks, as it moves through the transaction state machine and there is no API that exposes the transactional locks to the northbound agents.

When the transaction engine wants to take a lock for a transaction (for example when entering the validate state) it first checks that no other transaction has the lock. Then it checks that no user session has a global lock on that datastore. Finally each dataprovider is invoked by its trans_lock() callback.

In contrast to the implicit transactional locks, some northbound agents expose explicit access to the global locks. This is done a bit differently by each agent.

The management API exposes the global locks by providing maapi_lock() and maapi_unlock() functions (and the corresponding maapi_lock_partial() maapi_unlock_partial() for partial locking). Once a user session is established (or attached to) these functions can be called.

In the CLI the global locks are taken when entering different configure modes as follows:

The global locks are then kept by the CLI until either the configure mode is exited, or in the case of commit confirmed <timeout> the lock is released when it returns.

The Web UI behaves in the same way as the CLI (it presents three edit tabs called "Edit private", "Edit exclusive", and "Edit shared" which corresponds to the CLI modes described above).

The NETCONF agent translates the <lock> operation into a request for the global lock for the requested datastore. Partial locks are also exposed through the partial-lock rpc.

Implementing the lock() and unlock() callbacks is not required of an external dataprovider. ConfD will never try to initiate the trans_lock() state transition (see the transaction state diagram in Section 7.5, “User sessions and ConfD Transactions”) towards a data provider while a global lock is taken - so the reason for a dataprovider to implement the locking callbacks is if someone else can write (or lock for example to take a backup) to the data providers database.

CDB ignores the lock() and unlock() callbacks (since the data-provider interface is the only write interface towards it).

CDB has its own internal locks on the database. The running and startup datastore each has a single write and multiple read locks. It is not possible to grab the write-lock on a datastore while there are active read-locks on it. The locks in CDB exists to make sure that a reader always gets a consistent view of the data (in particular it becomes very confusing if another user is able to delete configuration nodes in between calls to get_next() on YANG list entries).

During a transaction trans_lock() takes a CDB read-lock towards the transactions datastore and write_start() tries to release the read-lock and grab the write-lock instead.

A CDB external client (usually referred to as an MO, managed object) implicitly takes a CDB read-lock between cdb_start_session() and cdb_end_session() on the specified datastore (running or startup). This means that while an MO is reading, a transaction can not pass through write_start() (and conversely a CDB reader can not start while a transaction is in between write_start() and commit() or abort()).

The Operational store in CDB does not have any locks. ConfD's transaction engine can only read from it, and the MO writes are atomic per write operation.

When a session tries to modify a data store that is locked in some way, it will fail. For example, the CLI might print:

admin@host% commit
Aborted: the configuration database is locked
[error][2009-06-11 16:27:21]
        

Since some of the locks are short lived (such as a CDB read lock), ConfD can be configured to retry the failing operation for a short period of time. If the data store still is locked after this time, the operation fails.

To configure this, set /confdConfig/commitRetryTimeout in confd.conf.

On a typical system, the configuration data resides in ConfD's database CDB. Some of the parameters in the configuration are intended for the target OS environment, such as the IP address of the management interface. The OS reads this information from its own configuration files, such as /etc/conf.d/net. This means that the application typically reads this data from CDB, and generates configuration files needed by the system before starting them. If a manager changes one of these parameters, the application subscribes to changes in CDB, regenerates the files, and restarts the system daemons. This mechanism can also be used for the configuration of ConfD itself. The application must subscribe to changes to any parameter affecting ConfD (such as management IP address), update the ConfD configuration file confd.conf, and then instruct ConfD to reload it.

ConfD comes bundled with a small example tool which can be used to patch confd.conf files: $CONFD_DIR/src/confd/tools/xmlset.c. This tool uses the light-weight Expat XML Parser (http://expat.sourceforge.net/).

This example changes confd.conf to disable the Web UI:

$ xmlset C false confdConfig webui enabled < confd.conf

This example changes confd.conf to removes the encryptedStrings container:

$ xmlset R confdConfig encryptedStrings < confd.conf

In the ConfD distribution in the $CONFD_DIR/src/confd/dyncfg directory the confd_dyncfg.yang YANG module is included. The module defines the namespace http://tail-f.com/ns/confd_dyncfg/1.0 which contains all the ConfD configuration parameters that can be modified in runtime. I.e. it is a subset of the namespace that defines the ConfD configuration file (confd.conf).

To enable the feature of storing ConfD's configuration in CDB the setting /confdConfig/runtimeReconfiguration has to be set to namespace in the configuration file. This instructs ConfD to read all its "static" configuration from the configuration file, and then load the rest of the configuration from the confd_dyncfg namespace (which must be served by CDB). A requirement is that the confd_dyncfg.fxs is in ConfD's loadPath. It is also advisable to have a suitable _init.xml file in ConfD's CDB directory.

The best way to understand how to use this feature is the example confdconf/dyncfg in the bundled example collection.

In most cases the interesting use of this feature is to be able to expose a particular aspect of ConfD's configuration to the end-user and hide the rest. This can be achieved by combining the use of the --export none flag when compiling the confd_dyncfg.yang module with the use of the symlink feature (exactly how they work are explained in Section 10.7, “Hidden Data”). The snmpa/6-dyncfg example in the example collection shows how to expose a small subset of the SNMP agent configuration (as well as some minor aspects of the CLI parameters) in a private namespace.

For example, if we want to be able to expose the ConfD's built-in SNMP agents listen port as an end-user configurable as the leaf /sys/snmp-port, we could write a YANG model like this:

container sys {
        tailf:symlink snmp-port {
          tailf:path "/dyncfg:confdConfig/dyncfg:snmpAgent/dyncfg:port";
        }
      }

When a transaction containing changes to /confdConfig is committed ConfD will pick up the changes made and act accordingly. Thus there is no longer a need for confd --reload except for closing/re-opening of log-files (as described above) or to update the fxs files for sub-agents.

When /confdConfig/runtimeReconfiguration is set to namespace, any settings in confd.conf for the parameters that exist in the confd_dyncfg namespace are ignored, with one exception: the configuration under /confdConfig/logs. This configuration is needed before CDB has started, and ConfD will therefore initially use the settings from confd.conf, with the CDB settings taking precedence once CDB has started (i.e. when the transition to phase1 is completed).

We can set up ConfD to use a different IPC mechanism than TCP for the client library connections, as well as for the communication between ConfD nodes in a HA cluster. This can be useful e.g. in a chassis system where ConfD runs on a management blade, while the managed objects run on data processing blades that may not have a TCP/IP implementation.

There are several requirements that must be fulfilled by such an IPC mechanism:

For ConfD to be able to use this mechanism without knowledge of address format etc, we must provide C code in the form of a shared object, which is dynamically loaded by ConfD. The interface between ConfD and the shared object code is defined in the ipc_drv.h file in the $CONFD_DIR/src/confd/ipc_drv directory in the release. The shared object must be named ipc_drv_ops.so and installed in the $CONFD_DIR/lib/confd/lib/core/confd/priv directory of the ConfD installation, see the sample Makefile in the ipc_drv directory. The interface is implemented via the confd_ext_ipc_init() function. This function must be provided by the shared object, and it must return a pointer to a callback structure defined in the shared object:

struct confd_ext_ipc_cbs {
    int (*getaddrinfo)(char *address,
                       int *family, int *type, int *protocol,
                       struct sockaddr **addr, socklen_t *addrlen,
                       char **errstr);
    int (*socket)(int family, int type, int protocol, char **errstr);
    int (*getpeeraddr)(int fd, char **address, char **errstr);  /* optional */
    int (*connect)(char *address, char **errstr);
    int (*bind)(char *address, char **errstr);
    void (*unbind)(int fd);                                     /* optional */
};

The structure must provide (i.e. have non-NULL function pointers for) either both of the getaddrinfo() and socket() callbacks, or both of the connect() and bind() callbacks - it may of course provide all of them. The getpeeraddr() and unbind() callbacks are optional. If both getaddrinfo() and socket() are provided, the shared object can also be used by applications using the C APIs to connect to ConfD (see e.g. the confd_cmd.c source code in the $CONFD_DIR/src/confd/tools directory).

All the callbacks except unbind() can report an error by returning -1, and in this case optionally provide an error message via the errstr parameter. If an error message is provided, errstr must point to dynamically allocated memory - ConfD will free it through a call to free(3) after reporting the error.

Two examples using this interface are provided in the $CONFD_DIR/src/confd/ipc_drv directory. One of them (ipc_drv_unix.c) uses AF_UNIX sockets, and implements only the connect(), bind(), and unbind() callbacks. The other (ipc_drv_etcp.c) actually uses standard AF_INET/AF_INET6 TCP sockets just like the "normal" ConfD IPC - this can be meaningful if we need to set some non-standard socket options such as Linux SO_VRF for all IPC sockets. This example implements the getaddrinfo(), socket(), and getpeeraddr() callbacks.

An older version of this interface (also defined in ipc_drv.h) used a confd_ipc_init() function and a struct confd_ipc_cbs callback structure. This interface is deprecated, but will continue to be supported. The main differences are that the old interface lacks the getaddrinfo(), socket(), and getpeeraddr() callbacks, and that any error message would be provided via a static errstr structure element.

To enable the use of this alternate IPC mechanism for the client library connections, we need to set /confdConfig/confdExternalIpc/enabled to "true" in confd.conf. This causes any settings for /confdConfig/confdIpcAddress/ip and /confdConfig/confdIpcAddress/port to be ignored, and we can instead specify the address to use in /confdConfig/confdExternalIpc/address. The address is given in text form, and ConfD passes it to the getaddrinfo(), bind(), and/or connect() callbacks without any interpretation.

If we want to use the alternate IPC for the inter-node HA communication, we can in the same way set /confdConfig/ha/externalIpc/enabled and /confdConfig/ha/externalIpc/address in confd.conf. Additionally the HA API uses a struct that holds a node address:

struct confd_ha_node {
    confd_value_t nodeid;
    int af;               /* AF_INET | AF_INET6 | AF_UNSPEC */
    union {               /* address of remote note */
        struct in_addr ip4;
        struct in6_addr ip6;
        char *str;
    } addr;
    char buf[128];        /* when confd_read_notification() and        */
                          /* confd_ha_status() populate these structs, */
                          /* if type of nodeid is C_BUF, the pointer   */
                          /* will be set to point into this buffer     */
    char addr_buf[128];   /* similar to the above, but for the address */
                          /* of remote node when using external IPC    */
                          /* (from getpeeraddr() callback for slaves)  */
};

When this struct is used to specify the address of the master in the confd_ha_beslave() call, the af element should be set to AF_UNSPEC, and the str element of the addr union should point to the text form of the master node's address. When the struct is used to deliver information from ConfD, in the HA event notifications and the result of a confd_ha_status() call, af will also be set to AF_UNSPEC, but str will be NULL for slave nodes unless a peer address has been provided via the getpeeraddr() callback.

The client changes we need to do are analogous to those listed in the table above for the case of using a different IP address and/or port for TCP - the differences are:

As noted above, confd_cli is provided as source, so we can alternatively modify it to support the alternate IPC mechanism "natively". This is also the case for confd_cmd, confd_load, and maapi.

By default, the clients connecting to the ConfD IPC port are considered trusted, i.e. there is no authentication required, and we rely on the use of 127.0.0.1 for /confdConfig/confdIpcAddress/ip to prevent remote access. In case this is not sufficient, it is possible to restrict the access to the IPC port by configuring an access check.

The access check is enabled by setting the confd.conf element /confdConfig/confdIpcAccessCheck/enabled to "true", and specifying a filename for /confdConfig/confdIpcAccessCheck/filename. The file should contain a shared secret, i.e. a random character string. Clients connecting to the IPC port will then be required to prove that they have knowledge of the secret through a challenge handshake, before they are allowed access to the ConfD functions provided via the IPC port.

To provide the secret to the client libraries, and inform them that they need to use the access check handshake, we have to set the environment variable CONFD_IPC_ACCESS_FILE to the full pathname of the file containing the secret. This is sufficient for all the clients mentioned above, i.e. there is no need to change application code to support or enable this check.

CDB keeps its data in two files A.cdb and C.cdb. If ConfD is stopped, these two files can simply be copied, and the copy is then a full backup of CDB. If ConfD is running, we cannot copy the files, but need to use confd --cdb-backup file to copy the two CDB files into a backup file (in gzipped tar format).

Furthermore, if neither A.cdb nor C.cdb exists in the configured CDB directory, CDB will attempt to initialize from all files in the CDB directory with the suffix ".xml".

Thus, there exists two different ways to reinitiate CDB from a previous known good state, either from .xml files of from a CDB backup. The .xml files would typically be used to reinstall "factory defaults" whereas a CDB backup could be used in more complex scenarios.

When ConfD starts and fails to initialize, the following exit codes can occur:

If the ConfD daemon starts normally, the exit code is 0.

If CDB is reinitialized to factory defaults, it may not be possible to reach the machine over the network. The only way to reconfigure the machine is through a CLI login over the serial console.

If the AAA database is broken, ConfD will start but with no authorization rules loaded. This means that all write access to the configuration is denied. The ConfD CLI can be started with a flag confd_cli --noaaa which will allow full unauthorized access to the configuration. Usage of the ConfD cli with this flag can possibly be enabled for some special UNIX user which can only login over the serial port. Thus --noaaa provides a way to reconfigure the box although the AAA database is broken.

ConfD attempts to handle all runtime problems without terminating, e.g. by restarting specific components. However there are some cases where this is not possible, described below. When ConfD is started the default way, i.e. as a daemon, the exit codes will of course not be available, but see the --foreground option in the confd(1) manual page.

When the system is updated, ConfD executes a two phase commit protocol towards the different participating databases including CDB. If a participant fails in the commit() phase although the participant succeeded in the prepare phase, the configuration is possibly in an inconsistent state.

When ConfD considers the configuration to be in a inconsistent state, operations will continue. It is still possible to use NETCONF, the CLI and all other northbound management agents. The CLI has a different prompt which reflects that the system is considered to be in an inconsistent state and also the Web UI shows this:

  -- WARNING ------------------------------------------------------
  Running db may be inconsistent. Enter private configuration mode and
  install a rollback configuration or load a saved configuration.
  ------------------------------------------------------------------
        

It is slightly more involved using the NETCONF agent. The NETCONF transaction which resulted in a failed commit will fail, but following that the only way to see that the system is considered to be in an inconsistent state is by reading the data defined by tailf-netconf-monitoring.

The MAAPI API has two interface functions which can be used to set and retrieve the consistency status. This API can thus be used to manually reset the consistency state. Apart from this, the only way to reset the state to a consistent state is by reloading the entire configuration.

Some examples are dependent on features that might only be available on Linux. Before such examples can run, they would have to be ported.

$ python
...
>>> import paramiko
>>>

If you have trouble starting or running ConfD, the examples or the clients you write, here are some troubleshooting tips.

The Erlang API to ConfD is implemented as an Erlang/OTP application called econfd. This application comes i two flavours. One is builtin in ConfD in order to support applications running in the same Erlang VM as ConfD. The other is a separate library which is included in source form in the ConfD release, in the $CONFD_DIR/erlang directory. Building econfd as described in the $CONFD_DIR/erlang/econfd/README file will compile the Erlang code and generate the documentation.

This API can be used by applications written in Erlang in much the same way as the C and Java APIs are used, i.e. code running in an Erlang VM can use the econfd API functions to make socket connections to ConfD for data provider, MAAPI, CDB, etc access. However the API is also available internally in ConfD, which makes it possible to run Erlang application code inside the ConfD daemon, without the overhead imposed by the socket communication.

There is little or no support for testing and debugging Erlang code executing internally in ConfD, since ConfD provides a very limited runtime environment for Erlang in order to minimize disk and memory footprints. Thus the recommended method is to develop Erlang code targeted for this by using econfd in a separate Erlang VM, where an interactive Erlang shell and all the other development support included in the standard Erlang/OTP releases are available. When development and testing is completed, the code can be deployed to run internally in ConfD without changes.

For information about the Erlang programming language and development tools, please refer to www.erlang.org and the available books about Erlang (some are referenced on the web site).

All application code SHOULD use the prefix "ec_" for module names, application names, registered processes (if any), and named ets tables (if any), to avoid conflict with existing or future names used by ConfD itself.

The Erlang code is packaged into applications which are automatically started and stopped by ConfD if they are located at the proper place. ConfD will search the load path as defined by /confdConfig/loadPath for directories called erlang-lib. The structure of such a directory is the same as a standard lib directory in Erlang. The directory may contain multiple Erlang applications. Each one must have a valid .app file. See the Erlang documentation of application and app for more info.

The following config settings in the .app file are explicitly treated by ConfD:

When the application is started, one of its processes should make initial connections to the ConfD subsystems, register callbacks etc. This is typically done in the init/1 function of a gen_server or similar. While the internal connections are made using the exact same API functions (e.g. econfd_maapi:connect/2) as for an application running in an external Erlang VM, any Address and Port arguments are ignored, and instead standard Erlang inter-process communication is used. The internal_econfd/embedded_applications/transform example in the bundled collection shows a transform written in Erlang and executing internally in ConfD.

An alternate way (the old way) of running custom code in the Erlang VM of ConfD is to load single Erlang modules (as opposed to use proper applications). When ConfD starts, specifically when phase0 is reached, ConfD will search the load path as defined by /confdConfig/loadPath for compiled Erlang modules, i.e. *.beam files. The modules that are found will be loaded, unless the module name conflicts with an existing ConfD module. If there is a module name conflict, ConfD will terminate with an error message and exit code 21. The -on_load() directive can be used to spawn a process that makes initial connections to the ConfD subsystems, registers callbacks, sets up supervision if desired, etc. The internal_econfd/single_modules/transform example in the bundled collection shows a transform written in Erlang and executing internally in ConfD.

The --printlog option to confd, which prints the contents of the ConfD errorLog, is normally only useful for Tail-f support and developers, but it may also be relevant for debugging problems with application code running inside ConfD. The errorLog collects the events sent to the OTP error_logger, e.g. crash reports as well as info generated by calls to functions in the error_logger(3) module. Another possibility for primitive debugging is to run confd with the --foreground option, where calls to io:format/2 etc will print to standard output. Printouts may also be directed to the developer log by using econfd:log/3.

While Erlang application code running in an external Erlang VM can use basically any version of Erlang/OTP, this is not the case for code running inside ConfD, since the Erlang VM is evolving and provides limited backward/forward compatibility. To avoid incompatibility issues when loading the beam files, the Erlang compiler erlc in same version the ConfD distribution should be used.

ConfD provides the VM, erlc and the kernel, stdlib, and crypto OTP applications.

We can implement user-defined types with Erlang code in a manner similar to what is described for C in the section called “USER-DEFINED TYPES” in confd_types(3). In the econfd API, we populate a #confd_type_cbs{} record and register it using econfd_schema:register_type_cbs/1. For an application running inside ConfD, this registration will have the same effect as using a shared object in the C API, i.e. the callback functions will be used internally by ConfD for doing string <-> value translation and syntax validation.

Callbacks for user-defined types may in general be required to be registered very early in the ConfD startup, in particular default values specified in the YANG data model will be translated from string form to internal representation when the corresponding .fxs file is loaded. A really early start of the application is achieved by using the early_phase0 as confd_start_phase in the application .app file. An application started in this early phase should not have e.g. registration of normal data provider callbacks, since ConfD is not prepared to handle such registrations at this early point in the startup. The internal_econfd/embedded_applications/user_type example shows how the callbacks can be implemented in Erlang.

An alternate way (the old way) of defining ConfD user-defined-types in Erlang is to load a single module (as opposed to use a proper application). By giving a module implementing such callbacks a name starting with "ec_user_type" (i.e. file name ec_user_type*.beam), we can tell ConfD that it should be loaded early enough for default value translation. The internal_econfd/single_modules/user_type example shows how the callbacks can be implemented in Erlang. It uses this naming convention to be able to handle the translation of a default value specified in the data model.