Chapter 9. Semantic validation

ConfD stores device configuration data. Some device configuration data is truly critical for the correct operations of the device. Misconfiguring a network device may lead to a situation where the device is no longer connected to the network. Before committing configuration data it is crucial to ensure that the new configuration is correct.

Another benefit with a guaranteed correct configuration, is that application software which reads the configuration data need not check the validity of the configuration.

ConfD has support for several different levels of validation. We have:

A YANG model is a schema. It has a number of constructs the define the structure of the model as a whole as well as type constrains on individual leaves.

Structure enforcing statements include constructs like container presence statements, leaf mandatory statements, leaf-list min and max elements statements and list min and max elements statements.

Each leaf has a either a built-in primitive type, e.g. integer, string, boolean etc, or a derived type e.g. a union, enumeration, boundary restriction, or regular expression pattern

The ConfD CLI and Web UI use this information to guide the operator what is possible to configure.

Before going for semantic validation we should also make sure that our need for validation can not be satisfied by any of the integrity constraint constructs available in the YANG model modeling language:

Read more about integrity constraints in the Chapter 3, The YANG Data Modeling Language chapter as well in http://www.ietf.org/rfc/rfc6020.txt.

Using XPath expressions it is possible to express constraints on values in virtually any way. XPath is complicated and requires a bit of work to learn. It is well worth the effort though, since writing declarative constraints on the data model - in the data model - is better than writing C code that executes outside of ConfD.

Semantic validation in ConfD extends the validation functionality to allow for verification of constraints that can not be expressed by the above constructs. It is important to realize that the basic concept is the same, though. The task of semantic validation is to make sure that the new configuration satisfies some set of logical constraints before it is allowed to be committed.

With a command centric view of configuration, validation may be thought of as checking the validity of operator actions vis-a-vis existing configuration. This tends to lead to complex and error prone code, since there will often be a large number of combinations of actions and configuration values that need to be checked, and some "corner cases" can easily be overlooked. Furthermore covering multiple user interfaces, such as NETCONF and Web UI in addition to CLI, with the same validation code will be an almost impossible task.

In contrast, ConfD's data model centric validation concept, i.e. checking the validity of the configuration that will be the result of those actions, allows for clear and concise validation code that rejects an invalid configuration regardless of which commands or other operations that were used to (attempt to) create it.

Attempting to validate the operations instead of the resulting configuration can also lead to problems with loading config backups or doing rollbacks. The old configuration that should be applied as a result of such actions is obviously valid (as long as the logical constraints have not changed), but validation logic that rejects specific changes to the configuration may still result in that configuration being rejected.

An additional benefit of the "logical constraints" approach to validation is the possibility of "off-line" validation - i.e. a complete configuration can be loaded onto a device that is not in production use, and the validation code can give its verdict about its validity, even though the sequence of operations that would lead to this configuration may not even be known.

Since ConfD provides access to the complete new configuration inside a transaction for the purpose of semantic validation, it is generally straightforward to implement constraints that are expressed in terms of relations between configuration nodes that must hold true for any valid configuration. Identifying and formulating those constraints is thus the first thing we must do when implementing semantic validation.

There is however one case where using the validation functionality to check operator actions can be useful, namely when we want to warn the operator of undesirable consequences - e.g. "If you change this value, the system will reboot". This is not validation in the sense of verifying correctness, since the new configuration will be valid too and should not be rejected - but the validation code is still the appropriate place to implement such functionality. To specifically inspect changes from the current configuration to the new configuration, the MAAPI API provides functions to iterate over all or a subset of the changes, or if CDB is used, current configuration values can be read via the CDB API.

In a manner similar to how we use callpoints to register callback functions which read and write data in external databases, we use named validation points to define which code is responsible for the validation of different parts of the configuration. However unlike callpoints, validation points do not form a hierarchy where they "take over" responsibility from validation points higher up in the XML tree.

The validation code can reject the data, accept it, or accept it with a warning. If a warning is produced, it will be displayed for interactive users (e.g. through the CLI or Web UI). The user may choose to abort or continue to commit the transaction.

Validation callbacks are typically assigned to individual leafs or containers in the YANG model, but this is mostly a matter of organization and modularization of the validation code. In some cases it may even be feasible to use a single validation callback, e.g. on the top level node of the configuration. In such a case, this callback is responsible for the validation of all values and their relationships throughout the configuration.

A validation callback is only invoked if its validation point is for an element that exists in the new configuration. This may be surprising, but it is a logical consequence of the "validate the configuration, not the operations" concept of ConfD validation - we can not be asked to validate something that does not exist. Since it sometimes leads to the question "How can I prevent deletion of an element?", an example may be useful:

container notification {
  leaf protocol {
    type enumeration {
        enum SNMP;
        enum SMTP;
        enum NETCONF;
    }
    mandatory true;
  }
  leaf smtp-server {
    type inet:host;
  }
}
        

Here we can configure a notification protocol, and optionally the address of an SMTP server - we want it to be optional, since it is not needed unless the notification protocol is SMTP. However if protocol SMTP and a server has been configured, the SMTP server element must not be deleted. But if we assign a validation point to that element, the callback will not get invoked on deletion, since the element does not exist in the new configuration!

The solution is in the previous section - we must identify and formulate the logical constraint. In this case it is "If notification protocol SMTP is configured, an SMTP server must also be configured".

The easiest way to enforce this is through an XPath expression as in:

container notification {
   leaf protocol {
     must ". != 'SMTP' or ../smtp-server" {
       error-message "Must specify smtp-server";
     }
     type enumeration {
     ......
        

Alternatively, if we us C code logic to achieve the same thing to make sure that our callback is invoked, we assign the validation point to the "protocol" element. The validation callback will then get the value of this element as a parameter, and the implementation of the constraint check will just be:

if (CONFD_GET_ENUM_VALUE(newval) == nsprefix_SMTP &&
    maapi_exists(maapi_socket, tctx->thandle, "/notification/smtp-server") != 1) {
    confd_trans_seterr(tctx, "SMTP server must be configured");
    return CONFD_ERR;
}
        

See the examples below for the API details. Note well that since it is based on the logical constraint, this single expression also covers the other required case of "operational validation", i.e. setting of the notification protocol to SMTP - it will be rejected unless an SMTP server has been configured.

Next we describe how to connect user defined C code to the validation process. We start off with a really simple YANG model.

module mtest {
  namespace "http://tail-f.com/ns/example/mtest";
  prefix mtest;
  container mtest {
    leaf a_number {
      type int64;
      default 42;
    }
    leaf b_number {
      type int64;
      default 7;
    }
  }
}

We wish to ensure that the integer value /mtest/a_number is bigger than /mtest/b_number. We use a YANG model annotation file to specify the validation point. This is a good technique to use if wish to keep out data models clean of any Tail-f extensions.

module mtest.annot {
  namespace "http://tail-f.com/ns/example/mtest.annot";
  prefix mtesta;

  import mtest {
    prefix m;
  }

  import tailf-common {
    prefix tailf;
  }

  tailf:annotate "/m:mtest/m:a_number" {
    tailf:validate vp1;
  }
}

We define a validation point on /mtest/a_number called vp1. This instructs ConfD that whenever ConfD needs to validate the XML data element associated with /mtest/a_number, ConfD should call an external process which has registered itself under the validation point vp1 using the libconfd interface. If no process is registered, the validation will fail and it will not be possible to commit any changes.

We continue with the necessary C code to implement the relevant parts of the external process. The complete code for this can be found in examples.confd/validate/c in the ConfD examples collection. The validation code will be called during the validation phase of a ConfD transaction, i.e. before any write operations have been performed. The C code must install three different callback functions.

The init() and stop() callbacks are installed through the API call confd_register_trans_validate_cb() and the individual validation point callbacks are installed through consecutive calls to confd_register_valpoint_cb() for each defined validation point.

We start by creating three sockets to ConfD. We need two sockets for the callback machinery and one MAAPI socket.

            #include "confd_lib.h"
#include "confd_dp.h"
#include "confd_maapi.h"

/* include generated ns file */
#include "mtest.h"

int debuglevel = CONFD_DEBUG;

static int ctlsock;
static int workersock;
static int maapi_socket;
static struct confd_daemon_ctx *dctx;

            
    confd_init("more_a_than_b", stderr, debuglevel);

    if ((dctx = confd_init_daemon("mydaemon")) == NULL)
        confd_fatal("Failed to initialize confd\n");

    if ((ctlsock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open ctlsocket\n");

    addr.sin_addr.s_addr = inet_addr("127.0.0.1");
    addr.sin_family = AF_INET;
    addr.sin_port = htons(CONFD_PORT);

    OK(confd_load_schemas((struct sockaddr*)&addr, sizeof(struct sockaddr_in)));

    /* Create the first control socket, all requests to */
    /* create new transactions arrive here */
    if (confd_connect(dctx, ctlsock, CONTROL_SOCKET,
                      (struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0) {
        confd_fatal("Failed to confd_connect() to confd \n");
    }

    /* Also establish a workersocket, this is the most simple */
    /* case where we have just one ctlsock and one workersock */
    if ((workersock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open workersocket\n");
    if (confd_connect(dctx, workersock, WORKER_SOCKET,(struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0)
        confd_fatal("Failed to confd_connect() to confd \n");

                if ((*maapi_sock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open socket\n");

    if (maapi_connect(*maapi_sock, (struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0)
        confd_fatal("Failed to confd_connect() to confd \n");

          

The above code connects three times. We need the control socket and the worker socket for the C callbacks. This works precisely the same way as when C callbacks are installed for the external data provider API. Thus the request from ConfD to invoke the init() callback will arrive on the control socket whereas the subsequent requests to invoke the individual validate() callbacks as well as the finishing request to invoke stop() will arrive on the designated worker socket.

The MAAPI socket will be used to attach the running transaction to a MAAPI socket.

All three sockets are connected to the same port number.

Next step is to continue with the installation of the callbacks:

            struct confd_trans_validate_cbs vcb;
struct confd_valpoint_cb valp1;

static void OK(int rval)
{
    if (rval != CONFD_OK) {
        fprintf(stderr, "more_a_than_b.c: error not CONFD_OK: %d : %s \n",
                confd_errno, confd_lasterr());
        abort();
    }
}

                vcb.init = init_validation;
    vcb.stop = stop_validation;
    confd_register_trans_validate_cb(dctx, &vcb);

    valp1.validate = validate;
    strcpy(valp1.valpoint, "vp1");
    OK(confd_register_valpoint_cb(dctx, &valp1));

    OK(confd_register_done(dctx));

          

Note the call to confd_register_done() after the callback registrations - this is required, to tell ConfD that we have completed our registrations. The actual callbacks look like:

            static int init_validation(struct confd_trans_ctx *tctx)
{
    OK(maapi_attach(maapi_socket, mtest__ns, tctx));
    confd_trans_set_fd(tctx, workersock);
    return CONFD_OK;
}

static int stop_validation(struct confd_trans_ctx *tctx)
{
    OK(maapi_detach(maapi_socket, tctx));
    return CONFD_OK;
}

static int validate(struct confd_trans_ctx *tctx,
                    confd_hkeypath_t *keypath,
                    confd_value_t *newval)
{
    int64_t b_val;
    int64_t a_val;
    int64_t newval_a;

    /* we validate that a_number > b_number  */

    newval_a = CONFD_GET_INT64(newval);

    /* this switch is not necessary in this case; we know that we're
       called for a_number only.  the switch is useful when the same
       code is used to validate multiple objects. */
    switch (CONFD_GET_XMLTAG(&(keypath->v[0][0]))) {
    case mtest_a_number:
        OK(maapi_get_int64_elem(maapi_socket, tctx->thandle, &b_val,
                                "/mtest/b_number"));
        OK(maapi_get_int64_elem(maapi_socket, tctx->thandle, &a_val,
                                "/mtest/a_number"));
        /* just an assertion to show that newval == /mtest/a_number */
        /* in this transaction */

        assert(CONFD_GET_INT64(newval) == a_val);
        if (newval_a == 88) {
            /* This is how we get to interact with the CLI/webui */
            confd_trans_seterr(tctx, "Dangerous value: 88");
            return CONFD_VALIDATION_WARN;
        }
        else if (newval_a >  b_val) {
            return CONFD_OK;
        }
        else {
            confd_trans_seterr(tctx, "a_number is <= b_number ");
            return CONFD_ERR;
        }
        break;

    default: {
        char ebuf[BUFSIZ];
        sprintf(ebuf, "Unknown tag %d",
                CONFD_GET_XMLTAG(&(keypath->v[0][0])));
        confd_trans_seterr(tctx, ebuf);
        return CONFD_ERR;
    } /* default case */
    } /* switch */
}

          

The switch on the keypath exemplifies that we really get the keypath populated with the path leading to the textual element being validated. We can thus have the same validation point validate different XML data elements.

The init callback attaches to MAAPI and the global variable maapi_socket is used to read data from the transaction. All MAAPI functions use a "transaction handle"; this handle is available inside the instantiated struct confd_trans_ctx *tctx structure.

Finally we have a poll loop where we dispatch requests to invoke C callbacks on the control socket and the worker socket.

                while (1) {
        struct pollfd set[2];
        int ret;

        set[0].fd = ctlsock;
        set[0].events = POLLIN;
        set[0].revents = 0;

        set[1].fd = workersock;
        set[1].events = POLLIN;
        set[1].revents = 0;

        if (poll(&set[0], 2, -1) < 0) {
            perror("Poll failed:");
            continue;
        }

        if (set[0].revents & POLLIN) {
            if ((ret = confd_fd_ready(dctx, ctlsock)) == CONFD_EOF) {
                confd_fatal("Control socket closed\n");
            } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
                confd_fatal("Error on control socket request\n");
            }
        }
        if (set[1].revents & POLLIN) {
            if ((ret = confd_fd_ready(dctx, workersock)) == CONFD_EOF) {
                confd_fatal("Worker socket closed\n");
            } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
                confd_fatal("Error on worker socket request\n");
            }
        }
    }

         

In the above example we also showed how to issue a warning as opposed to a validation failure. Hadn't it been for that, it would have been considerably easier to express the same validation as an XPath expression. Thus we attach a must statement to the mtest container as:

container mtest {
  must "a_number > b_number" {
    error-message "a_number is <= b_number";
  }

         

When CDB first starts or upgrades the database it creates a special transaction which, when committed, will invoke validation. An external validation point (written e.g. in C) has to be registered before these transactions are committed, otherwise starting ConfD will fail. Starting ConfD and external applications in a synchronized way is accomplished using ConfD start phases (see the Advanced Topics chapter). To avoid this extra complexity use the --ignore-initial-validation option when starting ConfD (useful during development).

In a validation point it might be desirable to access CDB to validate a value against the old value of the parameter (or some other parameter) in the configuration. Using the normal cdb calls this works fine in the normal case, but when CDB is initializing there are no old values. The cdb_get_phase() call can be used to check for this case, (see the confd_lib_cdb(3) manual page for details).

In general, validation code for a particular element in the configuration may read any other part of the configuration, and accept or reject the configuration based on that. I.e. the outcome of the validation may actually depend on other configuration elements than the one the validation point is assigned to - and for correct operation, the validation code must be executed when any element it depends on has been modified. As ConfD can not make any assumptions about these dependencies, it takes the safe default of always invoking all callbacks (for existing elements) on every configuration change.

It is possible to declare these dependencies explicitly in the YANG model. This can be a significant optimization, but it is strictly an optimization, i.e. a validation callback implementing a logical constraint verification will always return the same result for a given configuration, it doesn't matter if it is invoked unnecessarily due to lack of a dependency declaration in the YANG model. On the other hand an incorrect dependency declaration, that omits some dependency, can allow changes that lead to an invalid configuration. Thus if dependency declarations are used, it is critical that they are correct, and in particular that they are updated as needed if the validation logic of the callback is changed.

There can be multiple dependency declarations for a validation point. Each declaration consists of a dependency element specifying a configuration subtree that the validation code is dependent upon. If any element in any of the subtrees is modified, the validation callback is invoked. A subtree can be specified as an absolute path or as a relative path.

The relative path '.' is often used to declare that the validation code needs to be run whenever the current element or an element below it is modified. However note that per above, routinely specifying '.' as the only dependency for all validation points is a dangerous practice - if the validation logic actually depends on elements outside the subtree of the validation point, an invalid configuration may go undetected. Also, for a leaf element, having '.' as the only dependency is almost always wrong - if the validation really depends only on the leaf itself, it is likely that it could be expressed as a constraint in the YANG model instead of via a validation callback.

As described above, if a dependency is not declared, it defaults to a single dependency on the root of the configuration tree (/), which means that the validation code is executed when any configuration element is modified.

If dependencies are declared on a leaf element, an implicit dependency on the leaf itself is added.

As an example, consider the /mtest/a_number validation above. The element a_number has validation code attached to it, and this code depends on element b_number. Thus, this code has to be executed whenever a_number or b_number is modified. To specify this, we can do:

leaf a_number {
  type int64;
  default 42;
  tailf:validate vp1 {
    tailf:dependency '../b_number';
  }
}
        

Here we specified the validation point with its dependency sub-element directly in the YANG model - it is of course possible to use an annotation file in this case too.

It is also possible, and recommended for performance reasons, to specify dependencies in must statements:

leaf a_number {
  type int64;
  default 42;
  must ". > ../b_number" {
    tailf:dependency '../b_number';
  }
}
        

The compiler gives a warning if a must statement lacks a tailf:dependency statement, and it cannot derive the dependency from the expression. The options --fail-on-warnings or -E TAILF_MUST_NEED_DEPENDENCY can be given to force this warning to be treated as an error.

Configuration policies is an optional mechanism by which the operator of a ConfD-based system can define its own custom validation rules. A configuration policy enforces custom validation rules on the configuration data. These rules assert that the user-defined conditions are always true in committed data. If a configuration change is done such that a policy rule would evaluate to false, the configuration change is rejected by the system.

As an example, an operator might define a configuration policy that bgp must never be disabled on a device, or define a policy that the MTU on SONET interfaces must be greater than 2048.

The data model for configuration policies is defined in tailf-configuration-policy.yang, in the directory $CONFD_DIR/src/confd/configuration_policy/. The YANG model contains a Also included in this directory is a pre-compiled .fxs file, and a Makefile that can be modified as necessary, for example to compile the fxs file with a --export parameter to confdc.

To enable this optional feature, put the tailf-configuration-policy.fxs in the load path to ConfD.

container protocols {
  container bgp {
    presence "enables bgp";
    // BGP config goes here...
  }
}
        

admin@host% configure

admin@host% set policy rule chk-bgp expr "/protocols/bgp"

admin@host% set policy rule chk-bgp error-message "bgp must be enabled"

admin@host% commit
Commit complete.
        

admin@host% delete protocols bgp

admin@host% commit
Aborted: bgp must be enabled
        

admin@host> set autowizard false

admin@host> configure

admin@host% set policy rule chk-sonet-mtu

admin@host% edit policy rule chk-sonet-mtu

admin@host% set foreach "/interface[type='sonet']"

admin@host% set expr "mtu >= 2048"

admin@host% set error-message "Sonet interface {name} has MTU {mtu}, must be at least 2048"

admin@host% top

admin@host% commit
Commit complete.
[ok][2010-11-02 09:39:00]
        

mbj@x15% set interface so-1/0 type sonet mtu 4096

mbj@x15% set interface so-1/1 type sonet mtu 4096

mbj@x15% set interface so-1/2 type sonet mtu 1024

mbj@x15% validate
Failed: Sonet interface so-1/2 has MTU 1024, must be at least 2048