Blackboard architecture¶

The Blackboard system is targeted as the front-end for a great variety of applications, providing an architecture that integrates experience formation and active behaviour from a set of different functional modules. These modules can work on different levels of abstraction, independently from each other or in collaboration, in a bottom-up or top-down manner. A key feature of this system should be its ability to evolve, so that easy modification, exchange and/or extension of modules can be achieved within a scalable architecture.

This document will provide you an overview about the blackboard architecture that is used together with the knowledge sources in order to provide such a system.

Blackboard system architectural considerations¶

In order to implement this integrative and system-wide view, some core attributes of the system have been established as follows.

Building a flexible system¶

The system we develop is a platform, i.e. it provides functionality to execute other functionality. While the target functionality is clear – auditory and multi-modal experience formation, scene understanding and exploration – it involves many different problems, each with many possible solutions. We therefore design the system with extension in mind, trying not to constrain possible functionality of modules.

In particular, the blackboard system allows the plugging-in of different knowledge sources. Knowledge sources are modules that define their own functionality, to be executed in the organised frame of our system. They define by themselves which data they need for execution and which data they produce – the blackboard system provides the tools for requesting and storing this data, but does not care about the actual contents (while the knowledge sources do not need to care about where and how data is stored). It is also important that the blackboard system has no static knowledge of what types of knowledge sources are available. So long as knowledge sources follow a certain implementation scheme, independent of their actual functionality they can register dynamically (i.e. at runtime) as a module in the blackboard system. Thus, a library of knowledge sources can be built during this project that can be extended arbitrarily, without need to modify the blackboard system. Implementors of new modules need only be concerned with implementing their functionality.

The Two!Ears architecture has been designed and implemented using an object-oriented approach. Accordingly, the implementation scheme knowledge sources must adhere to is provided in the form of an abstract class). Additionally, to enable creation of new knowledge sources that depend on auditory signals without needing to hard-code a signal request into the system, an auditory front-end dependent knowledge source superclass has been developed.

Building a dynamic system¶

Key to providing the described flexibility is to neither hard-code lists of usable knowledge sources nor the interactions between them. Hard-coded (or static) lists and dependencies would be overly restrictive – the system must be open to dynamic change.

At the same time, flexibility for extension is not the only cause for needing a dynamic system. The system is intended to be an active system that does not only work in a signal processing bottom-up manner, but also in a cognitive top-down manner. Modules must therefore be allowed to change the system setup at runtime. This means that it is essential for our system to be equipped with functionality for dynamic module instantiation, registration and removal. This also implies the need for on-the-fly rewiring of the communication links between modules.

Dynamic system construction¶

To ensure an easy-to-use system, we implemented as the main class a wrapper that integrates the different main components, and hides their connections where possible. This main class is called BlackboardSystem, since the blackboard is the central component of our platform. This is an excerpt of its definition:

class BlackboardSystem
    properties
        blackboard;
        blackboardMonitor;
        scheduler;
        robotConnect;
        dataConnect;
    methods
        BlackboardSystem()
        setRobotConnect( robotConnect )
        setDataConnect( connectorClassName )
        buildFromXml( xmlName )
        addKS( ks )
        createKS( ksClassName, ksConstructArgs )
        numKSs()
        run()

The engine of our system is distributed across the BlackboardSystem, Blackboard, BlackboardMonitor and Scheduler classes, with the BlackboardSystem class holding instances of the latter three. These four classes each have genuine responsibilities: the BlackboardSystem integrates the framework parts, responsible for constructing and setting up the system. The blackboard is the central storage of functional data and knowledge sources. It holds a data map that saves arbitrary knowledge source data along time, together with methods to add and recall data from within knowledge source code. Additionally, the knowledge sources themselves are put into blackboard storage by the BlackboardSystem.

The BlackboardMonitor is responsible for creating bindings on demand between knowledge sources, by instantiating event listeners. It keeps track of these bindings and maintains the agenda of knowledge sources. The Scheduler is the executive component of the system. While the BlackboardMonitor keeps control of the knowledge sources in the agenda, the Scheduler decides about the order of those knowledge sources to be executed. It does that based on the attentional priorities of the knowledge sources. Fig. 31 shows the system class diagram.

../../_images/blackboard-system-class-diagram.png

Fig. 31 Class diagram of the whole blackboard system. The BlackboardSystem class is the integrative system component holding the other modules and giving access to system functionality.

An example of an XML-configured blackboard system is shown below. Two identity knowledge sources are connected to the Auditory front-end, and triggering an identity decision knowledge source.

<blackboardsystem>
    <dataConnection Type="AuditoryFrontEndKS"/>

    <KS Name="baby" Type="IdentityKS">
        <Param Type="char">baby</Param>
        <Param Type="char">6687829ce1a73694a1ce41c7c01dec1b</Param>
    </KS>
    <KS Name="femaleSpeech" Type="IdentityKS">
        <Param Type="char">femaleSpeech</Param>
        <Param Type="char">6687829ce1a73694a1ce41c7c01dec1b</Param>
    </KS>
    <KS Name="idDec" Type="IdDecisionKS">
        <Param Type="int">0</Param>
        <Param Type="int">1</Param>
    </KS>

    <Connection Mode="replaceOld" Event="AgendaEmpty">
        <source>scheduler</source>
        <sink>dataConnect</sink>
    </Connection>
    <Connection Mode="replaceOld">
        <source>dataConnect</source>
        <sink>baby</sink>
        <sink>femaleSpeech</sink>
    </Connection>
    <Connection Mode="replaceParallel">
        <source>baby</source>
        <source>femaleSpeech</source>
        <sink>idDec</sink>
    </Connection>
</blackboardsystem>

Several core functionalities are provided through the BlackboardSystem class:

Connecting a Robotic platform or the Binaural simulator to the Blackboard system. The connected robot or simulator must implement the robot interface for delivering audio ear signals and commanding movement and head rotation. The blackboard system and all its components including the knowledge sources get access to the audio stream and robot actions through this connection (setRobotConnect method).
Setting the type of the module that integrates with the Auditory front-end, instantiating it and connecting it to the Blackboard system. This module is a knowledge source itself, responsible for processing the ear signals into cues (such as interaural time differences) needed by other knowledge sources. The Auditory front-end itself is connected to the Robotic platform or the Binaural simulator in order to obtain the ear signals. (setDataConnect method).
Instantiating and adding knowledge sources. These knowledge sources must inherit from AbstractKS (see abstract knowledge source) or AuditoryFrontEndDepKS (see Section Auditory signal dependent knowledge source superclass: AuditoryFrontEndDepKS) to be able to be interfaced and run by the system. Knowledge sources that inherit from AuditoryFrontEndDepKS automatically get connected with the Auditory front-end by the system in order to place their signal/cue requests.

Adding and instantiating knowledge sources can take place both before or while running the system; it can be done from outside the system or from inside knowledge sources. This enables the development of top-down controlling knowledge sources from higher cognitive experts running in the system (addKS or createKS method).
The start-up configuration of the system can completely be defined by an XML file; the system is then constructed before running by loading this file. Of course this configuration can be changed dynamically while executing the system. The XML description needs at least a dataConnection node specifying the type of the Auditory front-end module; then, it can also contain KS nodes with parameters to construct knowledge sources, and Connection nodes that specify event bindings between knowledge sources. See the code listing from above for an example (buildFromXml method).
Starting the execution of the system. This triggers the blackboard system to request data from the robot/binaural simulator connection, and subsequent action by the Auditory front-end and all knowledge sources that are connected. The system will not stop execution before the Robotic platform or the Binaural simulator sends an ending signal (run method).

Dynamic blackboard memory¶

The Blackboard class holds the central data repository of the platform. It stores the knowledge sources and any shared data, in particular the output of the knowledge sources (e.g. estimates of the location of a sound source). It is accessible to all knowledge sources; and it not only stores current data, but keeps track of the history of this data in order to enable knowledge sources to work on time series data.

Importantly, the Blackboard is flexible about data categories, which do not have to be hard-coded into the system. Knowledge sources decide on their own and at runtime what to add and what to request. Thus, the system does not need to be changed in order to implement new knowledge sources that work with new data categories. Of course, knowledge sources can only read data categories that are actually stored in the blackboard by other knowledge sources (or themselves).

The following listing shows an excerpt of the Blackboard interface:

class Blackboard
        KSs;
        signals;
        currentSoundTime;
    methods
        Blackboard()
        addData( dataLabel, data, append, time )
        getData( dataLabel, reqSndTime )
        getLastData( dataLabel, time )
        getNextData( dataLabel, time )
        getDataBlock( dataLabel, blockSize_s )

Prominently featured are methods to add and access data:

addData: lets knowledge sources add data to the blackboard storage. The data category has to be named in dataLabel, data hands over the actual data to store. append is an optional flag indicating whether to overwrite or append data at the same time step (there might, for example, be several source identity hypotheses per time step, but only one source number hypothesis might be allowed). time specifies the time point under which this data shall be stored. It is optional and, if not set, defaults to the current time.
getData: lets knowledge sources read data from the blackboard storage. dataLabel indicates the data category requested, reqSndTime the time point of interest. getLastData, getNextData and getDataBlock are special cases of getData for retrieving the last data, the next data after a particular point in time, or a whole data block of length blockSize_s.

The following is an example from the implementation of the IdDecisionKS class:

idHyps = obj.blackboard.getData( ...
             'identityHypotheses', obj.trigger.tmIdx ).data;
%...
%find the most likely identity hypothesis -> maxProbHyp
%...
obj.blackboard.addData( ...
    'identityDecision', maxProbHyp, false, obj.trigger.tmIdx );

Let us assume that we have an instantiation called bbs of the Blackboard system. Now we would like to see the currently available data in its memory. For that you can run:

>> bbs.blackboard.getDataLabels()

or specify explicitly a time for which you would like to see the available data:

>> bbs.blackboard.getDataLabels(bbs.blackboard.currentSoundTimeIdx-1)

Additionally, the blackboard is used as a storage for pointers to signals from the Auditory front-end requested by knowledge sources inheriting from AuditoryFrontEndDepKS. The actual memory in which these signals are stored for recall is implemented in the Auditory front-end through circular buffers.

Dynamic blackboard interactions¶

Knowledge sources can communicate information through the flexible blackboard storage. However, adding data to the blackboard does not trigger other knowledge sources to be executed. Such interaction – triggering knowledge source execution – is done through an event system. Specifically, knowledge sources do not actually trigger execution of other knowledge sources (since they are decoupled and have no “knowledge” of each other), but knowledge sources make a request to be triggered upon the firing of particular events.

Each knowledge source has a standard event it can trigger, KsFiredEvent, inherited from AbstractKS. Beyond that, every knowledge source class is free to define as many additional events as reasonable for its task. Knowledge sources cause the events themselves through a call to notify as in the following example, in which the knowledge source induces an event and attaches a BlackboardEventData object holding the time that it was triggered:

notify( 'KsFiredEvent', BlackboardEventData(obj.trigger.tmIdx) );

The blackboard system a priori is totally ignorant of which events exist (clear responsibilities principle, open to extension). It also does not monitor any events by default, until knowledge sources request to be triggered by an event. This request is done through the method bind provided by the BlackboardMonitor class, whose interface is (partially) listed in the following excerpt:

class BlackboardMonitor
    properties
        pastAgenda;
        executing;
        agenda;
    methods
        BlackboardMonitor()
        bind( sources, sinks, addMode, eventName )

The bind method connects the sinks knowledge sources to event eventName (optional, defaults to KsFiredEvent) caused by the sources knowledge sources. addMode specifies how the BlackboardMonitor shall handle adding the triggered knowledge sources into the agenda. It understands the following modes, illustrated in Fig. 32:

add: Add the triggered knowledge source to the end of the agenda, regardless of whether or not there is already a (not yet executed) knowledge source instantiation of this sink in the agenda from a former triggering.
replaceOld: Replace old knowledge source instantiations of this sink in the agenda with the new one. Only instantiations of the sink triggered by the same source and same event are replaced. This is an important mode for knowledge sources where processing current data is more important than processing all data.
replaceParallel: Replace knowledge source instantiations of this sink from the same time point of parallel sources in the agenda with the new one. Only instantiations of the sink triggered at the same time and by the same event are replaced. This mode avoids sinks being unnecessarily executed several times with the same information.
replaceParallelOld: Replace old or current knowledge source instantiations of this sink triggered by parallel sources in the agenda with the new one. Only instantiations of the sink triggered by the same event are replaced. This mode is a combination of the replaceOld and replaceParallel modes.

It should be noted that the addMode only affects triggered knowledge source instantiations in the agenda, i.e. those that are not executed yet. As soon as a knowledge source is executed, it is removed from the agenda (first in executing, afterwards in pastAgenda).

../../_images/blackboard-event-system.png

Fig. 32 The different possibilities of event binding between knowledge sources with the blackboard system.

Dynamic blackboard scheduler¶

The scheduler is the component of the blackboard system that actually executes the knowledge sources – but first, it schedules them, that is, it decides the order in which knowledge sources waiting in the agenda get executed. This order is rescheduled after every execution of a knowledge source, since the conditions determining the order may have changed, or new knowledge sources may be present in the agenda that are more urgent.

The following factors influence the order of execution of knowledge sources:

Knowledge sources have a property called attentional priority. Knowledge sources with higher priority get executed before knowledge sources with lower priority. This priority can be set by the knowledge source itself, by other knowledge sources or from outside the system. The BlackboardMonitor provides a method for setting focus on a knowledge source (increasing its priority), along with the option to propagate this higher priority down along the dependency chain of this knowledge source. The dependency chain is determined by the event bindings.
Knowledge sources must implement a method canExecute, that returns whether or not the knowledge source can execute at this moment, and which is called by the scheduler if the knowledge source is first on the scheduling list. If it cannot execute, the knowledge source can decide whether to remain in the agenda or be removed from it.
Knowledge sources define a maximum invocation frequency, that cannot be exceeded. It is a maximum frequency, because knowledge sources get not necessarily executed periodically, since they are triggered by events, but not by timers. The scheduler checks whether the last execution time was long enough ago before considering the knowledge source for execution. Until then, it remains in the agenda.

This listing shows the relevant parts of the interface with respect to influencing the scheduling:

class BlackboardMonitor
    methods
        focusOn( ks, propagateDown )
        resetFocus()

class AbstractKS
    properties
        invocationMaxFrequency_Hz;
    methods (Abstract)
        canExecute()
        execute()
    methods
        focus()
        unfocus()