Blackboard architecture¶
The Blackboard system is targeted as the front-end for a great variety of applications, providing an architecture that integrates experience formation and active behaviour from a set of different functional modules. These modules can work on different levels of abstraction, independently from each other or in collaboration, in a bottom-up or top-down manner. A key feature of this system should be its ability to evolve, so that easy modification, exchange and/or extension of modules can be achieved within a scalable architecture.
This document will provide you an overview about the blackboard architecture that is used together with the knowledge sources in order to provide such a system.
Blackboard system architectural considerations¶
In order to implement this integrative and system-wide view, some core attributes of the system have been established as follows.
Building a flexible system¶
The system we develop is a platform, i.e. it provides functionality to execute other functionality. While the target functionality is clear – auditory and multi-modal experience formation, scene understanding and exploration – it involves many different problems, each with many possible solutions. We therefore design the system with extension in mind, trying not to constrain possible functionality of modules.
In particular, the blackboard system allows the plugging-in of different knowledge sources. Knowledge sources are modules that define their own functionality, to be executed in the organised frame of our system. They define by themselves which data they need for execution and which data they produce – the blackboard system provides the tools for requesting and storing this data, but does not care about the actual contents (while the knowledge sources do not need to care about where and how data is stored). It is also important that the blackboard system has no static knowledge of what types of knowledge sources are available. So long as knowledge sources follow a certain implementation scheme, independent of their actual functionality they can register dynamically (i.e. at runtime) as a module in the blackboard system. Thus, a library of knowledge sources can be built during this project that can be extended arbitrarily, without need to modify the blackboard system. Implementors of new modules need only be concerned with implementing their functionality.
The Two!Ears architecture has been designed and implemented using an object-oriented approach. Accordingly, the implementation scheme knowledge sources must adhere to is provided in the form of an abstract class). Additionally, to enable creation of new knowledge sources that depend on auditory signals without needing to hard-code a signal request into the system, an auditory front-end dependent knowledge source superclass has been developed.
Building a dynamic system¶
Key to providing the described flexibility is to neither hard-code lists of usable knowledge sources nor the interactions between them. Hard-coded (or static) lists and dependencies would be overly restrictive – the system must be open to dynamic change.
At the same time, flexibility for extension is not the only cause for needing a dynamic system. The system is intended to be an active system that does not only work in a signal processing bottom-up manner, but also in a cognitive top-down manner. Modules must therefore be allowed to change the system setup at runtime. This means that it is essential for our system to be equipped with functionality for dynamic module instantiation, registration and removal. This also implies the need for on-the-fly rewiring of the communication links between modules.
Dynamic system construction¶
To ensure an easy-to-use system, we implemented as the main class a wrapper that integrates the different main components, and hides their connections where possible. This main class is called BlackboardSystem, since the blackboard is the central component of our platform. This is an excerpt of its definition:
class BlackboardSystem
properties
blackboard;
blackboardMonitor;
scheduler;
robotConnect;
dataConnect;
methods
BlackboardSystem()
setRobotConnect( robotConnect )
setDataConnect( connectorClassName )
buildFromXml( xmlName )
addKS( ks )
createKS( ksClassName, ksConstructArgs )
numKSs()
run()
The engine of our system is distributed across the BlackboardSystem
,
Blackboard
, BlackboardMonitor
and Scheduler
classes, with the
BlackboardSystem
class holding instances of the latter three. These four
classes each have genuine responsibilities: the BlackboardSystem
integrates
the framework parts, responsible for constructing and setting up the system. The
blackboard is the central storage of functional data and knowledge sources. It
holds a data map that saves arbitrary knowledge source data along time, together
with methods to add and recall data from within knowledge source code.
Additionally, the knowledge sources themselves are put into blackboard storage
by the BlackboardSystem
.
The BlackboardMonitor
is responsible for creating bindings on demand between
knowledge sources, by instantiating event listeners. It keeps track of these
bindings and maintains the agenda of knowledge sources. The Scheduler
is
the executive component of the system. While the BlackboardMonitor
keeps
control of the knowledge sources in the agenda, the Scheduler
decides about
the order of those knowledge sources to be executed. It does that based on the
attentional priorities of the knowledge sources.
Fig. 31 shows the system class diagram.
An example of an XML-configured blackboard system is shown below. Two identity knowledge sources are connected to the Auditory front-end, and triggering an identity decision knowledge source.
<blackboardsystem>
<dataConnection Type="AuditoryFrontEndKS"/>
<KS Name="baby" Type="IdentityKS">
<Param Type="char">baby</Param>
<Param Type="char">6687829ce1a73694a1ce41c7c01dec1b</Param>
</KS>
<KS Name="femaleSpeech" Type="IdentityKS">
<Param Type="char">femaleSpeech</Param>
<Param Type="char">6687829ce1a73694a1ce41c7c01dec1b</Param>
</KS>
<KS Name="idDec" Type="IdDecisionKS">
<Param Type="int">0</Param>
<Param Type="int">1</Param>
</KS>
<Connection Mode="replaceOld" Event="AgendaEmpty">
<source>scheduler</source>
<sink>dataConnect</sink>
</Connection>
<Connection Mode="replaceOld">
<source>dataConnect</source>
<sink>baby</sink>
<sink>femaleSpeech</sink>
</Connection>
<Connection Mode="replaceParallel">
<source>baby</source>
<source>femaleSpeech</source>
<sink>idDec</sink>
</Connection>
</blackboardsystem>
Several core functionalities are provided through the BlackboardSystem
class:
Connecting a Robotic platform or the Binaural simulator to the Blackboard system. The connected robot or simulator must implement the robot interface for delivering audio ear signals and commanding movement and head rotation. The blackboard system and all its components including the knowledge sources get access to the audio stream and robot actions through this connection (
setRobotConnect
method).Setting the type of the module that integrates with the Auditory front-end, instantiating it and connecting it to the Blackboard system. This module is a knowledge source itself, responsible for processing the ear signals into cues (such as interaural time differences) needed by other knowledge sources. The Auditory front-end itself is connected to the Robotic platform or the Binaural simulator in order to obtain the ear signals. (
setDataConnect
method).Instantiating and adding knowledge sources. These knowledge sources must inherit from
AbstractKS
(see abstract knowledge source) orAuditoryFrontEndDepKS
(see Section Auditory signal dependent knowledge source superclass: AuditoryFrontEndDepKS) to be able to be interfaced and run by the system. Knowledge sources that inherit fromAuditoryFrontEndDepKS
automatically get connected with the Auditory front-end by the system in order to place their signal/cue requests.Adding and instantiating knowledge sources can take place both before or while running the system; it can be done from outside the system or from inside knowledge sources. This enables the development of top-down controlling knowledge sources from higher cognitive experts running in the system (
addKS
orcreateKS
method).The start-up configuration of the system can completely be defined by an XML file; the system is then constructed before running by loading this file. Of course this configuration can be changed dynamically while executing the system. The XML description needs at least a
dataConnection
node specifying the type of the Auditory front-end module; then, it can also containKS
nodes with parameters to construct knowledge sources, andConnection
nodes that specify event bindings between knowledge sources. See the code listing from above for an example (buildFromXml
method).Starting the execution of the system. This triggers the blackboard system to request data from the robot/binaural simulator connection, and subsequent action by the Auditory front-end and all knowledge sources that are connected. The system will not stop execution before the Robotic platform or the Binaural simulator sends an ending signal (
run
method).
Dynamic blackboard memory¶
The Blackboard
class holds the central data repository of the platform. It
stores the knowledge sources and any shared data, in particular the output of
the knowledge sources (e.g. estimates of the location of a sound source). It is
accessible to all knowledge sources; and it not only stores current data, but
keeps track of the history of this data in order to enable knowledge sources to
work on time series data.
Importantly, the Blackboard
is flexible about data categories, which do not
have to be hard-coded into the system. Knowledge sources decide on their own and at runtime what to add and what
to request. Thus, the system does not need to be changed in order to implement
new knowledge sources that work with new data categories. Of course, knowledge
sources can only read data categories that are actually stored in the blackboard
by other knowledge sources (or themselves).
The following listing shows an excerpt of the Blackboard
interface:
class Blackboard
KSs;
signals;
currentSoundTime;
methods
Blackboard()
addData( dataLabel, data, append, time )
getData( dataLabel, reqSndTime )
getLastData( dataLabel, time )
getNextData( dataLabel, time )
getDataBlock( dataLabel, blockSize_s )
Prominently featured are methods to add and access data:
addData
- lets knowledge sources add data to the blackboard storage. The data category
has to be named in
dataLabel
,data
hands over the actual data to store.append
is an optional flag indicating whether to overwrite or append data at the same time step (there might, for example, be several source identity hypotheses per time step, but only one source number hypothesis might be allowed).time
specifies the time point under which this data shall be stored. It is optional and, if not set, defaults to the current time. getData
- lets knowledge sources read data from the blackboard storage.
dataLabel
indicates the data category requested,reqSndTime
the time point of interest.getLastData
,getNextData
andgetDataBlock
are special cases ofgetData
for retrieving the last data, the next data after a particular point in time, or a whole data block of lengthblockSize_s
.
The following is an example from the implementation of the IdDecisionKS
class:
idHyps = obj.blackboard.getData( ...
'identityHypotheses', obj.trigger.tmIdx ).data;
%...
%find the most likely identity hypothesis -> maxProbHyp
%...
obj.blackboard.addData( ...
'identityDecision', maxProbHyp, false, obj.trigger.tmIdx );
Let us assume that we have an instantiation called bbs
of the Blackboard system.
Now we would like to see the currently available data in its memory. For that
you can run:
>> bbs.blackboard.getDataLabels()
or specify explicitly a time for which you would like to see the available data:
>> bbs.blackboard.getDataLabels(bbs.blackboard.currentSoundTimeIdx-1)
Additionally, the blackboard is used as a storage for pointers to signals from
the Auditory front-end requested by knowledge sources inheriting from
AuditoryFrontEndDepKS
. The actual memory in which these signals are stored
for recall is implemented in the Auditory front-end through circular buffers.
Dynamic blackboard interactions¶
Knowledge sources can communicate information through the flexible blackboard storage. However, adding data to the blackboard does not trigger other knowledge sources to be executed. Such interaction – triggering knowledge source execution – is done through an event system. Specifically, knowledge sources do not actually trigger execution of other knowledge sources (since they are decoupled and have no “knowledge” of each other), but knowledge sources make a request to be triggered upon the firing of particular events.
Each knowledge source has a standard event it can trigger, KsFiredEvent
,
inherited from AbstractKS
. Beyond that, every knowledge source class is free
to define as many additional events as reasonable for its task. Knowledge
sources cause the events themselves through a call to notify
as in the
following example, in which the knowledge source induces an event and attaches a
BlackboardEventData
object holding the time that it was triggered:
notify( 'KsFiredEvent', BlackboardEventData(obj.trigger.tmIdx) );
The blackboard system a priori is totally ignorant of which events exist (clear
responsibilities principle, open to extension). It also does not monitor any
events by default, until knowledge sources request to be triggered by an event.
This request is done through the method bind
provided by the
BlackboardMonitor
class, whose interface is (partially) listed in the
following excerpt:
class BlackboardMonitor
properties
pastAgenda;
executing;
agenda;
methods
BlackboardMonitor()
bind( sources, sinks, addMode, eventName )
The bind
method connects the sinks
knowledge sources to event
eventName
(optional, defaults to KsFiredEvent
) caused by the sources
knowledge sources. addMode
specifies how the BlackboardMonitor
shall
handle adding the triggered knowledge sources into the agenda. It understands
the following modes, illustrated in Fig. 32:
add
- Add the triggered knowledge source to the end of the agenda, regardless of whether or not there is already a (not yet executed) knowledge source instantiation of this sink in the agenda from a former triggering.
replaceOld
- Replace old knowledge source instantiations of this sink in the agenda with the new one. Only instantiations of the sink triggered by the same source and same event are replaced. This is an important mode for knowledge sources where processing current data is more important than processing all data.
replaceParallel
- Replace knowledge source instantiations of this sink from the same time point of parallel sources in the agenda with the new one. Only instantiations of the sink triggered at the same time and by the same event are replaced. This mode avoids sinks being unnecessarily executed several times with the same information.
replaceParallelOld
- Replace old or current knowledge source instantiations of this sink
triggered by parallel sources in the agenda with the new one. Only
instantiations of the sink triggered by the same event are replaced. This
mode is a combination of the
replaceOld
andreplaceParallel
modes.
It should be noted that the addMode
only affects triggered knowledge source
instantiations in the agenda, i.e. those that are not executed yet. As soon as
a knowledge source is executed, it is removed from the agenda (first in
executing
, afterwards in pastAgenda
).
Dynamic blackboard scheduler¶
The scheduler is the component of the blackboard system that actually executes the knowledge sources – but first, it schedules them, that is, it decides the order in which knowledge sources waiting in the agenda get executed. This order is rescheduled after every execution of a knowledge source, since the conditions determining the order may have changed, or new knowledge sources may be present in the agenda that are more urgent.
The following factors influence the order of execution of knowledge sources:
- Knowledge sources have a property called attentional priority. Knowledge
sources with higher priority get executed before knowledge sources with lower
priority. This priority can be set by the knowledge source itself, by other
knowledge sources or from outside the system. The
BlackboardMonitor
provides a method for setting focus on a knowledge source (increasing its priority), along with the option to propagate this higher priority down along the dependency chain of this knowledge source. The dependency chain is determined by the event bindings. - Knowledge sources must implement a method
canExecute
, that returns whether or not the knowledge source can execute at this moment, and which is called by the scheduler if the knowledge source is first on the scheduling list. If it cannot execute, the knowledge source can decide whether to remain in the agenda or be removed from it. - Knowledge sources define a maximum invocation frequency, that cannot be exceeded. It is a maximum frequency, because knowledge sources get not necessarily executed periodically, since they are triggered by events, but not by timers. The scheduler checks whether the last execution time was long enough ago before considering the knowledge source for execution. Until then, it remains in the agenda.
This listing shows the relevant parts of the interface with respect to influencing the scheduling:
class BlackboardMonitor
methods
focusOn( ks, propagateDown )
resetFocus()
class AbstractKS
properties
invocationMaxFrequency_Hz;
methods (Abstract)
canExecute()
execute()
methods
focus()
unfocus()