CPN Tools 4 Extensions: Part 4: Advanced Communication and Debugging

In this part we look at more advanced communication.  Previous parts assume little knowledge of the inner workings of CPN Tools, and allow you to make extensions almost as if you wrote regular Java programs.  Lying under all the abstractions is a very versatile communication protocol, which allows very elaborate extensions.

This part gives an introduction to getting started doing more advanced communication.  We also introduce the protocol debugger, which makes experimenting with advanced communication much simpler.  This is part 4 of a series on extensions for CPN Tools 4; it assumes you have read part 1, but it may also be useful to read parts 2 and 3 to get a feel for what is possible.  Note that this is just an introduction; using advanced communication requires you do much more work than the previous parts.  Also, we cannot provide any support for this.

Advanced communication has 3 usage scenarios: GUI callbacks, package inspection and filtering, and handling extension commands.  Only the two first are interesting for most developers as the third requires changes to the CPN Tools GUI and is quite likely to be restricted to the author of this blog.  We shall therefore focus on these first two.

GUI Callback (opcode = 3)

Here we are concerned with communication pattern 8 from part 1 ((I told you I expected you to read that.)).  For reference, here it is:

Screen Shot 2013-09-29 at 15.39.51

We are dealing with a extension-initiated call to the CPN Tools GUI.  Before we go on, let us look at what a request (and response) looks like.

Package format

In CPN Tools, all packages are in the BIS format.  This is short for Boolean, Integer, String.  Everything is encoded as a sequence of those values.  The underlying communication platform takes care of serializing and transporting this.  We notice that the send operation of the Channel class takes and returns a Packet.  A Packet is shared among Access/CPN and the extension server.  The packet implements the BIS package format and a serialization to binary for transmission.  Simplified, the Package has this interface:

[java] public class Packet implements Serializable {
public Packet(int command) { … }
public Packet(int opcode, int command) { … }

public Packet(int opcode, List<? extends Object> parameters) { … }

public Packet(int opcode, String data) { … }
public Packet(String data) { … }

public void addBoolean(boolean bv) { … }
public void addInteger(int iv) { … }
public void addString(String sv) { … }

public int getCommand() { … }
public int getOpcode() { … }

public void reset() { … }

public boolean getBoolean() { … }
public int getInteger() { … }
public String getString() { … }

public String getData() { … }

public List<Object> getParameters() { … }
public Class<?> getReturnType() { … }

public void send(DataOutputStream out) throws IOException { … }
public void receive(DataInputStream in) throws IOException { … }

Each package has an opcode, which helps the simulator finding out what to do with each packet. By convention, most packets also have a command. The first two constructors (ll. 2-3) construct a package using an opcode and a command. The simpler version (l. 2) assumes opcode 9 (which is used for communication patterns 4, 9a and 9b). The GUI callback messages use opcode 3.

The constructor in line 5 is used to create packages for communication pattern 6, and the constructors in lines 7-8 allow us to use pattern 5 (which we get back to later).

The add methods (ll. 10-12) allow us to construct packets for sending, and the methods in lines 14-21 allow us to read packets. getData (l. 21) is for reading packets for pattern 5, and getParameters/getReturnType are for reading packets for pattern 6.

send and receive (ll. 28-29) are used for serializing packets and should never be called manually.

To support the GUI callback pattern (pattern 8), we need the constructor in line 3, the add methods (ll. 10-12), and the getters (ll. 17-21).

Package contents and creation

An overview of all callback messages can be seen at http://cpntools.org/cpn2000/callback_messages. As the banner says, this information is normally for internal use; using the advanced communication patterns dig soo deeply inside CPN Tools extreme caution is advised. Also, (at the time of writing,) the information may contain planned commands, incomplete descriptions, and wrong descriptions.

All descriptions contain a command number and a description of which parameters are required. Let’s look at the description of the “Create Canvas” command:

Create canvas (command = 2)

[raw] Extra call parameters:
blist= saved, raise
ilist= nil
slist= name
Return value:
blist= nil
ilist= cmd=1
slist= canvas id

From the name, we get the command number (2). We thus create a package for doing so as in line 1 below. We use the constructor (l. 3 above) to create a package with opcode 3 (GUI callback) and command 2 (Create Canvas).

Next, we need to add content to the package. We use the add methods (ll. 10-12 above) for this. The order we use to add values of the same type matters, but it does not matter whether we add booleans, integers or strings first. Sometimes we even need to mix this.

The description says that the Create Canvas package takes two booleans (blist), the first describing whether the page should be saved (i.e., whether or not it is a canvas page as described in part 3). the second boolean indicates whether the page should be opened automatically or just created in the index. These are added in lines 2-3 below.

The description says that the package takes no integers (ilist = nil) and one string (slist). The string is added completely analogously to the booleans in line 4 below.

[java] Packet p = new Packet(3, 2);
p.addBoolean(false); // saved
p.addBoolean(true); // show
p.addString(“Hello World”); // name

p = channel.send(p);

int tag = p.getInteger();
assert tag == 1;
String id = p.getString();

Having successfully created our package, we are ready to send it. If we use the AbstractExtension (from part 2), our extension has a channel field which can be used (after setChannel) as in line 6. Note that packages should be treated as functional, i.e., they should not be changed after initial creation and should only in certain instances be reused ((Read: should never be reused.)). We therefore assign the result of send back to p in line 6 as it returns a fresh Packet instance.

Next comes the decoding of the result. This is read in the same way as the input, except there may be multiple cases. Most packages will have an integer indicating the success or failure of the command. In almost all cases should the tag contain the value 1. In lines 9-10 we read and check the tag value. A value other than 1 typically means that the simulator or GUI has been shut down and is almost always fatal.

Before reading data, we call the reset method (l. 8). This is not necessary in this case, but is whenever we want to read a packet again; packets can only be read sequentially from the front.

Finally, we read the canvas id string result (l. 11) as specified in the return values. This value can then be used for future callbacks.

Protocol Debugger

Writing code as above is necessary whenever we want to do communication in our extension. It is a bit unhandy for exploration of the package format, however. Sometimes it is easier to understand the data by seeing data from a live execution than to see it from the abstract documentation (especially if the documentation is inaccurate). For this reason, CPN Tools 4 ships with a protocol debugger. This is not installed by default but is an option during installation:

Screen Shot 2013-11-05 at 20.44.21

Having the debugger installed, we get a new Tool palette, Development, which yields a Debug tool.

Screen Shot 2013-11-05 at 20.23.26

Applying this to the background, opens up the debugger. Configured to send the same packet we went thru for the code example, it looks like this:

Screen Shot 2013-11-05 at 20.24.57

From the top left, we have configured it as follows: first, we have configured it to send GUI messages.  Second, we have entered the command number 2.  In the first list, we can enter our booleans; simply click true or false to add the desired value.  Select a value and click remove to remove it or clear to clear the list of booleans.  The next box allows us to enter integers, add and remove them, and clear the list.  We have not entered any.  The last list allows us to enter strings, add and remove them and clear the list.  We have added one string.

Below that are controls for dealing with the constricted packet.  We can send or clear the data.  Clearing corresponds to clearing all lists and removing the command at the top right.  Pressing send yields this:

Screen Shot 2013-11-05 at 20.26.14

In the background we see that a new canvas sheet (yellow) with the desired name has been opened — just like it said on the tin. We also see that the bottom text field of the debugger contains data. It shows the response of sending the command we just constructed. As expected, we see a single integer with value 1 and a single string which contains an identifier we can use to later draw on our canvas.

This illustrates how to send simple GUI messages; we can use the debugger for testing a sequence of commands and construct packets programatically. Now let us take a look at some of the more complex packages.

Sending Commands (opcode = 9)

Sending commands using communication pattern 4 is as easy as sending GUI commands using pattern 8.  In fact, from the point of view of extensions, this is exactly the same except the packets are different.  To recap, pattern 4 looks like this:

Screen Shot 2013-09-29 at 15.28.53

The only difference is who handles the command. In fact, the GUI command pattern 8 above look more complex, so why did we start with them instead of the simpler pattern 4? The reason is that all the complexity is hidden, so the only thing users need to concern themselves with is the actual package construction, and the simulator command packages are significantly more complex than the GUI callbacks.

You can see an overview of simulator command packages at http://cpntools.org/cpn2000/apn_ml_protocol_manual#typical_usage_of_ml-engine. Simulator commands have an extra layer of command structure; the command refers to a main group of commands, and each is separated into subcommands. Here, we typically only want to deal with command 500 (simulation); sometimes we may wish to tinker with the options available with command 200, and in rare cases we may wish to tinker with the model, adding/changing declarations using command 300, changing the net structure using command 400, and messing with monitors using command 450. We should never have to tinker with the boot options (command 100) and extension commands (command 10000). The state-space protocol is not complete and only works under very specific conditions. I already mentioned that using this API is not supported, and this goes doubly for any simulator command, heck make that 5 times for all commands but simulation commands.

Having said this, you can really mess around with things using these commands. You can add and remove places and transitions, reroute arcs, and completely make the view the simulator has of the model differ from what the GUI sees.

As a simple example, we can use command 500, 2 (command 500, subcommand 2) from here. For reference, the documentation for this states:

2: In this case the simulator returns the simulation time and step number.

[raw] Extra call parameters:
blist= nil
ilist= nil
slist= nil
Return value:
blist= nil
ilist= TERMTAG=1
slist= sim-time,sim-step

The format is the same as for the GUI commands with the exception that the command number now is a subcommand, so we have to use 500 for the command and add the number 2 before any other integer parameters. As all parameter lists are empty, the command looks as follows after sending:

Screen Shot 2013-11-05 at 21.39.48

We see we have changed the package type to command. We have a command number of 500 and have added a single integer 2. As all lists are empty, we do not add more values. After sending, we get the response at the bottom; we see that both the model time and step number are 0. These are sent as strings as they have no upper bound (and can hence not be represented as integers) and the time value may also be a real value instead of an (unbounded) integer.

Note that the protocol performs none or very little sanity checks. This means that sending off a wrong command can crash the GUI or simulator or cause either to lock up. All extensions are expected to adhere 100% to the protocol.

It is also important to notice that just because simulator state is updated, the GUI is not necessarily updated; extensions are responsible for this themselves. For example, if a marking is changed, the extension should check enabling of surrounding transitions and inform the GUI vis callback command 1 ((It is possible that the model scraper structure will be extended to handle this automatically in the future, but for now extensions have to do this thermselves.)).

More on the Debugger

The debugger is extensible and contains a couple other tabs. Those are mostly for my use, but may be useful for others and are therefore included. The Scraper tab follows the model scraper we talked about in part 2:

Screen Shot 2013-11-05 at 21.46.13

It shows all pages along with all their places, transitions and arcs at the top. The middle field shows all update events sent from the scraper. This is all updated live as the model is changed.

The Demos tab contains a number of simple demos — mostly test code. See more about that in this video, which also shows off the debugger:


Filtering Commands

Here, we deal with communication patterns 9a and 9b. In fact, 9a is just a simplified version of 9b, so let’s quickly recap that:

Screen Shot 2013-09-29 at 15.53.32

We can request to get such updates on the prefilter and handle methods of the Extension (see part 1) using the addSubscription method of the AbstractExtension (see part 2). The addSubscription takes a Command, which takes two parameters; now we are in a position to fully understand those parameters: the first is the command and the second is the subcommand. This means, we can subscribe to messages of any desired type. We can even subscribe to messages to another extension.

After adding a subscription, we receive notifications on the prefilter method and the handle method (ll. 8-9 of the Extension interface in part 1). These receive packages. The prefilter method is allowed to return a new packet, thereby modifying the request sent from the GUI, and the handle method may return a new response. Note that the response MUST be valid for the packet received originally to not confuse the GUI!

This is an extremely powerful mechanism, that allows extensions to observe, alter and extend what the GUI is doing. For example, the model scraper from part 2 only subscribes to messages 400, 2, which includes information about the structure of the model. The Declare extension (used to implement the Declare functionality) subscribes to and filters commands for going to initial state, for checking enabling, and for executing transitions. This allows the extension to restrict enabled transitions without changes to the user interface. Finally, the time interval estension (which implements time intervals) listens for commands 400, 2 (syntax check page), and checks if the time annotation contains a range. If it does, it creates a new type and variable (command 300), alters the transition to instead use the new variable in the time expression, and changes the guard so the variable can reliably be bound using manual binding and during state space analysis. All this with no change to simulator nor GUI. It is easy to envision using this to implement bounded places, ordered places, transition fusion, etc.


Here, we have looked at the Packet class and how it can support communication patterns 4, 6 and 9a/b. We have shown how to create such packages programmatically and also how to make them using the debugger. We have seen where to find information about the package formats, and scraped the surface of what is possible using this mechanism.

The mechanism is very powerful and also very difficult to use. Documentation may not be completely up to date, and some calls make undocumented assumptions. Unfortunately, we cannot support use of this part of the API ((We can, however, provide paid consultancy for this; contact for a quote.)). Of use may be the code decoding messages on the simulator part here, and the code parforming the main syntax check loop in CPN Tools here (the CPNetMLCheckerThread) and the code responsible for constructing most packages in the GUI here.

We are nearing the end of the introduction to CPN Tools 4 series; I’ve only planned one more part which lists all extensions available at the time of writing and which of the techniques they use.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.