How do I specify the compare function of the scoreboard in Cocotb?

I want to extend the Endian Swapper example of Cocotb, so that, it also checks the contents of the packages outputted by the device under test (DUT). In the provided example code, the model function which generates the expected output appends the unmodified input transaction to the list of expected outputs. This list is given as a parameter to the scoreboard.

To understand how the scoreboarding works and why the model function did not append the byte-swapped transactions, I introduced a design error in the DUT. Within the following code block of endian_swapper.vhdl

if (byteswapping = '0') then
    stream_out_data      <= stream_in_data;
else
    stream_out_data      <= byteswap(stream_in_data);
end if;

I just inverted the if condition in the first line to: (byteswapping /= '0').

After re-running the testbench, I would have expected that the test fails, but it still passes:

#  62345.03ns INFO     cocotb.regression                         regression.py:209  in handle_result                   Test Passed: wavedrom_test
#  62345.03ns INFO     cocotb.regression                         regression.py:170  in tear_down                       Passed 33 tests (0 skipped)
#  62345.03ns INFO     cocotb.regression                         regression.py:176  in tear_down                       Shutting down...

It seems, that the compare function is missing in the creation of the scoreboard:

self.scoreboard = Scoreboard(dut)
self.scoreboard.add_interface(self.stream_out, self.expected_output)

It should have a third parameter in the call of add_interface, but this parameter is undocumented.

So, how do I specify this compare function, so that, also the package content is checked?

I am using QuestaSim for simulation and executed the testbench with make SIM=questa. I also clean-upped the build directory between the runs.

Solution

If I apply the following diff when using Icarus the tests fail as expected:

diff --git a/examples/endian_swapper/hdl/endian_swapper.sv b/examples/endian_swapper/hdl/endian_swapper.sv
index 810d3b7..a85db0d 100644
--- a/examples/endian_swapper/hdl/endian_swapper.sv
+++ b/examples/endian_swapper/hdl/endian_swapper.sv
@@ -119,7 +119,7 @@ always @(posedge clk or negedge reset_n) begin
             stream_out_startofpacket <= stream_in_startofpacket;
             stream_out_endofpacket   <= stream_in_endofpacket;

-            if (!byteswapping)
+            if (byteswapping)
                 stream_out_data      <= stream_in_data;
             else 
                 stream_out_data      <= byteswap(stream_in_data);

I don't have access to Questa but I'll see what happens on a VHDL simulator. My instinct would be to double check that you ran make clean after making the change and check that Questa isn't caching the built RTL libraries somehow.

You are correct that there are some undocumented keyword arguments to the add_interface method of the scoreboard:

compare_fn can be any callable function
reorder_depth is an integer to permit re-ordering of transactions

If you supply a compare_fn it will be called with the transaction when it's received by a monitor, however this is quite a primitive mechanism. It's not scalable and only there for historical reasons (hence un-documented).

A better way is to subclass the Scoreboard class and define a custom compare method according to the following prototype:

def compare(self, got, exp, log, strict_type=True):
    """
    Common function for comparing two transactions.
    Can be re-implemented by a subclass.
    """

where got and exp are the received and expected transactions and log is a reference to the logger instance of the monitor (to provide more meaningful messages).