vunit, what are reset conditions between test case

I'm very confused about vunit testing, especially the link between tests and the way they are reset.

Please take a look at next minimal example:

device under test

Device has one inner state that latch on 1 when input goes to 1

library IEEE;
use IEEE.STD_LOGIC_1164.all;
use IEEE.NUMERIC_STD.all;

entity dut is
    port(
        A : in  std_logic;
        B : out std_logic := '0'
    );
end entity;

architecture RTL of dut is
begin
    -- will latch when A goes to '1'
    -- stay 1 forever
    B <= '1' when A = '1';
end architecture;

testbench

test1 : Dut inner state to '1'
test2 : Dut has been reset in between test1 and test2
test3 : Does not set dut inner state to '1' because C, which is set at start of process, but somehow reset to initial state (1) before test.
test4 : Does not set dut inner state to '1' because D, which is set at start of process, but somehow reset to initial state (U) before test.
test5 : Does not set dut inner state to '1' because E, which is set at start of loop, but somehow reset to initial state (U) before test. Which seems contrary to the the documentation (or most likely, I don't get it)
test6 : Dut inner state is set, F is not reset.

library IEEE;
use IEEE.STD_LOGIC_1164.all;
use IEEE.NUMERIC_STD.all;

library vunit_lib;
use vunit_lib.run_pkg.all;
use vunit_lib.check_pkg.all;

library libdut;
use libdut.all;

entity some_tb is
    generic (runner_cfg : string);
end entity;

architecture arch of some_tb is
    signal A, B, C : std_logic := '0';
    signal D, E, F : std_logic;
begin
    invdut: entity dut port map (A => A, B => B);

    F <= '1';

    main: process begin
        test_runner_setup(runner, runner_cfg);
        C <= '1';
        D <= '1';

        while test_suite loop
            E <= '1';

            if run("test 1") then
                check_equal(B, '0', "test 1.1");
                A <= '1';
                wait for 1 ns;
                check_equal(A, '1', "test 1.2");
            elsif run("test 2") then
                check_equal(A, '0', "test 2.1");
                check_equal(B, '0', "test 2.2");
            elsif run("test 3") then
                A <= C;
                wait for 1 ns;
                check_equal(B, '0', "test 3.1");
            elsif run("test 4") then
                A <= D;
                wait for 1 ns;
                check_equal(A, 'U', "test 4.1");
                check_equal(B, '0', "test 4.2");
            elsif run("test 5") then
                A <= E;
                wait for 1 ns;
                check_equal(A, 'U', "test 5.1");
                check_equal(B, '0', "test 5.2");
            elsif run("test 6") then
                A <= F;
                wait for 1 ns;
                check_equal(A, '1', "test 6.1");
                check_equal(B, '1', "test 6.2");
            end if;
        end loop;

        test_runner_cleanup(runner); -- Simulation ends here
        wait;
    end process;
end architecture;

Vunit file

#!/usr/bin/env python3

from pathlib import Path
from vunit import VUnit

SRC_PATH = Path(__file__).parent / "src"
TEST_PATH = Path(__file__).parent / "test"

def create_testsuite(project):
    lib = project.add_library("libdut")
    lib.add_source_files(SRC_PATH / "*.vhdl")

    libuart_test = project.add_library("libdut_test")
    libuart_test.add_source_files(TEST_PATH / "*.vhdl")

The question(s)

This seems very random to me. I will ask multiple question into one, because, most likely, they are tightly correlated.

When are the signal reset ? Is every test run in a different simulation or are they "in series" with chances of side effect ? In which foremost case, why is there a loop ?

Solution

By default VUnit will run each test case in a separate simulation but it is possible to change this behavior using the run_all_in_same_sim attribute. The rationale for this is described here: https://vunit.github.io/run/user_guide.html#running-test-cases-independently

Having the loop allow for this option but there are also other reasons as explained in https://vunit.github.io/run/user_guide.html#running-a-vunit-testbench-standalone. In general, https://vunit.github.io/run/user_guide.html is a good source for the details about VUnit execution.

The reason your signal assignments do not work as you expect is due to delta cycles. When A is assigned E in test 5 it is done in the same delta cycle as E is assigned 1 since nothing between the two assignments consume time. The new value of E is not present when A is assigned E. Instead A gets the previous value of E (U).

The difference between test 5 and 6 is that F is assigned 1 before the test_runner_setup procedure completes. A procedure can consume time and in this case it does, at least delta cycles. When A is assigned F the new value of F is present.

I'm not sure how well you know the concept of delta cycles and the difference between signals and variables. If not, I suggest that you look into that.