Cryptic NSInternalInconsistencyException when running unit tests in Xcode 9 GM

I'm running my iOS app's unit tests on Xcode 9 GM, and a couple of them are failing with a weird NSInternalInconsistencyException, complaining that some test assertions cannot be reported, because the implicated tests have no associated XCTestRun object. I'm using OCMockito + OCHamcrest for mocking and call verification.

For demonstration purposes, let's say my app is called MyTestApp, and I have a test class FooTest (which inherits from XCTestCase). In -setUp, I create and wire together the various mock objects for the tests, and in -tearDown I set them to nil just in case.

Here's an example of the exception I'm getting:

2017-09-19 13:23:01.852729-0700 xctest[17006:5392130] *** Assertion failure in -[FooTest recordFailureWithDescription:inFile:atLine:expected:], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-13201/Sources/XCTestFramework/Core/XCTestCase.m:308
/Users/fooUser/Workspaces/MyTestApp/tests/FooTest.mm:46: error: -[FooTest testSomething] : failed: caught "NSInternalInconsistencyException", "Unable to report test assertion failure 'Expected 1 matching invocation, but received 0' from /Users/fooUser/Workspaces/MyTestApp/tests/FooTest.mm:135 because it was raised inside test case -[FooTest testSomethingElse] which has no associated XCTestRun object. This may happen when test cases are constructed and invoked independently of standard XCTest infrastructure."
(
    0   CoreFoundation                      0x000000010255d1cb __exceptionPreprocess + 171
    1   libobjc.A.dylib                     0x0000000101ebff41 objc_exception_throw + 48
    2   CoreFoundation                      0x0000000102562362 +[NSException raise:format:arguments:] + 98
    3   Foundation                          0x0000000101827089 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 193
    4   XCTest                              0x0000000101d96875 -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:] + 518
    5   MyAppUnitTests                     0x000000011d7f2f5a -[HCXCTestFailureReporter executeHandlingOfFailure:] + 154
    6   MyAppUnitTests                     0x000000011d7f2b73 -[HCTestFailureReporter handleFailure:] + 67
    7   MyAppUnitTests                     0x000000011d64ec9a MKTFailTest + 217
    8   MyAppUnitTests                     0x000000011d64c3f4 -[MKTExactTimes verifyData:] + 254
    9   MyAppUnitTests                     0x000000011d64f2ba -[MKTBaseMockObject verifyInvocation:usingVerificationMode:] + 137
    10  MyAppUnitTests                     0x000000011d64f20b -[MKTBaseMockObject handlingVerifyOfInvocation:] + 115
    11  MyAppUnitTests                     0x000000011d64f15a -[MKTBaseMockObject forwardInvocation:] + 64
    12  CoreFoundation                      0x00000001024dfed8 ___forwarding___ + 760
    13  CoreFoundation                      0x00000001024dfb58 _CF_forwarding_prep_0 + 120
    14  MyAppUnitTests                     0x000000011c40b486 -[FooTest setUp] + 294
    15  XCTest                              0x0000000101d97b39 __24-[XCTestCase invokeTest]_block_invoke_3 + 31
    16  XCTest                              0x0000000101d97809 __24-[XCTestCase invokeTest]_block_invoke + 271
    17  XCTest                              0x0000000101ddff45 -[XCUITestContext performInScope:] + 183
    18  XCTest                              0x0000000101d976ef -[XCTestCase invokeTest] + 141

Several things of note:

The unit tests are not initializing XCTestCase or any XCTest "infrastructure" classes outside of the XCTest framework.
The remaining statements of the affected unit tests actually gets executed, up to the point of the exception.
The implicated line of code looks like: [verify(_mockTestObject) description];
The stack trace is lying about where the problem occurs - it says that it's in -setUp, but from what I can tell, OCMockito simply reports expectation failures at the beginning of the next unit test's run. The problem location (FooTest.mm:135) looks like the above verify() call.

In fact, all unit tests that fail with this exception fail because we're trying to verify that -[NSObject description] has been called on one mock object or another. Removing all instances of verifying that call makes the tests pass.

I searched on Google for other instances of this particular NSInternalInconsistencyException, and that didn't yield any results. I'm not sure in what way -[NSObject description] is different from any other method that may be called on a mock object - it's possible that the problem isn't really there either, but that it just manifests that way. I was also trying to find if there are any debug/diagnostic options I can turn on for XCTest so I could maybe get more verbose logging information about test execution, but I haven't found anything like that either.

Any ideas for where I should look next? Thanks a lot!

Solution

I experienced a similar problem while working on some big suite of legacy code tests. The question is: are your tests with XCTExpectation working fine when you run them separately? If so, this means that some of the tests that are executed before your tests with NSInternalInconsistencyException are dispatching XCTest related methods in a way which executes them after the relevant test finishes.

it looks like (example):

Test1 -> dispatches asynchronously "block" which executes XCTFail

Test1 -> finishes

XCTFail executed (but Test1 passes, as it finished without "fail") on the main or other thread.

Test2 -> tests something with XCTExpectation -> NSInternalInconsistencyException

Apple docs don't provide much information about internal guts of XCTest, but I'm pretty sure that this is the issue. Try following troubleshooting steps to pin down tests "conflicting" (the bad ones which do asynchronous stuff without XCTestExpectation, like use methods with completion handlers which are eventually doing XCTest assertions, fails etc.):

Do binary search in your suite: disable half of the tests and run the suite with your FooTest.
If your test suite runs fine re-enable half of the disabled tests. If test suite runs with exception, re-enabled all disabled tests and disable other half.
Repeat step 1 then with respect to smaller amount of tests left.
Finally you will end up with tests which cause this exception.

In my case there were multiple tests causing this conflict with XCTestExpectation, hence the search was quite pesky (several hours for a suite of 1000+ XCTestCases, so around 5k tests).

Then investigate thoroughly what happens in the test which is conflicting with your test.