I have added macrobenchmark module with StartupTimingMetric
and FrameTimingMetric
metrics long time ago and my setup was working fine. Since then I have not been running benchmarks due to some reasons. Recently, had time to go back and update the user flows and run the benchmarks again. But my FrameTimingMetric
benchmarks started to fail at step of running measureBlock
. UiAutomator
couldn't find any object by resource id inside of measureBlock
but could see all of them in setupBlock
. I'm using Compose and necessary composables have testTag
added.
It was working fine a year ago when I have added benchmarks initially, but now they stopped. In the end I managed to fix the problem by removing startupMode = StartupMode.COLD
. I have no idea why this flag is affecting the visibility of composables in measureBlock
only, while setupBlock
works just fine.
Could someone help me understand why this is happening?
// NOT working case
rule.measureRepeated(
packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
?: throw Exception("targetAppId not passed as instrumentation runner arg"),
metrics = listOf(FrameTimingMetric()),
compilationMode = compilationMode,
startupMode = StartupMode.COLD,
iterations = 10,
setupBlock = {
pressHome()
startActivityAndWait()
// other setup methods
// "MyList" is visible here and list var is NOT null
// val list = device.findObject(By.res("MyList"))
},
measureBlock = {
// "MyList" is NOT visible here and list var IS null
val list = device.findObject(By.res("MyList"))
}
)
// Working case
rule.measureRepeated(
packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
?: throw Exception("targetAppId not passed as instrumentation runner arg"),
metrics = listOf(FrameTimingMetric()),
compilationMode = compilationMode,
// startupMode = StartupMode.COLD, // <- REMOVE this line
iterations = 10,
setupBlock = {
pressHome()
startActivityAndWait()
// other setup methods
// "MyList" is visible here and list var is NOT null
// val list = device.findObject(By.res("MyList"))
},
measureBlock = {
// "MyList" is visible here and list var is NOT null
val list = device.findObject(By.res("MyList"))
}
)
If StartupMode.COLD
is used, the app process is killed between the
execution of setupBlock
and measureBlock
to allow for app preparation
without starting the process.
If you need the process to remain active, use
StartupMode.WARM
, which restarts activities without restarting the process, or
set startupMode
to null
and call killProcess()
within the setupBlock
.
You can subscribe to b/278214396 to get more updates about this in the future.