amazon-web-servicesautomated-testsappiumaws-device-farm

Is it possible to modify the Network Profile during a test case on Device Farm?


I am building a web app that runs in-browser. This web app is a PWA and should continue to work normally if the device goes "offline".

I am trying to write Appium tests which I want to run on AWS Device Farm to validate this behaviour.

The AWS Device Farm Docs suggest that it is possible to "Change Network Conditions During Your Test" and points to the use of the CreateNetworkProfile. However when I use this API although I can create a new network profile, I can't find a way to activate the profile midway through a test.

Here is an example of a simple test:

const webdriverio = require('webdriverio')
const assert = require('chai').assert
const {
  CreateNetworkProfileCommand,
  DeviceFarmClient
} = require('@aws-sdk/client-device-farm')

const offlineProfile = new CreateNetworkProfileCommand({
  projectArn:
    'arn:aws:devicefarm:us-west-2:XXXXXXXXXXXX:project:XXXXXXXXXXXXXXXXXXX',
  name: 'offline',
  uplinkBandwidthBits: 0,
  downlinkBandwidthBits: 0
})
const onlineProfile = new CreateNetworkProfileCommand({
  projectArn:
    'arn:aws:devicefarm:us-west-2:XXXXXXXXXXXX:project:XXXXXXXXXXXXXXXXXXX',
  name: 'online',
  uplinkBandwidthBits: 104857600,
  downlinkBandwidthBits: 104857600
})

const credentials = {
  accessKeyId: '',
  secretAccessKey: '',
  sessionToken: ''
}

const deviceFarmClient = new DeviceFarmClient({
  region: 'us-west-2',
  credentials
})

describe('Test Online and Offline', function () {
  it('Should go to Google then go offline then fail to get to Bing', async function () {
    const client = await webdriverio.remote({
      capabilities: {
        automationName: 'XCUITest',
        platformName: process.env.DEVICEFARM_DEVICE_PLATFORM_NAME,
        deviceName: process.env.DEVICEFARM_DEVICE_NAME,
        platformVersion: process.env.DEVICEFARM_DEVICE_OS_VERSION,
        browserName: 'Safari'
      },
      path: '/wd/hub',
      host: process.env.APPIUM_HOST || 'localhost',
      port: process.env.APPIUM_PORT || 4723,
      logLevel: 'info'
    })

    await client.url('https://www.google.com')
    const title = await client.getTitle()
    assert.equal(title, 'Google')
    console.log('Online check completed')

    const response = await deviceFarmClient.send(offlineProfile)
    console.log(response)
    // At this point I need the device to switch to using the new "offlineProfile" NetworkProfile
    console.log("Network switched to 'offline'")

    await client.url('https://www.bing.com')
    const title2 = await client.getTitle()
    assert.equal(title2, 'Bing')  // This step is expected to fail because the device should be offline

    const response2 = await deviceFarmClient.send(onlineProfile)
    console.log(response2)
    // At this point I need the device to switch to using the new "onlineProfile" NetworkProfile to clean up

    await client.deleteSession()
  })
})

Is it possible to do what I want to do with Device Farm? If it is, how would I go about switching the active Network Profile mid-test?


Solution

  • I have received an answer to this from AWS Support - unfortunately dynamic changes to the Network Profile are not supported (at time of writing).

    Support response:

    AWS Device Farm service allows selecting a custom Network Profile only at the time of scheduling the test run and not when tests are in running state. The custom Network Profile created in Device Farm can only be selected to use in "ScheduleRun" API [1] by specifying the "networkProfileArn" parameter. Hence, we have to manage the network configurations through our test scripts or any supported library.

    As per the above statement from service team; I am sorry to say but, dynamic Network Profile handling for our custom profile is yet not supported. So, we have an option of using "Mobile JSON Wire Protocol Specification" API provided by Selenium framework and control setting of the network connection of a device.