pythonpython-3.xpytestpython-asyncioxdist

Delayed statement execution and test result without holding testing

(I am a non technical QA so please pardon my technical ignorance)

Libraries:

  1. pytest==6.2.4
  2. mysql-connector-python==8.0.23
  3. selenium==3.141.0
  4. psycopg2==2.8.6
  5. assertpy==1.1
  6. requests==2.25.1
  7. sshtunnel==0.4.0

(I don't mind switching to another python test framework if it solve my problem)

Test Execution Environment: Currently locally and triggered manually. In future, through CI pipeline.

Use Case:

I would be running E2E tests using selenium and then checking few things in DB. Some checks would be immediate while some checks would be delayed for some known duration.

These checks are delayed because the API under test creates a job which execute after 20min to insert/update something in DB table.

API is tightly coupled with frontend cookies so cannot separately test this functionality through API unit/integration testing.

These tests are compliance related so cannot ignore them. For now manually testing them once a week. It is a real pain in ...

Let's say if, I have 35 such tests and each automation test needs to wait for 20 mins then the total duration of test execution is very high.

Consider a Scenario:

I have few test files in which there are 5 tests. At the end of each test method there are 4 assertions. The last 2 assertion needs me to verify something in a DB table after 'x' mins. For each test method, the waiting duration for each last 2 assertion could be varying (>=15min) and this would be known beforehand.

Now I have tried,

  1. time.sleep(duration) This just holds up entire testing.

  2. Even concurrent/parallel execution does not help because when they reach last 2 assertion statements in each test method then they all have to wait for a duration and it just halts testing for given time.

  3. Thought of using RabbitMQ with Celery and scheduling those assertions but by the time these assertions would be executed, pytest would have assigned all tests a status: (Pass/Fail). So if test1 was marked as Pass and after 10 mins, after the execution of assertion statement it Fails then I would not be able to update status of test1 in pytest.

Current Workaround:

For each test method, I store required ids in a file and manually check them after 30mins.

I have few things in mind:

If I am able to:

  1. Delay or schedule execution of a statement in a test method. (Such statements would be the last few statements in a test method)

  2. Hold Test results for the completion of execution of delayed statement.

  3. Update the Result of such tests or test method after delayed assertions are over.

  4. Start with next test method after scheduling these delayed statements when running parallel tests. (It is always understood that such delayed statements would be the last few statements of test method)

Is this doable and achievable?

Other References: https://github.com/pytest-dev/pytest/issues/9012#issuecomment-900851702 (I really did not get that)

Any docs for reference, pseudocode, article, blogs where similar problem has been solved would be appreciated.

Solution

  • We should break this 1 test into 2 tests.

    1. 1st test should drive UI till something is added up in the queue and verify the same.

    2. 2nd test should check that the message was processed correctly.