Currently it is a known issue that the tests are failing constantly and they are not being maintained properly. They are also fragile, dying off to unlucky timouts errors (443) which is just noise. not helpful.
I have currently disabled the cron job for the tests as is just pollutes our inboxes, however it still runs on-push. The pytest.yml workflow created real requests every run which makes it fragile, slow, and therefore unreliable. But it does have the benefit of helping to catch any changes to the scratch website/api.
i think we should have a bunch of mock tests which ensure a known json "REST response" results in the correct Scratchattach objects being outputted. These are to catch regressions in the data unpacking logic.
I then think that we should have some other tests that run less frequently on a cron job, not on push, which make real web requests to scratch, e.g. once a week. This will catch changes to the scratch website/api.
However I am not sure of the best way to do these mock tests. If you use the monkeypatching method, you can overwrite the requests.get/post/put/delete methods and then return mock responses. However, with the aim of asynchronous features, this may have to be also done for aiohttp.
Currently it is a known issue that the tests are failing constantly and they are not being maintained properly. They are also fragile, dying off to unlucky timouts errors (443) which is just noise. not helpful.
I have currently disabled the cron job for the tests as is just pollutes our inboxes, however it still runs on-push. The
pytest.ymlworkflow created real requests every run which makes it fragile, slow, and therefore unreliable. But it does have the benefit of helping to catch any changes to the scratch website/api.i think we should have a bunch of mock tests which ensure a known json "REST response" results in the correct Scratchattach objects being outputted. These are to catch regressions in the data unpacking logic.
I then think that we should have some other tests that run less frequently on a cron job, not on push, which make real web requests to scratch, e.g. once a week. This will catch changes to the scratch website/api.
However I am not sure of the best way to do these mock tests. If you use the monkeypatching method, you can overwrite the
requests.get/post/put/deletemethods and then return mock responses. However, with the aim of asynchronous features, this may have to be also done for aiohttp.