Reduce test CI script duration

[enzoevers]

The test pipeline currently spends a lot of time compiling https://github.com/google/fuzztest.git. It should be seen if this time can be reduced.