Feature #5446
opencorrelate git version to ttcn3 tests
0%
Description
This happens a lot: we improve ttcn3 testing and enhance osmo-foo master, and then
obviously the latest binaries cannot possibly pass the tests. We invent and managa
shims in jenkins.sh and ttcn3 config to not do something or other on latest.
A way to not have this burden would be that the ttcn3 test suite for a program
is correlated to the git version being tested, for example if the ttcn3 is kept
in the same git tree as the program. We should use the 'latest' version of
ttcn3 tests for a latest binary. With an implicit correlation, we can always
use exactly the tests that match the specific git revision that was built, no
matter if it was released or we're just rebasing a local branch.
I think in the long run it would save us a lot of grunt work, and it would
definitely save a lot of code cruft to make specific parts of the tests
optional.
Related issues
Updated by neels about 2 years ago
- Related to Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_msc added
Updated by laforge about 2 years ago
A way to not have this burden would be that the ttcn3 test suite for a program
is correlated to the git version being tested, for example if the ttcn3 is kept
in the same git tree as the program.
I think this approach was ok if we assume "perfect" test coverage at the time
we tag a release. In reality, it is not.
The problem with that is that we don't get increased/fixed tests for 'latest'.
So if we discover a bug/problem and introduce a new test case, that test doesn't
get executed on 'latest' and we don't even see that it doesn't pass.
How can we reconcile those two?