Project

General

Profile

Actions

Feature #5446

open

correlate git version to ttcn3 tests

Added by neels about 2 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
02/07/2022
Due date:
% Done:

0%

Spec Reference:

Description

This happens a lot: we improve ttcn3 testing and enhance osmo-foo master, and then
obviously the latest binaries cannot possibly pass the tests. We invent and managa
shims in jenkins.sh and ttcn3 config to not do something or other on latest.

A way to not have this burden would be that the ttcn3 test suite for a program
is correlated to the git version being tested, for example if the ttcn3 is kept
in the same git tree as the program. We should use the 'latest' version of
ttcn3 tests for a latest binary. With an implicit correlation, we can always
use exactly the tests that match the specific git revision that was built, no
matter if it was released or we're just rebasing a local branch.

I think in the long run it would save us a lot of grunt work, and it would
definitely save a lot of code cruft to make specific parts of the tests
optional.


Related issues

Related to OsmoBSC - Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_mscResolvedfixeria12/05/2021

Actions
Actions #1

Updated by neels about 2 years ago

  • Related to Bug #5337: ttcn3-bsc-test: leaked struct bsc_subscr in BSC_Tests.TC_no_msc added
Actions #2

Updated by laforge about 2 years ago

A way to not have this burden would be that the ttcn3 test suite for a program
is correlated to the git version being tested, for example if the ttcn3 is kept
in the same git tree as the program.

I think this approach was ok if we assume "perfect" test coverage at the time
we tag a release. In reality, it is not.

The problem with that is that we don't get increased/fixed tests for 'latest'.
So if we discover a bug/problem and introduce a new test case, that test doesn't
get executed on 'latest' and we don't even see that it doesn't pass.

How can we reconcile those two?

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)