Project

General

Profile

Bug #3812

stream_test fails depending on where it is run

Added by neels about 2 months ago. Updated about 1 month ago.

Status:
Stalled
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
02/21/2019
Due date:
% Done:

0%

Spec Reference:

Description

msuraev, your new stream_test fails for me.
On my home laptop sporadically, on my office box reliably.

 1: stream_test                                     FAILED (testsuite.at:8)

▶ cat tests/testsuite.dir/1/testsuite.log 
#                             -*- compilation -*-
1. testsuite.at:4: testing stream_test ...
../../../src/libosmo-netif/tests/testsuite.at:8: $abs_top_builddir/tests/stream/stream_test
--- experr    2019-02-21 16:12:24.956776809 +0100
+++ /n/s/dev/make/libosmo-netif/tests/testsuite.dir/at-groups/1/stderr    2019-02-21 16:12:33.984665754 +0100
@@ -12,8 +12,7 @@
 autoreconnecting test step 6 [client OK, server OK], FD reg 1

 autoreconnecting test step 5 [client OK, server OK], FD reg 1
-[ CONNECTED] osmo_stream_cli_recv(): connection closed with srv
-[      NONE] osmo_stream_cli_reconnect(): retrying in 9 seconds...
+connection closed with srv

 autoreconnecting test step 4 [client OK, server OK], FD reg 0

@@ -37,7 +36,6 @@
 non-reconnecting test step 2 [client OK, server OK], FD reg 1

 non-reconnecting test step 1 [client OK, server OK], FD reg 1
-[ CONNECTED] osmo_stream_cli_recv(): connection closed with srv
-[      NONE] osmo_stream_cli_reconnect(): not reconnecting, disabled.
+connection closed with srv

-non-reconnecting test step 0 [client OK, server OK], FD reg 0
+non-reconnecting test step 0 [client OK, server OK], FD reg 1
1. testsuite.at:4: 1. stream_test (testsuite.at:4): FAILED (testsuite.at:8)
log.txt log.txt 35.7 KB osmith, 03/13/2019 01:19 PM

History

#1 Updated by msuraev about 2 months ago

  • Status changed from New to In Progress

#2 Updated by msuraev about 2 months ago

Can you attach your config.log? I'm unable to reproduce this yet.

#3 Updated by msuraev about 2 months ago

Also, please add output of "ldd ./tests/stream/stream_test" - I'm unable to reproduce this on your office box under my user so it could be smth specific to your build.

#4 Updated by neels about 2 months ago

ldd uncovered the cause:

In a successful run, stream_test links libosmo-netif.so from the build tree.
In a failing run, stream_test links a libosmo-netif.so from /usr/local/lib.

The only difference between a succesful and a failing run, from the same build tree, is:
in a failing run, env contains LD_LIBRARY_PATH=/usr/local/lib

Somehow, setting LD_LIBRARY_PATH changes the preference of which library gets linked.
It should always link the library just now built in the source tree, regardless of the environment variables.
Instead, pointing an LD_LIBRARY_PATH to /usr/local/lib makes it prefer the installed library.

We need to find a way to use the build-tree library even if LD_LIBRARY_PATH is set.
Why? Consider a user is installing to a custom ~/prefix/lib, then merely passing LD_LIBRARY_PATH pointing there would break the build.

#5 Updated by neels about 2 months ago

How to reproduce:

- build and install libosmo-netif tag 0.4.0
- export LD_LIBRARY_PATH=/usr/local/lib
- 'make check' in current master of libosmo-netif (fe3527da2ac691c961f767c97d70bfe00d1e4d10)

The combination of an older library installed and the LD_LIBRARY_PATH in the env causes the test failure.

#6 Updated by neels about 2 months ago

  • Priority changed from Immediate to Normal

A workaround is to remove the system-installed libosmo-netif, hence downgrading priority

#7 Updated by msuraev about 2 months ago

  • Priority changed from Normal to Low

The combination of an older library installed and the LD_LIBRARY_PATH in the env causes the test failure.

Note: this might happen with any test in any of osmocom projects. In general I think the "solution" would be to not use LD_LIBRARY_PATH together with 'make check'. We should double-check with libtool docs to see how it's meant to interact with LD_LIBRARY_PATH.

#8 Updated by neels about 2 months ago

msuraev wrote:

I think the "solution" would be to not use LD_LIBRARY_PATH together with 'make check'. We should double-check with libtool docs to see how it's meant to interact with LD_LIBRARY_PATH.

That's a workaround. If the user needs to adjust the environment to be able to find installed library files and is then unable to do 'make check', your workaround doesn't work.
A solution would be something that allows setting whichever LD_LIBRARY_PATH and still have 'make check' work.
I'd prefer a way to make Makefile.am stronger than LD_LIBRARY_PATH, but in some projects, instead of linking library files, we directly compile .c files into the test binary.

#9 Updated by neels about 2 months ago

  • Assignee deleted (msuraev)
  • Priority changed from Low to Normal

#10 Updated by osmith about 1 month ago

I ran into this issue even without having LD_LIBRARY_PATH set. Here is how to reproduce it, run this script in the local libosmo-netif.git checkout.

#!/bin/sh -ex
: make distclean
env

git checkout 0fe9b3664a8d0b74d5e1c67593b350d67de3f98f # previous commit
autoreconf -fi
./configure
make clean
make check
make install

git checkout fa7df8726001146c902d3a93e2a4b425fa9024b0 # current master
autoreconf -fi
./configure
make clean
make check

The second "make check" fails, full output is attached.

testsuite.log contains:

5. testsuite.at:29: testing amr_test ...
./testsuite.at:32: $abs_top_builddir/tests/amr/amr_test
stderr:
/home/user/code/osmo-dev/src/libosmo-netif/tests/amr/.libs/amr_test: symbol lookup error: /home/user/code/osmo-dev/src/libosmo-netif/tests/amr/.libs/amr_test: undefined symbol: osmo_amr_oa_to_bwe

ldd shows that everything links against /usr/local/lib:

$ ldd tests/amr/.libs/amr_test
        linux-vdso.so.1 (0x00007ffcc8d11000)
        libtalloc.so.2 => /usr/lib/x86_64-linux-gnu/libtalloc.so.2 (0x00007fa3a203d000)
        libosmogsm.so.11 => /usr/local/lib/libosmogsm.so.11 (0x00007fa3a1ddb000)
        libosmocore.so.12 => /usr/local/lib/libosmocore.so.12 (0x00007fa3a1bb2000)
        libosmonetif.so.6 => /usr/local/lib/libosmonetif.so.6 (0x00007fa3a19a2000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa3a1603000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa3a13ff000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fa3a1066000)
        libsctp.so.1 => /usr/lib/x86_64-linux-gnu/libsctp.so.1 (0x00007fa3a0e63000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa3a2455000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa3a0c49000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fa3a09e4000)
        libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007fa3a07b0000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fa3a059d000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fa3a0366000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fa3a0131000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fa39feae000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fa39fca5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa39fa88000)

#11 Updated by osmith about 1 month ago

It turns out, that at least in Debian 9, the following file exists by default:

$ sudo cat /etc/ld.so.conf.d/libc.conf
# libc default configuration
/usr/local/lib

#12 Updated by msuraev about 1 month ago

I'm not sure it's actually related: I've got problem with amr-test despite not having anything in /usr/local/lib on ubuntu.
And adding "amr_amr_test_LDFLAGS = -no-install" to tests/Makefile.am fixed it for me. The same flag is used with stream tests so I think it's entirely different issue.

#13 Updated by msuraev about 1 month ago

  • Status changed from In Progress to Stalled

#14 Updated by osmith about 1 month ago

msuraev: using -noinstall fixes it for me too, created a patch: https://gerrit.osmocom.org/#/c/libosmo-netif/+/13241

You are right, then the stream_test issue is a completely different one.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)