Project

General

Profile

Feature #4550

Integrate RPM spec files into Osmocom repositories and build them on OBS

Added by osmith 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
05/13/2020
Due date:
% Done:

100%

Spec Reference:

Description

The spec files from roox will be integrated into the Osmocom source repositories:

https://build.opensuse.org/project/show/home:mnhauke:osmocom:latest

I will put them in contrib (e.g. contrib/libosmocore.spec).

(Creating this issue, so I can reference it in the commit messages, more details will follow.)

osmocom-nightly-opensuse.sh osmocom-nightly-opensuse.sh 1.01 KB script from roox, that runs as cronjob to update https://build.opensuse.org/project/show/home:mnhauke:osmocom:nightly osmith, 05/13/2020 08:04 AM

Related issues

Related to Cellular Network Infrastructure - Feature #4563: install test for centos packages (like the debian install test)Resolved05/25/2020

Related to Cellular Network Infrastructure - Feature #4564: Run ttcn-3 testsuite with centos packagesResolved05/25/2020

History

#1 Updated by osmith 5 months ago

OBS provides Source Services, which can do various source related tasks, such as downloading sources, extracting files, adjusting the version in spec files.

laforge asked me to look into, if we could use this with our builds to extract the .spec files once they are in the Osmocom repositories.

Right now, our osmocom-*-packages.sh scripts in osmo-ci.git create source tarballs and upload them to OBS. I've considered changing this to generate a _service file instead. But this would break the debian builds, as we make changes to the debian subdirs before generating the source tarballs (e.g. conflicting latest/nightly packages #2640, checkout_copy_debian8_jessie()). This would not be possible if the source was cloned with instructions in _service files instead.

So I'm adjusting the existing scripts to simply add the .spec file next to the source tarball.

#2 Updated by osmith 5 months ago

  • % Done changed from 0 to 20

I've imported all existing .spec files to contrib/$project.spec in branch osmith/rpm of each related Osmocom git repository.

osmo-ci.git: scripts/osmocom-*-packages.sh: upload the .spec file next to the source tarball now, and set the right version and source tarball name in the spec file (also branch osmith/rpm).

I'm testing everything in my own OBS namespace first, before submitting the patches to gerrit:
https://build.opensuse.org/project/show/home:osmith42

Unfortunately, the CentOS machines are not as responsive as the Debian ones; packages seem to get stuck in a state ("finished") for a long time before the build continues.

#3 Updated by osmith 5 months ago

Also there's a new script "osmocom-obs-linkpac.sh" in osmo-ci.git, to link the packages from official OpenSUSE repositories into the Osmocom repositories (e.g. systemd-rpm-macros). This only needs to run once after creating a repository (either official one or in home:USER for testing).

#4 Updated by laforge 5 months ago

On Wed, May 13, 2020 at 02:21:56PM +0000, osmith [REDMINE] wrote:

I've imported all existing .spec files to contrib/$project.spec in branch osmith/rpm of each related Osmocom git repository.

It may make sense to include them in EXTRA_DIST to make sure they are part
of any tarball release we mage (in fact, the same is true for the existing Debian
packaging information, i.e. debian/*)

#5 Updated by osmith 5 months ago

Okay, I'll look into EXTRA_DIST once most packages are building.

Regarding systemd-rpm-macros, I found that in centos 8, the systemd package provides the macros. Unfortunately, the same conflict did not appear with local testing, so I didn't catch this earlier.

[  125s] [359/387] installing systemd-239-29.el8
[  125s]     file /usr/lib/rpm/macros.d/macros.systemd from install of systemd-239-29.el8.x86_64 conflicts with file from package systemd-rpm-macros-4-3.1.noarch
[  125s] exit ...

So I'll put this in all packages that currently depend on systemd-rpm-macros.

%if 0%{?centos_ver}                                                                                                                                                                                                                                                             
BuildRequires:  systemd                                                                                                                                                                                                                                                         
%else                                                                                                                                                                                                                                                                           
BuildRequires:  systemd-rpm-macros                                                                                                                                                                                                                                             
%endif

#6 Updated by osmith 5 months ago

After fixing a lot of small issues, most packages are building in my OBS namespace now:

libasn1c
libdbi
libdbi-drivers
libgtpnl
libosmo-abis
libosmo-netif
libosmo-sccp
libosmocore
libsmpp34
ortp
osmo-bsc
osmo-bts
osmo-ggsn
osmo-hlr
osmo-iuh
osmo-mgw
osmo-pcu
osmo-sgsn
osmo-trx
osmo-msc
python-cheetah
uhd

#7 Updated by osmith 5 months ago

There is an unexpected error when trying to install packages (tested with osmo-trx-uhd):

/var/tmp/rpm-tmp.3e2APB: line 1: fg: no job control                                                                                                   
error: %prein(osmo-trx-uhd-1.2.0.34.0f87-7.1.x86_64) scriptlet failed, exit status 1                                                                  

Error in PREIN scriptlet in rpm package osmo-trx-uhd                                                                                                  
error: osmo-trx-uhd-1.2.0.34.0f87-7.1.x86_64: install failed 

According to research, this means that centos doesn't know the %prein macro:

I'm looking into disabling that macro in the spec files for centos8 now. It's a bit tricky, because the %prein macro is not directly mentioned as such, it must be an expansion from another macro.

Once I know which macro is responsible, I can add a stub like this:

%{!?service_add_pre:%define service_add_pre %dnl}

#8 Updated by osmith 5 months ago

$ rpm -pq --scripts file.rpm

shows, that the centos packages have unexpanded macros as scripts:

preinstall scriptlet (using /bin/sh):
%service_add_pre    osmo-trx-uhd.service
postinstall scriptlet (using /bin/sh):
%service_add_post   osmo-trx-uhd.service
preuninstall scriptlet (using /bin/sh):
%service_del_preun  osmo-trx-uhd.service
postuninstall scriptlet (using /bin/sh):
%service_del_postun osmo-trx-uhd.service

while the opensuse packages have the macros expanded properly:

rpm -pq --scripts osmo-trx-uhd-1.2.0.34.0f87-7.1.x86_64.rpm                                                                 
warning: osmo-trx-uhd-1.2.0.34.0f87-7.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8f2653be: NOKEY                                            
preinstall scriptlet (using /bin/sh):

# disable migration if initial install under systemd
[ -d /var/lib/systemd/migrated ] || mkdir -p /var/lib/systemd/migrated || :
if [ $1 -eq 1 ]; then
        for service in osmo-trx-uhd.service ; do
                sysv_service="${service%.*}" 
                touch "/var/lib/systemd/migrated/$sysv_service" || :
        done
...

I'll disable the related blocks in the spec files with if else macros.

#9 Updated by osmith 5 months ago

  • % Done changed from 20 to 30

Osmocom programs can now be installed on centos8 from my repository:

$ cd /etc/yum.repos.d/
$ wget https://download.opensuse.org/repositories/home:osmith42/CentOS_8_Stream/home:osmith42.repo
$ dnf install \
    osmo-bsc \
    osmo-bts \
    osmo-ggsn \
    osmo-hlr \
    osmo-iuh \
    osmo-mgw \
    osmo-msc \
    osmo-pcu \
    osmo-sgsn \
    osmo-trx \
    osmo-trx-uhd

I'm looking into EXTRA_DIST, and then I'll submit the first patches to gerrit. Once the patches are merged, and some more OBS configuration is done, we will have the packages available in the official Osmocom repositories.

#10 Updated by osmith 5 months ago

osmith wrote:

I'm looking into EXTRA_DIST, and then I'll submit the first patches to gerrit.

Since the debian dir is also not in EXTRA_DIST, I'll submit the patches first and then create a new patch for each repository where I add both the debian subdir and the spec file to EXTRA_DIST.

#12 Updated by osmith 5 months ago

  • % Done changed from 30 to 50

#13 Updated by laforge 5 months ago

On Fri, May 15, 2020 at 09:40:04AM +0000, osmith [REDMINE] wrote:

I'll disable the related blocks in the spec files with if else macros.

shouldn't we rather try to make sure the respective macro definitions are available
at build time when building our packages? I would hope there is some way how we can
teach OBS to add some macros to the RPM build process? If so, we could probably simply
import the macros and not have to modify the spec files at all (or add ifdefs, ...)

#14 Updated by osmith 5 months ago

laforge wrote:

On Fri, May 15, 2020 at 09:40:04AM +0000, osmith [REDMINE] wrote:

I'll disable the related blocks in the spec files with if else macros.

shouldn't we rather try to make sure the respective macro definitions are available
at build time when building our packages? I would hope there is some way how we can
teach OBS to add some macros to the RPM build process? If so, we could probably simply
import the macros and not have to modify the spec files at all (or add ifdefs, ...)

I have tried to do just that, by building systemd-rpm-macros from suse for centos8. But this creates a conflict, there is already a /usr/lib/rpm/macros.d/macros.systemd file in centos8. It turns out, that centos8 is packaging the upstream version of the file, while opensuse has a custom version with added macros that do not exist upstream (suse version).

When comparing both files, I found that suse is doing some sysv_migration related stuff, see for example %service_add_pre that does not exist upstream.

In other words, the systemd-rpm-macros dependency and these %service_add_pre etc. macros are all suse specific. It's probably possible to add a systemd-rpm-macros stub file that makes the packages work on centos8, but this feels like a bad workaround. I think the suse specific stuff should rather be marked as such with the appropriate %ifs.

This may sound like an unmaintainable ifdef mess, but actually it's still pretty maintainable. For osmo-bts, for example:

+%if 0%{?suse_version}
 BuildRequires:  systemd-rpm-macros
+%else
+BuildRequires:  systemd
+%endif
+%if 0%{?suse_version}
 %pre    %service_add_pre    osmo-bts-trx.service
 %post   %service_add_post   osmo-bts-trx.service
 %preun  %service_del_preun  osmo-bts-trx.service
 %pre    virtual %service_add_pre    osmo-bts-virtual.service
 %post   virtual %service_add_post   osmo-bts-virtual.service
 %preun  virtual %service_del_preun  osmo-bts-virtual.service
 %postun virtual %service_del_postun osmo-bts-virtual.service
+%endif

#15 Updated by osmith 5 months ago

  • % Done changed from 50 to 60
Patches updated:
  • spec file renamed to .spec.in
  • using VERSION and autotools to fill it in
  • various build related fixes (e.g. in libosmocore using a pkgconf() name instead of hardcoding a dependency package name, so it works with both suse and centos)
  • osmo-ci.git: scripts adjusted to .spec.in change
  • osmo-ci.git: new script added for linking required dependencies to suse packages
  • some cosmetic changes (consistent e.g. URL)

https://gerrit.osmocom.org/q/topic:rpm+status:open

I've verified, that these .spec files build locally and will push them to the home:osmith42 test repo now. These should be good for master, after these patches are merged we can enable building in the centos RPMs in the official repository.

(The EXTRA_DIST change is not done yet, as written earlier I plan to make an extra commit for each repo, which also adds the debian subdir.)

#16 Updated by osmith 5 months ago

  • Related to Feature #4563: install test for centos packages (like the debian install test) added

#17 Updated by osmith 5 months ago

  • Related to Feature #4564: Run ttcn-3 testsuite with centos packages added

#18 Updated by osmith 5 months ago

  • % Done changed from 60 to 90

The EXTRA_DIST change is not done yet, as written earlier I plan to make an extra commit for each repo, which also adds the debian subdir.

Done now, and patches are merged.

rpm packages are building successfully for centos in the official nightly repository:

For opensuse, libosmo-sccp is currently failing. But this should be easy to fix:

[   77s] libosmo-mtp-devel.aarch64: E: lto-no-text-in-archive (Badness: 10000) /usr/lib64/libmtp.a
[   77s] libosmo-sigtran-devel.aarch64: E: lto-no-text-in-archive (Badness: 10000) /usr/lib64/libosmo-sigtran.a
[   77s] libosmo-sccp-devel.aarch64: E: lto-no-text-in-archive (Badness: 10000) /usr/lib64/libsccp.a
[   77s] libosmo-xua-devel.aarch64: E: lto-no-text-in-archive (Badness: 10000) /usr/lib64/libxua.a
[   77s] This archive does not contain a non-empty .text section.  The archive was not
[   77s] created with -ffat-lto-objects option.

The latest repository is prepared, and all dependencies (such as uhd) are building already. The Osmocom git tags for stable don't have the .spec.in files yet, so they won't be built until the next release. Then the OBS packaging script will pick up the .spec.in files automatically and start building the Osmocom rpms for latest as well.

Follow up issues for testing the packages:
  • #4563: install test for centos packages (like the debian install test)
  • #4564: Run ttcn-3 testsuite with centos packages

#19 Updated by osmith 5 months ago

osmith wrote:

For opensuse, libosmo-sccp is currently failing. But this should be easy to fix:
[...]

https://gerrit.osmocom.org/c/libosmo-sccp/+/18481

#20 Updated by roox 5 months ago

osmith wrote:

osmith wrote:

For opensuse, libosmo-sccp is currently failing. But this should be easy to fix:
[...]

https://gerrit.osmocom.org/c/libosmo-sccp/+/18481

It's the only package with static libraries (.a files).
Packaging static libraries should be avoided on SUSE systems ...
https://en.opensuse.org/openSUSE:Shared_library_packaging_policy#Best_Practices

I was wondering why I haven't had this issues for the nightly builds and it urned out that I had a workaround (besides some other things that might not be interesting for upstream (like rename libmtp, use system talloc)) in the following patch:
https://build.opensuse.org/package/view_file/home:mnhauke:osmocom:nightly/libosmo-sccp/0001-build-fixes.patch?expand=1

#21 Updated by osmith 5 months ago

roox wrote:

osmith wrote:

osmith wrote:

For opensuse, libosmo-sccp is currently failing. But this should be easy to fix:
[...]

https://gerrit.osmocom.org/c/libosmo-sccp/+/18481

It's the only package with static libraries (.a files).
Packaging static libraries should be avoided on SUSE systems ...
https://en.opensuse.org/openSUSE:Shared_library_packaging_policy#Best_Practices

I was wondering why I haven't had this issues for the nightly builds and it urned out that I had a workaround (besides some other things that might not be interesting for upstream (like rename libmtp, use system talloc)) in the following patch:
https://build.opensuse.org/package/view_file/home:mnhauke:osmocom:nightly/libosmo-sccp/0001-build-fixes.patch?expand=1

I saw that workaround and looked into upstreaming the static -> dynamic library change into the source tree. However, libsccp, libmtp and libxua are built as static libraries on purpose according to https://osmocom.org/projects/libosmo-sccp/wiki, because they are legacy and should not be used by newer code. So I've removed your patch here: https://gerrit.osmocom.org/c/libosmo-sccp/+/18278

#22 Updated by osmith 5 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)