Project

General

Profile

Feature #4345

build verification / gerrit for erlang projects

Added by laforge 22 days ago. Updated 10 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
jenkins
Target version:
-
Start date:
01/06/2020
Due date:
% Done:

100%

Spec Reference:

Description

We do have a number of Erlang projects in Osmocom. Some mostly historical (osmo_ss7, mgw_nat, ...) but also some current ones (osmo_gsup, osmo_dia2gsup).

We'd like to use the typical "gerrit" workflow for merging changes to those projects. That means we also need to have build verification to get the +V flag in gerrit.

I've created a docker image in https://gerrit.osmocom.org/c/docker-playground/+/16737 which has what it takes to do said build verification.

Unfortuantely it seems that our jenkins-jobs templates for build verification seem to be very much oriented towards the idea that all of those jobs run in one image only. Even indenepdent of this task, I would think it makes sense to extend this to permit executing on different images, as we may e.g. want to build on different distributions/versions at some point in the future.

the typical workflow for build verification would look like this:
  • checkout a given osmocom erlang project (e.g. osmo_gsup.git)
  • run rebar3 compile to build
  • run rebar3 eunit to execute unit tests

If those pass without any error exit code, the build passes.

There is no need for manually handling dependencies, as rebar3 will take care of recursively downloading + building any dependencies.


Checklist

  • get docker-playground patches merged
  • get osmo-ci patches merged
  • submit erlang patches
  • get erlang patches merged
  • add repos to gerrit, with working sync

History

#1 Updated by osmith 21 days ago

I've created a docker image in https://gerrit.osmocom.org/c/docker-playground/+/16737 which has what it takes to do said build verification.
Unfortuantely it seems that our jenkins-jobs templates for build verification seem to be very much oriented towards the idea that all of those jobs run in one image only. Even indenepdent of this task, I would think it makes sense to extend this to permit executing on different images, as we may e.g. want to build on different distributions/versions at some point in the future.

Currently all build jobs are using the "osmocom:deb9_amd64" image, and its Dockerfile is located in osmo-ci.git (in contrary to all other Dockerfiles, that we have in docker-playground.git).
It gets rebuilt by the update-osmo-ci-on-slaves job, which is triggered by changes in osmo-ci.git and calls docker/rebuild_osocom_jenkins_images.sh in osmo-ci.git.

To make multiple images available for gerrit verification, I suggest:
  • trigger the update-osmo-ci-on-slaves on changes in docker-playground.git, too (not just osmo-ci.git)
  • move "osmocom:deb9_amd64" to docker-playground.git (and possibly give it a more descriptive name?)
  • rebuild_osocom_jenkins_images.sh: have a list of relevant images, and build all of them from docker-playground.git
Suggestion for using the image in gerrit-verifications.yml:
  • below existing variables docker_run, docker_img, cmd, add new variables: docker_run_erlang, docker_img_erlang, cmd_erlang
  • repositories that don't use the default image would have "cmd: {cmd_erlang}" as attribute

run rebar3 compile to build
run rebar3 eunit to execute unit tests

How about we keep using contrib/jenkins.sh in each repository to define the exact build verification steps, holding these two commands in this case? Then it could be extended per repository as necessary.

laforge, what do you think about this proposal? If you like it, I'd be happy to go ahead with implementing it like this.

#2 Updated by laforge 21 days ago

On Tue, Jan 07, 2020 at 10:58:06AM +0000, wrote:

laforge, what do you think about this proposal? If you like it, I'd be happy to go ahead with implementing it like this.

Full ACK!

#3 Updated by osmith 14 days ago

  • Checklist item get docker-playground patches merged added
  • Checklist item get osmo-ci patches merged added
  • Checklist item submit erlang patches added
  • Checklist item get erlang patches merged added
  • Status changed from New to In Progress
  • % Done changed from 0 to 80

Patches submitted: https://gerrit.osmocom.org/q/topic:gerrit-erlang

I have patches ready for erlang repositories, that were recently updated (osmo_gsup, osmo_dia2gsup, osmo_ss7), to add a contrib/jenkins.sh file there. I will push these patches to gerrit, once the docker-playground and osmo-ci patches are merged, so we can see that gerrit verification works as it should.

#4 Updated by osmith 13 days ago

  • Checklist item get docker-playground patches merged set to Done
  • Checklist item get osmo-ci patches merged set to Done

#5 Updated by osmith 12 days ago

  • Checklist item add repos to gerrit, with working sync added

The patches to add jenkins.sh look like this:

--- /dev/null
+++ b/contrib/jenkins.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -ex
+
+rebar3 compile
+rebar3 eunit

However, the repositories did not exist in gerrit yet.

I've added them as empty repositories here:
https://gerrit.osmocom.org/admin/repos/q/filter:erlang

According to the wiki, it should sync the repositories automatically:
https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit#Adding-a-new-repository

I'll check back later if that worked.

#6 Updated by laforge 12 days ago

On Thu, Jan 16, 2020 at 02:39:53PM +0000, osmith [REDMINE] wrote:

However, the repositories did not exist in gerrit yet.

I've added them as empty repositories here:
https://gerrit.osmocom.org/admin/repos/q/filter:erlang

Unfortunately the naming of one repo is wrong: erlang/osmo_ss7 vs.
erlang/ss7. I also couldn't find a way to use the delete-project
plugin, despite it being installed and "running" :/

We will need erlang/{mgw_nat,osmo_cap,osmo_diameter2gsup,osmo_gsup,osmo_map,osmo_sccp}

According to the wiki, it should sync the repositories automatically:
https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit#Adding-a-new-repository

The sync works from gerrit to git.osmocom.org and github, but not the
other way around.

One needs to explicitly add gerrit as a second remote and push all branches/tags from git.osmocom.org
to gerrit.

#7 Updated by osmith 11 days ago

laforge wrote:

On Thu, Jan 16, 2020 at 02:39:53PM +0000, osmith [REDMINE] wrote:

However, the repositories did not exist in gerrit yet.

I've added them as empty repositories here:
https://gerrit.osmocom.org/admin/repos/q/filter:erlang

Unfortunately the naming of one repo is wrong: erlang/osmo_ss7 vs.
erlang/ss7. I also couldn't find a way to use the delete-project
plugin, despite it being installed and "running" :/

Looks like you found it, the repository is gone again :)

(for future reference: browse -> repositories -> $name -> commands -> delete project)

We will need erlang/{mgw_nat,osmo_cap,osmo_diameter2gsup,osmo_gsup,osmo_map,osmo_sccp}

created:
  • erlang/mgw_nat
  • erlang/osmo_dia2gsup
  • erlang/osmo_gsup
  • erlang/osmo_map
  • erlang/osmo_sccp
  • erlang/osmo_ss7

I could not find "erlang/osmo_cap" on git.osmocom.org.

https://git.osmocom.org/?q=erlang

Do you mean "erlang/signerl"?

According to the wiki, it should sync the repositories automatically:
https://osmocom.org/projects/cellular-infrastructure/wiki/Gerrit#Adding-a-new-repository

The sync works from gerrit to git.osmocom.org and github, but not the
other way around.

One needs to explicitly add gerrit as a second remote and push all branches/tags from git.osmocom.org
to gerrit.

Thanks, wiki page adjusted and manually pushed all repos listed above to gerrit.

#8 Updated by osmith 11 days ago

  • Checklist item submit erlang patches set to Done
  • Checklist item add repos to gerrit, with working sync set to Done

#9 Updated by osmith 11 days ago

  • % Done changed from 80 to 90

Pushed new patches:
https://gerrit.osmocom.org/q/topic:gerrit-erlang

They make gerrit verification work for:
  • erlang/osmo_gsup
  • erlang/osmo_dia2gsup
  • erlang/osmo_ss7

These are the repositories that were most recently updated. The steps for building the older erlang repositories seem to be more complicated (I've looked at mgw_nat's INSTALL, and it involves patching rebar?) - so I suggest that the respective contrib/jenkins.sh and gerrit-verifications.yml entries get added when there are new patches coming in. It should be trivial to add the gerrit-verifications.yml entry now.

#10 Updated by osmith 11 days ago

  • Checklist item get erlang patches merged set to Done
  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

#11 Updated by fixeria 10 days ago

Nice work! Dangschee, osmith!

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)