Project

General

Profile

Actions

Feature #3385

closed

Move project specific manuals from osmo-gsm-manuals to each respective git repository

Added by pespin over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
07/06/2018
Due date:
% Done:

100%

Spec Reference:

Description

Following subdirs in osmo-gsm-manuals.git should be moved under doc subdir of each respective git repository (osmo-bsc.git, osmo-bts.git, etc.)

OsmoBSC
OsmoBTS
OsmocomBB
OsmoGGSN
OsmoGSMTester
OsmoHLR
OsmoMGCP
OsmoMGW
OsmoMSC
OsmoNAT
OsmoNITB
OsmoPCU
OsmoSGSN
OsmoSTP
OsmoTRX

The idea is to keep the common parts and build related scripts (makefile, etc) in osmo-gsm-manuals.git, and then conditionally enable building them by using a configure flag (eg. --enable-man --with-man-path=/path/to/osmo-gsm-manuals.git/).

We can have most of the Makefile logic in a .mk file in osmo-gsm-manuals.git and include it in every repository, similar to what we do with osmo-release.mk from libosmocore.git:
In configure.ac:

dnl include release helper
RELMAKE='-include osmo-release.mk'
AC_SUBST([RELMAKE])

This way we can have user manual and VTY documentation up to date and sync with code, and force to have documentation changes together with code changes in the same gerrit patch before +2 it.

This will also allow to easily generate man pages from a specific .adoc file (which is also included in user manual).


Files

move-manuals.tar.gz move-manuals.tar.gz 1.26 KB osmith, 11/23/2018 03:23 PM

Checklist

  • add manuals with commit history to the project repos (run the attached script)
  • repos using autotools: integrate with configure.ac, toplevel Makefile.am etc.
  • repos not using autotools: rename Makefile.am to Makefile and adjust it slightly
  • configure jenkins to build all manuals in the gerrit build verification
  • configure jenkins to build and publish manuals when a commit is merged to master
  • delete project manuals from osmo-gsm-manuals.git
  • Osmocom Docker image: add dependencies for osmo-gsm-manuals
  • integrate manuals to osmocom-bb.git, openbsc.git, osmo-gsm-tester.git
  • contrib/jenkins.sh: build manuals and add --publish option
  • remove remaining project related files from osmo-gsm-manuals.git
  • jenkins job: master-builds.yml: run all project's contrib/jenkins.sh with --publish
  • build and publish manuals for repos without configure.ac in the root directory (osmocom-bb, openbsc, osmo-gsm-tester)
  • whitelist the jenkins bots SSH keys for uploading to the docs server
  • enable PUBLISH=1 again (temporarily disabled due to missing upload permissions)

Related issues

Related to OsmoBSC - Feature #3583: OsmoBSC manual: copy/move bts-examples chapter from OsmoNITBResolvedosmith09/20/2018

Actions
Related to Cellular Network Infrastructure - Bug #3725: Jenkins isn't using the credentials store for uploading to rita.osmocom.orgResolvedosmith12/12/2018

Actions
Related to Cellular Network Infrastructure - Bug #3726: Jenkins: build all Osmocom projects in docker, not only someResolvedosmith12/12/2018

Actions
Blocked by Cellular Network Infrastructure - Bug #3720: Jenkins can't upload to rita.osmocom.orgResolvedosmith12/06/2018

Actions
Precedes Cellular Network Infrastructure - Feature #3386: Generate man pages at build time from adoc filesNewosmith07/09/2018

Actions
Precedes Cellular Network Infrastructure - Feature #3695: generate VTY reference manuals from 'make' directlyRejectedosmith

Actions
Actions #1

Updated by pespin over 5 years ago

  • Precedes Feature #3386: Generate man pages at build time from adoc files added
Actions #2

Updated by neels over 5 years ago

  • Related to Feature #3583: OsmoBSC manual: copy/move bts-examples chapter from OsmoNITB added
Actions #3

Updated by osmith over 5 years ago

  • Status changed from New to In Progress
  • Assignee changed from 4368 to osmith

Assigning this to myself, because another issue that is assigned to me (#3386) depends on this one.

Actions #4

Updated by osmith over 5 years ago

The idea is to keep the common parts and build related scripts (makefile, etc) in osmo-gsm-manuals.git, and then conditionally enable building them by using a configure flag (eg. --enable-man --with-man-path=/path/to/osmo-gsm-manuals.git/).

pespin: I wonder if that works properly with packaging the man pages for Debian later on. We could also extend osmo-gsm-manuals.git with a "make install" target and write a .pc file, that points to the shared code (pkg-config allows custom variables). Then we could create a "osmo-gsm-manuals-dev" package as build time dependency for the other osmo packages. What do you think?

Actions #5

Updated by pespin over 5 years ago

Fine for me. Keep in mind that it may be interesting to have some Makefile installed by "make install" in osmo-gsm-manuals.git and reachable through pkg-config custom variable which can then be included as described in initial post. This way we avoid copy&paste same stuff on how to build adoc stuff into each project. Up to you to find best solution.

Actions #6

Updated by osmith over 5 years ago

I have researched how this issue relates to the Jenkins jobs.

Right now we have:
  • push to master:
    • osmo-gsm-manuals.git - jenkins job "master-osmo-gsm-manuals" (from master-builds.yml)
      • calls ./contrib/jenkins.sh --publish
      • builds the manuals for all projects
      • runs rsync to publish the "./out" folder
  • push to gerrit:
    • osmo-gsm-manuals.git - jenkins job "gerrit-osmo-gsm-manuals" (from gerrit-verifications.yml)
      • calls ./contrib/jenkins.sh
      • builds the manuals for all projects
After moving the manuals to the project folders, I think we need:
  • push to master:
    • osmo-gsm-manuals.git
      • build and publish the documentation of all projects
    • osmo-*.git
      • build and publish the documentation of that project
  • push to gerrit:
    • osmo-gsm-manuals.git
      • build the documentation of all projects
    • osmo-*.git
      • build the repo with its documentation
Actions #7

Updated by laforge over 5 years ago

On Tue, Oct 30, 2018 at 03:39:46PM +0000, osmith [REDMINE] wrote:

After moving the manuals to the project folders, I think we need:
  • push to master:
    • osmo-gsm-manuals.git
      • build and publish the documentation of all projects

that would be rather nice, but I would guess we could live without any
automatism here.

  • osmo-*.git
    • build and publish the documentation of that project

this part is mandatory.

I guess with the lines below you're referring to the gerrit build verification?

  • push to gerrit:
    • osmo-gsm-manuals.git
      • build the documentation of all projects
    • osmo-*.git
      • build the repo with its documentation

I guess in this case it would be sufficient to just verify for osmo-*.git when a commit happens
to that specific repo. I don't think it's strictly required to built-test osmo-*.git after any
change to osmo-gsm-manuals.git. I cannot really think of anything being committed to the 'common'
part that would break the builds right now (short of removing a file that others need), so we can
probably ignore that as a rather esoteric case.

In terms of the generated/published manuals, this new scheme woould also allow us
to build + publish manuals for specific versions, particularly the tagged versions.

So basically once we tag a given version of osmo-msc (e.g. 1.2.3) , it
should be possible to build the manuals for exactly that version of the
osmo-msc user manual. Ideally that would run automatically, but of
course it is also possible to trigger this manually.

One interesting questions is what kind of "common" part to use in that
case. Some pinned commit hash of osmo-gsm-manuals.git that's stored in osmo-msc.git
repository at the 1.2.3 tag? Or simply the masteer of
osmo-gsm-manuals.git?

Regards,
Harald

Actions #8

Updated by osmith over 5 years ago

  • % Done changed from 0 to 30

Thanks for the feedback!

Time for a status update. I've pushed my WIP code to osmith/move-manuals-to-project-repos branches in both osmo-msc.git and osmo-gsm-manuals.git. My plan is to make it work with MSC first (as proof of concept), then with all other projects.

  • moved osmo-gsm-manuals.git/OsmoMSC to osmo-msc.git/doc/man
  • osmo-gsm-manuals.git installs the common files to $prefix/share/osmo-gsm-manuals (e.g. /usr/local/share/osmo-gsm-manuals):
    $ autoreconf -fi
    $ ./configure
    $ make
    $ make install
    
  • a pkg-config file gets generated, with a custom "topdir" variable that points to $prefix/share/osmo-gsm-manuals
  • the new topdir variable is queried from pkg-config and used by the project's manual build Makefile.am instead of the old "TOPDIR := .."
  • "../build", "../common" was hardcoded in a lot of places, this was changed to use that variable where possible (asciidoc), or adjusted to use a symlink to $topdir/common (doxygen)
  • osmo-msc.git's pdf is building now (needs more polishing though, images are probably missing etc.)
  • osmo-msc.git builds the pdf only when --enable-man is specified at the configure line
  • the check for installed dependencies was moved from osmo-gsm-manuals.git's global Makefile to "osmo-gsm-manuals-check-depends" installed to $prefix/bin. It gets called inside osmo-msc.git's configure when --enable-man is set.
  • I have moved libosmocore.git/doc/vty/merge_doc.xsl to osmo-gsm-manuals.git/merge_doc.xsl. It was not used anywhere in libosmocore, but it is needed for building the manuals.

Also I've noticed that GIT_VERSION and GIT_DATE are problematic here:
https://git.osmocom.org/osmo-gsm-manuals/tree/build/Makefile.asciidoc.inc

GIT_VERSION := $(shell git describe --abbrev=4 --dirty --always --tags)
GIT_DATE := $(shell $(TOPDIR)/build/unix-time-to-fmt.py `git log -n 1 "--pretty=%at" ../.`)

This won't work if ./configure is running in a source folder extracted from a release tarball.

Actions #9

Updated by osmith over 5 years ago

  • % Done changed from 30 to 50
update:
  • all common pages from osmo-gsm-manuals.git can be built as standalone test now (to see if there are any syntax errors etc. in the common pages, and to see if the common build scripts are working)
  • made --enable-man work with out-of-tree builds
  • unix-time-to-fmt.py: use "unknown" if we're not in a git directory
  • both pdfs for msc get generated now, though there's content missing in the vty reference
Actions #10

Updated by osmith over 5 years ago

  • % Done changed from 50 to 60
update:
  • fixed missing pages in the vty-reference pdfs
  • make checks from asciidoc Makefile work again
  • update Makefile.*.inc text comments
  • remove --publish from contrib/jenkins.sh, add a "publish" target to the common Makefile that gets included in each project

And I've thought about creating a "osmo-gsm-manuals-dev" Debian package as part of this issue. But there doesn't seem to be any benefit to that until we can build the manuals as man pages (#3386). So instead of blowing up the patchset further with a new "debian"-folder, let's do that as part of the man pages issue.

I'll clean up the code, then submit the patches for review on Monday :)

Actions #11

Updated by osmith over 5 years ago

  • % Done changed from 60 to 70
Patches submitted:

The manuals are only added to MSC so far. When this is reviewed, I can add them to the other project repositories as well.

Actions #12

Updated by neels over 5 years ago

  • Precedes Feature #3695: generate VTY reference manuals from 'make' directly added
Actions #13

Updated by osmith over 5 years ago

  • % Done changed from 70 to 80

One patch in the patchset ("build manuals from the project repositories") did too many things at once:
https://gerrit.osmocom.org/#/c/osmo-gsm-manuals/+/11725/

I have split it up in multiple commits, and refactored the code so all the existing, project specific manuals build with autotools with the current folder structure before we finally move them. That will make the moving a lot easier, because we already know that all of them build. And it makes it easier to see that the changes don't break anything.

New patchset coming in tomorrow, my current code (that still needs clean up before pushing to gerrit) is in the "osmith/move-manuals-to-project-repos" branches in osmo-msc.git and osmo-gsm-manuals.git.

Actions #14

Updated by osmith over 5 years ago

The ~20 patches that change the buildsystem to autoconf/automake to get "make install" working have been merged (also with "make distcheck" support for consistency with the other Osmocom projects, so "make distcheck" will still work with ./configure --enable-manuals).

Jenkins tried to run ./jenkins.sh --publish and failed, I've patched the job so it doesn't do that anymore:
https://gerrit.osmocom.org/#/c/osmo-ci/+/11863/

Actions #15

Updated by osmith over 5 years ago

  • Checklist item add manuals with commit history to the project repos (run the attached script) added
  • Checklist item repos using autotools: integrate with configure.ac, toplevel Makefile.am etc. added
  • Checklist item repos not using autotools: rename Makefile.am to Makefile and adjust it slightly added
  • Checklist item configure jenkins to build all manuals in the gerrit build verification added
  • Checklist item configure jenkins to build and publish manuals when a commit is merged to master added
  • File move-manuals.tar.gz move-manuals.tar.gz added

neels: you proposed in here that we keep the git history when moving the files, bypassing gerrit. I've attached a script that does this. Please read it once, edit the commit message if you like, and if you think it is fine, would you like to run it and push the changes to master of each repository? On my laptop, this takes a bit more than four minutes to move all manuals with the history.

The checklist items added are the last missing pieces to finally finishing this up.

Actions #16

Updated by osmith over 5 years ago

  • Checklist item delete project manuals from osmo-gsm-manuals.git added
Actions #17

Updated by osmith over 5 years ago

In argreement with Neels, I have executed the script and pushed the commits to an extra branch called osmith/move-manuals-with-history:

Left: dir name in osmo-gsm-manuals.git
Right: project git repo name

OsmoBSC => osmo-bsc
OsmoBTS => osmo-bts
OsmoGGSN => osmo-ggsn
OsmoGSMTester => osmo-gsm-tester
OsmoHLR => osmo-hlr
OsmoMGW => osmo-mgw
OsmoMSC => osmo-msc
OsmoNITB => openbsc
OsmoPCU => osmo-pcu
OsmoSGSN => osmo-sgsn
OsmoSIPConnector => osmo-sip-connector
OsmoSTP => libosmo-sccp
OsmoTRX => osmo-trx
OsmocomBB => osmocom-bb

These dirs have not been moved, as they are obsolete. They only contained the vty reference, which could be easily restored anyway.

OsmoMGCP: NOT MOVING
OsmoNAT: NOT MOVING

Actions #18

Updated by osmith over 5 years ago

  • Checklist item add manuals with commit history to the project repos (run the attached script) set to Done
  • Checklist item repos using autotools: integrate with configure.ac, toplevel Makefile.am etc. set to Done
  • Checklist item repos not using autotools: rename Makefile.am to Makefile and adjust it slightly set to Done
Actions #19

Updated by osmith over 5 years ago

neels: thanks for applying the patch to these repositories:

OsmoBSC => osmo-bsc
OsmoBTS => osmo-bts
OsmoGGSN => osmo-ggsn
OsmoHLR => osmo-hlr
OsmoMGW => osmo-mgw
OsmoMSC => osmo-msc
OsmoPCU => osmo-pcu
OsmoSGSN => osmo-sgsn
OsmoSIPConnector => osmo-sip-connector
OsmoSTP => libosmo-sccp
OsmoTRX => osmo-trx

These following are still missing. They don't have a configure.ac in the top level, but the manuals can be built with cd doc/manuals; make as the commit message in their osmith/move-manuals-with-history branches says. Can you also merge those?

OsmoNITB => openbsc
OsmocomBB => osmocom-bb
OsmoGSMTester => osmo-gsm-tester

Actions #20

Updated by osmith over 5 years ago

  • Checklist item install osmo-gsm-manuals dependencies on all relevant buildbots added

Patches for deletion of the moved manual dirs have been submitted:
https://gerrit.osmocom.org/#/q/topic:move-manuals+status:open

Regarding build of manuals in CI, it seems that the buildbots, on which the project related builds run, do not have all dependencies installed:

https://jenkins.osmocom.org/jenkins/job/gerrit-osmo-bsc/a1=default,a2=default,a3=default,a4=default,label=osmocom-gerrit-debian9/1877/console

Binary 'xsltproc' not found in path, please install libxslt.
configure: error: "missing dependencies!" 

EDIT: the buildbots have all dependencies, but some jobs are running in Docker containers, and these don't have the dependencies. I can patch this though. Here's some documentation of how it all fits together: https://osmocom.org/projects/osmocom-servers/wiki/Jenkins_build_verification_jobs

Actions #21

Updated by osmith over 5 years ago

  • Checklist item delete project manuals from osmo-gsm-manuals.git set to Done
Actions #22

Updated by neels over 5 years ago

osmith wrote:

These following are still missing.

ah thanks, completed now

Actions #23

Updated by osmith over 5 years ago

  • Checklist item deleted (install osmo-gsm-manuals dependencies on all relevant buildbots)
  • Checklist item Osmocom Docker image: add dependencies for osmo-gsm-manuals added
  • Checklist item integrate manuals to osmocom-bb.git, openbsc.git, osmo-gsm-tester.git added
  • Checklist item contrib/jenkins.sh: build manuals and add --publish option added
  • Checklist item remove remaining project related files from osmo-gsm-manuals.git added
  • Checklist item jenkins job: master-builds.yml: run all project's contrib/jenkins.sh with --publish added
  • % Done changed from 80 to 90

Getting closer to the finish line, new patches up for review.

https://gerrit.osmocom.org/#/q/topic:move-manuals+status:open

Actions #24

Updated by osmith over 5 years ago

  • Checklist item Osmocom Docker image: add dependencies for osmo-gsm-manuals set to Done
  • Checklist item remove remaining project related files from osmo-gsm-manuals.git set to Done
Actions #25

Updated by osmith over 5 years ago

  • Checklist item build and publish manuals for repos without configure.ac in the root directory (osmocom-bb, openbsc, osmo-gsm-tester) added
  • Checklist item whitelist the jenkins bots SSH keys for uploading to the docs server added
  • Checklist item enable PUBLISH=1 again (temporarily disabled due to missing upload permissions) added
  • Checklist item configure jenkins to build all manuals in the gerrit build verification set to Done
  • Checklist item configure jenkins to build and publish manuals when a commit is merged to master set to Done
  • Checklist item integrate manuals to osmocom-bb.git, openbsc.git, osmo-gsm-tester.git set to Done
  • Checklist item contrib/jenkins.sh: build manuals and add --publish option set to Done
  • Checklist item jenkins job: master-builds.yml: run all project's contrib/jenkins.sh with --publish set to Done
Update: I've noticed several other things and fixed them along the way:
  • osmo-mgw was built twice, with and without transcoding support, although it doesn't support that (leftover from old code)
  • docker arguments were formatted differently in gerrit-verifications.yml and master-builds.yml, which made it harder to check both files side-by-side
  • redundant environment variable specifications were confusing
  • openbsc.git's manuals had wrong names (e.g. osmo-nat instead of osmo-bsc-nat)
  • wrong distcheck variable was used in Makefile.am, so it could not be overruled in contrib/jenkins.sh

The patches for extending the contrib/jenkins.sh files went through several iterations (in which some of the points above came up), and are finally in master of the respective Osmocom git repositories. The Jenkins jobs are adjusted to pass environment variables for building and publishing manuals. I've updated the TODO list.

Actions #26

Updated by laforge over 5 years ago

  • Blocked by Bug #3720: Jenkins can't upload to rita.osmocom.org added
Actions #27

Updated by osmith over 5 years ago

  • Checklist item whitelist the jenkins bots SSH keys for uploading to the docs server set to Done
  • Checklist item enable PUBLISH=1 again (temporarily disabled due to missing upload permissions) set to Done
Actions #28

Updated by osmith over 5 years ago

Actions #29

Updated by osmith over 5 years ago

  • Related to Bug #3725: Jenkins isn't using the credentials store for uploading to rita.osmocom.org added
Actions #30

Updated by osmith over 5 years ago

  • Related to Bug #3726: Jenkins: build all Osmocom projects in docker, not only some added
Actions #31

Updated by osmith over 5 years ago

  • Checklist item build and publish manuals for repos without configure.ac in the root directory (osmocom-bb, openbsc, osmo-gsm-tester) set to Done
Actions #32

Updated by osmith over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

Finished \o/

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)