Project

General

Profile

Feature #2819

migrate to python3

Added by msuraev almost 2 years ago. Updated 2 days ago.

Status:
In Progress
Priority:
High
Assignee:
Target version:
-
Start date:
01/07/2018
Due date:
% Done:

90%

Spec Reference:

Description

Python 2 is scheduked for complete deprecation by upstream in 2 years - https://pythonclock.org/
We should migrate all remaining python 2 code to python 3.

The code in osmo-python-tests iscompatible with both python 2 and 3 as of 2f266e0cec10e32db8781a8fb788b0432c33d6d9.

This allows to migrate code which uses it (ctrl and vty tests) to python 3. Once there're no more python 2 code depending on osmo-python-tests, we can drop support for python 2 from osmo-python-tests.


Checklist

  • osmo-python-tests (and related changed in osmo-* repos using external tests)
  • osmo-gsm-manuals
  • docker image used with jenkins scripts (osmo-ci)

History

#1 Updated by laforge 5 months ago

  • Priority changed from Normal to Low

#2 Updated by laforge 13 days ago

  • Assignee set to osmith
  • Priority changed from Low to High

I belive most of this has been done meanwhile, but maybe a good idea to check the status and update this ticket, including a check list with what's still missing [if any]?

#3 Updated by osmith 10 days ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

This should be straight forward, it seems that all scripts are already python3 compatible. So we only need to drop the python2 support.

However, I'm having problems with the debian packaging; override_dh_auto_install in debian/control does not appear to be working. When building the package, it still calls setup.py with "python" instead of "python3".

#4 Updated by osmith 9 days ago

  • Checklist item osmo-python-tests added
  • Checklist item osmo-gsm-manuals added
  • % Done changed from 50 to 60

Patch submitted for osmo-python-tests: https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16500

osmo-gsm-manuals needs to be fixed too, it depends on python-pychart at least (instead of python3-pychart).

#5 Updated by osmith 9 days ago

osmo-gsm-manuals needs to be fixed too, it depends on python-pychart at least (instead of python3-pychart).

Turns out, that there is no python3-pychart package. Upstream of pychart seems to be dead, and nobody ported it to python3.

The good news is, that we are barely using it in the manuals. The only place I can find, where we use it, is in the osmux reference, the chart for "Evaluation: Expected traffic savings". Furthermore it seems that generating this chart was broken for some time - the pdf only shows the text "osmux-reference__8.png" where it should be.

So I will create an svg from the image once, and then drop the python2 dependency.

#6 Updated by pespin 9 days ago

Hi, with regards to osmux chart being broken see https://osmocom.org/issues/4193

#7 Updated by osmith 9 days ago

  • Checklist item docker image used with jenkins scripts (osmo-ci) added
  • % Done changed from 60 to 90

Hi, with regards to osmux chart being broken see https://osmocom.org/issues/4193

Thanks for the pointer! As side-effect, we can close this issue once the patches are merged.

Patches submitted:

https://gerrit.osmocom.org/q/topic:drop-py2

#8 Updated by osmith 8 days ago

  • Checklist item osmo-gsm-manuals set to Done

All patches were merged, but osmo-bsc master started failing.

I was under the impression, that all previous scripts in osmo-python-tests were already working with python 3. But as it turns out, this isn't true.

So I've started to write fixes for the python3 issues, and found that besides a few small fixes in osmo-python-tests, there is also a vty_test_runner.py file in various osmo-* repositories, that is still depending on python2. Clearly this isn't a tiny fix that could be fast-tracked through gerrit to make everything work again, but rather multiple small fixes that should be properly reviewed.

Therefore I've reverted the osmo-ci (docker) and osmo-python-tests patches, so master works again and I can finish the python2 removal next week.

https://gerrit.osmocom.org/c/osmo-ci/+/16513
https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16512

#9 Updated by osmith 8 days ago

  • % Done changed from 90 to 70

#10 Updated by osmith 5 days ago

I'm going to test all repositories, that have ./configure --enable-external-tests, with the python3 version, and fix the code as necessary where it breaks.

So far I've adjusted osmo-python-tests.git and osmo-bsc.git, so the python3 version of the tests for osmo-bsc.git are passing successfully.

WIP branches (of both repositories): osmith/fix-python3

#11 Updated by osmith 4 days ago

  • % Done changed from 70 to 80

I've fixed up and tested:

  • osmo-bsc
  • openbsc
  • libosmo-sccp
  • osmo-sgsn
  • osmo-msc
  • osmo-sip-connector
  • osmo-mgw
  • osmo-hlr
  • osmo-pcap (found that external tests are not running, see #4317)
Remaining repositories with --enable-external-tests (I will continue there tomorrow):
  • libosmocore

#12 Updated by osmith 3 days ago

  • % Done changed from 80 to 90

I've done more fixes, tested that everything builds on OBS (own namespace), found more bugs, fixed those, then submitted the patches.

Almost all patches depend on this osmo-python-tests change:
https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16540

All other patches should be merged quickly afterwards, as soon as the jenkins slaves have installed the new osmo-python-tests:
https://gerrit.osmocom.org/q/topic:drop-py2

Except for this osmo-ci patch, which will drop python2 from the docker image. This can be merged at a later point, and I can do some additional testing with repositories that are not using osmo-python-tests, as described in the gerrit comment. I've set the patch to WIP:
https://gerrit.osmocom.org/c/osmo-ci/+/16554

#13 Updated by osmith 2 days ago

  • Checklist item changed from osmo-python-tests to osmo-python-tests (and related changed in osmo-* repos using external tests)
  • Checklist item osmo-python-tests (and related changed in osmo-* repos using external tests) set to Done

Everything merged but the "drop python2" patch for osmo-ci.git. As discussed, I'll do this in the new year in case of potential fallout.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)