Project

General

Profile

Feature #2819

migrate to python3

Added by msuraev over 3 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
01/07/2018
Due date:
% Done:

100%

Spec Reference:

Description

Python 2 is scheduked for complete deprecation by upstream in 2 years - https://pythonclock.org/
We should migrate all remaining python 2 code to python 3.

The code in osmo-python-tests iscompatible with both python 2 and 3 as of 2f266e0cec10e32db8781a8fb788b0432c33d6d9.

This allows to migrate code which uses it (ctrl and vty tests) to python 3. Once there're no more python 2 code depending on osmo-python-tests, we can drop support for python 2 from osmo-python-tests.


Checklist

  • osmo-python-tests (and related changed in osmo-* repos using external tests)
  • osmo-gsm-manuals
  • docker image used with jenkins scripts (osmo-ci)

Associated revisions

Revision 76c6c504 (diff)
Added by max about 3 years ago

Use python 3 for utilities

There're no python2-specific code in there so we can switch right away
without waiting till 2020 for python 2 deprecation.

Related: OS#2819
Change-Id: I8d34aed124b00c5dd2ab1bcc84bbfa8c620282cc

Revision 0999d417 (diff)
Added by osmith over 1 year ago

Drop python2 and pychart dependencies

Python 2 is EOL at the end of 2019, so don't depend on it anymore.
Remove pychart support, because upstream is dead and there is no python3
version. We were only using it for one graph, and that has been
replaced in osmo-bsc I36b721f895caee9766528e14d854b6aa2a2fac85.

Depends: osmo-bsc I36b721f895caee9766528e14d854b6aa2a2fac85
Related: OS#2819
Change-Id: I57522e57e39682d6d123af4584785d327361e5a3

Revision f659f5e5 (diff)
Added by osmith over 1 year ago

doc: osmux: use generated svg instead of pychart

Replace python 2 code using pychart to draw a graph in
osmux-reference.adoc with the generated svg file. The upstream of
pychart is dead, there is no python 3 version, and python 2 is EOL at
the end of 2019.

This is the only time we ever made use of pychart in osmo-gsm-manuals,
so with this change, we can just drop the dependency.

I've generated the chart by saving the python code in chart.py, then:
$ ./chart.py --format=svg --font-size=3 > chart.svg

Related: OS#2819, OS#4193
Depends: osmo-ci I754b133d77743582bd84c33c74ecc9eb9ca4c0ef
Change-Id: I36b721f895caee9766528e14d854b6aa2a2fac85

Revision dc8451de (diff)
Added by osmith over 1 year ago

osmoappdesc.py, tests: switch to python 3

Make build and external tests work with python3, so we can drop
the python2 dependency.

This should be merged shortly after osmo-python-tests was migrated to
python3, and the jenkins build slaves were (automatically) updated to
have the new osmo-python-tests installed.

Related: OS#2819
Depends: osmo-python-tests I3ffc3519bf6c22536a49dad7a966188ddad351a7
Change-Id: I344c49001fba23bdcfdef06ab174c52b60edd01c

Revision e1f0844a (diff)
Added by osmith over 1 year ago

debian, osmoappdesc.py, tests: switch to python 3

Make build and external tests work with python3, so we can drop
the python2 dependency.

This should be merged shortly after osmo-python-tests was migrated to
python3, and the jenkins build slaves were (automatically) updated to
have the new osmo-python-tests installed.

Related: OS#2819
Depends: osmo-python-tests I3ffc3519bf6c22536a49dad7a966188ddad351a7
Change-Id: I438ca0c4b8e7957d0f347a5b2f5c4cb93f9325e6

Revision 14ec634a (diff)
Added by osmith over 1 year ago

osmoappdesc.py, tests: switch to python 3

Make build and external tests work with python3, so we can drop
the python2 dependency.

This should be merged shortly after osmo-python-tests was migrated to
python3, and the jenkins build slaves were (automatically) updated to
have the new osmo-python-tests installed.

Related: OS#2819
Depends: osmo-python-tests I3ffc3519bf6c22536a49dad7a966188ddad351a7
Change-Id: I8c07d99c1bc9f0383e4bce17544e0998998cc54d

Revision 16441ffb (diff)
Added by osmith over 1 year ago

debian, asn1tostruct.py: switch to python3

Make build work with python3 and drop the python2 dependency.

Related: OS#2819
Change-Id: Idb9d7a349ddb7619c784823f740648c7956bae5e

Revision a8a646af (diff)
Added by osmith over 1 year ago

osmoappdesc.py, tests: switch to python 3

Make build and external tests work with python3, so we can drop
the python2 dependency.

This should be merged shortly after osmo-python-tests was migrated to
python3, and the jenkins build slaves were (automatically) updated to
have the new osmo-python-tests installed.

Related: OS#2819
Depends: osmo-python-tests I3ffc3519bf6c22536a49dad7a966188ddad351a7
Change-Id: I344c49001fba23bdcfdef06ab174c52b60edd01c

Revision a47d37c9 (diff)
Added by osmith over 1 year ago

debian, utils: switch to python 3

Make build and external tests work with python3, so we can drop
the python2 dependency.

This should be merged shortly after osmo-python-tests was migrated to
python3, and the jenkins build slaves were (automatically) updated to
have the new osmo-python-tests installed.

Related: OS#2819
Depends: osmo-python-tests I3ffc3519bf6c22536a49dad7a966188ddad351a7
Change-Id: I84ef43f700e125c7a65f92347f12844e07e65655

History

#1 Updated by laforge almost 2 years ago

  • Priority changed from Normal to Low

#2 Updated by laforge over 1 year ago

  • Assignee set to osmith
  • Priority changed from Low to High

I belive most of this has been done meanwhile, but maybe a good idea to check the status and update this ticket, including a check list with what's still missing [if any]?

#3 Updated by osmith over 1 year ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

This should be straight forward, it seems that all scripts are already python3 compatible. So we only need to drop the python2 support.

However, I'm having problems with the debian packaging; override_dh_auto_install in debian/control does not appear to be working. When building the package, it still calls setup.py with "python" instead of "python3".

#4 Updated by osmith over 1 year ago

  • Checklist item osmo-python-tests added
  • Checklist item osmo-gsm-manuals added
  • % Done changed from 50 to 60

Patch submitted for osmo-python-tests: https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16500

osmo-gsm-manuals needs to be fixed too, it depends on python-pychart at least (instead of python3-pychart).

#5 Updated by osmith over 1 year ago

osmo-gsm-manuals needs to be fixed too, it depends on python-pychart at least (instead of python3-pychart).

Turns out, that there is no python3-pychart package. Upstream of pychart seems to be dead, and nobody ported it to python3.

The good news is, that we are barely using it in the manuals. The only place I can find, where we use it, is in the osmux reference, the chart for "Evaluation: Expected traffic savings". Furthermore it seems that generating this chart was broken for some time - the pdf only shows the text "osmux-reference__8.png" where it should be.

So I will create an svg from the image once, and then drop the python2 dependency.

#6 Updated by pespin over 1 year ago

Hi, with regards to osmux chart being broken see https://osmocom.org/issues/4193

#7 Updated by osmith over 1 year ago

  • Checklist item docker image used with jenkins scripts (osmo-ci) added
  • % Done changed from 60 to 90

Hi, with regards to osmux chart being broken see https://osmocom.org/issues/4193

Thanks for the pointer! As side-effect, we can close this issue once the patches are merged.

Patches submitted:

https://gerrit.osmocom.org/q/topic:drop-py2

#8 Updated by osmith over 1 year ago

  • Checklist item osmo-gsm-manuals set to Done

All patches were merged, but osmo-bsc master started failing.

I was under the impression, that all previous scripts in osmo-python-tests were already working with python 3. But as it turns out, this isn't true.

So I've started to write fixes for the python3 issues, and found that besides a few small fixes in osmo-python-tests, there is also a vty_test_runner.py file in various osmo-* repositories, that is still depending on python2. Clearly this isn't a tiny fix that could be fast-tracked through gerrit to make everything work again, but rather multiple small fixes that should be properly reviewed.

Therefore I've reverted the osmo-ci (docker) and osmo-python-tests patches, so master works again and I can finish the python2 removal next week.

https://gerrit.osmocom.org/c/osmo-ci/+/16513
https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16512

#9 Updated by osmith over 1 year ago

  • % Done changed from 90 to 70

#10 Updated by osmith over 1 year ago

I'm going to test all repositories, that have ./configure --enable-external-tests, with the python3 version, and fix the code as necessary where it breaks.

So far I've adjusted osmo-python-tests.git and osmo-bsc.git, so the python3 version of the tests for osmo-bsc.git are passing successfully.

WIP branches (of both repositories): osmith/fix-python3

#11 Updated by osmith over 1 year ago

  • % Done changed from 70 to 80

I've fixed up and tested:

  • osmo-bsc
  • openbsc
  • libosmo-sccp
  • osmo-sgsn
  • osmo-msc
  • osmo-sip-connector
  • osmo-mgw
  • osmo-hlr
  • osmo-pcap (found that external tests are not running, see #4317)
Remaining repositories with --enable-external-tests (I will continue there tomorrow):
  • libosmocore

#12 Updated by osmith over 1 year ago

  • % Done changed from 80 to 90

I've done more fixes, tested that everything builds on OBS (own namespace), found more bugs, fixed those, then submitted the patches.

Almost all patches depend on this osmo-python-tests change:
https://gerrit.osmocom.org/c/python/osmo-python-tests/+/16540

All other patches should be merged quickly afterwards, as soon as the jenkins slaves have installed the new osmo-python-tests:
https://gerrit.osmocom.org/q/topic:drop-py2

Except for this osmo-ci patch, which will drop python2 from the docker image. This can be merged at a later point, and I can do some additional testing with repositories that are not using osmo-python-tests, as described in the gerrit comment. I've set the patch to WIP:
https://gerrit.osmocom.org/c/osmo-ci/+/16554

#13 Updated by osmith over 1 year ago

  • Checklist item changed from osmo-python-tests to osmo-python-tests (and related changed in osmo-* repos using external tests)
  • Checklist item osmo-python-tests (and related changed in osmo-* repos using external tests) set to Done

Everything merged but the "drop python2" patch for osmo-ci.git. As discussed, I'll do this in the new year in case of potential fallout.

#14 Updated by osmith over 1 year ago

  • Checklist item docker image used with jenkins scripts (osmo-ci) set to Done
  • % Done changed from 90 to 100

#15 Updated by osmith over 1 year ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)