Project

General

Profile

Actions

Bug #5099

closed

old docker images on build slaves

Added by laforge almost 3 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
03/31/2021
Due date:
% Done:

100%

Spec Reference:

Description

I just saw this on the gtp0-deb9build slave:

root@deb9build:/# docker image ls
REPOSITORY                                                TAG                 IMAGE ID            CREATED             SIZE
osmocom-build/osmo-sgsn-latest                            latest              8c7f026f5ad0        23 minutes ago      134MB
osmocom-build/osmo-pcu-latest-centos8                     latest              3ab91889ca69        29 minutes ago      316MB
osmocom-build/ttcn3-sgsn-test                             latest              1ff4ec8cbf8c        45 minutes ago      1.65GB
osmocom-build/osmo-sgsn-latest-centos8                    latest              17475bf527e4        47 minutes ago      319MB
osmocom-build/ttcn3-pcu-test                              latest              652075108812        59 minutes ago      1.2GB
osmocom-build/osmo-pcu-master                             latest              bf3fadfe9fb7        About an hour ago   1.02GB
osmocom-build/debian-stretch-build                        latest              91cf921f318e        About an hour ago   977MB
osmocom-build/osmo-msc-latest                             latest              6c48b7fe8eda        About an hour ago   134MB
osmocom-build/osmo-stp-latest                             latest              fcc427824a0d        About an hour ago   131MB
osmocom-build/debian-stretch-obs-latest                   latest              453f3b17ddbe        About an hour ago   124MB
osmocom-build/ttcn3-msc-test                              latest              fa11be13057b        About an hour ago   1.74GB
osmocom-build/osmo-msc-latest-centos8                     latest              4799e37ad5a2        2 hours ago         320MB
osmocom-build/osmo-stp-latest-centos8                     latest              fd3d1cead57c        2 hours ago         316MB
osmocom-build/ttcn3-ggsn-test                             latest              54e7e4aef0e0        2 hours ago         978MB
osmocom-build/osmo-ggsn-latest-centos8                    latest              79b357f5f86d        2 hours ago         316MB
osmocom-build/centos8-obs-latest                          latest              11f48d65f824        2 hours ago         276MB
osmocom-build/debian-stretch-titan                        latest              abba1aa5ff27        6 hours ago         858MB
registry.osmocom.org/osmocom-build/debian-stretch-titan   latest              abba1aa5ff27        6 hours ago         858MB
osmocom-build/osmo-gsm-tester                             latest              c30e4e596697        8 hours ago         3.99GB
osmocom-build/debian-buster-erlang                        latest              1756fe124d7f        8 hours ago         1.15GB
debian                                                    stretch             8dce81298a14        9 hours ago         101MB
registry.osmocom.org/debian                               stretch             8dce81298a14        9 hours ago         101MB
debian                                                    sid                 e3cf4dcf91a6        9 hours ago         124MB
registry.osmocom.org/debian                               sid                 e3cf4dcf91a6        9 hours ago         124MB
debian                                                    buster              109b803c6a30        9 hours ago         114MB
registry.osmocom.org/debian                               buster              109b803c6a30        9 hours ago         114MB
osmocom-build/debian-buster-jenkins                       latest              adf00923ab6e        9 hours ago         3.48GB
osmocom-build/debian-stretch-jenkins                      latest              ef1a11362fd8        11 hours ago        2.31GB
osmocom-build/ttcn3-ns-test                               latest              0ff9ff9ebf10        12 hours ago        1.2GB
osmocom-build/osmo-ns-master                              latest              20a63d9c4ad0        12 hours ago        1.05GB
osmocom-build/ttcn3-cbc-test                              latest              461099dfb0a5        20 hours ago        1GB
osmocom-build/ttcn3-gbproxy-test                          latest              73940bd1a3f1        20 hours ago        1.33GB
osmocom-build/ttcn3-smlc-test                             latest              36eb5c383d31        20 hours ago        1.28GB
osmocom-build/ttcn3-remsim-test                           latest              a13283bbe3ef        21 hours ago        905MB
osmocom-build/ttcn3-sip-test                              latest              1b7f2812b31e        23 hours ago        967MB
osmocom-build/osmo-sip-master                             latest              ccfc46d03546        23 hours ago        988MB
osmocom-build/ttcn3-hlr-test                              latest              a83463032594        24 hours ago        1.21GB
osmocom-build/osmo-hlr-master                             latest              dc701a212f65        24 hours ago        1GB
osmocom-build/ttcn3-bsc-test                              latest              4b5d7fd9fa58        24 hours ago        1.31GB
osmocom-build/ttcn3-mgw-test                              latest              557b037a0216        24 hours ago        930MB
osmocom-build/osmo-bts-master-centos8                     latest              5f4d0fc6a477        2 days ago          694MB
osmocom-build/osmo-bsc-master-centos8                     latest              abfb4b3a0d5b        2 days ago          727MB
osmocom-build/osmo-smlc-latest                            latest              abd15a776a43        3 days ago          131MB
osmocom-build/osmo-sip-latest                             latest              a0949be7a2a4        3 days ago          142MB
osmocom-build/osmo-hlr-latest                             latest              e9d964b560ff        4 days ago          131MB
osmocom-build/osmo-bts-latest                             latest              270e0affd24d        4 days ago          132MB
osmocom-build/osmo-bsc-latest                             latest              1087b44d4696        4 days ago          135MB
osmocom-build/osmo-mgw-latest                             latest              86af08c170c7        4 days ago          131MB
registry.osmocom.org/osmocom-build/debian-stretch-titan   <none>              4e18b7aeb7aa        4 days ago          858MB
debian                                                    <none>              0a3cbbaa2b82        4 days ago          101MB
registry.osmocom.org/debian                               <none>              0a3cbbaa2b82        4 days ago          101MB
debian                                                    jessie              3aaeab7a4777        4 days ago          129MB
registry.osmocom.org/debian                               jessie              3aaeab7a4777        4 days ago          129MB
debian                                                    <none>              463adba1ec3f        4 days ago          114MB
registry.osmocom.org/debian                               <none>              463adba1ec3f        4 days ago          114MB
registry.osmocom.org/osmocom-build/fpga-build             latest              07a8933f89e6        6 days ago          5.39GB
osmocom-build/debian-buster-build                         latest              4da065519a9e        8 days ago          1.6GB
osmocom-build/osmo-cbc-master-centos8                     latest              9da1c7441e76        8 days ago          722MB
osmocom-build/osmo-gbproxy-master-centos8                 latest              f95cb930af43        8 days ago          689MB
osmocom-build/osmo-cbc-latest-centos8                     latest              9cf204289dff        9 days ago          316MB
osmocom-build/osmo-remsim-master-centos8                  latest              68d695ac2002        9 days ago          881MB
osmocom-build/osmo-cbc-master                             latest              ddf7e48e4dc3        10 days ago         1.02GB
osmocom-build/osmo-smlc-master                            latest              a4a3c5ef30db        10 days ago         995MB
osmocom-build/ttcn3-sccp-test                             latest              80c35db4d4bf        10 days ago         942MB
osmocom-build/ttcn3-stp-test                              latest              cf25fd7cf879        10 days ago         942MB
osmocom-build/ttcn3-bts-test                              latest              dac37dfb1b56        10 days ago         1.16GB
osmocom-build/osmocom-bb-host-master                      latest              844a2b686262        10 days ago         1GB
osmocom-build/osmo-bts-master                             latest              57fced9a14ca        10 days ago         1.01GB
osmocom-build/osmo-bsc-master                             latest              194970b60880        10 days ago         1.03GB
osmocom-build/osmo-mgw-master                             latest              61cbdf4786a1        11 days ago         1.01GB
osmocom-build/osmo-msc-master                             latest              f64fa59d9d0d        11 days ago         1.05GB
osmocom-build/osmo-stp-master                             latest              07bfaa5d1404        11 days ago         1.02GB
osmocom-build/osmo-pcu-master-centos8                     latest              4886ff13bdf2        11 days ago         705MB
osmocom-build/osmo-ggsn-master                            latest              979fa0b7d80b        11 days ago         1.27GB
osmocom-build/nplab-m3ua-test                             latest              8dca1ee7f0db        11 days ago         1.11GB
osmocom-build/sigtran-tests                               latest              41b6a4153d6e        11 days ago         1.11GB
osmocom-build/osmo-sip-master-centos8                     latest              0a0d67edebbb        11 days ago         682MB
osmocom-build/osmo-cbc-latest                             latest              4d19ca8f3883        2 weeks ago         136MB
osmocom-build/osmo-sip-latest-centos8                     latest              a2745ddbff8a        2 weeks ago         317MB
osmocom-build/osmo-mgw-master-centos8                     latest              8bc5847fc56f        2 weeks ago         694MB
osmocom-build/osmo-pcu-latest                             latest              f8a6e1d67e20        2 weeks ago         131MB
osmocom-build/osmo-ggsn-latest                            latest              7a61292c477e        2 weeks ago         609MB
osmocom-build/osmo-gbproxy-latest-centos8                 latest              da0f04f85f2b        2 weeks ago         316MB
osmocom-build/osmo-smlc-latest-centos8                    latest              50dd8221a594        2 weeks ago         316MB
osmocom-build/osmo-bts-latest-centos8                     latest              a6e18427df16        2 weeks ago         317MB
osmocom-build/osmo-bsc-latest-centos8                     latest              186406e066dd        2 weeks ago         318MB
osmocom-build/osmo-msc-master-centos8                     latest              758c9fd4c01f        2 weeks ago         730MB
osmocom-build/osmo-stp-master-centos8                     latest              2075653264c8        2 weeks ago         711MB
osmocom-build/osmo-ggsn-master-centos8                    latest              6dd7f322546d        2 weeks ago         688MB
osmocom-build/centos8-build                               latest              16d559d15d5f        2 weeks ago         617MB
debian                                                    <none>              26deccaae877        2 weeks ago         101MB
registry.osmocom.org/debian                               <none>              26deccaae877        2 weeks ago         101MB
debian                                                    <none>              dc2eddc15825        2 weeks ago         114MB
registry.osmocom.org/debian                               <none>              dc2eddc15825        2 weeks ago         114MB
osmocom-build/osmo-hlr-latest-centos8                     latest              a9ec196dcfed        3 weeks ago         305MB
osmocom-build/osmo-hlr-master-centos8                     latest              e1078dc1f9a3        3 weeks ago         684MB
osmocom-build/osmo-sgsn-master-centos8                    latest              b430dafdd6ac        3 weeks ago         698MB
osmocom-build/osmo-gbproxy-latest                         latest              b4957dcd786c        3 weeks ago         131MB
osmocom-build/osmo-remsim-master                          latest              7b84eebfe8e9        3 weeks ago         1.18GB
osmocom-build/osmo-sgsn-master                            latest              f8a6bbdb2519        3 weeks ago         1.03GB
osmocom-build/osmo-remsim-latest                          latest              c6b2ae1ff5f5        3 weeks ago         137MB
registry.osmocom.org/osmocom-build/debian-stretch-titan   <none>              8565b6d28098        4 weeks ago         864MB
osmocom-build/ttcn3-bscnat-test                           latest              6671d002dbbe        4 weeks ago         1.18GB
osmocom-build/osmo-nitb-master                            latest              541fe1807ea3        4 weeks ago         1.08GB
registry.osmocom.org/osmocom-build/debian-stretch-titan   <none>              951ff8746d52        6 weeks ago         858MB
registry.osmocom.org/debian                               <none>              f00be68857e6        7 weeks ago         101MB
debian                                                    <none>              f00be68857e6        7 weeks ago         101MB
registry.osmocom.org/debian                               <none>              f9d06e40f53a        3 months ago        101MB
debian                                                    <none>              f9d06e40f53a        3 months ago        101MB
registry.osmocom.org/centos                               centos8             300e315adb2f        3 months ago        209MB

I'm not exactly sure how we wnd up with months-old docker images. Maybe because some builds are only triggered rarely on some slaves? In any case we should have some automatic method to delete all old (> 1 week0 images related to 'master' as surely they serve no purpose anymore. For 'latest' its probably a bit more difficult to understand when to time those out.

The problem with older images is that they depend on older versions of the base debian images, which quickly adds up in terms of storage space consumption.

Actions #1

Updated by laforge almost 3 years ago

  • Subject changed from old docker images onbuild slaves to old docker images on build slaves
Actions #2

Updated by osmith almost 3 years ago

A generic solution would be sorting by last used date, and delete everything that was not used by N weeks.

This was proposed here in 2014, but never implemented: https://github.com/moby/moby/issues/4237

At the end of the issue, 3rd party solutions are posted...

https://github.com/stepchowfun/docuum

This one looks quite nice, the idea is that you run it as daemon (e.g. with systemd):

$ docuum --threshold '30 GB'

and it makes sure that at most 30 GB are used for docker images, if it exceeds the size, it deletes the images that were not used for the longest time.

A downside is, that it is not packaged in debian.

If you like this solution, I could create an ansible rule to add it.

Actions #3

Updated by osmith almost 3 years ago

FWIW, I've installed docuum on my laptop and it works great so far.

[2021-04-12 16:15:56 +02:00 INFO] Waking up…
[2021-04-12 16:15:56 +02:00 INFO] Updating last-used timestamp for image `sha256:a5737f342378564b5f898173b3a68e8f8050e11d3861ef087e2af0fdb40ecd91`…
[2021-04-12 16:15:57 +02:00 INFO] Docker images are using `19.81 GB`, which is within the limit of `20.00 GB`.
[2021-04-12 16:15:57 +02:00 INFO] Going back to sleep…
[2021-04-12 16:17:04 +02:00 INFO] Waking up…
[2021-04-12 16:17:04 +02:00 INFO] Updating last-used timestamp for image `sha256:a5737f342378564b5f898173b3a68e8f8050e11d3861ef087e2af0fdb40ecd91`…
[2021-04-12 16:17:05 +02:00 INFO] Docker images are currently using `20.16 GB` but the limit is `20.00 GB`. Some images will be deleted.
[2021-04-12 16:17:05 +02:00 INFO] Deleting image `sha256:fb20f2fd1071d5c70f5d0a08299eaffd93f5b19d0fb4759b13c3017e4d199d62`…
Untagged: user/ttcn3-bsc-test:latest
Deleted: sha256:fb20f2fd1071d5c70f5d0a08299eaffd93f5b19d0fb4759b13c3017e4d199d62
[2021-04-12 16:17:07 +02:00 INFO] Deleting image `sha256:c6128192acfc0ec0b2fc314ad0f837d56f44d5f2f06d6e3aae31b39d42b492dd`…
Deleted: sha256:c6128192acfc0ec0b2fc314ad0f837d56f44d5f2f06d6e3aae31b39d42b492dd
Deleted: sha256:c375de2e0c331cba37f6cda76a80525c4fe0e0a6f6fff0d127768d784bed0460
[2021-04-12 16:17:08 +02:00 INFO] Deleting image `sha256:2436d43a34a4eb30a8b5800c510c65d42f809a5d90e180f01b8305ce7b68f674`…
Deleted: sha256:2436d43a34a4eb30a8b5800c510c65d42f809a5d90e180f01b8305ce7b68f674
[2021-04-12 16:17:08 +02:00 INFO] Deleting image `sha256:97f2f5d252ef0e26213c4ec9dd8726c1cd3033db30ebf26e5f55242fd8e0dff8`…
Deleted: sha256:97f2f5d252ef0e26213c4ec9dd8726c1cd3033db30ebf26e5f55242fd8e0dff8
Deleted: sha256:b91026cccb5d8cef3e0a688ffcf818eafbf9dba23e4f0ca8a295093aa043b884
[2021-04-12 16:17:09 +02:00 INFO] Docker images are now using `19.72 GB`, which is within the limit of `20.00 GB`.
[2021-04-12 16:17:09 +02:00 INFO] Going back to sleep…
[2021-04-12 16:17:09 +02:00 INFO] Waking up…
[2021-04-12 16:17:09 +02:00 INFO] Updating last-used timestamp for image `sha256:c42347edd619e8826b208bdf6e7ee892eb389ed59563d3e098efbec775f0e4f5`…
[2021-04-12 16:17:10 +02:00 INFO] Docker images are using `19.72 GB`, which is within the limit of `20.00 GB`.
[2021-04-12 16:17:10 +02:00 INFO] Going back to sleep…
Actions #4

Updated by laforge over 2 years ago

  • Priority changed from High to Normal

osmith wrote in #note-2:

If you like this solution, I could create an ansible rule to add it.

yes, please.

Actions #5

Updated by osmith about 2 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 90
Actions #6

Updated by osmith about 2 years ago

  • Status changed from In Progress to Resolved
Actions #7

Updated by osmith about 2 years ago

  • % Done changed from 90 to 100
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)