Bug #4555
host2 disk space running low
100%
Description
/dev/md2 438G 377G 39G 91% /
- 199 GB are jenkins (build history / artefacts)
- 23GB is the deb8build jenkins slave lxc
- 79GB is the deb9build jenkins slave lxc
- 29GB are our docker image layers
We could add hard disks / SSDs to the server, but those are (rented) relatively expensive at ~ 8-10 EUR per month, which we'd have to double for RAID-1. It would be much more economic to upgrade from the AX60 to an AX51-NVMe (2x 1TB storage, faster CPU), which has only an EUR 5 per month price increase. However, that would mean migrating all data to a new machne and then finally switch over after everything is migrated.
History
#3 Updated by laforge 8 months ago
What's quite interesting is the difference in 'du' output within the debian9 lxc and outside of it:
root@deb9build-ansible:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/md2 438G 371G 45G 90% /
root@host2 /var/lib/lxc # du --max-depth=1 -h 78G ./deb9build-ansible
#4 Updated by laforge 7 months ago
- Status changed from New to In Progress
- % Done changed from 0 to 10
the problem is becoming more critical. I've inquired wih Hetzner about an upgrade to AX51-NVMe.
The biggest consumer of space are the BTS tests at about 1.8 to 2 GB per build. We keep 35 at the moment, ant we test master+latest for debian+centos, adding to almost 100GB at the moment already only for the BTS test.
As an interim measure, I am bzip'ing all those pcap files of the BTS tests.
#5 Updated by roh 7 months ago
- Priority changed from Normal to Urgent
disk ran full tonight.
redmine for osmocom.org stalled out, so i tried to reload the docker compose unit which failed (looped)
i scraped off 2% reserved space of /dev/md2 via tune2fs to get it up again - this is urgent now.
/dev/md2 438G 415G 8.9G 98% /
#8 Updated by laforge 6 months ago
- Status changed from In Progress to Resolved
- % Done changed from 10 to 100
all services migrated to new host2. The new machine still has a different IP address (178.63.23.163) and the old machine is running TCP + UDP port forwarding.
On Tuesday or Wednesday morning 8am, the IPs get migrated, and the old machine will disappear. I'm taking a last backup of the old 'host2' right now.
Resolving this ticket as the disk space problem is now absent.
I also re-compressed all pcap.bz2 to pcap.gz for convenience reasons.
/dev/md2 906G 397G 464G 47% /