Project

General

Profile

Feature #5083

migrate jenkins master to machine with lots of HDD storage

Added by laforge about 1 month ago. Updated 11 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
jenkins
Target version:
-
Start date:
03/20/2021
Due date:
% Done:

100%

Spec Reference:

Description

With the increaes of test jobs, the storage requirements of our jenkins master (for the build artefacts archive) are exploding. Right now its 528GB. Most of it is the BTS test jobs, and interestingly not their pcap files anymore but the merged TTCN3 test logs.

97G     TTCN3-centos-bts-test
36G     TTCN3-centos-bts-test-latest
98G     ttcn3-bts-test
78G     ttcn3-bts-test-latest

Those text files would nicely compress (one example file from > 80MB to 2.6 MB), but jenkins seems to neither decompress on the fly, nor send the correct mime type to the browser to enable the browser to show uncompressed text.

So rather than patching jenkins or coming up with other non-standard methods that affect usability, the best is to move to larger storage.

Unfortunately adding disks to Hetzner servers is disproportionally expensive. For adding two 4TB disks to a server we can actually just as well get another dedicated server with those disks in it.

So my plan is to actually rent another dedicated server with 2x 4TB RAID-1 storage and run nothing but the jenkins master there. The build slaves would still be NVMe SSD based, so the slow storage of the master shouldn't really matter.


Related issues

Related to Osmocom.org Servers - Bug #4862: replicate download.opensuse.org content somewhere on *.osmocom.orgResolved11/18/2020

Associated revisions

Revision 1da1fefb (diff)
Added by fixeria about 1 month ago

library/PCUIF_Types: get rid of version 9 compatibility glue

Change-Id: Ia9f366ca1fdad700a90ca3367e43523f7bac39a1
Related: OS#5083

Revision 204fe621 (diff)
Added by fixeria about 1 month ago

library/PCUIF_Types: use PADDING attribute for 'PCUIF_Message'

PCUIFv9 compatibility has been dropped in [1], so now we can tell
TITAN's RAW codec what kind of padding to expect in received
messages and to append to encoded messages. This eliminates
thousands of warnings about unhandled tail octets.

Change-Id: Id5ab587f7984a91f3603efc5d427aa4cb4fa6757
Related: [1] Ia9f366ca1fdad700a90ca3367e43523f7bac39a1
Related: OS#5083

Revision ec9e981c (diff)
Added by fixeria about 1 month ago

BTS_Tests: do not connect to PCUIF socket if not used

The PCUIF connection involves a lot of frequent messages, such as
the TIME.ind and since recently DATA.ind with len=0. As a result,
the test suite logs are getting unreadable due to lots of coding
warnings and port queueing notifications.

This change is aimed to improve the situation a bit, by establishing
the PCUIF connection only for those test cases which actually use it.

Side effects:

  • TC_pcu_socket_verify_info_ind becomes reliable, because the
    PCUIF establishment is done after the RSL bootstrapping;
  • TC_pcu_socket_connect_multi starts to fail, because it used
    to pass due to timeout, since not all messages are handled
    in the 'alt' statement.

Change-Id: I09ccb65ce94a41ffdad4e93da650c3f32d422af4
Related: OS#5083

History

#1 Updated by laforge about 1 month ago

  • Subject changed from migrate jenkins master to machien with lots of HDD storage to migrate jenkins master to machine with lots of HDD storage

#2 Updated by fixeria about 1 month ago

... and interestingly not their pcap files anymore but the merged TTCN3 test logs.

I think we can also reduce the size of logs by eliminating tons of PCUIF enc/dec messages, please see:

https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/23461 library/PCUIF_Types: use PADDING attribute for 'PCUIF_Message'
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/23465 BTS_Tests: do not connect to PCUIF socket if not used

#3 Updated by laforge 11 days ago

  • Status changed from New to Resolved
  • % Done changed from 0 to 100

migration has been performed shortly after midnight CEST on April 12, 2021.

#4 Updated by laforge 11 days ago

  • Related to Bug #4862: replicate download.opensuse.org content somewhere on *.osmocom.org added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)