Project

General

Profile

Bug #4862

replicate download.opensuse.org content somewhere on *.osmocom.org

Added by laforge 11 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
laforge
Category:
-
Target version:
-
Start date:
11/18/2020
Due date:
% Done:

100%

Spec Reference:

Description

as we can see right now. {download,mirrors,build}.opensuse.org are all down.

This means we can not build any of our container images, nor can any users install pre-built osmocom patches.

We should at least have an alternative mirror on osmocom.org.

Checklist

  • move to dedicated uid
  • export via rsync
  • export via https
  • reference it from wiki so it can be found

Related issues

Related to Osmocom.org Servers - Feature #5083: migrate jenkins master to machine with lots of HDD storageResolved03/20/2021

Associated revisions

Revision 7d7a06b7 (diff)
Added by pespin 11 months ago

common: Introduce OSMOCOM_REPO_MIRROR param

This allows easily changing to another osmocom repo base URL if the
default one is down.

Related: OS#4862
Change-Id: I8010b08f3dabacfb3c13a44eece6c7a490e0742e

History

#1 Updated by laforge 11 months ago

https://ftp.uni-stuttgart.de/opensuse/repositories/network:/osmocom:/ seems to be a current mirror, in case anyone needs it.

#2 Updated by pespin 11 months ago

See patch:
https://gerrit.osmocom.org/c/docker-playground/+/21239 common: Introduce OSMOCOM_REPO_MIRROR param

With it, one can for instance:
cd ttcn3-pcu-test
OSMOCOM_REPO_MIRROR="http://ftp.uni-stuttgart.de/opensuse" ./jenkins.sh

#3 Updated by laforge 6 months ago

  • Related to Feature #5083: migrate jenkins master to machine with lots of HDD storage added

#4 Updated by laforge 6 months ago

now that we have host3.osmocom.org with 4TB RAID-1 storage, we should be able to host both amirror of the OBS builds, as well as an archive of them.

#5 Updated by laforge 6 months ago

Something like rsync://62.146.92.210/buildservice-repos/network:/osmocom: works to rsync the packages. The size is quite small:

2.4M    ./pcsc
293M    ./latest
3.1M    ./mingw:
8.0K    ./tdm:
4.0K    ./next
8.0K    ./nitb-split:
175M    ./nightly

#6 Updated by laforge 6 months ago

  • Status changed from Stalled to In Progress
  • % Done changed from 0 to 30

I'm testing the following script now on host3:

#!/bin/bash
set -e -x

# base directory on the local side
BASE_DIR=/home/laforge/osmocom-obs-mirror/
# sync remote where to find the osmocom packages
REMOTE=rsync://ftp.gwdg.de/pub/opensuse/repositories/network:/osmocom:

cd "$BASE_DIR" 

RSYNC_ARGS="-av --delete" 
DATE=`date +%Y%m%d-%H%M%S`

# create output directory
DIR="$BASE_DIR/$DATE" 
mkdir -p "$DIR" 

PREVIOUS="$BASE_DIR/previous" 
if [ -d "$PREVIOUS" ]; then
        RSYNC_ARGS+=" --link-dest=$PREVIOUS" 
fi

# finally, perform rsync
rsync $RSYNC_ARGS "$REMOTE"/ "$DIR"/

# update 'previous' for the next run
rm "$PREVIOUS" 
ln -sf "$DATE" "$PREVIOUS"

#7 Updated by laforge 6 months ago

  • Checklist item move to dedicated uid added
  • Checklist item export via rsync added
  • Checklist item export via https added
  • Checklist item reference it from wiki so it can be found added

the script seems to be working fine. de-duplication works as expected. Currently using 19GB for 10 days of history (twice daily).

Need to make this available publicly via at least https and rsync and document it for people to find

#8 Updated by laforge 6 months ago

  • Checklist item move to dedicated uid set to Done
  • Checklist item export via rsync set to Done
  • % Done changed from 30 to 60
$ rsync rsync://host3.osmocom.org/obs-mirror/
drwxr-xr-x          4,096 2021/04/22 09:54:05 .
lrwxrwxrwx             15 2021/04/22 00:23:20 previous
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111633
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111812
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111848
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-111930
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-112218
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210412-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210413-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210413-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210414-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210414-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210415-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210415-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210416-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210416-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210417-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210417-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210418-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210418-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210419-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210419-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210420-002301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210420-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210421-002302
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210421-122301
drwxr-xr-x          4,096 2020/11/19 22:48:52 20210422-002301

#9 Updated by laforge 6 months ago

  • Checklist item export via https set to Done

#10 Updated by laforge 6 months ago

  • Checklist item reference it from wiki so it can be found set to Done
  • Status changed from In Progress to Resolved
  • % Done changed from 60 to 100

Linked from wiki at Latest_Builds and Nightly_Builds

#11 Updated by osmith 5 months ago

Script updated:
  • "previous" -> ".previous" (so it does not show up in dir listing)
  • "rm -f" instead of "rm", so it doesn't stop if the previous symlink does not exist and never creates it (as we just ran into this)

#12 Updated by osmith 5 months ago

Updated:
  • only archive directories mentioned in obs-mirror-include.txt
  • add "|| true" after rsync command so it doesn't stop there if a dir from the list was not found
  • add test-include-list.sh

#13 Updated by osmith 4 months ago

As discussed, I've added a new cronjob to delete nightly archives older than one month.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)