Project

General

Profile

Bug #4083

git.osmocom.org: error: garbage at end of loose object

Added by osmith 4 months ago. Updated about 8 hours ago.

Status:
New
Priority:
Normal
Assignee:
sysmocom
Category:
-
Target version:
-
Start date:
07/01/2019
Due date:
% Done:

0%

Spec Reference:

Description

Right now, I can't clone libosmo-sccp from https://git.osmocom.org/libosmo-sccp. It fails with:

$ git -C /home/user/code/osmo-dev/src/osmo-ci/_temp/repos clone https://git.osmocom.org/libosmo-sccp
Cloning into 'libosmo-sccp'...
error: garbage at end of loose object 'a70b4599f63b50c4d654eb9e1b1a099a7c8cdc36'
fatal: loose object a70b4599f63b50c4d654eb9e1b1a099a7c8cdc36 (stored in /home/user/code/osmo-dev/src/osmo-ci/_temp/repos/libosmo-sccp/.git/objects/a7/0b4599f63b50c4d654eb9e1b1a099a7c8cdc36) is corrupt

I have noticed this multiple times with various repositories in the past week.

A few minutes later, it is working again. So maybe this is related to some sync job?

History

#2 Updated by laforge 3 months ago

There is a "synch job" and that's the gerrit replication job that pushes/mirrors
gerrit.osmocom.org to git.osmocom.org. We're using the built-in gerrit replication
service for this, which in turn AFAIK just does a git+ssh push, which shouldn't introduce
any races. After all, it happens quite frequently that somebody clones a public git
repo while somebody else is pushing something.

However, pulling/cloning over https means there's a nginx reverse proxy between the server
and the client, and there may be some problem introduced here.

It would be useful to have exact timestamps to check if there's anything in the log. So
maybe it's worth to modify related scripts to print a timestamp before doing the clone?

The interesting bit from our config is that the http+https access to git.osmocom.org
are actually redirected to cgit.osmocom.org

To be honest, I never really looked too deep into how git over http works, given that
in the past I've only used it with the native git protocol and with ssh. For a long
time AFAIR it was even discouraged to use http* with it.

#3 Updated by osmith 3 months ago

Thanks for looking into it.

It would be useful to have exact timestamps to check if there's anything in the log. So
maybe it's worth to modify related scripts to print a timestamp before doing the clone?

https://gerrit.osmocom.org/c/osmo-ci/+/14894

#4 Updated by osmith 3 months ago

laforge, here is one set of timestamps:

+ date '+%Y-%m-%d %H:%M:%S'
2019-07-29 07:01:59
+ git clone https://git.osmocom.org/libosmo-sccp
Cloning into 'libosmo-sccp'...
error: garbage at end of loose object '70751fce24f989d7334cba180e34ca65840ed0f9'
fatal: loose object 70751fce24f989d7334cba180e34ca65840ed0f9 (stored in /home/osmocom-build/jenkins/workspace/Osmocom_OBS_nightly/nightly-3g_QnWQVB/repo/libosmo-sccp/.git/objects/70/751fce24f989d7334cba180e34ca65840ed0f9) is corrupt
+ date '+%Y-%m-%d %H:%M:%S'
2019-07-29 07:02:10
+ exit 1

https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly/173/console

#5 Updated by osmith about 2 months ago

+ date '+%Y-%m-%d %H:%M:%S'
2019-08-29 01:26:58
+ git clone https://git.osmocom.org/osmo-gsm-manuals
Cloning into 'osmo-gsm-manuals'...
error: garbage at end of loose object 'f880d707aa636080def299450b13901904614449'
fatal: loose object f880d707aa636080def299450b13901904614449 (stored in /home/osmocom-build/jenkins/workspace/Osmocom_OBS_nightly/nightly-3g_6IdnDB/repo/osmo-gsm-manuals/.git/objects/f8/80d707aa636080def299450b13901904614449) is corrupt
+ date '+%Y-%m-%d %H:%M:%S'
2019-08-29 01:27:11

https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly/204/console

#6 Updated by osmith about 12 hours ago

+ date '+%Y-%m-%d %H:%M:%S'
2019-10-21 01:26:55
+ git clone https://git.osmocom.org/osmo-gsm-manuals
Cloning into 'osmo-gsm-manuals'...
error: garbage at end of loose object '51c7d508314156e6677972975c8a12a9224ccbe4'
fatal: loose object 51c7d508314156e6677972975c8a12a9224ccbe4 (stored in /home/osmocom-build/jenkins/workspace/Osmocom_OBS_nightly/nightly-3g_7tHS0x/repo/osmo-gsm-manuals/.git/objects/51/c7d508314156e6677972975c8a12a9224ccbe4) is corrupt
+ date '+%Y-%m-%d %H:%M:%S'
2019-10-21 01:27:07

https://jenkins.osmocom.org/jenkins/job/Osmocom_OBS_nightly/259/console

#7 Updated by laforge about 10 hours ago

I cannot reproduce it here. From which build slave[s] is this happening? Maybe it relates
to a specific git client version?

#8 Updated by osmith about 8 hours ago

laforge wrote:

I cannot reproduce it here. From which build slave[s] is this happening? Maybe it relates
to a specific git client version?

All three errors linked above happened on build2-deb9build-ansible. I've also experienced this problem on my local machine (also running Debian 9, currently git 2.11.0 installed), but very rarely. Maybe it is related to the git version.

When I try to pull / clone again at a later point (maybe a few minutes, I did not watch the time), it works again.

It would be useful to have exact timestamps to check if there's anything in the log. So
maybe it's worth to modify related scripts to print a timestamp before doing the clone?

The log output above contains timestamps. Is there anything useful in the logs?

All in all, this problem appears very rarely so it does not seem worth spending much time on it.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)