Project

General

Profile

Bug #4774

occasional 504 error when accessing git.osmocom.org

Added by laforge about 2 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
10/03/2020
Due date:
% Done:

30%

Spec Reference:

Description

we occasionally get '504' timeout errors when accessing git.osmocom.org. pespin reported this yesterday.

Architecturally, it's cgit (in apache) behind a nginx reverse proxy.

In order to debug this further, I enabled logging of timing in nginx. One day later, I can see:

around 23:30 UTC, we actually exceed the 60s proxy timeout and return 504, which is probably sever overload due to too many build jobs causing I/O starvation at that time?

02/Oct/2020:23:29:14 +0000] "GET /dect/libpcap/log/README.tru64 HTTP/1.1" 504 183 "-" "The Knowledge AI" 60.004 60.004 .
Oct/2020:23:29:28 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 60.003 60.004 .
Oct/2020:23:29:42 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 59.999 60.000 .
[02/Oct/2020:23:29:45 +0000] "GET /dect/linux-2.6/plain/fs/ceph/mds_client.h HTTP/1.1" 504 183 "http://git.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 60.003 60.004 .
02/Oct/2020:23:30:03 +0000] "GET /wireshark/commit/?id=311b1ee70072531352262f3a47181e89472e3109 HTTP/1.1" 504 585 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.132 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 60.007 60.009 .
[02/Oct/2020:23:30:17 +0000] "GET /wireshark/plain/epan/wmem/wmem_miscutl.c HTTP/1.1" 504 183 "http://cgit.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 60.000 60.001 .
02/Oct/2020:23:30:24 +0000] "GET /dect/dedected/log/kismet-dect-v2 HTTP/1.1" 504 183 "-" "The Knowledge AI" 60.003 60.001 .
[02/Oct/2020:23:30:49 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 59.998 60.001 .
[02/Oct/2020:23:30:50 +0000] "GET /dect/linux-2.6/plain/fs/cifs/cifs_debug.c HTTP/1.1" 504 183 "http://git.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 59.999 60.001 .
Oct/2020:23:30:50 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 59.999 60.001 .
Oct/2020:23:30:57 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 60.003 60.005 .
[02/Oct/2020:23:31:07 +0000] "GET /erlang/osmo_sccp/ HTTP/1.1" 504 183 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)" 60.005 60.004 .
02/Oct/2020:23:31:10 +0000] "GET /op25/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.28.0.windows.1" 60.000 60.004 .
[02/Oct/2020:23:31:11 +0000] "GET /atmel-asf-projects/commit/?id=3634cc8e5d3124e3cf6cf46c0f1d5385ff511426 HTTP/1.1" 504 183 "-" "The Knowledge AI" 60.000 60.000 .
Oct/2020:23:31:11 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.1" 504 183 "-" "git/2.17.1" 60.001 60.000 .
[02/Oct/2020:23:31:22 +0000] "GET /wireshark/plain/epan/wmem/wmem_miscutl.h HTTP/1.1" 504 183 "http://cgit.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 60.002 60.004 .
02/Oct/2020:23:31:34 +0000] "GET /dect/libpcap/log/README.tru64 HTTP/1.1" 504 183 "-" "The Knowledge AI" 59.998 60.000 .

but then we also have plenty of 504 within milliseconds:

[03/Oct/2020:01:02:58 +0000] "GET /plugins/gitiles/gapk/+log/f624d18067a34815ebb2e6332bac6aaa2385a5de/Makefile.am?format=JSON HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.006 0.008 .
[03/Oct/2020:01:48:05 +0000] "GET /plugins/gitiles/libosmocore/+/dc3be0a6533be782c76f23ce653f4fc683b3b4f1/.gitignore?format=TEXT HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.002 0.000 .
[03/Oct/2020:01:52:16 +0000] "GET /meta-telephony/objects/1b/3362f0bd99362b85f911d20fef6b8327dbf825 HTTP/1.1" 200 504 "-" "git/1.9.1 (Linux) git-repo/2.9" 0.003 0.004 .
[03/Oct/2020:02:13:53 +0000] "GET /docker-playground/plain/sua-test/Dockerfile?id=58e97d1f0b68eaf57a19fb852846763a53b0d19e HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" 0.006 0.004 .
[03/Oct/2020:02:14:34 +0000] "GET /osmo-trx/plain/INSTALLATION?id=441d82add9afbc75dc3e9fe45a80ce3ff8f8b15e HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" 0.004 0.004 .
[03/Oct/2020:02:23:51 +0000] "GET /plugins/gitiles/gapk/+log/a0c4d21bf60ecd77069a7f7e0203e46f133bfbfa/Makefile.am?format=JSON HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.006 0.008 .
[03/Oct/2020:02:42:52 +0000] "GET /qemu/plain/block-migration.h?id=ac531cb6e542b1e61d668604adf9dc5306a948c0 HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 0.004 0.004 .
[03/Oct/2020:02:54:50 +0000] "GET /gr-osmosdr/objects/2a/e3fdbc220df4896a87f9fdb1bacc8a6c08c849 HTTP/1.1" 200 504 "-" "git/2.20.1" 0.013 0.012 .
[03/Oct/2020:03:16:35 +0000] "GET /gr-osmosdr/objects/2a/e3fdbc220df4896a87f9fdb1bacc8a6c08c849 HTTP/1.1" 200 504 "-" "git/2.28.0" 0.003 0.004 .
[03/Oct/2020:03:22:11 +0000] "GET /plugins/gitiles/libosmocore/+/07f1103782a94090c2cef46de8a3f6d03ddfeef7/.gitignore?format=TEXT HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.002 0.000 .
[03/Oct/2020:03:26:58 +0000] "GET /gr-osmosdr/objects/2a/e3fdbc220df4896a87f9fdb1bacc8a6c08c849 HTTP/1.1" 200 504 "-" "git/2.25.1" 0.003 0.004 .
[03/Oct/2020:04:13:54 +0000] "GET /gr-osmosdr/objects/2a/e3fdbc220df4896a87f9fdb1bacc8a6c08c849 HTTP/1.1" 200 504 "-" "git/2.17.1" 0.007 0.008 .
[03/Oct/2020:06:49:28 +0000] "GET /osmo-pcu/plain/.gitignore?id=2db0f08e08d074dc2876b780c76f7261851e7296 HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" 0.006 0.008 .
[03/Oct/2020:06:51:00 +0000] "GET /qemu/plain/hw/framebuffer.h HTTP/1.1" 200 504 "https://git.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 0.006 0.004 .
[03/Oct/2020:07:00:38 +0000] "GET /osmo-uecups/plain/Makefile.am HTTP/1.1" 200 504 "-" "The Knowledge AI" 0.006 0.004 .
[03/Oct/2020:07:37:53 +0000] "GET /plugins/gitiles/osmo-gsm-tester/+/8f4f17437d8926b8692b46ac2caac94e86d39f87?format=JSON HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.006 0.008 .
[03/Oct/2020:07:51:22 +0000] "GET /docker-playground/plain/sua-test/Dockerfile?id=9d3b695d461096d8150033007495ce5f95c25f11 HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" 0.003 0.000 .
[03/Oct/2020:08:33:58 +0000] "GET /ptrkrysik/test_data/tree/vf_call6_a725_d174_g5_Kc1EF00BAB3BAC7002.cfile HTTP/1.1" 504 183 "-" "The Knowledge AI" 60.002 60.001 .
[03/Oct/2020:08:34:07 +0000] "GET /osmocom-lcs/commit/?id=8dbd19e6d8bfea5c1e1913e1f4015d5aab66d305 HTTP/1.1" 504 183 "-" "The Knowledge AI" 60.003 60.004 .
[03/Oct/2020:08:55:24 +0000] "GET /jenkins/job/OpenBSC/ws/openbsc/git-version-gen/*fingerprint*/?auto_refresh=true HTTP/1.1" 404 504 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)" 0.001 0.000 .
[03/Oct/2020:09:23:43 +0000] "GET /qemu/plain/block-migration.h?id=e7d81004e486b0e80a674d164d8aec0e83fa812f HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 0.005 0.008 .
[03/Oct/2020:09:39:13 +0000] "GET /plugins/gitiles/libosmo-abis/+/25c83611be697f53925184658cdbff72de58c8c6?format=JSON HTTP/1.1" 200 504 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" 0.011 0.012 .


Related issues

Related to Osmocom.org Servers - Bug #4083: git.osmocom.org: error: garbage at end of loose objectNew07/01/2019

History

#1 Updated by laforge about 2 months ago

the format above was generated by the following nginx conig:

looking at the cgit/nginx log output:

172.18.0.20 - - [02/Oct/2020:23:29:11 +0000] "GET /atmel-asf-projects/commit/?id=3634cc8e5d3124e3cf6cf46c0f1d5385ff511426 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [02/Oct/2020:23:29:12 +0000] "GET /wireshark/plain/epan/wmem/wmem_map_int.h HTTP/1.0" 504 183 "http://cgit.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 
172.18.0.20 - - [02/Oct/2020:23:29:14 +0000] "GET /dect/libpcap/log/README.tru64 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [02/Oct/2020:23:29:28 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:29:42 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:29:45 +0000] "GET /dect/linux-2.6/plain/fs/ceph/mds_client.h HTTP/1.0" 504 183 "http://git.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 
172.18.0.20 - - [02/Oct/2020:23:30:03 +0000] "GET /wireshark/commit/?id=311b1ee70072531352262f3a47181e89472e3109 HTTP/1.0" 504 585 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.132 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
172.18.0.20 - - [02/Oct/2020:23:30:17 +0000] "GET /wireshark/plain/epan/wmem/wmem_miscutl.c HTTP/1.0" 504 183 "http://cgit.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 
172.18.0.20 - - [02/Oct/2020:23:30:24 +0000] "GET /dect/dedected/log/kismet-dect-v2 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [02/Oct/2020:23:30:49 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:30:50 +0000] "GET /dect/linux-2.6/plain/fs/cifs/cifs_debug.c HTTP/1.0" 504 183 "http://git.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 
172.18.0.20 - - [02/Oct/2020:23:30:50 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:30:57 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:31:07 +0000] "GET /erlang/osmo_sccp/ HTTP/1.0" 504 183 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)" 
172.18.0.20 - - [02/Oct/2020:23:31:10 +0000] "GET /op25/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.28.0.windows.1" 
172.18.0.20 - - [02/Oct/2020:23:31:11 +0000] "GET /atmel-asf-projects/commit/?id=3634cc8e5d3124e3cf6cf46c0f1d5385ff511426 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [02/Oct/2020:23:31:11 +0000] "GET /rtl-sdr/info/refs?service=git-upload-pack HTTP/1.0" 504 183 "-" "git/2.17.1" 
172.18.0.20 - - [02/Oct/2020:23:31:22 +0000] "GET /wireshark/plain/epan/wmem/wmem_miscutl.h HTTP/1.0" 504 183 "http://cgit.osmocom.org/" "Mozilla/5.0 (compatible; Adsbot/3.1)" 
172.18.0.20 - - [02/Oct/2020:23:31:34 +0000] "GET /dect/libpcap/log/README.tru64 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [03/Oct/2020:08:33:58 +0000] "GET /ptrkrysik/test_data/tree/vf_call6_a725_d174_g5_Kc1EF00BAB3BAC7002.cfile HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [03/Oct/2020:08:34:07 +0000] "GET /osmocom-lcs/commit/?id=8dbd19e6d8bfea5c1e1913e1f4015d5aab66d305 HTTP/1.0" 504 183 "-" "The Knowledge AI" 
172.18.0.20 - - [03/Oct/2020:08:34:09 +0000] "GET /uboot-mt623x/tree/board/sbc8560/u-boot.lds?id=3afac79ec27b91df185f090b31dad9620779f440 HTTP/1.0" 504 183 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)" 

We can see that the cgit-nginx itself also generates '504' codes. Probably if the CGI doesn't produce output in time? I've increased that timeout to 120s, just in case.

#2 Updated by laforge about 2 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

#3 Updated by laforge about 2 months ago

  • Related to Bug #4083: git.osmocom.org: error: garbage at end of loose object added

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)