Project

General

Profile

Feature #2927

Use traceback instead of inspect to speed up log.py

Added by zecke 9 days ago.

Status:
New
Priority:
Urgent
Assignee:
-
Target version:
-
Start date:
02/10/2018
Due date:
% Done:

0%

Spec Reference:

Description

I wanted to use log.Origin and self.log inside the "osmo_ms_driver" for the massive parallel (2*10k processes) "mobile". I have issues starting this amount of processes within 60s.

The cruel pit seems to be that the logging code wants to look at the stack trace to format the caller/linenumber correctly. It is done using the "inspect" module and this recurses down to stat. Please consider removing the usage of inpsect and just use the "traceback" module. It will give us access to filename+lineno+method name. I hope it is enough for debugging things.

Breakpoint 1, ___xstat64 (vers=3, name=0xb71b5ce8 "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_ms_driver/starter.py", buf=0xbfffe3cc) at ../sysdeps/unix/sysv/linux/xstat64.c:33
33    ../sysdeps/unix/sysv/linux/xstat64.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  <built-in method stat of module object at remote 0xb796df54>
  File "/usr/lib/python3.5/linecache.py", line 74, in checkcache
    stat = os.stat(fullname)
  File "/usr/lib/python3.5/traceback.py", line 354, in extract
    linecache.checkcache(filename)
  File "/usr/lib/python3.5/traceback.py", line 207, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_gsm_tester/log.py", line 279, in get_src_from_caller
    caller = extract_stack(limit=levels_up+1)
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_gsm_tester/log.py", line 80, in _log
    src = get_src_from_caller(src + 1)
  (frame information optimized out)
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_gsm_tester/log.py", line 466, in log
    log(*messages, _origin=self, _src=_src, **named_items)
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_ms_driver/starter.py", line 39, in start
    self.log("Starting process")
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_ms_driver/ul_test.py", line 68, in launch
    proc = ms.start(loop)
  File "/media/sf_source/gsm/osmo-gsm-tester/src/osmo_ms_driver/__main__.py", line 60, in <module>
    test.launch(loop)
  <built-in method exec of module object at remote 0xb799f874>
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  (frame information optimized out)

Also available in: Atom PDF