HowTo: Install and use py-spy on UCS

Knowledge Base Community
,
1

Purpose

py-spy is a sampling profiler for Python applications. It inspects a running Python process and shows which functions it is executing, without requiring application code changes or a service restart.

On UCS, it can help investigate performance issues in Python-based services such as the Univention AD Connector, for example:

  • High CPU usage during synchronization.
  • Slow processing of directory changes.
  • A connector that appears to be stuck.

Use record to create a flame graph, top for a live view, and dump for a snapshot of Python thread stacks. Profiling complements connector logs; it does not replace them.

Requirements and scope

  • Root access to the UCS host running the target process.
  • An x86-64 system for the wheel used below.
  • Python 3 with the python3-venv package available.
  • Access to GitHub to download the wheel.

This article uses py-spy 0.4.2. The installation transcript below was supplied from a UCS system in Version 5.2-7

Operational note: py-spy is designed for low-overhead profiling, but profiling is not impact-free. By default, it briefly pauses the target process while collecting samples. Use bounded recording periods and monitor the service.

1. Install py-spy

Run the installation from /root/univention-support, as shown in the transcript. The directory must already exist. Shell prompts and command output are included for reference and must not be entered as commands.

Important: The final line contains the PID placeholder <pid>. Use the corrected recording command in section 3 instead.

wget https://github.com/benfred/py-spy/releases/download/v0.4.2/py_spy-0.4.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl

Get the necessary packages and install py-spy:

root@dc01:~/univention-support# univention-install python3-venv
root@dc01:~/univention-support# python3 -m venv venv

For a new shell session, activate the existing environment:

root@dc01:~/univention-support# source venv/bin/activate
(venv) root@dc01:~/univention-support# python3 -m pip install py_spy-0.4.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Processing ./py_spy-0.4.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Installing collected packages: py-spy
Successfully installed py-spy-0.4.2

The virtual environment contains the profiler only. The target service does not need to use this environment.

2. Identify the target process

For the UCS 5.2 AD Connector, the launcher starts Python with the module univention.connector.ad.main.

List matching processes:

pgrep -af 'univention[.]connector[.]ad[.]main'

Check the full command line and select the Python process belonging to the connector instance you want to investigate. If multiple instances are running, profile the relevant instance separately.

In the examples below, 12345 is an example PID. Replace it with the actual PID before running a command.

3. Record a flame graph

Record the selected process for 120 seconds:

py-spy record --pid 12345 --output ad-connector-profile.svg --duration 120
Option Meaning
--pid / -p PID of the running Python process
--output / -o Output file
--duration / -d Recording duration in seconds

The default output format is an SVG flame graph. The .svg extension makes the file type explicit; the original filename graph0 would not change the default format.

Capture the profile while the problem is occurring. Transfer the resulting file securely to your workstation and open it in a browser.

Interpret the result

  • Wider frames represent a larger share of the collected samples.
  • Vertically stacked frames represent the call hierarchy.
  • The horizontal axis is not a chronological timeline.
  • A wide frame identifies a frequently sampled code path, not necessarily the root cause.

Idle threads are excluded by default. If the connector appears to be waiting or hanging, take a stack snapshot or include idle threads in a separate recording:

py-spy record --pid 12345 --output ad-connector-idle.svg --duration 120 --idle

Profiles recorded with --idle include waiting threads and must not be interpreted as CPU usage alone.

4. Inspect live activity or capture a stack snapshot

Live view

py-spy top --pid 12345

This displays a continuously updated overview of sampled functions. Press Ctrl+C to exit the profiler.

Stack snapshot

py-spy dump --pid 12345

This prints the current Python stack for each thread and can help locate where an apparently stuck process is waiting.

To save the snapshot:

py-spy dump --pid 12345 > ad-connector-stacks.txt

Avoid --locals unless explicitly required: local variables can contain credentials or other sensitive data.

Troubleshooting and handling results

  • Permission denied: Run as root on the UCS host. Additional security restrictions may still prevent access; do not disable them indiscriminately.
  • Process not found: Check the PID again. A service restart changes the PID.
  • Few or no samples: Confirm that the problem occurred during recording. For waiting processes, use dump or record with --idle.
  • Python version or interpreter errors: Check compatibility with py-spy 0.4.2 before proceeding.

For AD Connector investigations, correlate the recording period with:

  • /var/log/univention/connector-ad.log
  • /var/log/univention/connector-ad-status.log

Treat profiles and stack dumps as potentially sensitive diagnostic data. Share them only through approved channels.

When finished, leave the profiler’s virtual environment:

deactivate

This affects only the current shell and does not stop or restart the connector.

References

1 Like