Debugging Guide

This guide provides systematic debugging procedures for diagnosing issues with License Monitor and License Server Detail.

Debug Mode

Enabling Debug Output

Command Line

# Run License Monitor in debug mode
/usr/local/bin/license_monitor \
    --config /etc/license-monitor/config.toml \
    --debug \
    --level debug

Environment Variable

export LICENSE_MONITOR_LOG_LEVEL=debug
systemctl restart license-monitor

Config File

config.toml

[daemon]
log_level = "debug"
console_log_level = "debug"

Debug Log Levels

Level	Use Case
error	Production, errors only
warn	Production, warnings and errors
info	Normal operation visibility
debug	Troubleshooting
trace	Deep debugging (verbose)

Request Tracing

Using Request IDs

Every API request includes a correlation ID:

# Make request and capture ID
curl -v -H "X-API-Key: KEY" http://localhost:8080/api/health 2>&1 | \
    grep "X-Request-ID"

# Output: < X-Request-ID: req-abc123

# Find in logs
grep "req-abc123" /var/log/license-monitor/license_monitor.log

Adding Custom Request ID

# Send custom request ID
curl -H "X-API-Key: KEY" \
     -H "X-Request-ID: my-trace-123" \
     http://localhost:8080/api/health

# Find in logs
grep "my-trace-123" /var/log/license-monitor/license_monitor.log

Network Debugging

Connection Testing

1. Test DNS resolution

   nslookup license-server.example.com
   dig license-server.example.com

2. Test TCP connectivity

   nc -zv license-server.example.com 27000
   telnet license-server.example.com 27000

3. Test HTTP connectivity

   curl -v http://localhost:8080/api/health

4. Capture network traffic

   tcpdump -i any port 8080 -w capture.pcap
   # Analyze with Wireshark

HTTP Debug

# Verbose HTTP request
curl -v -H "X-API-Key: KEY" http://localhost:8080/api/licenses

# Include timing
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/api/health

# curl-format.txt:
#     time_namelookup:  %{time_namelookup}s\n
#     time_connect:     %{time_connect}s\n
#     time_appconnect:  %{time_appconnect}s\n
#     time_pretransfer: %{time_pretransfer}s\n
#     time_starttransfer: %{time_starttransfer}s\n
#     time_total:       %{time_total}s\n

Process Debugging

Process Status

# Find process
pgrep -a license_monitor

# Check process details
ps aux | grep license_monitor

# Check open files
lsof -p $(pgrep license_monitor)

# Check network connections
ss -tlnp | grep license_monitor
netstat -tlnp | grep license_monitor

Resource Usage

# Real-time monitoring
top -p $(pgrep license_monitor)

# Memory details
cat /proc/$(pgrep license_monitor)/status | grep -E "(VmSize|VmRSS|VmPeak)"

# File descriptors
ls -la /proc/$(pgrep license_monitor)/fd | wc -l
cat /proc/$(pgrep license_monitor)/limits | grep "open files"

strace Debugging

# Trace system calls
strace -p $(pgrep license_monitor) -f 2>&1 | tee strace.log

# Trace specific calls
strace -p $(pgrep license_monitor) -e trace=network

# Trace file operations
strace -p $(pgrep license_monitor) -e trace=file

Configuration Debugging

Validate Configuration

# Check TOML syntax
/usr/local/bin/license_monitor --config /etc/license-monitor/config.toml --validate

# Or with Python
python3 -c "import toml; toml.load('/etc/license-monitor/config.toml')"

Print Effective Configuration

# Show merged configuration
/usr/local/bin/license_monitor --config /etc/license-monitor/config.toml --dump-config

Environment Variable Debugging

# Show all relevant env vars
env | grep -E "(LICENSE_MONITOR|OTEL|LOG)"

# Check systemd environment
systemctl show license-monitor --property=Environment

License Server Debugging

Test License Commands

# Test FlexLM
lmstat -a -c 27000@server

# Test RLM
rlmstat -a -c 5053@server

# Test with timeout
timeout 30 lmstat -a -c 27000@server

Parse Script Debugging

# Test parse script manually
lmstat -a -c 27000@server | python3 /etc/license-monitor/servers/parse.py

# Debug parse script
lmstat -a -c 27000@server | python3 -c "
import sys
import json

data = sys.stdin.read()
print('Input length:', len(data))
print('First 500 chars:', data[:500])
"

Database Debugging

Convex Debugging

# Check Convex connection
npx convex dashboard

# View Convex logs
npx convex logs

# Check function execution
npx convex functions

Browser Debugging

Developer Tools

1. Open DevTools: Press F12

2. Check Console tab for JavaScript errors

3. Check Network tab:

   • Filter by XHR/Fetch for API calls
   • Filter by WS for WebSocket
   • Look for red (failed) requests

4. Check Application tab:

   • Cookies for session data
   • Local/Session Storage for cached data

WebSocket Debugging

// In browser console
const ws = new WebSocket('ws://localhost:8080/ws/logs?api_key=KEY');
ws.onopen = () => console.log('Connected');
ws.onmessage = (e) => console.log('Message:', e.data);
ws.onerror = (e) => console.log('Error:', e);
ws.onclose = (e) => console.log('Closed:', e.code, e.reason);

Creating Debug Reports

Collecting Information

debug-report.sh

#!/bin/bash
REPORT_DIR="/tmp/license-monitor-debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$REPORT_DIR"

echo "Collecting system info..."
uname -a > "$REPORT_DIR/system.txt"
cat /etc/os-release >> "$REPORT_DIR/system.txt"

echo "Collecting service status..."
systemctl status license-monitor > "$REPORT_DIR/service-status.txt" 2>&1

echo "Collecting recent logs..."
journalctl -u license-monitor -n 500 > "$REPORT_DIR/journal.log" 2>&1
tail -500 /var/log/license-monitor/license_monitor.log > "$REPORT_DIR/app.log" 2>&1

echo "Collecting config (sanitized)..."
sed 's/api_key = .*/api_key = "[REDACTED]"/' /etc/license-monitor/config.toml > "$REPORT_DIR/config.toml"

echo "Collecting network info..."
ss -tlnp > "$REPORT_DIR/network.txt"
iptables -L -n > "$REPORT_DIR/firewall.txt" 2>&1

echo "Creating archive..."
tar czf "$REPORT_DIR.tar.gz" -C /tmp "$(basename $REPORT_DIR)"

echo "Debug report: $REPORT_DIR.tar.gz"

Caution

Always sanitize debug reports before sharing. Remove API keys, passwords, and sensitive data.

Next Steps

• Log Analysis - Understanding log formats
• Common Issues - Frequent problems
• Performance - Performance debugging