You had SigNoz running, data was coming in, everything looked fine, then something broke. Traces stopped showing up, ClickHouse started pushing your disk, or a Docker Compose change took down half the stack. This is what production observability really looks like, and where SigNoz deployment troubleshooting starts to matter.

This guide focuses on what happens after real traffic hits your system and things fail in new ways. You will see how to handle the main problems:

• OTLP ingestion issues
• ClickHouse startup loops and disk pressure
• Missing traces from bad exporter configs
• Components failing after Docker Compose changes

If you haven’t set up SigNoz yet, start with the self-hosted SigNoz installation guide before continuing SigNoz deployment troubleshooting.

Why SigNoz Breaks in Production

When you first install SigNoz, everything works on low volume. The problems come later, when your services generate real spans, your ClickHouse fills up, or someone changes a config file, and suddenly nothing comes in.

SigNoz deployment troubleshooting in production means looking at three layers:

• Instrumentation: Is your app actually generating trace data?
• Ingestion: Is the OTel Collector receiving and forwarding data?
• Storage: Is ClickHouse healthy and able to write?

Knowing which layer is broken cuts your debugging time in half. This guide covers all three.

First Step: Check Your SigNoz Containers

Before changing any config, always start by checking the health of your containers. A single unhealthy container can silently kill data flow across your whole stack.

Run this to see all container statuses:

docker compose ps -a

You’re looking for anything that shows unhealthy, exited, or a high restart count. The most common culprits are signoz-clickhouse, signoz-otel-collector, and the query service.

To see detailed logs from a specific container, you can use:

docker compose logs -f signoz-otel-collector
docker compose logs -f signoz-clickhouse

Pay attention to error messages, connection refused errors, and anything that says failed to export. These logs tell you exactly where data is failing.

Note: SigNoz needs at least 4 GB of RAM available to Docker. If containers keep restarting with no clear error, try increasing Docker’s memory limit first.

Fix OTLP Ingestion Issues

This is the most common problem in SigNoz deployment troubleshooting. Your app sends data, but nothing shows up in the UI. Here’s how to fix the issue.

Verify the Collector is Listening

The OTel Collector listens on two ports:

• 4317: OTLP gRPC
• 4318: OTLP HTTP

Check if these ports are active:

netstat -tuln | grep -E '4317|4318'

If they’re not showing as LISTEN, the collector isn’t running or isn’t bound correctly. Check your docker-compose.yaml to make sure the ports are exposed:

ports:
  - "4317:4317"   # gRPC
  - "4318:4318"   # HTTP

Test Connectivity from Your App

From the machine running your app, run a quick connectivity check:

curl -v http://<signoz-host>:4318/v1/traces

If you get a connection refused or timeout, the problem is network-level, a firewall rule, a Docker network config, or the collector isn’t running.

Enable Debug Logging on the Collector

When you can reach the port but traces still don’t appear, enable debug logs to see exactly what the collector is receiving:

service:
  telemetry:
    logs:
      level: debug

Add this to your otel-collector-config.yaml, then restart:

docker compose restart signoz-otel-collector
docker compose logs -f signoz-otel-collector

Look for lines like “Traces received” or errors like “failed to export“. This is the fastest way to confirm whether data is arriving at all.

Wrong Endpoint URL

One of the most common SigNoz deployment troubleshooting mistakes is a wrong endpoint URL in the exporter config. For self-hosted SigNoz, your app’s exporter should point to:

gRPC: http://<host>:4317
HTTP: http://<host>:4318

A common mistake is mixing up gRPC and HTTP ports. If your exporter is configured for OTLP HTTP but pointing at port 4317, it will fail silently or with a confusing error. Always double-check that the protocol matches the port.

Fix the OTel Collector Config

A bad collector config is one of the main reasons SigNoz deployment troubleshooting gets really confusing. The collector will start but silently drop data if the pipeline is misconfigured.

The Minimum Working Config

Here is a minimal collector config that works with self-hosted SigNoz:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  otlp:
    endpoint: signoz-otel-collector:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

The most important thing here is the tls.insecure: true setting. Without it, the collector will try to make a TLS connection to SigNoz’s internal collector and fail.

Add a Debug Exporter to Confirm Data Flow

If you’re not sure whether trace data are even reaching the pipeline, add a debug exporter temporarily:

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: signoz-otel-collector:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]

Now check the collector logs. If you see trace data in the debug output but nothing in SigNoz, the problem is in the exporter, the endpoint is wrong, or the internal SigNoz collector isn’t accepting the data. If you see nothing in debug, the problem is in instrumentation.

Check the Pipeline is Wired Correctly

A pipeline that’s missing a receiver, processor, or exporter reference will silently fail. Make sure every component declared in receivers:, processors:, and exporters: is also referenced in service.pipelines. It’s a common copy-paste mistake.

For more details on all receiver, processor, and exporter options, see the official OpenTelemetry Collector Configuration.

Fix Missing Traces

Missing traces are a classic SigNoz deployment troubleshooting scenario. The setup looks right, the collector seems healthy, but traces don’t appear in the UI.

Step 1: Confirm Instrumentation is Working

Set this environment variable in your app before starting it:

export OTEL_TRACES_EXPORTER=console

If you don’t see traces printed to stdout, your instrumentation is not working at all. This is an app-level problem, not a SigNoz problem. Fix your SDK setup before going further.

Step 2: Verify the service.name Attribute

SigNoz groups traces by service.name. If this attribute is missing or different from what you expect, your traces are there, just grouped under a different service. Check the Traces tab and look for unexpected service names.

Every instrumented app must set this:

# Python example
resource = Resource(attributes={"service.name": "my-api-service"})

# Environment variable
export OTEL_SERVICE_NAME=my-api-service

Step 3: Check for Sampling

If you added a head-based sampler to your collector to reduce volume, you may have accidentally set the sample rate too low. A probabilistic sampler set to 0.01 means only 1% of traces get through. Check your processor config:

processors:
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 10  # This means 90% of traces are dropped

Temporarily set this to 100% during debugging, then tune it back when data is confirmed.

Step 4: Time Range in the UI

It sounds obvious, but check the time range in the SigNoz Traces UI. If your app only generated traces 30 minutes ago and the filter is set to “Last 15 minutes“, you won’t see them. Widen the time window first before assuming data is missing.

Fix ClickHouse Startup Loop and Crash Recovery

ClickHouse startup failures are a serious part of SigNoz deployment troubleshooting. When ClickHouse gets stuck in a restart loop or refuses to start, SigNoz has nowhere to write data.

The Bootstrap Hang (histogram-quantile)

On first startup, SigNoz runs an init container that downloads a helper binary (histogram-quantile) from GitHub. If your server can’t reach GitHub, due to a firewall or network restriction, this job hangs forever, and everything else waits.

Check if this is your problem:

docker compose logs signoz-telemetrystore-clickhouse-user-scripts

If you see the container stuck on a download, fix your network access to GitHub or manually pre-download the binary and mount it into the container.

ClickHouse Unhealthy After Restart

If ClickHouse starts as unhealthy after a system reboot or a Docker restart, it’s often because parts files got corrupted during the previous shutdown. The fix is to trigger a restore:

# Shell into the ClickHouse container
docker exec -it signoz-clickhouse bash

# Create the force restore flag
touch /var/lib/clickhouse/flags/force_restore_data

# Exit and restart
exit
docker compose restart signoz-clickhouse

This tells ClickHouse to attempt data recovery on the next startup. It will scan and repair broken parts. Watch the logs during this process; you’ll see messages about parts being fixed.

ClickHouse and “max_suspicious_broken_parts” Error

Sometimes, ClickHouse refuses to start because it finds too many broken parts and stops to protect your data. The log will show something like “Too many broken parts“.

You can override this by increasing the limit in your ClickHouse config:

<clickhouse>
  <merge_tree>
    <max_suspicious_broken_parts>100</max_suspicious_broken_parts>
  </merge_tree>
</clickhouse>

Set this only while recovering. Return it to the default (10) after ClickHouse starts and repairs itself.

Fix High Disk Usage in ClickHouse

High disk usage starts as a slow problem but turns into a serious SigNoz deployment troubleshooting issue when the disk reaches 100%, and ClickHouse stops writing new data.

If you’re running high telemetry volume on a shared server, consider a dedicated server hosting to give ClickHouse dedicated resources and avoid competing with other workloads.

Find What’s Eating Your Disk

First, connect to ClickHouse and run this query to find the biggest databases:

SELECT
    database,
    sum(bytes_on_disk)/1024/1024/1024 AS size_gb
FROM system.parts
GROUP BY database
ORDER BY size_gb DESC;

If you see the system database near the top, the problem is ClickHouse’s own internal logs, not your application data.

The Hidden Storage Killer: system Tables

ClickHouse stores internal logs in tables like zookeeper_log, query_log, and part_log. These grow without limit by default. You can find the exact culprit with:

SELECT
    table,
    formatReadableSize(sum(data_compressed_bytes)) AS compressed_size,
    sum(rows) AS total_rows
FROM system.parts
WHERE database = 'system' AND active = 1
GROUP BY table
ORDER BY sum(data_compressed_bytes) DESC;

This limits internal logs to 7 days and prevents endless growth.

Set TTL on Application Data Tables

For your traces, metrics, and logs, set a reasonable TTL that matches your retention needs:

-- Set trace data to expire after 30 days
ALTER TABLE signoz_traces.distributed_signoz_index_v3
MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

For most teams, 7 to 30 days for traces and 30 to 90 days for metrics is a good starting point. You can tune this as you learn your actual usage patterns.

Fix Components Failing After Compose Changes

This is a common but overlooked part of SigNoz deployment troubleshooting. You update your docker-compose.yaml, restart the stack, and suddenly, things that worked before are broken.

Orphan Containers

When you change service names or remove services from Compose, old containers can delay and conflict with new ones. Always use the --remove-orphans flag:

docker compose down --remove-orphans
docker compose up -d

Volume Data Mismatch

If you deleted and recreated a container but kept the old Docker volume, the new container might be reading stale data or schema from a previous version. Check your named volumes:

docker volume ls | grep signoz

If you’re doing a fresh reinstall after a major version change, you may need to remove old volumes:

docker compose down -v   # WARNING: This deletes all volume data
docker compose up -d

Only do this if you intentionally want to start with a clean database.

Port Conflicts After Config Changes

After editing Compose files, check if any new port bindings conflict with existing services on the host:

sudo ss -tuln | grep -E '4317|4318|3301|9000'

Port 9000 is used by ClickHouse, and port 3301 is the SigNoz UI. If anything else on the host is using these ports, SigNoz containers will fail to bind and won’t start properly.

Validate the Full SigNoz Pipeline

When you’ve fixed individual components but still aren’t sure everything is working, do a full pipeline validation. This is the final step in any serious SigNoz deployment troubleshooting session.

1. Send a Test Trace Manually:

You can use telemetrygen to send a test trace directly to your collector:

docker run --rm \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:latest \
  traces \
  --otlp-endpoint <your-host>:4317 \
  --otlp-insecure \
  --duration 5s

If this test trace shows up in SigNoz, your collector and ClickHouse are working. The problem is in your app’s instrumentation. If the test trace doesn’t show up, the problem is at the collector or storage level.

2. Check the Query Service:

The query service is what the SigNoz frontend talks to. If ClickHouse is healthy but the UI shows nothing, check the query service logs:

docker compose logs -f signoz-query-service

Look for connection errors to ClickHouse, schema migration failures, or authentication errors. Sometimes after a version upgrade, the query service needs to run migrations before it can read from the new schema.

Scaling SigNoz to More Servers

When your telemetry volume grows, SigNoz deployment troubleshooting often runs into hardware limits. ClickHouse uses a lot of memory on purpose because it caches data to make queries fast.

On a shared VPS, this puts constant pressure on RAM and disk.

If telemetry is pushing your current server to disk and memory limits, a better option is to move ClickHouse to its own machine or switch to a PerLod dedicated server where the resources are not shared. This removes one of the biggest bottlenecks in high‑volume SigNoz setups.

For a related approach to centralized log infrastructure using lightweight collectors, see this guide on Grafana Alloy and Loki for Docker logs. It shows how to offload log collection from your app containers using a separate pipeline, which reduces load on SigNoz itself.

Conclusion

SigNoz deployment troubleshooting is mostly about checking the right places in the right order. Most problems fit into three simple groups:

• The app is not sending any telemetry.
• The OTel Collector is misconfigured or not reachable.
• ClickHouse is under heavy load or has broken data parts.

Start by checking container health and turning on debug logs in the collector
Then confirm your pipelines are wired correctly in the config.

For ClickHouse, use the system.parts table to see which tables are using the most disk, and add TTL rules before the disk is full. For missing traces, make sure service.name is set and that your sampling settings are not dropping too much data.

Work through these layers one by one, and your SigNoz stack will return to a healthy state.

FAQs