SigNoz Troubleshooting: OTLP, ClickHouse & Missing Traces

You had SigNoz running, data was coming in, everything looked fine, then something broke. Traces stopped showing up, ClickHouse started pushing your disk, or a Docker Compose change took down half the stack. This is what production observability really looks like, and where SigNoz deployment troubleshooting starts to matter.

This guide focuses on what happens after real traffic hits your system and things fail in new ways. You will see how to handle the main problems:

• OTLP ingestion issues
• ClickHouse startup loops and disk pressure
• Missing traces from bad exporter configs
• Components failing after Docker Compose changes

If you haven't set up SigNoz yet, start with the self-hosted SigNoz installation guide before continuing SigNoz deployment troubleshooting.

When you first install SigNoz, everything works on low volume. The problems come later, when your services generate real spans, your ClickHouse fills up, or someone changes a config file, and suddenly nothing comes in.

SigNoz deployment troubleshooting in production means looking at three layers:

• Instrumentation: Is your app actually generating trace data?
• Ingestion: Is the OTel Collector receiving and forwarding data?
• Storage: Is ClickHouse healthy and able to write?

Knowing which layer is broken cuts your debugging time in half. This guide covers all three.

Before changing any config, always start by checking the health of your containers. A single unhealthy container can silently kill data flow across your whole stack.

Run this to see all container statuses:

You're looking for anything that shows unhealthy, exited, or a high restart count. The most common culprits are signoz-clickhouse, signoz-otel-collector, and the query service.

To see detailed logs from a specific container, you can use:

Pay attention to error messages, connection refused errors, and anything that says failed to export. These logs tell you exactly where data is failing.

Note: SigNoz needs at least 4 GB of RAM available to Docker. If containers keep restarting with no clear error, try increasing Docker's memory limit first.

This is the most common problem in SigNoz deployment troubleshooting. Your app sends data, but nothing shows up in the UI. Here's how to fix the issue.

The OTel Collector listens on two ports:

• 4317: OTLP gRPC
• 4318: OTLP HTTP

Check if these ports are active:

If they're not showing as LISTEN, the collector isn't running or isn't bound correctly. Check your docker-compose.yaml to make sure the ports are exposed:

From the machine running your app, run a quick connectivity check:

If you get a connection refused or timeout, the problem is network-level, a firewall rule, a Docker network config, or the collector isn't running.

When you can reach the port but traces still don't appear, enable debug logs to see exactly what the collector is receiving:

Add this to your otel-collector-config.yaml, then restart:

Look for lines like "Traces received" or errors like "failed to export". This is the fastest way to confirm whether data is arriving at all.

One of the most common SigNoz deployment troubleshooting mistakes is a wrong endpoint URL in the exporter config. For self-hosted SigNoz, your app's exporter should point to:

A common mistake is mixing up gRPC and HTTP ports. If your exporter is configured for OTLP HTTP but pointing at port 4317, it will fail silently or with a confusing error. Always double-check that the protocol matches the port.

A bad collector config is one of the main reasons SigNoz deployment troubleshooting gets really confusing. The collector will start but silently drop data if the pipeline is misconfigured.

Here is a minimal collector config that works with self-hosted SigNoz:

The most important thing here is the tls.insecure: true setting. Without it, the collector will try to make a TLS connection to SigNoz's internal collector and fail.

If you're not sure whether trace data are even reaching the pipeline, add a debug exporter temporarily:

Now check the collector logs. If you see trace data in the debug output but nothing in SigNoz, the problem is in the exporter, the endpoint is wrong, or the internal SigNoz collector isn't accepting the data. If you see nothing in debug, the problem is in instrumentation.

A pipeline that's missing a receiver, processor, or exporter reference will silently fail. Make sure every component declared in receivers:, processors:, and exporters: is also referenced in service.pipelines. It's a common copy-paste mistake.

For more details on all receiver, processor, and exporter options, see the official OpenTelemetry Collector Configuration.

Missing traces are a classic SigNoz deployment troubleshooting scenario. The setup looks right, the collector seems healthy, but traces don't appear in the UI.

Set this environment variable in your app before starting it:

If you don't see traces printed to stdout, your instrumentation is not working at all. This is an app-level problem, not a SigNoz problem. Fix your SDK setup before going further.

SigNoz groups traces by service.name. If this attribute is missing or different from what you expect, your traces are there, just grouped under a different service. Check the Traces tab and look for unexpected service names.

Every instrumented app must set this:

If you added a head-based sampler to your collector to reduce volume, you may have accidentally set the sample rate too low. A probabilistic sampler set to 0.01 means only 1% of traces get through. Check your processor config:

Temporarily set this to 100% during debugging, then tune it back when data is confirmed.

It sounds obvious, but check the time range in the SigNoz Traces UI. If your app only generated traces 30 minutes ago and the filter is set to "Last 15 minutes", you won't see them. Widen the time window first before assuming data is missing.

ClickHouse startup failures are a serious part of SigNoz deployment troubleshooting. When ClickHouse gets stuck in a restart loop or refuses to start, SigNoz has nowhere to write data.

On first startup, SigNoz runs an init container that downloads a helper binary (histogram-quantile) from GitHub. If your server can't reach GitHub, due to a firewall or network restriction, this job hangs forever, and everything else waits.

Check if this is your problem:

If you see the container stuck on a download, fix your network access to GitHub or manually pre-download the binary and mount it into the container.

If ClickHouse starts as unhealthy after a system reboot or a Docker restart, it's often because parts files got corrupted during the previous shutdown. The fix is to trigger a restore:

This tells ClickHouse to attempt data recovery on the next startup. It will scan and repair broken parts. Watch the logs during this process; you'll see messages about parts being fixed.

Sometimes, ClickHouse refuses to start because it finds too many broken parts and stops to protect your data. The log will show something like "Too many broken parts".

You can override this by increasing the limit in your ClickHouse config:

Set this only while recovering. Return it to the default (10) after ClickHouse starts and repairs itself.

High disk usage starts as a slow problem but turns into a serious SigNoz deployment troubleshooting issue when the disk reaches 100%, and ClickHouse stops writing new data.

If you're running high telemetry volume on a shared server, consider a dedicated server hosting to give ClickHouse dedicated resources and avoid competing with other workloads.

First, connect to ClickHouse and run this query to find the biggest databases:

If you see the system database near the top, the problem is ClickHouse's own internal logs, not your application data.

ClickHouse stores internal logs in tables like zookeeper_log, query_log, and part_log. These grow without limit by default. You can find the exact culprit with:

This limits internal logs to 7 days and prevents endless growth.

For your traces, metrics, and logs, set a reasonable TTL that matches your retention needs:

For most teams, 7 to 30 days for traces and 30 to 90 days for metrics is a good starting point. You can tune this as you learn your actual usage patterns.

This is a common but overlooked part of SigNoz deployment troubleshooting. You update your docker-compose.yaml, restart the stack, and suddenly, things that worked before are broken.

When you change service names or remove services from Compose, old containers can delay and conflict with new ones. Always use the --remove-orphans flag:

If you deleted and recreated a container but kept the old Docker volume, the new container might be reading stale data or schema from a previous version. Check your named volumes:

If you're doing a fresh reinstall after a major version change, you may need to remove old volumes:

Only do this if you intentionally want to start with a clean database.

After editing Compose files, check if any new port bindings conflict with existing services on the host:

Port 9000 is used by ClickHouse, and port 3301 is the SigNoz UI. If anything else on the host is using these ports, SigNoz containers will fail to bind and won't start properly.

When you've fixed individual components but still aren't sure everything is working, do a full pipeline validation. This is the final step in any serious SigNoz deployment troubleshooting session.

1. Send a Test Trace Manually:

You can use telemetrygen to send a test trace directly to your collector:

If this test trace shows up in SigNoz, your collector and ClickHouse are working. The problem is in your app's instrumentation. If the test trace doesn't show up, the problem is at the collector or storage level.

2. Check the Query Service:

The query service is what the SigNoz frontend talks to. If ClickHouse is healthy but the UI shows nothing, check the query service logs:

Look for connection errors to ClickHouse, schema migration failures, or authentication errors. Sometimes after a version upgrade, the query service needs to run migrations before it can read from the new schema.

When your telemetry volume grows, SigNoz deployment troubleshooting often runs into hardware limits. ClickHouse uses a lot of memory on purpose because it caches data to make queries fast.

On a shared VPS, this puts constant pressure on RAM and disk.

If telemetry is pushing your current server to disk and memory limits, a better option is to move ClickHouse to its own machine or switch to a PerLod dedicated server where the resources are not shared. This removes one of the biggest bottlenecks in high-volume SigNoz setups.

For a related approach to centralized log infrastructure using lightweight collectors, see this guide on Grafana Alloy and Loki for Docker logs. It shows how to offload log collection from your app containers using a separate pipeline, which reduces load on SigNoz itself.

SigNoz deployment troubleshooting is mostly about checking the right places in the right order. Most problems fit into three simple groups:

• The app is not sending any telemetry.
• The OTel Collector is misconfigured or not reachable.
• ClickHouse is under heavy load or has broken data parts.

Start by checking container health and turning on debug logs in the collector
Then confirm your pipelines are wired correctly in the config.

For ClickHouse, use the system.parts table to see which tables are using the most disk, and add TTL rules before the disk is full. For missing traces, make sure service.name is set and that your sampling settings are not dropping too much data.

Work through these layers one by one, and your SigNoz stack will return to a healthy state.