The infrastructure dashboard showed "Green across the board. Uptime: 99.97%." Yet a customer in Germany couldn't complete checkout for 40 minutes. P95 latency spiked to 14 seconds. No alerts fired. The monitoring measured the wrong metrics at the wrong thresholds. The company was blind to customer impact while believing everything was fine.
Observability is not about having dashboards. It is about asking questions and getting answers before your customers do.
Why this matters
Investors evaluate whether teams can detect customer impact quickly, trace failing requests across stacks within minutes, route alerts to informed personnel, and answer unanticipated production questions.
What investors assess
- Three Pillars: Metrics, logs, and traces must interconnect via correlation IDs, not exist separately.
- Signal Quality: Alerts should reflect customer impact (checkout latency exceeding SLO thresholds), not system internals (CPU at 80%).
- Structured Logs: JSON queryable records by tenant/request/service within 30 seconds—plain text logs signal maturity gaps.
- Distributed Tracing: Coverage across service boundaries including databases, external APIs, and async queues.
- SLOs as North Star: Service-level objectives shift from reactive to proactive monitoring using error budget consumption rates.
Stage-based expectations
- Seed/Early A: Basic metrics, centralized logging, Sev-1 alerts. Founders should articulate visibility gaps.
- Series B/Growth: Structured logging, distributed tracing on critical paths, per-service SLOs, context-aware alert routing with runbooks.
- Control Buyouts: Six months incident data correlated with observability artifacts; validation that alerts predicted problems.
Recommended patterns
- USE/RED Methods: Track utilization, saturation, errors (resources) and rate, errors, duration (services).
- Correlation IDs: Carry request identifiers through all service calls, queues, jobs, logs.
- Synthetic Monitoring: Simulate user transactions across regions to detect degradation pre-customer impact.
- Contextual Alerts: Link alerts to dashboards, runbooks, on-call contacts.
Actionable practices for revenue-critical journeys
- Define SLOs; track error budget weekly
- Build dashboards showing login, onboarding, core value action, billing
- Instrument top ten external dependencies (timeouts, errors, latency percentiles)
- Monthly alert hygiene review (remove alerts firing >10x without action)
- Track and publish on-call load per engineer
Red flags extending negotiations
- Infrastructure-layer dashboards lacking customer journey views
- Absent distributed tracing; manual log correlation without shared request identifiers
- Unstructured plain text logs
- SLOs defined but unmeasured; ignored error budgets
- On-call concentration on one–two engineers (key-person risk)
- Observability gaps at external boundaries (third-party APIs, payment processors, queues)
Two or more flags typically trigger remediation conditions or pricing discounts.
Glossary
- RED method: Rate, errors, duration—service-level metrics that reflect user experience.
- USE method: Utilization, saturation, errors—resource-level metrics.
- SLO: Service Level Objective—the target availability or performance a service commits to.
- Error budget: The allowed unreliability before an SLO is violated.
- Distributed tracing: End-to-end request flow visibility across multiple services.
- Correlation ID: A unique identifier following a request through all systems.
- Synthetic monitoring: Automated tests simulating user behavior to detect degradation.
- Alert fatigue: Loss of alert effectiveness due to too many false or irrelevant alerts.
Your turn
What observability gap hit you hardest? An outage your dashboards didn't catch? A P99 latency spike no alert predicted? Share the story—and what you learned from it.
Next in the Playbook
In Edition 25 we'll dive into Performance Engineering & Load Testing. The data you collect through observability becomes the input for understanding where to optimize. Stay tuned!
Originally published on the Tech Due Diligence Playbook newsletter on LinkedIn.