Case Study: Building an AI-Powered Observability Platform on AWS

This case study walks through a platform we designed, built, and operate: a multi-tenant, AI-powered observability SaaS for engineering teams — built entirely on AWS. It is a useful reference for anyone planning a data-heavy, multi-tenant product, because it touches almost every hard problem in that space: high-volume telemetry ingestion, tenant isolation you can actually prove, a database migration driven by unit economics, and an AI layer that does real work rather than decorating a dashboard.

The problem

Observability pricing is broken for small and mid-size engineering teams. The dominant vendors charge per gigabyte ingested, which means a team's monitoring bill grows with traffic rather than with value — and a single noisy deploy can produce a four-figure surprise. The product thesis: correlate findings across the tools a team already runs, add an AI intelligence layer on top, and charge a flat subscription. The engineering consequence of that thesis is unforgiving — the platform's own infrastructure cost per tenant has to stay low and predictable, or the pricing model collapses.

Architecture

Ingestion: per-tenant OpenTelemetry collectors

Telemetry arrives through per-tenant OpenTelemetry Collector deployments running on Amazon EKS. Each collector stamps every span, log record, and metric with a tenant_id resource attribute before anything is written. That decision — stamping identity at the edge, per tenant, rather than trusting application code to label its own data — is the foundation the rest of the isolation story stands on.

Storage: ClickHouse, self-hosted on EKS

Telemetry lands in ClickHouse: columnar storage, sub-second aggregations over billions of rows, and a natural fit for metrics and logs. We initially used a managed ClickHouse cloud service to move fast — and then the unit economics asserted themselves. The managed trial burned roughly $270 in twelve days on pure compute, with no idle suspend, for a workload that was still pre-revenue. That number doesn't work under flat-subscription pricing.

So we migrated to self-hosted ClickHouse on EKS, run by a Kubernetes operator, with two storage tiers: hot data on gp3 EBS volumes and cold data tiered automatically to Amazon S3 by TTL policy. TLS termination uses an in-cluster certificate authority managed by cert-manager. The migration was verified live — collectors and backend cut over with no data loss, rebuilding from live traffic — and the cost profile changed from "metered by someone else's margin" to "a node we control."

Tenant isolation: enforced by the database, not the application

Most SaaS products enforce tenant isolation in application code — a WHERE tenant_id = ? the developers promise to always include. We consider that insufficient for a product whose entire content is customers' operational data. Here, every tenant gets a distinct ClickHouse user with a row policy that scopes it to its own tenant_id at the database layer. The query builder's tenant predicate still exists, but as a second line of defense — if application code has a bug, the database itself refuses to return another tenant's rows.

Write credentials are separated the same way: each tenant's collector uses its own INSERT-only database user, because a shared write credential is a cross-tenant spoofing surface. And critically, isolation was proven empirically, not by design review: we query as tenant A and confirm tenant B's rows do not come back, as a repeatable test, after every infrastructure change. A security claim you haven't tried to falsify is a hope, not a control.

The rest of the platform

• Backend: Go, running on EKS behind a shared Application Load Balancer, deployed with Helm.
• Frontend: a React dashboard with a findings-first design — incidents, services, and exploration views.
• Auth: Amazon Cognito with a custom in-app sign-in flow.
• Billing: Stripe subscriptions with checkout, customer portal, and webhooks.
• Control plane: a separate Postgres RDS instance for tenant configuration — deliberately isolated from the telemetry path.
• Secrets: AWS Secrets Manager end to end, synced into the cluster; per-tenant credentials are fetched at runtime, never baked into config.

The AI layer: Amazon Bedrock doing real work

The AI features are the product's reason to exist, and all of them run on Amazon Bedrock — no third-party AI vendor in the data path, which matters to customers who are already cautious about where their logs go. The shipped capabilities include:

• Natural-language log querying — plain-English questions translated into structured queries against the telemetry store.
• One-click log summarization — an incident report generated from a window of error logs.
• Semantic log search — embedding-based search that finds issues by meaning when you don't know the right keyword.
• Anomaly detection and predictive alerting — statistical methods (Z-score, MAD, rate-of-change) plus time-to-threshold forecasting, so alerts fire before a limit is breached rather than after.
• Alert correlation and root-cause analysis — related alerts grouped into one incident with an AI explanation of the likely cause and next steps.
• Infrastructure auto-discovery — the platform scans a customer's AWS account (ECS, EC2, RDS, Lambda, load balancers, queues) and generates monitoring configuration and IaC for the gaps it finds.

A design principle worth stealing: start with statistical methods, layer in ML where it earns its place, and treat the LLM as the interface and the analyst — not as a magic box the whole product depends on.

Results

• Analytics cost moved from a metered external service to a self-hosted cluster with S3 cold storage — the difference between a pricing model that works and one that doesn't.
• Cross-tenant isolation is provable on demand, enforced in the database and re-verified empirically after every infrastructure change.
• Sub-second query performance over telemetry at volume, on a single well-configured node.
• Full-signal instrumentation of real production services flows through the platform end to end — traces, logs, and metrics, with PII redaction enforced at a single chokepoint in the log pipeline.

What this project generalizes to

If you're building a multi-tenant data product, three lessons from this build apply almost universally. First, enforce isolation where the data lives, not just where your code queries it — and test it like an attacker would. Second, watch unit economics from day one; the right database choice is inseparable from the pricing model. Third, AI features are credible when they sit on a disciplined data foundation — the model is the easy part.

Cloud Development Group designed, built, and operates this platform end to end: architecture, Kubernetes and data infrastructure, the AI layer, and the ongoing cost engineering. If you're planning something in this shape — multi-tenant, data-heavy, AI on top — we've made the mistakes already, and we'd be glad to help you skip them.