Firewall telemetry is high-volume, write-once, and almost never updated after the fact. That profile is exactly what column-store OLAP databases like ClickHouse are built for — and exactly what row-store operational databases are not.
ClickHouse is an open-source OLAP (online analytical processing) database. Instead of storing each row contiguously like a traditional operational database, it stores each column contiguously. For workloads that scan and aggregate — "how many events per source IP in the last 30 days" — that layout means only the relevant columns are read off disk, not entire rows of unrelated fields.
Firewall verdicts, DNS queries, and traffic events are written once and essentially never updated — a natural fit for ClickHouse's MergeTree storage engine, which is optimized for high-throughput inserts.
Counting, summing, and computing distinct values over millions or billions of rows completes in milliseconds to low seconds — the exact operations behind trend dashboards and reports.
ClickHouse scales by adding nodes and sharding data, so growing event volume from more gateways or longer retention windows doesn't require re-architecting the pipeline.
Queries are written in familiar SQL, so building custom reports or feeding a BI tool doesn't require learning a proprietary query language.
| Characteristic | PostgreSQL / SQLite (operational) | ClickHouse (analytical) |
|---|---|---|
| Primary use | Rule state, threat indicators, config — small, frequently-updated rows | Historical event streams — large volume, write-once |
| Query pattern | Point lookups by ID, small transactional writes | Aggregations across millions/billions of rows |
| Contention risk | Analytical scans would compete with live rule-lookup traffic | Isolated from the operational path entirely |
| Growth profile | Bounded — rules and indicators don't grow unbounded | Unbounded — designed for continuously growing event history |
L3/L4 packet verdicts — allow, block, drop — with source/destination, rule matched, and timestamp.
L7 enforcement decisions from the application-layer engine, correlated against the same time axis as firewall events.
Query volume by domain, client, and resolver — useful for both security investigation and DNS capacity planning.
Periodic network traffic summaries pulled on a scheduled cron, giving longer-horizon bandwidth and host activity trends.
Kubernetes CIS compliance observation events, stored alongside network telemetry for unified reporting.
All sources are routed through NATS JetStream before landing in ClickHouse, so ingestion survives restarts without dropping events.
Most firewall products treat event history as a byproduct, not a first-class store — and the difference shows up the moment you need to ask a question spanning more than a day or two.
| Common approach | Where it breaks down |
|---|---|
| Flat log files, rotated and discarded | Once a log file rotates out, that history is gone — there's no way to ask "what happened three months ago" because the data simply doesn't exist anymore |
| Events written into the operational database | Row-store databases tuned for fast rule lookups slow down or contend with live traffic decisions once historical event tables grow into the tens of millions of rows |
| Shipping raw logs to a generic log platform | General-purpose log search tools are built for text search, not columnar aggregation — a "sum bytes by source IP over 90 days" query is exactly what they're worst at |
| No cross-source correlation | Firewall logs, DNS logs, and application logs living in separate silos mean tracing a single request end-to-end requires manually stitching together three different tools |
ClickHouse's columnar design isn't a generic choice — it's specifically suited to the append-only, aggregate-query-heavy pattern that firewall and DNS telemetry produces.
Analytics queries run against ClickHouse, not the operational rules database — heavy historical reporting never competes with live rule-lookup performance.
Firewall, application-layer, DNS, traffic, and compliance events all land in the same store on the same time axis — cross-source correlation is a SQL join, not a manual cross-reference project.
NATS JetStream-backed ingestion means a brief ClickHouse or Ramen outage doesn't silently drop events — history stays complete even through restarts and maintenance windows.