How to handle high volume, high cardinality event streams

High volume and high cardinality meter events are defined by events that occur in high volume (billions or more per day) with events carrying dimension values that are high in cardinality (i.e. have many unique values).

As an example, consider a meter which measures the monthly active users and has a dimension ‘billingID’. For a single Customer ID there could be over 100 million unique billingIDs per day (number of unique values to the BillingID dimension). This would be an example of a high volume, high cardinality meter. To support use cases involving these meters, we have two options for the metering pipeline performance and capabilities, depending on the use case requirements and budget.

Realtime high-volume, high cardinality meter pipeline

To process a real time, time series aggregation by dimension values, with a meter pipeline having this ratio of volume to high cardinality is a heavy lift, but it is possible using Amberflo. To account for the heavier load and backend resource consumption, we’ve created the High Cardinality Meter Events pricing tier of $4,500 per 1 billion events.

High throughput, unique meter pipeline

With the goal of keeping the costs low and making this processing viable and cost-effective for all customers, we have designed and implemented a net-new Meter Processing Pipeline. This new pipeline lowers the costs significantly but does involve some tradeoffs (relative to the original realtime meter processing pipeline).

Capability Realtime high volume, high cardinality pipeline High throughput, unique meter pipeline
Freshness of data in usage queriesRealtimeLess than 4 hours
Backfilling meter eventsSupportedNot supported. The events will be reported with the current ingest time and not with the event time. This can be adjusted manually.
Filtering rules (correcting meter events after ingestion)Supported via API to filter/cancel events by timestamp, customerID, etc.Not supported. Can be done one-off manually.
Event-level tracingSupportedNot supported
Pricing plans and metered invoices SupportedSupport coming soon
Relative cost HigherLower (discounted)


Return to FAQs

See related from Metering and Event Ingestion: