How to handle high volume, high cardinality event streams
High volume and high cardinality meter events are defined by events that occur in high volume (billions or more per day) with events carrying dimension values that are high in cardinality (i.e. have many unique values).
As an example, consider a meter which measures the monthly active users and has a dimension ‘billingID’. For a single Customer ID there could be over 100 million unique billingIDs per day (number of unique values to the BillingID dimension). This would be an example of a high volume, high cardinality meter. To support use cases involving these meters, we have two options for the metering pipeline performance and capabilities, depending on the use case requirements and budget.
To process a real time, time series aggregation by dimension values, with a meter pipeline having this ratio of volume to high cardinality is a heavy lift, but it is possible using Amberflo. To account for the heavier load and backend resource consumption, we’ve created the High Cardinality Meter Events pricing tier of $4,500 per 1 billion events.
With the goal of keeping the costs low and making this processing viable and cost-effective for all customers, we have designed and implemented a net-new Meter Processing Pipeline. This new pipeline lowers the costs significantly but does involve some tradeoffs (relative to the original realtime meter processing pipeline).
See related from Metering and Event Ingestion:
