How to handle high volume, high cardinality event streams
High volume and high cardinality meter events are defined by events that occur in high volume (billions or more per day) with events carrying dimension values that are high in cardinality (i.e. have many unique values).
As an example, consider a meter which measures the monthly active users and has a dimension ‘billingID’. For a single Customer ID there could be over 100 million unique billingIDs per day (number of unique values to the BillingID dimension). This would be an example of a high volume, high cardinality meter. To support use cases involving these meters, we have two options for the metering pipeline performance and capabilities, depending on the use case requirements and budget.
Realtime high-volume, high cardinality meter pipeline
To process a real time, time series aggregation by dimension values, with a meter pipeline having this ratio of volume to high cardinality is a heavy lift, but it is possible using Amberflo. To account for the heavier load and backend resource consumption, we’ve created the High Cardinality Meter Events pricing tier of $4,500 per 1 billion events.
High throughput, unique meter pipeline
With the goal of keeping the costs low and making this processing viable and cost-effective for all customers, we have designed and implemented a net-new Meter Processing Pipeline. This new pipeline lowers the costs significantly but does involve some tradeoffs (relative to the original realtime meter processing pipeline).
| Capability | Realtime high volume, high cardinality pipeline | High throughput, unique meter pipeline |
| Freshness of data in usage queries | Realtime | Less than 4 hours |
| Backfilling meter events | Supported | Not supported. The events will be reported with the current ingest time and not with the event time. This can be adjusted manually. |
| Filtering rules (correcting meter events after ingestion) | Supported via API to filter/cancel events by timestamp, customerID, etc. | Not supported. Can be done one-off manually. |
| Event-level tracing | Supported | Not supported |
| Pricing plans and metered invoices | Supported | Support coming soon |
| Relative cost | Higher | Lower (discounted) |
See related from Metering and Event Ingestion:
- How to measure a multi-step process using a duration meter
- How to track seat-based objects in Amberflo
- How to meter the number of active seats or users for a customer
- How to understand dimensions in Amberflo
- How to verify the accuracy of meter data
- How to track and bill for "high watermark" usage
- How to handle erroneous or incorrect meters sent to Amberflo
- How to find the meter-id for retrieving and deleting a specific meter
- How to delete meters that are no longer in use