How to handle high volume, high cardinality event streams

high volume and high cardinality meter events are defined by events that occur in high volume (billions or more per day) with events carrying dimension values that are high in cardinality (i e have many unique values) as an example, consider a meter which measures the monthly active users and has a dimension ‘billingid’ for a single customer id there could be over 100 million unique billingids per day (number of unique values to the billingid dimension) this would be an example of a high volume, high cardinality meter to support use cases involving these meters, we have two options for the metering pipeline performance and capabilities, depending on the use case requirements and budget realtime high volume, high cardinality meter pipeline to process a real time, time series aggregation by dimension values, with a meter pipeline having this ratio of volume to high cardinality is a heavy lift, but it is possible using amberflo to account for the heavier load and backend resource consumption, we’ve created the high cardinality meter events pricing tier of $4,500 per 1 billion events high throughput, unique meter pipeline with the goal of keeping the costs low and making this processing viable and cost effective for all customers, we have designed and implemented a net new meter processing pipeline this new pipeline lowers the costs significantly but does involve some tradeoffs (relative to the original realtime meter processing pipeline) 📘 faqs and how tos docid\ bq14rede1rrwst00mbikx see related from metering and event ingestion how to measure a multi step process using a duration meter docid\ jejnqgjekdtlycfyxzsjj how to track seat based objects in amberflo docid\ k8g0vmh6adgukb3dmezay how to meter the number of active seats or users for a customer docid\ vyzt7z7cdkrhjyl avl68 how to understand dimensions in amberflo docid\ ai7gyga p lhj8fwzca f how to verify the accuracy of meter data docid\ ne5rit1fjkfpatdbiej s how to track and bill for "high watermark" usage docid\ g l0lgdqyvfnccrsv2xbr how to handle erroneous or incorrect meters sent to amberflo docid\ awzcv2cw6swss2ffmcl58 how to find the meter id for retrieving and deleting a specific meter docid\ ty1vub6lz6u3fo nbfgb6 how to delete meters that are no longer in use docid\ x x7d16c6dwqrz zfdyvk