Metering Design Principles

core design principles of the amberflo cloud metering service platform a core design principle of the cloud metering service platform is guaranteed accuracy to achieve this, the system must be designed from the ground up with three key attributes accuracy idempotency data deduplication 1\ accuracy the data must be fully accurate to allow the metering system to serve as the single source of truth for usage and consumption other systems can rely on the completeness and correctness of amberflo’s metering data for critical functions such as invoicing, billing, and reporting 2\ idempotency each record must be processed once and only once it must not be skipped it must not be processed more than once it must be processed exactly one time, regardless of any system failures or retries example if a valid meter event is ingested and the system fails during processing, amberflo ensures that the event is still processed exactly once this eliminates the risk of dropped events or duplicate billing the idempotency guarantee is built into the amberflo cloud metering service by default 3\ data deduplication data deduplication is a technique for eliminating duplicate copies of repeated data this can be a common occurrence, depending on the source generating the meters, and it is challenging to guarantee deduplication if the processing system (cloud metering service platform) itself is a highly distributed stateless system in a distributed system, any part of the system or node can fail at any time, and if it fails mid stream as data is undergoing transformation or processing, it makes it very difficult to "guarantee" data de duplication in cloud metering, it is common for the source (client generating the meters) to send duplicate meters because the cloud metering service serves as the system of record and the single source of truth for usage and consumption data, duplicate records must be deduped data deduplication is a core platform tenet to deliver accurate usage data to downstream systems such as pricing, billing, and others amberflo cloud metering service platform provides you with an out of the box data deduplication guarantee here are some different ways how you can enforce data deduplication using amberflo the amberflo platform will not store duplicate data for the same meter record if you call the ingest api with the same record, the meter repository will only hold the first record and discard any subsequent records the deduplication key is based on all the meter attributes that means if one of the meter record attributes is different (either the record time or any dimension value) we will consider it as a different record 1\ unique id you can enforce deduplication by adding a unique identifier to each meter event using a dimension dimensions with unique id append({'name' 'unique id', 'value' str(uuid4())}) metering meter(options tenant, options meter name,int(options meter value), dimensions=dimensions with unique id) 2\ unique timestamp if we want to make sure we use one meter for the current hour/minute/second, we can set the timestamp of the request metering meter(options tenant, options meter name,int(options meter value), dimensions=dimensions,timestamp=str(int(round(time time() 1000)))) configurable deduplication logic amberflo allows you to customize the logic used to identify duplicates while the system defaults to using a generated unique id, you can configure it to use any dimension key that aligns with your data model if an ingested event has a duplicate identifier based on the configured logic, the event will be rejected real world example avoiding duplicate charges in nlp pipelines problem a vendor uses amberflo to meter the processing of text blocks for customer support cases using a text blocks processedmeter occasionally, the vendor must reprocess historical data due to onboarding workflows or failures in their machine learning pipeline customers should not be billed multiple times for the same block solution amberflo implemented configurable deduplication logic using the block id as the uniqueness key before processing, the pipeline checks whether the block id has already been ingested if it has, the event is discarded to prevent duplicate billing