MongoDB

ingest meters from mongodb you can use mongodb as the metering data source if your system is currently using mongodb, you can set up amberflo side by side to get metering data into amberflo without impacting the current system we recommend the following alternative methods to ingest data from mongodb using logstash logstash is an open source data collection engine with real time pipelining capabilities logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice to this end, we can take advantage of buffering, inputting, outputting, and filtering abilities from logstash by adding a mongodb input and s3 output plugin you can use either the jdbc input plugin or the mongodb input plugin and write to the s3 folder provisioned by amberflo elastic logstash https //www elastic co/guide/en/logstash/current/plugins inputs jdbc html https //github com/phutchins/logstash input mongodb query mongodb using code you can query mongodb with its sdk and use the amberflo sdk to ingest the data for example, in python you can use pymongo to query mongodb and ingest the result using the amberflo python sdk for getting only new data, you will persist the last timestamp/id used amberflo handles deduplication use mongodb's change data stream with the amberflo sdk, you can use the mongodb real time data stream to act upon every insert https //developer mongodb com/quickstart/python change streams/ python sdk set up virtual environment and install sdk \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# setup usage api client and get a usage report from metering usage import (aggregationtype, take, timegroupinginterval, timerange, usageapiclient, create usage query) usage api = usageapiclient(os environ get("api key")) \# set up virtual environment and install sdk \# \# $ python3 m virtualenv venv \# $ venv/bin/activate \# $ pip install amberflo metering python import os \# sign up for free at https //ui amberflo io/ to get an api key api key = os environ\["amberflo api key"] \# set up customer api client and create a customer from metering customer import customerapiclient, create customer payload customer api = customerapiclient(os environ get("api key")) message = create customer payload( customer id="customer 123", customer name="customer 123", \# email is optional customer email="customer 123\@example com", \# traits are optional they can be used as filters or aggregation buckets traits={ "region" "us east 1", }, ) customer = customer api add or update(message) \# set up ingest api client and ingest a meter from time import time from metering ingest import create ingest client ingest api = create ingest client(api key=api key) ingest api meter( meter api name="api calls", meter value=1, meter time in millis=int(time() 1000), customer id=customer\["customerid"], ) \# shut down ingest api client ingest api shutdown() \# setup usage api client and get a usage report from metering usage import (aggregationtype, take, timegroupinginterval, timerange, usageapiclient, create usage query) usage api = usageapiclient(os environ get("api key")) since two days ago = timerange(int(time()) 60 60 24 2) query = create usage query( meter api name="api calls", aggregation=aggregationtype sum, time grouping interval=timegroupinginterval day, time range=since two days ago, group by=\["customerid"], usage filter={"customerid" \["customer 123"]}, take=take(limit=10, is ascending=false), ) report = usage api get(query)