I have an AI Gateway

amberflo integrates with third party ai gateways gateways act as the control point for all model traffic, capturing the metadata needed to understand usage, attribute costs, and govern ai workloads when connected to amberflo, a gateway becomes significantly more powerful its raw traffic is transformed into structured, attributed, real time usage data that unlocks budgets, guardrails, attribution, multi source aggregation, and ai governance across your organization amberflo supports connecting multiple gateways to a single amberflo account you can run different gateways for different teams, environments, or use cases, and all usage will be unified inside amberflo supported ai gateways litellm coming soon cloudflare apigee kong gateway this guide covers how to connect litellm when it is already deployed in your environment if you do not already have litellm running, or if your gateway is not currently supported, follow the guide below to deploy litellm from scratch and integrate it with amberflo  enable amberflo integration for an existing litellm deployment this guide assumes you already have litellm deployed and operational you will download the amberflo callback add a configuration entry to enable it add amberflo specific variables to your environment file update your deployment command redeploy and verify that usage flows into amberflo overview litellm handles and proxies llm requests amberflo provides real time usage metering, attribution, and cost governance the amberflo callback attaches to litellm and pushes meter events directly to the amberflo ingestion api once enabled, model usage appears immediately in the amberflo ai control tower it is important to note that the meter events are only metadata about the call to the llm the prompt and response never leave your vpc prerequisites you must already have a running litellm deployment (docker, kubernetes, vm, etc ) a postgres database used by litellm required for teams, virtual keys, organizations, and routing rules your existing yaml config file note litellm can technically run without postgres, but many required features (virtual keys, teams, routing rules) will not function postgres is required for the amberflo integration download the amberflo callback package amberflo has created a package to collect the necessary metadata and send it to amberflo download amberflo zip from amberflo unzip it into the same directory as your litellm config file your directory should look like this your directory/ ├── amberflo/ │ ├── init py │ └── (other amberflo callback files) ├── config yaml └── env note do not modify any callback source files add the amberflo callback to config yaml edit your litellm config and add litellm settings callbacks \ "amberflo litellm callback" if you already have other callbacks, add "amberflo litellm callback" as an additional entry this tells litellm to load the callback code from the amberflo/ directory add the amberflo environment variables amberflo provides the required variables and values for connecting the amberflo callback to your account you can find them in the connection wizard simply copy them and add them to your existing environment file the variables contain your amberflo api key the amberflo ingest endpoint the customer account identifier batch size, retry settings, and other callback related variables if you do not currently have an environment file you can create the env file in the same directory where you run your docker command and then copy in the values note do not commit this file to git optionally you can update the aflo hosted env value the string set as the value will be used to identify the instance of the ai gateway amberflo supports the ability to connect multiple ai gateways and this will allow you to filter the data based on which ai gateway instance the data is coming from update your deployment command (docker example) modify your existing docker command to include if you were not already using an environment file \ env file env a volume mount for the amberflo callback directory \ volume /amberflo /app/amberflo\ ro example docker run \\ \ env file env \\ \ volume /amberflo /app/amberflo\ ro \\ \ volume /config yaml /app/config yaml\ ro \\ \ publish 4000 4000 \\ ghcr io/berriai/litellm\ v1 79 0 stable \\ \ config /app/config yaml key points /amberflo maps to /app/amberflo inside the container litellm automatically discovers callbacks in this folder the env file supplies amberflo credentials and ingest information no additional installation is required redeploy the gateway stop your current litellm container and redeploy using the updated command test the integration step 1 call the gateway make sure to update the placeholder values in the curl command below \<your vm ip> \<your virtual key> \<model id> curl http //\<your vm ip> 4000/chat/completions \\ h "authorization bearer \<your virtual key>" \\ h "content type application/json" \\ d '{"model" "\<model id>","messages" \[{"role" "user", "content" "how are tokens calculated?"}]}' step 2 verify in amberflo once you've successfully made the api call in the previous step you should log in to amberflo and check the https //ui amberflo io/it spend/ai/summary you will begin to see your usage and cost show up there events should appear in near real time it can take up to 2 minutes for the first data to show up you may need to refresh the page automatic business unit creation amberflo automatically creates a new business unit the first time a virtual key is used mapping litellm team name → business unit name litellm team id → business unit id all future events for that key are attributed to that business unit you can rename business units later if needed the first time you send data for a particular team you will see only their litellm team id shown in the amberflo app if can take up to 5 minutes for the business unit to be fully created in amberflo