I need an AI Gateway

if you do not already have an ai gateway, this guide walks you through deploying litellm using docker and connecting it to amberflo litellm is the first supported gateway, and additional gateways will be available soon once deployed, the gateway will push real time usage events to amberflo, enabling attribution, budgets, cost guards, dashboards, and full ai governance and control prerequisites you will need docker installed and running an amberflo account a postgres database (required by litellm for saving providers, models, teams and virtual keys) create a working directory mkdir litellm gateway cd litellm gateway pull the litellm docker image and create a config pull the litellm proxy (ai gateway) image docker pull ghcr io/berriai/litellm\ v1 79 0 stable create a basic litellm config file in your folder and name it config yaml \# litellm proxy configuration (config yaml) \# general / proxy wide settings general settings litellm settings callbacks \ "amberflo litellm callback" download amberflo callback and set up environment file you can find the amberflo callback zip in the ai gateway setup wizard download amberflo zip file (callback package) move the files into your working directory and unzip unzip amberflo zip d your directory should now contain config yaml env amberflo/ init py (other callback files) creating an environment file an environment file stores configuration values that the gateway reads at startup it keeps sensitive or user specific settings separate from your main configuration so you do not hard code them into the yaml file follow these steps create a file named env in the same directory where you created your config yaml copy the values displayed in the wizard and paste them into the env file update any placeholders with the correct values for your setup save the file once saved, the gateway reads this file automatically when it starts you can review the details of each variable in the sections below, including what each key represents and how it is used litellm master key a required secret used to encrypt and decrypt sensitive fields stored in the litellm database this key must be a long, random string changing it will invalidate previously encrypted data litellm salt key a cryptographic salt used for hashing and securing stored values this must also be a long, random string do not reuse the same value across environments ui username the username for logging into the litellm admin ui this is the credential used for the web dashboard, not for model authentication ui password the password for logging into the litellm admin ui choose a strong, random password if this value changes, existing sessions become invalid database url the full postgres connection string used by litellm’s prisma client it must be unquoted and begin with postgresql // it defines the database host, port, user, password, and database name store model in db this allows you to add models using the admin ui instead of only via the config this should be set to true litellm master key= litellm salt key= ui username= ui password= database url= store model in db=true do not commit this file to git optionally you can update aflo hosted env value the string set as the value will be used to identify the instance of the ai gateway amberflo supports the ability to connect multiple ai gateways and this will allow you to filter the data based on which ai gateway instance the data is coming from create and configure the postgres database litellm requires postgres for teams virtual keys storing models create or provision a postgres instance construct your connection url postgresql //\<username> \<password>@\<host> \<port>/\<db name> make sure your postgres db is running and then update your environment file to include the connection string note if you are running the postgres db on your local machine you need you use host docker internal not localhost and the default port for postgres is 5432 database url postgresql //\<username> \<password>@\<host> \<port>/\<db name> start the litellm gateway container run docker run \\ \ env file env \\ \ volume /amberflo /app/amberflo\ ro \\ \ volume /config yaml /app/config yaml\ ro \\ \ publish 4000 4000 \\ ghcr io/berriai/litellm\ v1 79 0 stable \\ \ config /app/config yaml this will load the amberflo callback load your environment file connect litellm to postgres start the gateway on port 4000 test your integration check that the amberflo integration is working you can verify that the gateway is correctly sending meter events to amberflo before configuring any providers or models use the litellm master key that you created during deployment make a request to the gateway with a model name that does not exist the call will fail, but the failure still produces a meter event if the integration is set up correctly, amberflo will receive that event within one or two minutes and the demo data banner will disappear make sure to set the ip of your gateway and your master key in the sample curl below curl http //\<your vm ip> 4000/chat/completions \\ h "authorization bearer \<your master key>" \\ h "content type application/json" \\ d '{"model" "my model","messages" \[{"role" "user", "content" "how are tokens calculated?"}]}' what to check in the amberflo app, open the ai metering page look for a new entry under the lm api call error details meter if you see the event, the integration is working and you can continue to provider and model configuration in the next step complete ai gateway configuration litellm does not emit any usage until all required objects are created in the admin ui you must configure the gateway in the following order 1\ providers create a provider entry for each upstream service you plan to use examples include openai, anthropic, and bedrock add the required api keys or access tokens for each provider until a provider is created with valid credentials, no model that references it can run 2\ models define the models that the gateway will expose examples include gpt 5 or claude sonnet each model must reference one of the providers you created above this step establishes which upstream model is called, its model identifier, and any model level parameters 3\ teams create one or more teams a team is the entity to which usage and cost will be attributed inside amberflo this is how you can break down you ai spend and more specifically know who is using what and how much it is costing you 4\ virtual keys create a virtual key for testing and for any client that will issue requests through the gateway a virtual key authenticates the request and determines which team the usage belongs to without a virtual key, the gateway cannot be called and no usage will be recorded after completing these four steps, the gateway is fully configured and can emit usage events to amberflo ⚠️ go to the following page to learn how to set up a provider, model, team and virtual key they are required to test the integration with the ai gateway once you have completed those items you can complete the next steps docid\ wnokz5lhwnlhy5ysisv5h test the full integration step 1 call the gateway use a virtual key assigned to a team this is a sample curl command to call the litellm ai gateway you need to replace the following values your vm ip the public or private ip address of the machine running the litellm gateway if you are using docker locally, this is usually localhost if the gateway is on a vm, this is that vm’s ip your virtual key the virtual key you created in the litellm admin ui this key identifies the caller, determines which team the usage is attributed to, and is required for authentication model id the model identifier you defined when creating a model in the admin ui this is the exact string litellm expects (for example gpt 4o mini or claude 3 sonnet) if this does not match a configured model, the request will fail and no usage will be recorded curl http //\<your vm ip> 4000/chat/completions \\ h "authorization bearer \<your virtual key>" \\ h "content type application/json" \\ d '{"model" "\<model id>","messages" \[{"role" "user", "content" "how are tokens calculated?"}]}' step 2 verify in amberflo once you've successfully made the api call in the previous step you should log in to amberflo and check the https //ui amberflo io/it spend/ai/summary you will begin to see your usage and cost show up there events should appear in near real time it can take up to 2 minutes for the first data to show up you may need to refresh the page automatic business unit creation amberflo automatically creates a new business unit the first time a virtual key is used mapping litellm team name → business unit name litellm team id → business unit id all future events for that key are attributed to that business unit you can rename business units later if needed the first time you send data for a particular team you will see only their litellm team id shown in the amberflo app if can take up to 5 minutes for the business unit to be fully created in amberflo