LiteLLM Failover From AWS Bedrock to Azure Under One Model Name

A request for azure/gpt-6-sol on my LiteLLM gateway doesn't go to Azure anymore. It goes to AWS Bedrock, and Azure only sees it when Bedrock fails. The model name stayed the same on purpose, because every client in my homelab already had it baked in.

TL;DR: Six OpenAI model names each have two deployments in LiteLLM with the same model_name: a Bedrock entry and an Azure entry. Their order values come from two YAML anchors at the top of the config, so flipping the primary is a two-number change. A small script does the flip and rewrites the file in place, because sed -i and a single-file bind mount don't mix. I tested it with a broken primary and Azure answered, with x-litellm-attempted-fallbacks: 1 in the response.

The name is a contract, not a provider

Every LLM call from a homelab service goes through LiteLLM with its own scoped key. Open WebUI, Paperless, my note bot, and the scripts I use to ask a model for a second opinion all send a model name like azure/gpt-6-astra and nothing else. The azure/ prefix was accurate when I picked it.

On 2026-09-24 I moved six of those names to Bedrock: azure/gpt-6-astra, azure/gpt-6-sol, azure/gpt-6-luna, azure/gpt-5.6-sol, azure/gpt-5.6-terra and azure/gpt-5.6-luna. Renaming them would have meant touching every client, every Open WebUI preset, and every key's model list. Keeping the names meant touching one file. So the prefix now lies a little, and I'm fine with that. The response headers tell the truth when I need it.

About an hour and a half later I added the part this post is about: Azure didn't go away, it became the backup.

Two entries, one name

LiteLLM treats entries with the same model_name as deployments of one model. The order field in litellm_params sets the priority. Every request goes to order 1. When that call fails, the router sends the same request to order 2 right away. Trimmed from my config:

x-order-bedrock: &order_bedrock 1
x-order-azure: &order_azure 2
 
model_list:
  - model_name: "azure/gpt-6-sol"
    litellm_params:
      order: *order_azure
      model: "azure/responses/gpt-6-sol"
      <<: *azure_frontier
      api_version: "2025-04-01-preview"
 
  - model_name: "azure/gpt-6-sol"
    litellm_params:
      order: *order_bedrock
      model: "bedrock_mantle/openai.gpt-6-sol"
      allowed_openai_params: ["reasoning_effort", "verbosity"]

The anchors are the trick. None of the twelve entries has a literal 1 or 2 in it. They all point at *order_bedrock or *order_azure, and the numbers live in exactly two lines at the top. YAML only lets you use an anchor after it's defined, so those two keys sit above model_list. They're there only to hold the numbers.

The config already used anchors for the Azure credentials (<<: *azure_frontier merges the endpoint, key and API version), so this was the same pattern for a new purpose.

Five of the six models use the bedrock_mantle route, Bedrock's OpenAI-compatible endpoint, which takes reasoning_effort and verbosity per request. Astra isn't served there, so it goes through Bedrock Converse instead. More on what that costs below.

Not every error should fail over

The router doesn't fall back on a context-window error or a content-policy error. That's correct: the same prompt is the same size and says the same thing on Azure, so it would fail the same way, only slower. Failover is for a provider that's down or broken, not for a request that's wrong.

The test

I broke the primary on purpose and sent a normal request. Azure answered, and the response carried:

x-litellm-attempted-fallbacks: 1

The other header worth knowing is x-litellm-model-api-base. It shows which provider's endpoint served the call: the Bedrock mantle endpoint or the Azure resource. One exception: Astra on Bedrock Converse sends no such header, so for Astra the absence is the signal.

Switching the primary, and the bind-mount trap

Flipping the primary means swapping the two numbers and restarting LiteLLM. I wrote litellm-primary.sh bedrock|azure|status for it. The core of it:

cp $CFG $CFG.bak-primary
sed -E "s/^(x-order-bedrock: &order_bedrock) [0-9]+/\1 $b/; s/^(x-order-azure: &order_azure) [0-9]+/\1 $a/" \
  $CFG > /tmp/litellm-primary.yaml
cat /tmp/litellm-primary.yaml > $CFG
rm /tmp/litellm-primary.yaml
grep -E '^x-order-(bedrock|azure):' $CFG
docker restart LiteLLM

The obvious version is one sed -i. That doesn't work here. The config is a single-file bind mount into the container, and a single-file bind mount follows the inode, not the path. sed -i doesn't edit a file. It writes a new one and renames it over the old name. The host sees the change, and the container keeps its handle on the old inode.

So the script writes the result to a temp file and cats it back into the original. > truncates and rewrites the same inode, and the container sees the new text. The same rule covers any hand edit of that file: edit in place, never replace.

Two small things in there are deliberate. It keeps the previous file as config.yaml.bak-primary. And it prints the two anchor lines after the write, so a sed that matched nothing shows up as unchanged numbers instead of passing silently. After the restart it polls /health/liveliness every 3 seconds for up to 2 minutes, and if LiteLLM doesn't come back, it tells me where the old config is.

The caveats

Astra's effort is fixed on Bedrock. Converse rejects reasoning_effort for GPT models, so the Bedrock Astra entry drops the client's value and always sends reasoning: {effort: high}. On Azure, Astra follows whatever effort the client asks for. So the same model name behaves differently depending on who's primary. A quick question to Astra now costs a high-effort answer. The other five models act the same on both sides.

The cache goes cold on a switch. Each provider keeps its own prompt cache. All traffic goes to the primary, so its cache stays warm and the backup's is cold. The first requests after a switch or a failover pay the full input price. In a test that day, with Bedrock as the primary, Astra and Sol cached on their own: the second identical 6K-token call read the prompt from cache. That's worth a lot when my tools resend a long thread every turn, and it resets whenever traffic moves.

Bedrock is pickier about input. It rejects empty text blocks, which some clients send as empty assistant messages. modify_params: True in litellm_settings has LiteLLM clean those up before the call. The Bedrock entries also set drop_params: true and drop temperature, because the reasoning models reject it and some clients send it no matter what.

Not everything moved. The Azure deployments of the six models stay, and not only as the backup. Azure's model-router calls the Azure gpt-5.6 deployments itself. gpt-4.1, gpt-5, gpt-5-mini, gpt-5-nano, DeepSeek, embeddings, speech, image generation, TTS and rerank are all still Azure-only. The one follow-up I still owe myself: Azure token spend on the six moved models should drop to zero from 2026-09-24, and I'll check that in the next cost query instead of assuming it.

Why the keys matter here

Because every consumer has its own scoped LiteLLM key, none of them knew any of this happened. Nothing got a new URL, a new name, or a new secret. Spend is still logged per key, whichever cloud served the call.