Third-party "OpenAI-compatible" relays are now the default way to reach Chinese models from outside China. They are also the easiest place in your stack to be silently downgraded: you pay for a frontier model, the relay routes you to a smaller sibling, and nothing in the response says so.
This post is the test I would run as a buyer, written by someone who operates a relay (disclosure at the bottom). It is deliberately not "here is why we are trustworthy" — it is the procedure, plus our own numbers run against it.
Send the same long, boring string to both endpoints and compare usage.prompt_tokens. Different model families have different tokenizers, so a substitution usually shows up immediately as a count mismatch.
prompt = "The quick brown fox jumps over the lazy dog. " * 8 # relay: 164 prompt_tokens official: 164 prompt_tokens -> skew 0
This costs one request per side and rules out a whole class of swaps. It does not catch a same-family quantised variant, which is why you need the next signal.
Use prompts with exactly one correct answer and grade them with code, not another model: a JSON object returned byte-exact, the first eight primes, 17*23, an exact string, a fixed-length list.
Run each probe several times on both endpoints with temperature: 0. Then apply one rule:
The first public canary report on our relay scored four probes and averaged output similarity. One probe — "return the first eight primes as JSON" — passed on the relay and failed on the official DeepSeek endpoint, which pulled the mean down to 0.75 and read as "possibly downgraded".
The relay had passed every probe. The reference had a bad day. Mean similarity across few probes is a noisy statistic, and it penalises the honest side of the comparison. Grade per probe, on both sides, over repeats.
Model deepseek-v4-pro, relay aiapi-pro.com/v1 vs official api.deepseek.com/v1, 7 probes × 3 repeats per endpoint:
| Probe | Relay | Official | Downgrade? |
|---|---|---|---|
| exact JSON echo | 3/3 | 3/3 | no |
| first 8 primes as JSON | 3/3 | 3/3 | no |
| 17*23 | 3/3 | 3/3 | no |
| r's in "strawberry" | 1/3 | 0/3 | no |
| reply exactly "OK" | 3/3 | 3/3 | no |
| 1..10 space separated | 3/3 | 3/3 | no |
| JSON object with x,y | 3/3 | 3/3 | no |
Tokenizer fingerprint: prompt_tokens 164 on both sides, skew 0. The only non-perfect row is the strawberry probe, where both endpoints fail — the official model scored 0/3 and ours 1/3, so that row carries no fidelity signal in either direction.
The script, raw JSON and protocol are public: github.com/vvvvking/novai-canary. It is 42 API calls to reproduce.
git clone https://github.com/vvvvking/novai-canary cd novai-canary export RELAY_BASE=https://the-relay-you-are-testing/v1 export RELAY_KEY=... # usually a cheap or free-tier key export REF_BASE=https://api.deepseek.com/v1 export REF_KEY=... python canary.py
Point it at whatever model pair you actually care about. If the relay fails a probe the official endpoint passes, you have a lead worth investigating — not yet proof, but a concrete question to ask the operator.
109 Chinese models behind one OpenAI-compatible key, 5 of them free, no card required. Run the canary against our endpoint before you pay anything.
Get a free keyCompare it against the official provider on deterministic prompts at temperature 0, and only flag a downgrade when the relay fails something the official endpoint passes. Start with the tokenizer fingerprint — it costs one request per side.
No. The reference model is non-deterministic too. One reference-side failure over a handful of probes is enough to drag a mean score down and make a compliant relay look suspicious.
Out of scope for a canary. Fidelity, billing and retention are three separate questions; a test that answers the first says nothing about the other two.