# A RAG assistant and semantic search on Cloudflare's free plan: real costs and lessons learned

In a Cloudflare project I use an **AI assistant** (RAG: it answers from the site's own content) and a **semantic search** (the same retrieval, without the LLM), both on Workers AI and Vectorize, on the **free plan**.

This post covers what that costs in practice, what broke, and the design decisions that came out of it. All numbers are measured, not taken from a pricing calculator.

## The stack

|Part|What|
|-|-|
|Embeddings|`@cf/baai/bge-m3` (1024 dimensions, multilingual)|
|Answer model|`@cf/meta/llama-4-scout-17b-16e-instruct` (JSON mode)|
|Vector store|Vectorize, one index, **one namespace per site**|
|State|One Durable Object per site: rate limits, daily caps, an answer cache, sessions|
|Indexing and tests|GitHub Actions, embeddings on a self-hosted bge-m3 container|

Retrieval only ever queries that site's own namespace. Content is cut into chunks at its Markdown headings, at most 1,200 characters each, and each chunk carries its heading path so the embedding has context.

## 1. What things really cost

Workers AI bills in *neurons*. The rates I use come from Cloudflare's pricing page:

|Model|Input|Output|
|-|-|-|
|bge-m3|1,075 neurons / M tokens|n/a|
|llama-4-scout|24,545 neurons / M tokens|77,273 neurons / M tokens|

Measured in the dashboard, per request:

|Action|Neurons|
|-|-|
|One semantic search (one embedding of a short query)|**0.01**|
|One assistant answer (embedding + llama-4-scout)|**≈ 66**|
|One full run of my assistant test sets (≈ 38 questions)|**≈ 2,500**|

Two conclusions:

* **Search is practically free.** 10,000 searches a day cost about 100 neurons.
* **An answer is almost entirely LLM input.** The system prompt plus 5 retrieved chunks is roughly 2,500 to 3,500 tokens; the answer itself is short. If you need to save, retrieve fewer or smaller chunks before you look at the model.

Vectorize's free tier (5 M stored dimensions, 30 M queried dimensions a month) is not the bottleneck at this scale. Note how *queried* dimensions are counted: (stored vectors + queries) × dimensions. With about 600 vectors of 1024 dimensions that still leaves tens of thousands of queries a month.

## 2. The free allowance: account-wide, and the overrun lasts longer than you think

The free plan gives **10,000 neurons per day, for the whole account**: every Worker, every site, your test runs, the Playground.

What I observed when crossing it:

* **Below the limit**, the count resets at **00:00 UTC**, as documented.
* **Over the limit**, every model, embeddings included, answers `4006: you have used up your daily free allocation of 10,000 neurons`. The block does **not** end at 00:00 UTC: it lifts **24 hours after that day's first use**. If you started at 08:00 UTC and went over at 22:00, you wait until 08:00 UTC the next day.
* Meanwhile the dashboard happily showed **0 / 10,000** for the new day, while the API still returned 4006. Several Community threads and an open cloudflare-docs issue describe the same thing.
* The **Workers AI Playground** in the dashboard still answered with llama during the block. Don't use it to check whether your Workers can call a model; call the API, or look at your Worker's logs.

So the rule became: **never cross the real limit.** Everything below is built around that.

## 3. Your own test sets are the biggest consumer

I keep a test set per site: real questions with expected properties (answers in the right language, no invented phone numbers, the allergy disclaimer present, the off-topic question refused, the prompt injection ignored…). They run against the real models and the real index.

At first they ran on every merge that touched content or code. One busy day of merges and reruns used most of the allowance, and the live assistants hit the limit. A test run of ≈ 38 questions costs as much as 38 visitor questions, and on a small site that is more than a day's real traffic.

The decisions:

* **On a merge: re-index, and run only the cheap tests.** The search test set runs every time; it only embeds.
* **The assistant test sets run once a week, just before the daily reset**, Sunday 23:00 UTC. At that moment whatever visitors did not use that day would be lost anyway, so the tests may spend up to the Worker's own limit (9,000, see below), never more.
* **A run by hand has a checkbox** for the assistant tests, off by default, with a daytime ceiling of 6,000 so visitors keep room.
* **On a paid plan**, a single repository variable switches per-merge testing back on.

```yaml
on:
  push: { branches: [main], paths: ["content/**", "src/**"] }
  schedule:
    - cron: "0 23 * * 0"   # Sunday 23:00 UTC, an hour before the counter resets
  workflow_dispatch:
    inputs:
      assistant_tests:
        description: "Also run the AI assistant's test sets (~2,500 neurons)"
        type: boolean
        default: false
```

Two more things that saved neurons:

* **Incremental indexing.** The indexer reads the stored vectors, compares chunk text, heading and model, and only embeds chunks that changed. If only the content hash changed, it rewrites metadata with the old vector. A typo fix re-embeds one chunk, not the site.
* **After a content-only change, only the sites whose text changed get tested**, on the plans that test per merge.

## 4. Keep your own neuron counter, and stop before Cloudflare does

Workers AI has no "remaining budget" API. So the Workers count it themselves:

* One Durable Object for the **whole account** holds the count per UTC day. The other Worker reaches it through a cross-script Durable Object binding.
* Every call goes through a wrapper. Generation uses the token usage the model reports (or an estimate of characters ÷ 4), embeddings use the estimate, both priced with the table above.
* **Answers stop at 9,000.** That leaves a 10% margin for estimation errors.
* The test runs report their own spend to the same counter through an authenticated endpoint, so tests and visitors share one budget.

```ts
async generate(input) {
  const day = utcDay(now());
  if ((await budget.neuronsOn(day)) >= LIMIT) {
    await budget.switchOff(nextResumeTime(now()));   // 00:05 UTC
    throw new NeuronBudgetError();
  }
  const result = await guarded(() => models.generate(input)); // a real 4006 switches off too
  await budget.addNeurons(day, neuronsFor(model, usage(result)));
  return result;
}
```

Because it is an estimate, compare it now and then with the dashboard for the same UTC day. If they drift apart, lower the limit.

## 5. Fail visibly, recover on your own

The guiding rule: **better no assistant than a failing one.**

* **Hide before failing.** The widget asks a small status endpoint whether it may show (`GET …/status`, cached 5 minutes in the browser). While the assistants are switched off, the button does not appear at all.
* **A clear message mid-conversation.** A visitor who is already chatting when the limit is reached gets a "today's limit" message instead of a generic error. The search box says "not available right now".
* **Our own limit** switches the assistants off until **00:05 UTC**.
* **A real 4006** (the estimate was off, or something else spent neurons) switches them off for **4 hours**. After that, the first status request makes **one tiny probe call** (an embedding of `"ok"`, ≈ 0.01 neurons) before any visitor does: still 4006 means another 4 hours, success means back on.
* **Manual reset.** One authenticated REST call switches them back on at once, for example after checking the dashboard:

```bash
curl -X POST https://example.com/api/assistant/neurons \
  -H "Authorization: Bearer $BUDGET_KEY" \
  --data-binary '{"reset": true}'
```

(On Windows PowerShell, `curl -d '{"reset":true}'` loses its quotes. Use `--data-binary @file.json`.)

## 6. Index on your own embedding server, answer on Workers AI

This started as something else: a **live fallback**, built on a false conclusion. During the block, the logs said `Assistant failed at embedding: 4006`. So I tried the models in the dashboard's **Workers AI Playground**, and llama answered there without complaint. The Playground offered no embedding models to compare, so it looked like only the embedding model was blocked. It wasn't: the Playground is not a reliable test of your account's API quota. The fix I built on that conclusion: when Workers AI refused the embedding with 4006, the Worker would embed the visitor's question on my own bge-m3 server instead. It worked: the embedding step succeeded, and the log moved on to `Assistant failed at model: 4006`. The over-quota block covers **every** Workers AI model, so llama refused the answer one step later. **A fallback for half a pipeline is no fallback.** It only makes sense if every step has one (a second LLM provider too), and then it is a different architecture. I removed it from the Workers.

What stayed is the part that does pay off. The real reason now is **reliability**: the workflows must **always** be able to make their embeddings. When Workers AI is out of neurons, or blocked after an overrun, a re-index after a content change fails, and so do the test runs. The site's knowledge then stays out of date until the quota comes back. So I run **the same model**, BAAI/bge-m3, in Hugging Face *Text Embeddings Inference* (CPU build) on my own Azure Container Apps instance (2 vCPU, 4 GiB, scaled to zero, so no traffic means no cost). The model is baked into the image, so a start does not download 2.2 GB from the Hugging Face Hub. The GitHub workflow embeds everything there: indexing and the test questions. It works no matter what the Workers AI allowance says, and as a bonus it spends no neurons. The Workers keep using Workers AI for live queries.

* **The vectors are compatible.** Same model, normalized, cosine. I verified it both ways: an index embedded by Workers AI queried with container embeddings, and the reverse, both pass the same test sets. Compared directly, the same sentences score a cosine of 0.99995 or higher against another bge-m3 build (Ollama), so the small differences are just numeric noise.
* **A cold start takes 35 to 90 seconds**, and about 90 when it really starts from zero. Roughly 3 s of that is loading the model and 27 s is the server's warm-up; the rest is pulling the image and scheduling. That is fine for a workflow (wait for `GET /health`, up to 3 minutes, then embed; a warm call takes about a second), but it was another reason against the live fallback: far too slow for a visitor, who gives up long before.
* **Do not trim the batch token limit to start faster.** A lower `--max-batch-tokens` shortens the warm-up, but it truncates long inputs silently. I keep 2048.
* **The server prints its API key at startup.** Text Embeddings Inference logs its arguments, key included, at every start. Treat log access as key access, and redact that line before you paste logs anywhere.
* **Be gentle with a small container:**

  * batches of at most 32 texts (TEI's default);
  * sent **one after another**, not in parallel;
  * a 120 s timeout per call;
  * **retry on 5xx, timeout or dropped connection** (up to 3 attempts, 2 s and 5 s apart), because a replica can restart mid-job;
  * never retry a 4xx: a wrong key does not get better.
* **Fail loudly without the container.** When its URL or key is missing, the workflow logs it in one line ("Embeddings: Workers AI (EMBEDDING_URL is empty)"). One run silently spent neurons because a secret was missing; the log line made it obvious.

## 7. Semantic search without an LLM: the cheap tier

The search box reuses the retrieval: one embedding plus one Vectorize query, no generation. That makes it about 6,000× cheaper than an assistant answer, and a natural lower-priced product.

* **Entries, not chunks.** For search I index one vector per menu item, product or service, and one per page section, with the title, a snippet and the link in its metadata. A result can then open the item directly.
* **No least-bad guess.** Calibrate a score threshold on your own content. With bge-m3 on short, mostly Portuguese pages:

  * unrelated queries ("xyz qwerty", "best football team") scored up to **≈ 0.40**;
  * relevant ones mostly **0.50 and up**;
  * I use a **minimum of 0.45**, and drop results more than **0.10 below the best** match, so a clear hit stands alone.
* **Calibrate offline.** I did this with a quantized ONNX build of bge-m3 (`transformers.js`, `Xenova/bge-m3`, q8) on my laptop, at no neuron cost. Then I checked the same test cases against the real index in CI. Quantized and full-precision scores differ slightly, so keep the threshold a config var.
* **Never answer allergen questions from similarity.**
* **A metadata-free lookup trick.** One shared namespace has no configs at query time, but needs "all items tagged vegan". The indexer stores one extra vector per diet at a **fixed id** (`<namespace>:0`…`:3`), with the tagged items as JSON in its metadata. The Worker reads it with `getByIds`, no metadata index or similarity needed. These vectors have no URL, so a normal query never shows them as a result.
* **The usual hygiene:** an edge cache (1 hour) for repeated queries, a per-IP rate limit, a daily cap per site, and **no query text stored or logged**. The stats only count that a search happened.

## 8. Small things that cost me time

* **The Playground is not your API.** During a quota block it still answered (section 2), which led me to a wrong diagnosis (section 6).
* **The Workers AI binding and the REST API can disagree.** A model that still answered over REST (which my test runner used) was refused through the Worker's AI binding as deprecated (error 5028). A green test run did not prove the Worker could use the model. After a model change, check one live answer.
* **JSON mode is per model.** Sending `response_format` to a model without it returns 403 "This model doesn't support JSON Schema". Keep an allow-list, and parse JSON out of plain text as a fallback.
* **A Durable Object is a better rate limiter than KV on the free plan.** KV's free plan allows 1,000 writes a day for the whole account, and every question needs several. A Durable Object per site handles one request at a time, so its counts are exact.

## Take aways

* **Measure before you guess:** a search costs 0.01 neurons, an answer about 66, and your test suite more than your visitors.
* **The 10,000 free neurons are per account, not per Worker.** Crossing them blocks every model until 24 hours after that day's first use, so stay below with your own counter (stop at 90%).
* **Run expensive LLM tests weekly, just before the 00:00 UTC reset**; run the cheap tests on every change and make use of the remaining neurons (but do not go over the limit).
* **Hide the feature before it fails, recover on your own** (timer plus probe call), and keep a manual reset.
* **Embed for indexing and tests on your own container** with the same model, so the workflows always work, even when the Workers AI quota is used up or blocked. Answer visitors with Workers AI. A live embedding fallback does not help when the LLM step is blocked too.
* **Semantic search without an LLM** is a cheap, useful tier. Calibrate its threshold offline, and never let similarity answer allergy questions.

All of it runs on the free plan, and it is the free plan's hard limits that shaped the design. The upgrade to Workers Paid (US$ 5/month) is one config value away, and I am making it: there, going over the allowance is billed instead of blocked, which removes the 24-hour lockout. The counter, the hide-before-failing widget and the container stay useful either way. The container is for batch work only (indexing and tests); visitors never wait for it.

---

*Co-authored with Claude.*

If this was useful, [you can buy me a coffee](https://ko-fi.com/p47k0).

New articles: [follow via RSS](https://blog.koorevaar.com/feed.xml).