Cohere's Connectors: How Tool Results Get Merged Into Context

A connector moves retrieval to Cohere’s side of the wire. Instead of searching your data and passing the results in documents, you register an HTTP endpoint and Cohere calls it during the chat request. The mechanism is v1-only, and that is the first thing to know about it.

What a connector is

In the ordinary grounded pattern, retrieval is yours: you embed the query, search your index, and put the hits in the request. A connector inverts the control flow. You give Cohere a URL; on a chat request that names the connector, the model generates one or more search queries, Cohere issues a POST to your URL for each, and the documents your service returns are merged into the model’s context before it answers.

Cohere also operates one managed connector, web-search, which requires no service of your own. Everything else is a service you host and register through the connectors API.

The request shape

On the chat request, connectors are a list of objects each identified by id:

curl https://api.cohere.com/v1/chat \
  -H "Authorization: Bearer $CO_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "model": "command-r-plus-08-2024",
    "message": "What did we announce about the Utrecht store this month?",
    "connectors": [
      {"id": "web-search", "options": {"site": "example.com"}},
      {"id": "internal-wiki", "continue_on_failure": true}
    ]
  }'

Three fields on a connector entry are worth knowing:

• options — a free-form object passed through to your service, and for web-search the documented site key restricts results to one domain.
• continue_on_failure — when true, a connector that errors or times out is skipped and the answer is generated from whatever else was retrieved. When false, its failure fails the whole chat request. Defaulting this to true across every connector is the quiet way to ship a RAG system that silently degrades to ungrounded answers.
• user_access_token — for connectors that need to search as a specific user rather than as the application, so per-user permissions in the source system are honoured.

There is also search_queries_only. Set it and the API returns the queries the model would have run without running them or generating an answer — the cheapest way to debug a connector that is retrieving the wrong things, because it separates “the query was bad” from “the index answered badly”.

The contract your service must satisfy

A registered connector is a URL that accepts a POST at /search and returns documents. The request Cohere sends is minimal:

POST https://connector.example.com/search
Authorization: Bearer <the token you registered>
content-type: application/json

{"query": "Utrecht store announcement August 2026"}

The response must be an object with a results array:

{
  "results": [
    {
      "id": "wiki-9931",
      "title": "Utrecht store — August update",
      "text": "The Utrecht store moves to Oudegracht 210 on 1 September 2026.",
      "url": "https://wiki.example.com/utrecht-august"
    }
  ]
}

Beyond id, the field names are yours. They are also visible to the model, which is the part people underestimate: a field called text reads as content, a field called meta_blob_v2 reads as noise, and every field you return is billed as input tokens on every turn it survives in the context. Returning your entire row is expensive and makes the answer worse.

Latency is the other constraint. Your service sits inside the user’s chat request, so its p99 is added directly to time-to-first-token, and it may be called several times in one turn when the model generates multiple search queries.

How the results reach the model

Retrieved results are folded into the same document slot that the documents parameter fills. They are not pasted into the user message, and they are not a tool result. That has one very useful consequence: everything a connector returns is citable exactly like a document you passed by hand, and the ids it carries appear in the citations array on the response. The response also echoes the retrieved documents so a client can render sources without holding its own copy.

There is a second consequence that is easy to miss and expensive at scale: because retrieval happens inside the chat request, you cannot cache it. An identical question asked by two hundred users triggers two hundred calls to your service, and there is no layer between the model and your endpoint where a result can be reused. Caching has to live inside your connector — keyed on the query string Cohere sends — which is workable but means the cache is being keyed on a query the model wrote rather than on anything a user typed, and those vary more than you would expect for the same underlying question.

When the retrieved set is large enough to threaten the context window, prompt_truncation decides what happens. With "AUTO", low-relevance documents and old chat history are dropped to make the request fit; with "OFF", an over-length request is an error. On a connector-driven system, where you do not control how much comes back, that setting is doing considerably more work than its name suggests.

Registering and testing one

A connector is created once, through the connectors API, and the registration is where its authentication is decided:

curl https://api.cohere.com/v1/connectors \
  -H "Authorization: Bearer $CO_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "name": "Internal wiki",
    "url": "https://connector.example.com",
    "description": "Search the engineering and operations wiki.",
    "service_auth": {
      "type": "bearer",
      "token": "a-long-random-secret-you-generated"
    }
  }'

Cohere appends /search to the registered url and presents the token you supplied on every call. That token is the only thing standing between your internal index and the open internet, because the endpoint must be publicly reachable for Cohere to call it. Verifying it on every request — and rejecting rather than logging when it is absent — is not optional hardening; it is the whole access control model.

The registration returns an id, and that id is what goes in the connectors array on a chat request. Cohere validates the endpoint at registration time, so a connector that fails to create is usually a reachability or auth problem rather than a payload problem, and the error says which.

Test it in the right order, because a connector failure inside a chat request is opaque:

1. Call your own /search with curl and a hand-written query. This is the only step where you see your service’s real error messages.
2. Send a chat request with search_queries_only: true to see what queries the model generates from a user message. Surprising retrieval is very often a surprising query rather than a bad index.
3. Send a real chat request with continue_on_failure: false, so that a connector error fails loudly instead of producing a plausible ungrounded answer. Turn it back on only once you have decided what degraded service should look like.
4. Check the documents echoed on the response. If they are not what you expected to be retrieved, the problem is in your service; if they are right and the answer ignores them, it is a prompt problem.

v2 has no connectors

For a new integration, the migration is not hard and is arguably an improvement: run retrieval yourself and pass the results in documents, or expose your search as an ordinary tool and let the multi-step loop call it. Both put the retrieval step where you can see it, log it, cache it, and time it — three things that are structurally awkward when the search happens inside somebody else’s request. The tool route is documented in the multi-step tool use loop, and its results remain citable because Cohere treats tool outputs as documents.