Build a Slack Bot That Answers From Your Docs

The retrieval half of a Slack docs bot is the same retrieval you would build anywhere. The Slack half has three specific traps: a signed request you must verify, a three-second acknowledgement deadline that turns into duplicate answers when you miss it, and a permission model where the bot can read documents the person asking is not allowed to see.

How a Slack event reaches your code

With the Events API, Slack makes an HTTP POST to a URL you own. There is no connection you hold open and no polling. The sequence, once:

1. You create an app, add a bot user, and subscribe to events — app_mention is the right one to start with, because it fires only when somebody addresses the bot.
2. You give Slack a Request URL. Slack immediately POSTs {"type": "url_verification", "challenge": "..."} and expects the challenge value echoed back in the response body. Until you do, the URL is not saved.
3. You add OAuth scopes — at minimum something like app_mentions:read to receive mentions and chat:write to reply — and install the app to the workspace, which yields a bot token.
4. Thereafter every subscribed event arrives as a signed POST, and you reply by calling chat.postMessage.

Verifying the request

Your Request URL is a public endpoint. Anyone who finds it can post to it, so the signature check is not optional — it is the only thing standing between a URL and a bot that answers to strangers.

Slack signs the raw request body with your signing secret and sends the result in a header alongside a timestamp. The string that gets signed is the version, the timestamp and the body joined by colons.

# slack_verify.py
import hashlib, hmac, os, time

SIGNING_SECRET = os.environ["SLACK_SIGNING_SECRET"].encode()

def verify(headers, raw_body: bytes) -> bool:
    ts  = headers.get("X-Slack-Request-Timestamp", "")
    sig = headers.get("X-Slack-Signature", "")
    if not ts or not sig:
        return False
    if abs(time.time() - int(ts)) > 60 * 5:      # replay window
        return False
    basestring = b"v0:" + ts.encode() + b":" + raw_body
    expected = "v0=" + hmac.new(
        SIGNING_SECRET, basestring, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)

Two things break this in practice. You must hash the raw bytes — if your framework has already parsed and re-serialised the JSON, the signature will never match and the bug looks like a wrong secret. And hmac.compare_digest rather than ==, because a timing-variable comparison on a signature is a textbook finding in any security review.

The three-second rule

Slack expects a 200 quickly — the documented budget is three seconds — and it retries events it considers un-acknowledged. A retrieval-plus-generation round trip takes longer than that on a good day, so the naive implementation answers, times out, gets retried, and answers again. Users see the bot say the same thing three times.

The pattern is: acknowledge first, work afterwards, on a different thread or a queue.

# app.py — the shape that matters
import json, queue, threading

work = queue.Queue()

def handle_post(headers, raw_body):
    if not verify(headers, raw_body):
        return 401, b""
    body = json.loads(raw_body)

    if body.get("type") == "url_verification":
        return 200, body["challenge"].encode()

    # A retry of something we already have. Ack and drop.
    if headers.get("X-Slack-Retry-Num"):
        return 200, b""

    event = body.get("event", {})
    if event.get("type") == "app_mention" and not event.get("bot_id"):
        if seen(body.get("event_id")):        # idempotency, see below
            return 200, b""
        work.put(event)

    return 200, b""                            # ack within milliseconds

def worker():
    while True:
        event = work.get()
        try:
            answer, sources = answer_from_docs(event["text"], event["user"])
            post_message(channel=event["channel"],
                         thread_ts=event.get("thread_ts") or event["ts"],
                         text=answer)
        except Exception as e:
            post_message(channel=event["channel"],
                         thread_ts=event.get("thread_ts") or event["ts"],
                         text="I could not answer that one. (" + type(e).__name__ + ")")

threading.Thread(target=worker, daemon=True).start()

seen(event_id) is a table of event ids with a timestamp, and it is what makes the handler idempotent. Retry headers help but are not sufficient — a network blip can deliver the same event twice without one — and an idempotency key checked before the side effect is the general answer for every webhook you will ever receive.

Post a “thinking” message immediately if answers take more than a couple of seconds, then update it. Silence for eight seconds reads as broken, and people mention the bot again, which doubles your load exactly when it is slow.

Threading, and not talking to yourself

• Always reply in a thread. Pass thread_ts as the parent’s thread_ts if it exists and the message’s own ts otherwise. A bot that replies in-channel in a busy channel gets muted within a day.
• Ignore your own messages. If you subscribe to message events rather than only mentions, your own posts come back to you. Filter on the presence of bot_id, and additionally on your own bot user id, or you will build an infinite loop that is visible to the entire company.
• Read the thread for context, but bound it. Fetching the parent thread gives useful follow-up context; fetching a two-hundred-message thread puts two hundred messages in your prompt. Cap it at the last ten or a token budget, whichever binds first.

The permission model everyone forgets

This is the part that gets a bot uninstalled. Your bot holds one token with one set of scopes. Your document index holds everything you fed it. Neither knows anything about what the person who asked is allowed to read.

Three concrete leaks, all of which happen by default:

• The index is flat. If HR documents and engineering documents are in the same index, an engineer asking a plausible question gets an HR passage quoted back. The fix is filtering retrieval by the asker’s groups before the search, not after — post-filtering leaks through the snippet and through the fact that a result existed.
• The answer is posted where the question was. A question asked in a public channel gets an answer visible to everyone in it, including content the asker could see and the channel could not. Either restrict the bot to private contexts, or reply ephemerally, or refuse to quote restricted sources in public channels — but decide, rather than discovering.
• The bot user is a member. Adding the bot to a channel gives whatever holds its token the ability to read that channel’s history through the API. That is a legitimate capability and it is also an aggregation risk, because one process now sees more of the company’s conversation than any employee does.

The design that survives review: index documents with an explicit access group, resolve the asker to their groups at query time, and pass the group list into retrieval as a filter. The same problem in a multi-tenant product has the same shape and higher stakes.

A channel is untrusted input

If the bot reads thread context, anyone in the thread can write instructions into the bot’s prompt. If the bot can call tools — look up a customer, file a ticket, post to another channel — then a message is an attack surface. The defences are architectural, not textual: keep channel text in the user turn, never let retrieved or channel text choose a tool, and require an explicit human confirmation for anything that writes.

Cost and rate limits at company scale

A docs bot in one channel is free. The same bot installed company-wide has a different shape, and the arithmetic is worth doing before somebody posts the invite link in general.

800 employees, 4% mention the bot on a given working day
  = 32 questions/day, ~700/month

Per question:
  retrieval  1 embedding call, ~20 tokens
  answer     5 chunks x 300 + system 400 + thread context 300
             = 2,200 input tokens, ~250 output tokens

Monthly:  700 x 2,200 =  1.54M input tokens
          700 x   250 =  0.18M output tokens

At $0.15 / $0.60 per million: $0.23 + $0.11 = about $0.34 a month.

The bot is not the cost. The INDEX is: 5,000 internal documents at 6 chunks
each, re-embedded whenever the embedding model changes, plus whatever the
sync job costs to run. And the largest cost of all is nowhere in this
arithmetic — the engineer who maintains the connectors.

Rate limits are the operational constraint rather than money. Slack applies per-method limits to its Web API, and a bot that answers a burst of questions by posting several messages each can meet them quickly. Two habits keep you clear: one message per answer rather than a header, a body and a sources post; and a single-worker queue with a small delay, so bursts are smoothed rather than sent in parallel.

Handle the rejection properly when it comes. Respect any retry hint the API returns rather than retrying immediately, and never drop the message — queue it. A rate-limit response is guidance, not an error, and the failure mode of ignoring it is a bot that gets quieter under exactly the load that made it useful.

Before you install it anywhere real

1. Signature verification on, with the raw body. Test it by posting a forged request and confirming a 401.
2. Acknowledge in under a second, always. Measure it; do not assume it.
3. Event ids deduplicated for at least an hour.
4. Retrieval filtered by the asker’s groups, with a test that a user without a group cannot retrieve that group’s documents.
5. A per-day call budget and a kill switch that stops responses without a deploy — the same switch described in the autoresponder build.
6. Every answer logged with the question, the retrieved document ids and the asker, so “why did it say that” is answerable.