Build a Newsletter Digest Agent That Runs on Cron

A digest agent has one job that is genuinely hard: five outlets report the same announcement and the reader wants one entry, not five. That is clustering, not summarisation, and doing it before the model is called is also what keeps the daily cost in the region of small change — a claim this page derives rather than asserts.

What the job does, in order

1. Fetch each feed. Store new items by URL hash so a restart does not reprocess yesterday.
2. Filter with rules first: age, source, keyword blocklist. Free, and it removes a surprising fraction.
3. Embed the remaining titles and lead paragraphs — one batched call.
4. Cluster by similarity. Each cluster is one story, however many outlets covered it.
5. Rank clusters and keep the top N. This is the step that bounds the cost of everything after it.
6. Summarise each kept cluster in one call, given all its members.
7. Render and send. Store what was sent so tomorrow can suppress follow-ups on the same story.

The ordering is the design. Cluster before summarising and you pay for twelve summaries; summarise before clustering and you pay for sixty and then throw away forty-eight.

Fetching without a dependency

RSS and Atom are XML, and xml.etree.ElementTree is in the standard library. Feeds are frequently malformed, so parse defensively and skip rather than crash — one bad feed must not stop the digest.

# feeds.py
import hashlib, urllib.request
import xml.etree.ElementTree as ET

NS = {"atom": "http://www.w3.org/2005/Atom",
      "dc":   "http://purl.org/dc/elements/1.1/"}

def fetch_feed(url, timeout=20):
    req = urllib.request.Request(url, headers={"User-Agent": "digest/1.0"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        raw = r.read()
    try:
        root = ET.fromstring(raw)
    except ET.ParseError:
        return []                       # skip, log, carry on
    items = []
    for node in root.iter():
        tag = node.tag.split("}")[-1]
        if tag not in ("item", "entry"):
            continue
        title = text_of(node, ("title",))
        link  = link_of(node)
        summ  = text_of(node, ("description", "summary", "content"))
        if title and link:
            items.append({
                "id": hashlib.sha256(link.encode()).hexdigest()[:16],
                "title": title.strip(),
                "url": link,
                "summary": (summ or "")[:1200],
                "source": url,
            })
    return items

Iterating over every node and matching on the local tag name handles RSS and Atom with one code path and survives namespace declarations that vary between publishers. Cap the stored summary — some feeds carry the entire article, and you do not want a 20,000-token item in the clustering step.

The same story from five sources

Two passes, cheap then semantic, because they catch different things.

Pass one, near-identical text. Syndicated wire copy is often byte-identical or nearly so. A shingle-based Jaccard similarity over word trigrams catches it with no model call at all.

def shingles(text, n=3):
    w = [t for t in re.findall(r"[a-z0-9]+", text.lower()) if t]
    return {tuple(w[i:i + n]) for i in range(max(0, len(w) - n + 1))}

def jaccard(a, b):
    if not a or not b:
        return 0.0
    return len(a & b) / len(a | b)      # > 0.6 on trigrams = the same copy

Pass two, same event, different words. Embed title + first 200 characters and cluster greedily by cosine similarity. Single-link agglomeration is enough and it is fifteen lines.

def cluster(items, vecs, threshold=0.80):
    clusters = []                       # each: {"members": [...], "centroid": v}
    for it, v in zip(items, vecs):
        v = normalise(v)
        best, best_sim = None, 0.0
        for c in clusters:
            sim = dot(v, c["centroid"])
            if sim > best_sim:
                best, best_sim = c, sim
        if best is not None and best_sim >= threshold:
            best["members"].append(it)
            n = len(best["members"])
            best["centroid"] = normalise(
                [(c * (n - 1) + x) / n for c, x in zip(best["centroid"], v)])
        else:
            clusters.append({"members": [it], "centroid": v})
    return clusters

The threshold is the one number to tune, and tune it by reading output rather than by optimising anything. Too low and two unrelated stories about the same company merge; too high and the same press release appears three times. In practice 0.80 to 0.86 is the useful band for news titles with a modern embedding model, and it will differ for your sources — check twenty clusters by hand on day one.

Then suppress across days. Keep the centroid of every cluster you sent for the last week and drop a new cluster whose similarity to a sent one exceeds a slightly higher threshold. This is what stops the digest reporting the same acquisition every morning for four days. Embedding-based de-duplication has the same shape wherever you meet it.

Selection, before summarisation

Rank clusters by a formula, not by a model. The model is expensive and this decision does not need it.

score = (
    1.6 * log(1 + distinct_sources)     # five outlets covering it means something
  + 1.0 * source_weight_max             # your trusted-source weighting
  + 0.8 * keyword_match                 # topics the reader declared interest in
  - 0.5 * hours_old / 24
)

distinct_sources is the strongest signal in a digest and it is available only after clustering — which is another reason the order in the pipeline is what it is. Keep the top ten to fifteen; a digest with forty entries is not read.

The daily cost, derived

Here is the arithmetic, with every input something you can count in your own database on day one. The only figure you must supply is the price, which is why it appears as a symbol until the last line.

Inputs you can count:
  40 feeds x ~12 new items/day        = 480 items/day
  after rules filtering (~40% drop)   = 290 items/day
  clusters formed                     = ~180
  clusters kept for the digest        = 12
  mean members per kept cluster       = 2.4

Embedding step:
  290 items x ~120 tokens (title + 200 chars)   = 34,800 tokens

Summarisation step:
  12 clusters x (2.4 members x 250 tokens + 200-token prompt)
    = 12 x 800 in                               =  9,600 input tokens
  12 x 90 tokens out                            =  1,080 output tokens

Daily totals:  34,800 embedding tokens
                9,600 input tokens
                1,080 output tokens

Cost = 34,800 x P_embed + 9,600 x P_in + 1,080 x P_out   (per million)

Worked with a small model at $0.15 / $0.60 per million and embeddings at
$0.02 per million — substitute YOUR posted prices:

  embeddings  34,800/1e6 x 0.02 = $0.0007
  input        9,600/1e6 x 0.15 = $0.0014
  output       1,080/1e6 x 0.60 = $0.0006
                          TOTAL ~ $0.003 per day  ~ $0.09 per month

The one thing that breaks this budget is summarising before selecting. At 290 items instead of 12 clusters, the summarisation input goes from 9,600 tokens to roughly 90,000 — a twenty-fold increase in the expensive term, for output nobody reads.

Running it on cron without regret

# crontab -e
# 06:17 every weekday. Odd minute deliberately: every feed you poll is
# also being polled by everyone else on the hour.
17 6 * * 1-5 /usr/bin/flock -n /var/lock/digest.lock /opt/digest/run.sh >> /var/log/digest.log 2>&1

1. flock -n so a slow run cannot overlap the next one. Without it, a run that hangs on a dead feed produces two agents fetching the same feeds and two digests.
2. Idempotent by item id. Processing an item twice must be harmless. Then a crash mid-run is fixed by running it again, which is the entire recovery procedure.
3. A dead-man switch. Cron tells you nothing when a job does not run. Have the script report success to something that alerts on silence — a digest that quietly stopped three weeks ago is the characteristic failure of every cron job ever written.
4. Timeout every network call. One feed with a half-open connection will otherwise hold the job open until the next one starts, and flock will then silently skip that run. That applies to the model calls too — a request with no timeout can hang for a very long time.
5. Write the digest before sending it. Render to a file or a database row, then send. If sending fails you still have the digest and can resend without re-running anything.

For anything with more steps than this, a resumable state machine beats a cron script — but for a job whose recovery is “run it again”, cron plus a lock is genuinely the right answer.

Delivery, which fails silently

The digest is generated correctly and nobody receives it. This happens more often than any of the failure modes above, it produces no error, and the sender finds out weeks later.

1. Send from a domain you have configured for it. A message sent from a domain with no sender authentication set up will be filtered by most receivers, silently, with no bounce. If you do not know whether yours is configured, that is the first thing to check — the mail provider’s own documentation will list what it needs, and it is a DNS change rather than code.
2. Read the bounces. A hard bounce means the address is dead and must be removed; a soft bounce is temporary. A digest that keeps sending to a dead address for a year damages the sending reputation of everything else you send.
3. Send one message per recipient, not one with fifty addresses. Otherwise every reader sees the list, which is a data protection incident and an embarrassment in the same email.
4. Publish the digest somewhere as well as sending it. A URL per issue means a reader who did not receive it can still read it, and it gives you a delivery-independent record of what was sent on a given day.
5. Alert on a send that produced zero recipients as loudly as on an exception. A subscriber query that silently returns an empty list is the specific bug that ends a digest.

The general principle: the pipeline should be able to answer “did issue 214 reach Anna” from its own database, not from the mail provider’s dashboard. One row per issue per recipient, with a status, and the question becomes a query.

What makes a digest worth reading

• Say what happened, not what the article is about. “X acquired Y for £400m” beats “an article discussing the acquisition of Y”. Put that instruction in the prompt in those words.
• Name the sources on the cluster. “Reported by four outlets” is information, and it is the payoff for the clustering work.
• Link to the original of each member, not just the first one. Readers want the outlet they trust.
• Include what did not make it. A one-line “also: 34 items filtered, 9 clusters below the threshold” makes the digest auditable and stops the reader wondering what was hidden.
• Never let the summary assert something no member said. Ground it the same way as everywhere else in this cluster: the summary is written from the member texts and nothing else, and a claim not traceable to a member is a defect.