An EventBridge Rule That Triggers a Scheduled Model Pipeline

“Run the embedding job every night at three” is a two-line change and a set of assumptions that fail in the second month: the run that takes longer than the interval, the daylight-saving shift, the silent failure nobody sees because the target was invoked successfully.

Scheduler, not a scheduled rule

There are two ways to run something on a schedule in EventBridge. The older one is a rule on the default event bus with a schedule expression, targeting up to five resources. The newer one is EventBridge Scheduler, a separate service with its own API (aws scheduler ...) and its own resource type.

For a pipeline, Scheduler is the better default, for reasons that are concrete rather than aesthetic:

• Time zones. Scheduler evaluates a cron expression in an IANA time zone you specify. A scheduled rule is UTC only, so “3am local” drifts by an hour twice a year.
• Universal targets. Scheduler can call, in AWS’s stated figures, more than 270 services and over 6,000 API operations directly — including batch:SubmitJob — without a Lambda in between.
• Scale and cleanup. AWS documents a default quota of 10,000,000 schedules per Region, adjustable to billions, versus a much smaller rule budget on an event bus. And ActionAfterCompletion can delete a one-time schedule after it fires, which matters because completed one-time schedules still count against the quota.
• Flexible time windows. A window disperses invocations rather than firing a thousand schedules on the same second — useful precisely when the target is a rate-limited model API.

Creating the schedule

1. Create a role Scheduler can assume, trusting scheduler.amazonaws.com, with permission for exactly the one target action — batch:SubmitJob, or lambda:InvokeFunction on the one function ARN.
2. Create the schedule. --flexible-time-window is required even when you do not want one; pass Mode: OFF for exact timing.
3. Send failures somewhere. Scheduler delivers at least once and retries, and without a dead-letter queue a target that is failing produces no signal at all.

aws scheduler create-schedule \
  --name nightly-embed \
  --schedule-expression "cron(0 3 * * ? *)" \
  --schedule-expression-timezone "Europe/Amsterdam" \
  --flexible-time-window '{"Mode":"OFF"}' \
  --target '{
    "Arn": "arn:aws:scheduler:::aws-sdk:batch:submitJob",
    "RoleArn": "arn:aws:iam::111122223333:role/scheduler-invoke-batch",
    "Input": "{\"JobName\":\"nightly-embed\",\"JobQueue\":\"embed-queue\",\"JobDefinition\":\"embed-corpus\"}",
    "RetryPolicy": { "MaximumRetryAttempts": 2, "MaximumEventAgeInSeconds": 3600 },
    "DeadLetterConfig": { "Arn": "arn:aws:sqs:us-east-1:111122223333:schedule-dlq" }
  }'

The arn:aws:scheduler:::aws-sdk:batch:submitJob form is the universal target parameter: it names an SDK operation rather than a resource, and Input is the request body that operation expects, in its own casing. That last detail is the usual first failure — the SDK shape uses JobQueue, not the CLI’s --job-queue.

The cron dialect

Scheduler’s cron is not Unix cron and the differences bite. AWS documents six required fields, not five: cron(minutes hours day-of-month month day-of-week year). Beyond the extra year field:

• Day-of-week is 1–7 for SUN–SAT. In Unix cron, 1 is Monday. Every “it ran on the wrong day” report starts here.
• You cannot use * in both day-of-month and day-of-week — one of them must be ?. This is why the daily expression above reads 0 3 * * ? *.
• L, W and # are supported: 6L is the last Friday, 3W the weekday nearest the third, 3#2 the second Tuesday.
• All schedule types invoke with 60-second precision — a schedule set for 1:00 fires between 1:00:00 and 1:00:59 when no flexible window is set. Do not build anything that assumes the second.

Daylight saving has documented, specific behaviour that is worth knowing before it surprises you: on the spring-forward, a cron expression that lands on a time which does not exist is skipped for that day; on the fall-back, a time that occurs twice runs once. A job at 2:30am local time therefore does not run at all on one day a year. If the run is mandatory, schedule it in UTC, or outside the shift window, and accept the clock drift instead.

Whether the Lambda earns its place

The familiar pattern is schedule fires, Lambda runs, Lambda submits a job. With universal targets the Lambda is optional, and the question is whether it is doing work or just forwarding.

Delete it when the submission is static — same queue, same job definition, same parameters every night. A hop that only forwards is a hop that can fail, needs a role, needs logs and needs a timeout.

Keep it when something must be decided at fire time:

• The job’s parameters depend on the date or on a manifest that must be built — today’s S3 prefix, yesterday’s delta, the list of tenants due for reindexing.
• The run should be skipped conditionally, which is the overlap check below and the single best reason to keep it.
• The pipeline has more than one step with branching. At that point the right target is a Step Functions state machine rather than a Lambda — it gets you retry and catch semantics per step, which a Lambda would otherwise reimplement badly.

Overlap is your problem, not the scheduler’s

This is the part every scheduled-pipeline tutorial omits and every scheduled pipeline eventually hits. A schedule fires on time regardless of whether the previous run finished. There is no concurrency setting, no “skip if running” flag, and nothing in Scheduler that knows your job exists beyond having invoked something. A nightly embedding job that usually takes 40 minutes and one night takes 25 hours will have a second copy launched underneath it, both writing the same S3 prefix.

The three workable guards, in increasing order of robustness:

1. Check before submitting. In the Lambda, call aws batch list-jobs --job-queue embed-queue --job-status RUNNING plus RUNNABLE and STARTING, and return without submitting if a job with your name prefix is present. Cheap, readable, and racy under a flexible time window — adequate for a nightly job, not for one every five minutes.
2. Take a lock. A conditional DynamoDB write on a fixed key with a TTL is an actual mutual exclusion, and the TTL means a crashed run releases it rather than wedging the pipeline permanently.
3. Make overlap harmless. Shard by an idempotency key — a run id derived from the date, output objects checked before they are written — so a second copy finds the work done and exits. This is more work up front and the only one that also survives a manual re-run, which is the scenario that actually happens during an incident.

Whatever you choose, alarm on the absence of the run rather than on its failure. Scheduler’s own metrics tell you it invoked the target; they say nothing about whether the pipeline produced anything. An alarm on a business-level metric — documents embedded in the last 24 hours — catches the case where everything succeeded and nothing happened, which is the failure mode of a scheduled job that nobody is watching.