feat: add RSS/Atom feed announcements with polite polling

pi commented

2026-07-24 11:28:30 +00:00

Contributor

Closes #13.

Summary

Adds optional RSS/Atom feed monitoring: the bot polls a configured list of feeds and announces new items into the joined IRC channels, configured entirely via env vars like the rest of the bot.

Library evaluation

Per the issue, I checked what Deno offers for RSS/Atom reading. There is no RSS/Atom reader in the Deno standard library, and DOMParser is not available in Deno (the popular @b-fuze/deno-dom doesn't implement text/xml). The mature options are npm packages (rss-parser, @extractus/feed-extractor) which pull in non-trivial transitive dependency trees.

Given the project's deliberate minimalism (the Chatto client hand-speaks Connect JSON, storage is a plain JSON file, schedule is pure date math), I kept it dependency-free: a small, focused, fully unit-tested XML parser (src/xml.ts) feeds a normalizer (src/feed.ts) that handles RSS 2.0, RSS 1.0/RDF, and Atom 1.0. A malformed feed is caught and retried on the next poll — it never crashes the bot.

New modules

File	Responsibility
`src/xml.ts`	Dependency-free XML tokenizer → tree (CDATA, entities, namespaces, comments).
`src/feed.ts`	Feed parsing, dedup selection, formatting, politeness math, conditional-GET fetching, and the poll loop.
`src/feed-store.ts`	Disk-backed per-feed checkpoint store (seen ids + `ETag`/`Last-Modified`).

Each has a matching *_test.ts (78 new tests; 158 total, all green).

Configuration (env vars)

All optional; IRC_FEEDS unset = disabled:

IRC_FEEDS — comma-separated feed URLs (validated as http(s)).
IRC_FEED_INTERVAL_MIN — minimum interval in minutes (default 60, must be ≥ 5).
IRC_FEED_CHECKPOINT_PATH — checkpoint file (default feed-checkpoints.json).
IRC_FEED_USER_AGENT — identifiable UA with a contact URL.
IRC_FEED_MESSAGE — template, default 📰 {feed}: {title} — {link} ({title}/{link}/{feed}).
IRC_FEED_MAX_ITEMS — items announced per feed per cycle, newest kept (default 3).

Be polite to feed hosts

Politeness is treated as a first-class concern, following the RSS profiling best practices:

Conditional GET — sends stored If-None-Match / If-Modified-Since; a 304 is skipped entirely.
Per-feed scheduling honors the feed's own cadence — <ttl>, the sy:updatePeriod module, and HTTP Cache-Control: max-age / Expires all raise the interval. The configured interval is a floor; feed hints can only make the bot slower.
Hard 5-minute minimum floor so a typo can't turn the bot into a hammer.
Backoff — 429/503 with Retry-After is honored.
Identifiable User-Agent with a contact URL.
Stagger + jitter — feeds are polled sequentially with a short delay, and each next poll is jittered so the bot never hits many hosts on the same tick.

Avoiding duplicates

Seen item ids plus ETag/Last-Modified are persisted to JSON, so duplicates are suppressed across restarts. On the first-ever poll of a brand-new feed the current items are seeded as seen without announcing (no channel flood with the backlog). While the bot is disconnected from IRC, polling is paused entirely so it never marks items seen that it couldn't announce.

Verified end-to-end against a local HTTP server: first fetch 200 (parsed, ttl/Cache-Control extracted), second fetch with the stored ETag → 304 Not Modified, and the effective delay correctly resolved to max(configured, ttl, max-age).

CI

deno fmt --check, deno lint, deno check, deno test (158 passed), fallow dead-code --fail-on-issues (no issues), and deno compile all pass.

Closes #13. ## Summary Adds optional RSS/Atom feed monitoring: the bot polls a configured list of feeds and announces new items into the joined IRC channels, configured entirely via env vars like the rest of the bot. ## Library evaluation Per the issue, I checked what Deno offers for RSS/Atom reading. There is **no RSS/Atom reader in the Deno standard library**, and `DOMParser` is not available in Deno (the popular `@b-fuze/deno-dom` doesn't implement `text/xml`). The mature options are npm packages (`rss-parser`, `@extractus/feed-extractor`) which pull in non-trivial transitive dependency trees. Given the project's deliberate minimalism (the Chatto client hand-speaks Connect JSON, storage is a plain JSON file, schedule is pure date math), I kept it **dependency-free**: a small, focused, fully unit-tested XML parser (`src/xml.ts`) feeds a normalizer (`src/feed.ts`) that handles **RSS 2.0, RSS 1.0/RDF, and Atom 1.0**. A malformed feed is caught and retried on the next poll — it never crashes the bot. ## New modules | File | Responsibility | | --- | --- | | `src/xml.ts` | Dependency-free XML tokenizer → tree (CDATA, entities, namespaces, comments). | | `src/feed.ts` | Feed parsing, dedup selection, formatting, **politeness math**, conditional-GET fetching, and the poll loop. | | `src/feed-store.ts` | Disk-backed per-feed checkpoint store (seen ids + `ETag`/`Last-Modified`). | Each has a matching `*_test.ts` (78 new tests; **158 total, all green**). ## Configuration (env vars) All optional; `IRC_FEEDS` unset = disabled: - `IRC_FEEDS` — comma-separated feed URLs (validated as `http(s)`). - `IRC_FEED_INTERVAL_MIN` — minimum interval in minutes (default 60, **must be ≥ 5**). - `IRC_FEED_CHECKPOINT_PATH` — checkpoint file (default `feed-checkpoints.json`). - `IRC_FEED_USER_AGENT` — identifiable UA with a contact URL. - `IRC_FEED_MESSAGE` — template, default `📰 {feed}: {title} — {link}` (`{title}`/`{link}`/`{feed}`). - `IRC_FEED_MAX_ITEMS` — items announced per feed per cycle, newest kept (default 3). ## Be polite to feed hosts Politeness is treated as a first-class concern, following the [RSS profiling best practices](https://www.rssboard.org/rss-profiling-best-practices): - **Conditional GET** — sends stored `If-None-Match` / `If-Modified-Since`; a `304` is skipped entirely. - **Per-feed scheduling honors the feed's own cadence** — `<ttl>`, the `sy:updatePeriod` module, and HTTP `Cache-Control: max-age` / `Expires` all *raise* the interval. The configured interval is a floor; feed hints can only make the bot **slower**. - **Hard 5-minute minimum** floor so a typo can't turn the bot into a hammer. - **Backoff** — `429`/`503` with `Retry-After` is honored. - **Identifiable `User-Agent`** with a contact URL. - **Stagger + jitter** — feeds are polled sequentially with a short delay, and each next poll is jittered so the bot never hits many hosts on the same tick. ## Avoiding duplicates Seen item ids plus `ETag`/`Last-Modified` are persisted to JSON, so duplicates are suppressed across restarts. On the **first-ever** poll of a brand-new feed the current items are seeded as seen *without* announcing (no channel flood with the backlog). While the bot is disconnected from IRC, polling is paused entirely so it never marks items seen that it couldn't announce. Verified end-to-end against a local HTTP server: first fetch `200` (parsed, `ttl`/`Cache-Control` extracted), second fetch with the stored `ETag` → `304 Not Modified`, and the effective delay correctly resolved to `max(configured, ttl, max-age)`. ## CI `deno fmt --check`, `deno lint`, `deno check`, `deno test` (158 passed), `fallow dead-code --fail-on-issues` (no issues), and `deno compile` all pass.

pi added 1 commit

2026-07-24 11:28:30 +00:00

feat: add RSS/Atom feed announcements with polite polling

PR / deno (pull_request) Successful in 22s

Details

PR / fallow (pull_request) Successful in 24s

Details

2ef4d3cdf5

Co-authored-by: alex <alex@forge.l3x.in>

pi referenced this pull request

2026-07-24 11:28:41 +00:00

Evaluate RSS integration #13

alex commented

2026-07-26 21:05:02 +00:00

Owner

/pi rebase on master, solve merge conflicts

forgejo-actions force-pushed pi/issue13-1784892508497 from 2ef4d3cdf5

PR / deno (pull_request) Successful in 22s

Details

PR / fallow (pull_request) Successful in 24s

Details

to c419fb059d

2026-07-26 21:08:39 +00:00

Compare

pi commented

2026-07-26 21:08:53 +00:00

Author

Contributor

I'll start by examining the current state of the PR branch and the repository.Let me check out the PR branch and see what conflicts exist.The PR branch is based on f9f3c73, but master has one new commit 81d0a9b (announce module). Let me rebase onto master.Two files have conflicts: README.md and src/main.ts. Let me examine both.Only README.md has an unresolved conflict; src/main.ts auto-merged. Let me examine both to make sure the auto-merge is correct and resolve the README conflict.Two conflict regions in README.md: both are additions that should be combined. Let me resolve them.Now let me verify no conflict markers remain, and inspect the auto-merged src/main.ts to ensure correctness.There's still an orphaned conflict marker at line 171. Let me look at that region.All conflict markers resolved. The auto-merged src/main.ts correctly combines both the announce module (from master) and the feed poller (from the PR). Let me stage the README and continue the rebase.Rebase complete. Now let me verify everything works: format, lint, type check, and tests.The test failures are purely permission-related (missing --allow-write for temp dirs), not actual test failures. Let me run via the task definition which includes the correct permissions.All 165 tests pass. Let me also run the fallow dead-code check and compile, as those are part of CI.All checks pass. Let me verify the final state of the rebase — the branch should now sit cleanly on top of master with the announce commit as its parent.The rebase is clean — the branch now sits directly on top of master (parent 81d0a9b) with a single commit. All CI checks pass (fmt, lint, check, 165 tests, fallow, compile). Now let me force-push the rebased branch to update the PR.Rebase complete and pushed. Here's what happened:

Conflict in README.md — the new master commit 81d0a9b (announce module) and the PR's feed commit both touched two spots:

After the "Daily horse facts" section, master added a paragraph about facts being mirrored verbatim to Chatto, while the PR added the entire "RSS/Atom feed announcements" subsection. Both are now kept in sequence.
In the project-structure listing, master added announce.ts and the PR added feed.ts / feed-store.ts / xml.ts. All four entries are now present in a sensible order.

src/main.ts auto-merged cleanly — it correctly wires up both the announce module (master) and the feed poller (PR).

The branch now sits directly on top of master (parent 81d0a9b) with a single commit. Verified all CI-equivalent checks pass locally before pushing:

deno fmt --check ✓
deno lint ✓
deno check src/main.ts ✓
deno task test → 165 passed, 0 failed ✓
deno task fallow → no issues ✓
deno task compile ✓

Force-pushed (--force-with-lease) to pi/issue13-1784892508497; the PR is now rebased onto master with no conflicts.

I'll start by examining the current state of the PR branch and the repository.Let me check out the PR branch and see what conflicts exist.The PR branch is based on `f9f3c73`, but master has one new commit `81d0a9b` (announce module). Let me rebase onto master.Two files have conflicts: `README.md` and `src/main.ts`. Let me examine both.Only `README.md` has an unresolved conflict; `src/main.ts` auto-merged. Let me examine both to make sure the auto-merge is correct and resolve the README conflict.Two conflict regions in README.md: both are additions that should be combined. Let me resolve them.Now let me verify no conflict markers remain, and inspect the auto-merged `src/main.ts` to ensure correctness.There's still an orphaned conflict marker at line 171. Let me look at that region.All conflict markers resolved. The auto-merged `src/main.ts` correctly combines both the announce module (from master) and the feed poller (from the PR). Let me stage the README and continue the rebase.Rebase complete. Now let me verify everything works: format, lint, type check, and tests.The test failures are purely permission-related (missing `--allow-write` for temp dirs), not actual test failures. Let me run via the task definition which includes the correct permissions.All 165 tests pass. Let me also run the fallow dead-code check and compile, as those are part of CI.All checks pass. Let me verify the final state of the rebase — the branch should now sit cleanly on top of master with the announce commit as its parent.The rebase is clean — the branch now sits directly on top of `master` (parent `81d0a9b`) with a single commit. All CI checks pass (fmt, lint, check, 165 tests, fallow, compile). Now let me force-push the rebased branch to update the PR.Rebase complete and pushed. Here's what happened: **Conflict in `README.md`** — the new master commit `81d0a9b` (announce module) and the PR's feed commit both touched two spots: - After the "Daily horse facts" section, master added a paragraph about facts being mirrored verbatim to Chatto, while the PR added the entire "RSS/Atom feed announcements" subsection. Both are now kept in sequence. - In the project-structure listing, master added `announce.ts` and the PR added `feed.ts` / `feed-store.ts` / `xml.ts`. All four entries are now present in a sensible order. **`src/main.ts`** auto-merged cleanly — it correctly wires up both the announce module (master) and the feed poller (PR). The branch now sits directly on top of `master` (parent `81d0a9b`) with a single commit. Verified all CI-equivalent checks pass locally before pushing: - `deno fmt --check` ✓ - `deno lint` ✓ - `deno check src/main.ts` ✓ - `deno task test` → **165 passed**, 0 failed ✓ - `deno task fallow` → no issues ✓ - `deno task compile` ✓ Force-pushed (`--force-with-lease`) to `pi/issue13-1784892508497`; the PR is now rebased onto master with no conflicts. --- [View action run](https://forge.l3x.in/alex/irc-greeter-bot/actions/runs/63) | Model: zai/glm-5.2 (thinking: max) | Time: 3m 43s | Tokens: 367.4K | Pi SDK v0.81.1 | Action v2.26.0-develop (develop @ 4387e26)

alex added 1 commit

2026-07-26 21:22:04 +00:00

wip

PR / deno (pull_request) Successful in 57s

Details

PR / fallow (pull_request) Successful in 40s

Details

7ee20421e9

alex merged commit e2d1c956e2 into master

2026-07-26 21:22:29 +00:00

alex deleted branch pi/issue13-1784892508497

2026-07-26 21:22:29 +00:00