Refactor sources fetchers #10

Closed
opened 2026-10-23 09:09:53 +00:00 by alex · 2 comments
Owner

I believe most of source fetch scripts under scripts/ folder rewrite at least part or the same logic.

Let's factor that out into a shared lib to keep them DRY

I believe most of source fetch scripts under scripts/ folder rewrite at least part or the same logic. Let's factor that out into a shared lib to keep them DRY
Author
Owner

/pi new PR

/pi new PR
Contributor

The four update_*.cr scripts share the same fetch → dedupe → serialize → compare → write tail. Let me check the toolchain and then refactor.Now I'll create a shared library and refactor the four scripts.Now refactor each script:Created PR #11: #11 — "refactor(scripts): extract shared SourceData lib for source fetchers"

What it does:

  • New scripts/lib/source_data.cr with two helpers shared by all source fetchers:
    • SourceData.fetch(url, headers?) — upstream GET with a consistent abort-on-failure message.
    • SourceData.write_if_changed(target_path, payload, label) — pretty JSON serialization, change detection against the bundled src/data/*.json, mkdir_p + write, and the "Up to date (N …)" / "Updated …" logging.
  • All four update_{dilbert,monkeyuser,quotes,workchronicles}.cr scripts refactored onto it (net −63 lines), keeping their upstream-specific parsing/dedupe/normalization logic intact.

Validation: all four scripts compile, update_quotes.cr ran end-to-end against upstream, crystal tool format --check and ameba pass, and the unit suite is green (463 examples).


View action run | Model: zai/glm-5.3-flash (thinking: low) | Time: 4m 9s | Tokens: 184.9K | Cost: $0.01 | Pi SDK v1.4.0 | Action v2.30.1-develop (develop @ 4ea2232)

<!-- pi-coding-agent-comment --> The four `update_*.cr` scripts share the same fetch → dedupe → serialize → compare → write tail. Let me check the toolchain and then refactor.Now I'll create a shared library and refactor the four scripts.Now refactor each script:Created PR #11: https://forge.l3x.in/alex/botium/pulls/11 — "refactor(scripts): extract shared SourceData lib for source fetchers" **What it does:** - New `scripts/lib/source_data.cr` with two helpers shared by all source fetchers: - `SourceData.fetch(url, headers?)` — upstream GET with a consistent abort-on-failure message. - `SourceData.write_if_changed(target_path, payload, label)` — pretty JSON serialization, change detection against the bundled `src/data/*.json`, `mkdir_p` + write, and the "Up to date (N …)" / "Updated …" logging. - All four `update_{dilbert,monkeyuser,quotes,workchronicles}.cr` scripts refactored onto it (net −63 lines), keeping their upstream-specific parsing/dedupe/normalization logic intact. **Validation:** all four scripts compile, `update_quotes.cr` ran end-to-end against upstream, `crystal tool format --check` and `ameba` pass, and the unit suite is green (463 examples). --- [View action run](https://forge.l3x.in/alex/botium/actions/runs/86) | Model: zai/glm-5.3-flash (thinking: low) | Time: 4m 9s | Tokens: 184.9K | Cost: $0.01 | Pi SDK v1.4.0 | Action v2.30.1-develop (develop @ 4ea2232)
alex closed this issue 2026-10-23 09:29:18 +00:00
Sign in to join this conversation.
No labels
dependencies
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
alex/botium#10
No description provided.