refactor(scripts): extract shared SourceData lib for source fetchers #11

Merged
alex merged 2 commits from pi/issue10-1792746852837 into master 2026-10-23 09:29:18 +00:00
Contributor

Summary

The four scripts/update_*.cr source-fetch scripts each reimplemented the same tail end of logic. This factors the shared pieces into a small script-local library, scripts/lib/source_data.cr:

  • SourceData.fetch(url, headers?) — GET a URL, abort with a consistent error message on non-success status (previously duplicated in all four scripts with slightly different messages).
  • SourceData.write_if_changed(target_path, payload, label) — pretty-serialize the payload, compare against the bundled src/data/*.json, skip the write (and print "Up to date (N …)") when unchanged, otherwise mkdir_p + write and print "Updated …". Previously ~20 duplicated lines per script.

Each script keeps its own upstream-specific logic (parsing, dedupe, normalization, pagination, politeness pauses) — only the truly identical parts were extracted.

Validation

  • All four scripts compile individually (crystal build).
  • crystal run scripts/update_quotes.cr verified end-to-end: same output format and byte-identical JSON (the regenerated file only differed because upstream added quotes; the bundled data was left untouched).
  • crystal tool format --check passes, ameba reports 0 failures on scripts/, and the unit suite passes (463 examples).

Closes #10

## Summary The four `scripts/update_*.cr` source-fetch scripts each reimplemented the same tail end of logic. This factors the shared pieces into a small script-local library, `scripts/lib/source_data.cr`: - `SourceData.fetch(url, headers?)` — GET a URL, abort with a consistent error message on non-success status (previously duplicated in all four scripts with slightly different messages). - `SourceData.write_if_changed(target_path, payload, label)` — pretty-serialize the payload, compare against the bundled `src/data/*.json`, skip the write (and print "Up to date (N …)") when unchanged, otherwise `mkdir_p` + write and print "Updated …". Previously ~20 duplicated lines per script. Each script keeps its own upstream-specific logic (parsing, dedupe, normalization, pagination, politeness pauses) — only the truly identical parts were extracted. ## Validation - All four scripts compile individually (`crystal build`). - `crystal run scripts/update_quotes.cr` verified end-to-end: same output format and byte-identical JSON (the regenerated file only differed because upstream added quotes; the bundled data was left untouched). - `crystal tool format --check` passes, `ameba` reports 0 failures on `scripts/`, and the unit suite passes (463 examples). Closes #10
alex merged commit 4c86e9a088 into master 2026-10-23 09:29:18 +00:00
alex deleted branch pi/issue10-1792746852837 2026-10-23 09:29:18 +00:00
Sign in to join this conversation.
No reviewers
No labels
dependencies
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
alex/botium!11
No description provided.