generated from alex/base-template
refactor(scripts): extract shared SourceData lib for source fetchers #11
No reviewers
Labels
No labels
dependencies
No milestone
No assignees
2 participants
Notifications
Due date
No due date set.
Reference
alex/botium!11
Loading…
Reference in a new issue
No description provided.
Delete branch "pi/issue10-1792746852837"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
The four
scripts/update_*.crsource-fetch scripts each reimplemented the same tail end of logic. This factors the shared pieces into a small script-local library,scripts/lib/source_data.cr:SourceData.fetch(url, headers?)— GET a URL, abort with a consistent error message on non-success status (previously duplicated in all four scripts with slightly different messages).SourceData.write_if_changed(target_path, payload, label)— pretty-serialize the payload, compare against the bundledsrc/data/*.json, skip the write (and print "Up to date (N …)") when unchanged, otherwisemkdir_p+ write and print "Updated …". Previously ~20 duplicated lines per script.Each script keeps its own upstream-specific logic (parsing, dedupe, normalization, pagination, politeness pauses) — only the truly identical parts were extracted.
Validation
crystal build).crystal run scripts/update_quotes.crverified end-to-end: same output format and byte-identical JSON (the regenerated file only differed because upstream added quotes; the bundled data was left untouched).crystal tool format --checkpasses,amebareports 0 failures onscripts/, and the unit suite passes (463 examples).Closes #10