From feed to board

This is the actual pipeline that puts a posting on this board, step by step, with the code that does each step. Nothing is posted by hand; the board is a read-only view of what the pipeline collected.

1. Fetch. A scheduled job runs app/fetch_feeds.py --live --write once a day. It makes one request per Greenhouse board and at most five pages of twenty postings per Workday board, one second apart, all through the operation’s proxy. Every response is cached under app/cache/ with the date in the file name, so nothing is fetched twice in one day.
2. Filter. A posting is kept only when its title says wind, turbine or blade (TITLE_FILTER in fetch_feeds.py). Description text is never stored — the board keeps the posting’s facts only: title, employer, place, pay when stated, link, date.
3. Normalize. Places are put in one form — “Haskell, TX, US” becomes “Haskell, TX” (store.clean_location). Postings outside the US and Canada are rejected outright.
4. Deduplicate. Two keys: the posting’s URL, and employer + title + place. Employers re-post the same role under new requisition URLs; the second key collapses the copies (store.content_key).
5. Age window. Postings older than 60 days neither enter the store nor get served.
6. Serve. This site (app/serve.py) reads app/data/jobs.csv on every request and renders the page from it. There is no database and no stored HTML to go stale.
7. Re-check links. At the end of each run every stored apply link is re-checked on the same proxied path (fetch_feeds.check_apply_links). Workday rows are judged by today’s feed data, because Workday job pages refuse server-side readers even for live postings; Greenhouse rows by membership in the board’s API; every other employer gets one page fetch, checked for the employer’s own “posting expired” wording. Two consecutive failures and the posting comes off the board automatically; if more than half the checks fail in one run, the run itself is judged broken and nothing is removed. Removals are logged in app/data/removed-postings.log.

The latest run, in numbers

The numbers beside each step below come from that capture; a new capture replaces them (see RUNBOOK.md).

Run of

2026-10-16 (capture: app/runs/fetch-2026-10-16.log)

Boards fetched

4

Wind postings seen

22

Merged

0 added, 22 skipped (already listed, outside the window, or duplicates)

Apply links

151 judged, 7 failed, 1 removed, 1 left unknown

Employers whose career sites carry no readable feed are collected by sweep instead — the sources page says which path each employer takes.