Logbook of code changes

git · July 19, 2026, 10:39pm

github.com/earthobservations/wetterdienst

feat(knmi): add KNMI (Netherlands) observation provider (#1739)

committed 10:39PM - 19 Jul 26 UTC

Add the Royal Netherlands Meteorological Institute (KNMI) as an observation prov…ider with 10-minute, hourly and daily resolution, sourced from KNMI's modern Data Platform (NetCDF, API-key auth). Unlike other providers, KNMI's Data Platform has no per-station history endpoint: each file covers a single timestamp for the whole network, so a station time series means one file download per timestamp. Stations are enumerated per resolution (the 10-minute network is a superset of the hourly/daily one); rate-limit/transient retries mirror the AEMET pattern. The 10-minute resolution is KNMI's real-time feed (~30 min latency); the hourly/daily "validated" datasets lag ~2 days. The separate Actuele10mindataKNMIstations dataset is deprecated (frozen since Oct 2025) and intentionally not used. Notable specifics, all verified against live data: - Stations are exposed by their trailing WMO station number (De Bilt = 06260) rather than the full WSI. - KNMI's internal NetCDF "time" coordinate marks the *end* of the aggregation period; values are dated to the requested moment instead, matching the legacy daggegevens CSV service. - A start/end not on a resolution boundary is floored so the per-file request resolves; the framework trims the over-fetched edge. - Trace precipitation (encoded as -1) is normalized to 0. Files are read with h5netcdf directly into polars (no xarray/pandas): they are plain NetCDF4/HDF5 with unpacked float variables, so xarray's labeled-array machinery -- and its pandas dependency -- would be dead weight. The knmi extra therefore needs only h5netcdf, and is installed explicitly in CI (install.sh) and the Docker image. A free KNMI Data Platform API key is required: set WD_AUTH__KNMI=<key> or Settings(auth={"knmi": "<key>"}). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

git · July 19, 2026, 10:43pm

git · July 20, 2026, 3:23pm

git · July 20, 2026, 8:38pm

git · July 21, 2026, 3:17pm

github.com/panodata/macropipe

Chore(deps-dev): Bump ty from 0.0.60 to 0.0.61

committed 03:17PM - 21 Jul 26 UTC

dependabot[bot]

+1 -1

Bumps [ty](https://github.com/astral-sh/ty) from 0.0.60 to 0.0.61. - [Release no…tes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ty/compare/0.0.60...0.0.61) --- updated-dependencies: - dependency-name: ty dependency-version: 0.0.61 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

git · July 21, 2026, 9:29pm

git · July 21, 2026, 10:10pm

github.com/earthobservations/wetterdienst

perf(values): store aggregated result metadata columns as Enum

committed 10:10PM - 21 Jul 26 UTC

gutzbenj

+331 -101

Store the low-cardinality station_id, resolution, dataset and parameter columns …of aggregated value results (`.values.all()`) as polars Enum instead of String. These columns repeat on every row of tidy frames and account for roughly half their footprint; integer-backed Enum codes cut frame size by ~57% (50y daily, 100 stations, 14 params: 1.6 GiB -> 683 MiB). Categories are derived from the values actually present, so the cast never fails on provider-specific casing/humanization quirks (e.g. WSV emitting 'w' while the metadata declares 'W'). The streaming query() path is left as String; only the aggregated all() frame is cast. Provider value-result test expectations are updated to the Enum dtype. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 21, 2026, 10:16pm

git · July 22, 2026, 9:36am

git · July 22, 2026, 8:08pm

git · July 22, 2026, 10:39pm

github.com/earthobservations/wetterdienst

test(dwd/radar): skip BUFR tests on malformed payload instead of failing

committed 10:39PM - 22 Jul 26 UTC

gutzbenj

+12 -4

The site PE/PX250 BUFR "yesterday" tests assert the downloaded payload starts wi…th the BUFR magic header. DWD intermittently serves an incomplete/placeholder file for a timestamp whose data is not yet fully available, which is a data availability issue rather than a code regression. Treat a non-BUFR payload the same as an empty result (pytest.skip) so transient DWD data hiccups no longer turn CI red; valid BUFR payloads are still decoded and verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 23, 2026, 5:59am

github.com/earthobservations/wetterdienst

feat(fmi): add FMI (Finland) observation provider

committed 05:59AM - 23 Jul 26 UTC

gutzbenj

+953 -0

Add a key-less FMI (Finnish Meteorological Institute) observation provider cover…ing hourly and daily resolutions via FMI's open WFS stored queries. - station metadata from the fmi::ef::stations INSPIRE EF catalogue - observations from the ...::simple / ...::daily::simple stored queries, chunked into <=168h windows (FMI's sub-daily cap) with shared, de-duplicated boundary timestamps so no reading is lost regardless of endtime inclusivity - normalise FMI's -1 "no precipitation / no snow" sentinel to 0.0 - surface OWS ExceptionReport responses (HTTP 200) instead of silently treating them as empty - lxml parsing with entity resolution and network access disabled (XXE-safe) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 23, 2026, 5:59am

git · July 23, 2026, 6:45am

github.com/earthobservations/wetterdienst

perf(mosmix): stream zipped KML to cut parsing memory

committed 06:45AM - 23 Jul 26 UTC

gutzbenj

+42 -36

Parse the DWD MOSMIX/DMO KML by streaming straight from the zip handle instead o…f decompressing the whole ~600MB KML into memory. Combined with end-only iterparse events and pruning of already-parsed placemarks (clear + drop preceding siblings), peak memory is bounded by the compressed file size plus a single placemark rather than the full decompressed document. Same request (MOSMIX-S, one station) drops from ~1390MB to ~214MB peak RSS (~6.5x). Output is byte-identical; remote mosmix and dmo tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 23, 2026, 9:01am

github.com/earthobservations/wetterdienst

fix(noaa/ghcn): parse GHCN-hourly timestamps from the ISO date column

committed 09:01AM - 23 Jul 26 UTC

gutzbenj

+46 -8

The hourly (GHCNh) reader reconstructed each timestamp by concatenating the sepa…rate Year/Month/Day/Hour/Minute columns and parsing "%Y%m%d%H%M", which silently relies on every field being zero-padded. GHCNh already provides a ready-made ISO-8601 DATE column, so parse that directly (%Y-%m-%dT%H:%M:%S) -- simpler and robust against any unpadded rows. Also add an hourly regression test (the hourly path previously had no test coverage; only daily was tested), using a non-US station (Neustrelitz, Germany) with immutable historical data to also guard global coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 23, 2026, 8:49pm

git · July 23, 2026, 9:10pm

git · July 23, 2026, 9:37pm

github.com/earthobservations/wetterdienst

feat(chmi): add CHMI (Czechia) observation provider

committed 09:37PM - 23 Jul 26 UTC

gutzbenj

+944 -0

Add a key-less CHMI (Czech Hydrometeorological Institute) observation provider c…overing 10-minute, hourly, daily, monthly and annual resolutions via CHMI's open-data CSV portal. - station metadata from metadata/meta1.csv, collapsing per-station operational periods to one row (latest position, full date extent; active stations have a null end_date) - three file layouts handled: daily/monthly/annual as one flat CSV per (station, element); 10-minute and hourly as one CSV per (station, element, month) under a year sub-directory (so they require a date range) - three value formats: daily selects by TIMEFUNC (day-truncated), sub-daily keeps the native timestamp, monthly/annual select by TIMEFUNCTION+MDFUNCTION (mean of daily means/maxima/minima, total for precipitation) dated at the period start - parameters: 10-minute/daily air temperature, humidity, pressure, wind (+ daily extremes, precipitation, snow depth, sunshine); hourly dew point, precipitation and pressure; monthly/annual mean/max/min temperature and precipitation total Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

git · July 23, 2026, 9:51pm

git · July 24, 2026, 6:29am

github.com/earthobservations/wetterdienst

feat(rmi): add RMI (Belgium) observation provider (#1752)

committed 06:29AM - 24 Jul 26 UTC

gutzbenj

+1027 -0

Add the Royal Meteorological Institute of Belgium (RMI) automatic weather statio…n (AWS) network as a new observation provider, served through their key-less GeoServer WFS (GeoJSON) at 10-minute, hourly and daily resolution. - Query per station/date-range with a CQL filter and reshape the wide feature layout (one column per parameter) into the long value frame; sortBy=timestamp is required for startIndex paging on these key-less feature types. - Paging is driven by the response's numberMatched total (advancing by rows actually returned) and bounded by a runaway guard, so it can neither silently truncate on a server-side page cap nor loop forever against a misbehaving server. - Timestamps are true UTC instants; a shared helper parses them defensively (%.f tolerates fractional seconds). - The per-parameter validated flag in qc_flags is exposed as the quality code (1.0 validated, 0.0 not); stations output order is made deterministic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>