Logbook of code changes

git · July 16, 2026, 1:20am

git · July 16, 2026, 8:24am

feat(filesystem): add XML and YAML read support

committed 08:24AM - 16 Jul 26 UTC

Add XML and YAML as read formats for the filesystem source, on the same long-tai…l harness that carries MessagePack and CBOR. Both decode with their own library directly rather than through iterabledata, for format-specific safety and error-surfacing reasons. - XML: parse with a hardened lxml config (no entity resolution, no DTD load, no network, capped tree) that neutralizes XXE and entity-expansion; require a #tagname=<row-tag> reader hint naming the row element (the first consumer of the reader-hint channel); map attributes to @name, repeated children to lists, nested elements to objects. - YAML: load with yaml.safe_load_all so a !!python/object tag is rejected and a malformed file raises instead of loading zero rows; a list document expands to one row per element; normalize !!binary/!!set to portable types; accept both .yaml and .yml. - Harness: the whole-file eager-decoder seam now takes an options dict, and per-URI reader hints thread from the source URI to the decoder. CBOR and MessagePack are unaffected. - Both install with the existing optional 'iterable' extra (now also lxml and pyyaml); a base install fails a .xml/.yaml load with a precise pip-install hint and only advertises a format when its decoder imports. Refs #45.

git · July 16, 2026, 8:24am

git · July 16, 2026, 8:31am

git · July 16, 2026, 8:31am

git · July 16, 2026, 12:10pm

git · July 16, 2026, 12:10pm

git · July 16, 2026, 12:10pm

git · July 16, 2026, 12:10pm

git · July 16, 2026, 12:10pm

https://github.com/panodata/omniload/commit/d42bd7c011e6f5a49e11d5bfe6d246516754020e

git · July 16, 2026, 12:15pm

git · July 16, 2026, 3:47pm

git · July 16, 2026, 6:08pm

git · July 16, 2026, 9:08pm

github.com/earthobservations/wetterdienst

feat(aemet): add AEMET observation provider (#1736)

committed 09:08PM - 16 Jul 26 UTC

gutzbenj

+1465 -0

* feat(aemet): add AEMET observation provider Adds a new provider for Spain's A…EMET OpenData API, covering daily, monthly, and annual climatological values plus real-time (convencional) hourly observations, with API-key auth and rate-limit-aware retries via stamina. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): redact api_key from log messages, fix restapi coverage test CodeQL flagged clear-text logging of the AEMET api_key (embedded in request URLs and logged verbatim on failures/rate-limit warnings). Also updates test_restapi.py::test_coverage, which hardcoded the provider list and didn't yet include aemet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): send api_key as a request header instead of a URL query param The previous fix redacted api_key from log messages, but download_file() itself still logged the raw key-bearing URL internally (log.info), so the key could still leak into logs outside this module's own warnings. Confirmed AEMET accepts api_key via an "api_key" header (a wrong key sent this way gets the same JWT-validation error as the query-param form), matching how metno_frost already sends its credentials via headers rather than embedding them in URLs. With that, the key never touches a URL that gets constructed or logged, so the redact helper is no longer needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): broaden retry to cover transient network failures, not just 429 CI showed AEMET intermittently dropping connections outright (TLS handshake failures, timeouts), which download_file()'s own short built-in retry didn't clear and which previously fell straight through as a lost chunk. The rate-limit retry loop now also retries those (status in {408, 500, 503}), using an exponential backoff that starts short and grows toward the ~60s a 429 needs, instead of a fixed 60s wait for 429 only. Non-retryable failures (e.g. 404) are still returned immediately. Also documents the resulting behavior (calls can take noticeably longer under rate limiting or connectivity issues) in the provider docs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): retry on any failure except definitively bad requests The latest CI run showed a chunk failing without the retry loop even firing -- its status wasn't in the {429, 408, 500, 503} allow-list. Live probing confirmed AEMET's failure modes are too varied to enumerate (timeouts, hangs, and failures on the second-stage "datos" URL all surfaced with different statuses across a handful of runs). Flipped to a deny-list: retry everything except statuses that mean the request itself is invalid and retrying can't help (400/401/403/404). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): address Copilot review findings - Annual resolution no longer declares "humidity" as a valid parameter -- AEMET's annual aggregate never includes that field at all (unlike monthly), so it was advertising a parameter that could never return data. Split into a separate _ANNUAL_PARAMETERS list. - Move the AEMET-credentials skip from a module-level pytestmark to a per-test decorator on just the @pytest.mark.remote tests, so the local unit tests (chunking, retry behavior) still run without WD_AUTH__AEMET set (e.g. in forks without access to the CI secret). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): reduce retry budget, xfail live tests, add changelog entry Live AEMET outages have repeatedly outlasted even a generous retry budget across several CI runs, so a long retry (4 attempts, up to 60s backoff) was mostly wasted time rather than a real reliability win. Reduced to 2 short-backoff attempts (up to 15s), enough to smooth over brief blips without making a single failing call hang for minutes. The 5 live/remote AEMET tests are now xfail(strict=False), matching the existing ECCC precedent in this test suite for flaky third-party APIs -- a failure no longer blocks CI/merges on AEMET's own availability, while a pass is still recorded when AEMET is up. Verified live: 9 passed, 1 xfailed, 4 xpassed, exit 0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aemet): normalize non-UTC start/end date before formatting requests TimeseriesRequest.convert_timestamps() only tags naive datetimes as UTC -- it never converts an already-tz-aware non-UTC datetime. Since _collect_daily/_collect_monthly_or_annual formatted chunk_start/ chunk_end with a hardcoded "UTC" suffix, a caller passing e.g. a Europe/Madrid-tz datetime would silently have AEMET query the wrong time window (off by the local UTC offset). Both now explicitly convert to UTC first via ZoneInfo("UTC"), matching the convention already used elsewhere in this codebase (request.py, test file). Adds a regression test using a mocked _fetch_datos to assert the actual request URL reflects the correctly-converted UTC timestamps for a Madrid-tz input, without needing live credentials or network access. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

git · July 17, 2026, 10:13am

github.com/panodata/omniload

fix(incremental): perform a real SCD2 load for the scd2 strategy

committed 10:13AM - 17 Jul 26 UTC

hampsterx

+496 -3

- `--incremental-strategy scd2` now performs an SCD2 load: the validity columns … (`_dlt_valid_from` / `_dlt_valid_to`) are materialized, and a row that changes in the source is retired and succeeded rather than overwritten. Previously the option performed an SCD2 load on no path at all. dlt rejected it during extract for ordinary sources (`write_disposition=scd2` is not one of `skip`/`append`/`replace`/`merge`), and sources that own their write disposition ignored it silently. - Mechanism: `IncrementalStrategy` flattens dlt's two axes (write disposition x merge strategy) onto one, and `scd2` lives on the second. It is now mapped to dlt's structured `{"disposition": "merge", "strategy": "scd2"}`, next to the existing `delete+insert` unflattening, instead of being passed through as a write disposition dlt has no such value for. - SCD2 has dlt compare the whole source table against the records it holds open and retire whatever is missing from it. The combinations that misrepresent that table are now refused rather than left to rewrite history silently: - `--incremental-key`, `--sql-limit`, and `--yield-limit` each read back only part of the table, retiring every record they leave out while the source still holds it unchanged. - A `--mask` algorithm that draws a fresh value per run (`date_shift`, `noise`, `random`, `uuid`) rewrites the rows being compared, so every record reads as changed and is retired and re-inserted on every load. `masking.NON_DETERMINISTIC_ALGORITHMS` names them. - The Arrow-yielding SQL backends (`pyarrow`, `connectorx`) and the `mmap://` source carry no row hash, which dlt computes only for rows it receives as dicts, leaving `_dlt_id` unpopulated and the load dead in `normalize`. Populating it via `normalize.parquet_normalizer.add_dlt_id` is not an alternative, as those ids are random per run rather than a hash of the row, so every record would be retired and re-inserted on every load. SQL sources are read with `sqlalchemy` for SCD2 instead. A `query:` table is exempt from the backend rule, being read with sqlalchemy whichever backend is named, and the guide states that such an expression is itself the tracked relation. - `--primary-key` is not required for SCD2, as dlt hashes the whole row to spot a change. It stays optional and marks the natural key on the destination. - Unchanged: sources that manage their own incrementality still ignore the run-level strategy, and the filesystem family still rejects the key-based strategies. - The flaw predates the fork (`3174960`, 2024-07-12) and survived because the only `scd2` coverage was the filesystem reject path. The new tests exercise it on the sources where it applies, including that a re-run over unchanged data is a no-op rather than a retire-and-reinsert of every row.

git · July 18, 2026, 10:31am

git · July 18, 2026, 3:51pm

github.com/panodata/macropipe

Chore(deps-dev): Bump ty from 0.0.59 to 0.0.60

committed 03:51PM - 18 Jul 26 UTC

dependabot[bot]

+1 -1

Bumps [ty](https://github.com/astral-sh/ty) from 0.0.59 to 0.0.60. - [Release no…tes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ty/compare/0.0.59...0.0.60) --- updated-dependencies: - dependency-name: ty dependency-version: 0.0.60 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

git · July 19, 2026, 11:17am

git · July 19, 2026, 6:55pm

git · July 19, 2026, 7:34pm