committed 06:10PM - 04 May 26 UTC
WholeFileCacheFileSystem is not thread-safe: when download_files() uses
ThreadPo…olExecutor to download files in parallel, all threads share the
same filesystem instance (via NetworkFilesystemManager.filesystems class
dict). This causes a race condition in the in-memory cached_files metadata,
where _check_file() returns False after _get_cached_file_before_open()
completes, resulting in 'TypeError: cannot unpack non-iterable bool object'
at cached.py:716 (detail, fn = self._check_file(path)).
Fix: replace the class-level filesystems dict with threading.local() so
each thread gets its own filesystem instance with its own in-memory metadata.
The on-disk cache storage is still shared, providing cache reuse across runs.
Fixes test_dwd_observation_datasets_high_resolution failures in CI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>