Luftdatenpumpe: Problem exporting stations to file

Air quality data
,
1

Hi,

I need some advice on what else I might be doing wrong here.

luftdatenpumpe stations --source=postgresql://luftdatenpumpe@localhost/weatherbase --target=json.flex+stream://sys.stdout --target-fieldmap='key=station_id\|str,name=road_and_name_and_id' > $stationsfile
2022-12-12 21:14:54,690 [luftdatenpumpe.source.common        ] INFO   : Downloading without caching
2022-12-12 21:14:54,691 [luftdatenpumpe.commands             ] INFO   : Acquiring list of stations from network "ldi" with source "postgresql://luftdatenpumpe@localhost/weatherbase"
2022-12-12 21:14:54,716 [luftdatenpumpe.target.rdbms         ] INFO   : PostGIS version: POSTGIS="3.1.1 aaf4c79" [EXTENSION] PGSQL="130" GEOS="3.9.0-CAPI-1.16.2" PROJ="7.2.1" LIBXML="2.9.10" LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)"
2022-12-12 21:14:55,679 [luftdatenpumpe.commands             ] INFO   : Acquired #9499 stations
2022-12-12 21:14:55,679 [luftdatenpumpe.commands             ] INFO   : Will publish data to ['json.flex+stream://sys.stdout']
2022-12-12 21:14:55,679 [luftdatenpumpe.engine               ] INFO   : Configuring data sink "json.flex+stream://sys.stdout" with domain "stations"
2022-12-12 21:14:55,679 [luftdatenpumpe.engine               ] INFO   : Emitting to target data sinks, this might take some time
Traceback (most recent call last):
  File "/home/ldp/venv/bin/luftdatenpumpe", line 8, in <module>
    sys.exit(run())
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/commands.py", line 186, in run
    run_engine(options)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/commands.py", line 327, in run_engine
    engine.process(data)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/engine.py", line 86, in process
    target.flush(final=True)
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/target/stream.py", line 21, in flush
    self.handle.write(self.formatter(self.buffer))
  File "/home/ldp/venv/lib/python3.9/site-packages/luftdatenpumpe/target/json.py", line 43, in formatter
    value = station[str(key_right)]
KeyError: 'station_id\\'
2

It looks like the --target-fieldmap command-line option is interpreted incorrectly on the backslash character.

It is probably not your fault. Either Luftdatenpumpe is misbehaving, or maybe the Unix shell you are using needs a different input.

  • May I ask you which type of $SHELL you are using?
  • What happens if you omit the backslash character?
3

Report

It looks like the backslash character causes this problem, it works better when omitting it. I know I wrote that intentionally, because it was needed back then, but apparently not today. C’est la vie. The documentation has been fixed.

Kudos

Thank you very much for bringing this detail to our attention.

References

1 Like
4

Hi Andreas,

I use the normal Debian bash shell.
Without the backslash it works.

New Question:
In the postgresql database there are currently all German stations, because I have queried County=DE.
To fill the database only with the desired Frankfurt stations, I would have to empty the database again.
Should I simply delete the database and create a new one or is there a cleaner command?

(I am currently trying to get the Grafana plugin panodata-map-panel to work.
Grafana v8.5.15 claims that it cannot find the plugin. But it is in the PlugIn directory.) = allow_loading_unsigned_plugins in the grafana.ini was the solution

Thanks,
Oliver

5

Excellent. Thanks to your report, the corresponding fix for the documentation was Fix "Luftdaten-Viewer Grafana" documentation by amotl · Pull Request #49 · earthobservations/luftdatenpumpe · GitHub.

Wiping everything will be a definitive solution ;]. However, you may also be able to use the TRUNCATE SQL command, like

TRUNCATE TABLE ldi_stations, ldi_sensors;

Thank you very much. Documentation: Improve section about Grafana plugins · earthobservations/luftdatenpumpe@8ae53cf · GitHub will improve the documentation on this matter.