Panodata Community

Missing data of my luftdaten.info sensor with LDI station id 19575 from Veglie, Italy on the Grafana map

Hi @Andreas.Motl,
yesterday I inserted a new control unit in Veglie but i can’t see it. Did I do something wrong?
My control unit can sometimes be seen on the map and others not.
But is the measurement interval important?
How many seconds do you recommend me to insert?
Thank you
a.

Dear Andrea,

after checking our infrastructure and running the production of the station list manually like

luftdatenpumpe stations --network=ldi --reverse-geocode --target=${rdbms_uri} --country=IT --progress

we receive errors like

2019-10-29 13:47:31,444 [luftdatenpumpe.geo                  ] ERROR  : Reverse geocoding II failed: GeocoderUnavailable: Service not available. lat=44.68971596721, lon=10.63927710056
2019-10-29 13:47:32,444 [luftdatenpumpe.source.common        ] ERROR  : Failed computing humanized name for station Munch({'station_id': 16653, 'position': Munch({'latitude': 44.68971596721, 'longitude': 10.63927710056, 'altitude': 59.0, 'country': 'IT', 'exact_location': 1, 'indoor': 0, 'geohash': 'spzw4pe23ngg'})})
Traceback (most recent call last):
  File "/opt/luftdatenpumpe/luftdatenpumpe/source/common.py", line 107, in enrich_station
    country_code=station.position.get('country'),
  File "/opt/luftdatenpumpe/luftdatenpumpe/geo.py", line 55, in resolve_location
    location = reverse_geocode(latitude, longitude, country_code)
  File "</opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/decorator-4.4.0-py3.7.egg/decorator.py:decorator-gen-1>", line 2, in reverse_geocode
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/cache/region.py", line 1272, in get_or_create_for_user_func
    should_cache_fn, (arg, kw))
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/cache/region.py", line 879, in get_or_create
    async_creator) as value:
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/lock.py", line 186, in __enter__
    return self._enter()
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/lock.py", line 93, in _enter
    generated = self._enter_create(value, createdtime)
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/lock.py", line 179, in _enter_create
    return self.creator()
  File "/opt/luftdatenpumpe/.venv3/lib/python3.7/site-packages/dogpile.cache-0.7.1-py3.7.egg/dogpile/cache/region.py", line 839, in gen_value
    created_value = creator(*creator_args[0], **creator_args[1])
  File "/opt/luftdatenpumpe/luftdatenpumpe/geo.py", line 119, in reverse_geocode
    raise ValueError('Reverse geocoding failed for lat={}, lon={}'.format(latitude, longitude))
ValueError: Reverse geocoding failed for lat=44.68971596721, lon=10.63927710056

So, we conclude something is fishy there. We will have to look into that problem.

With kind regards,
Andreas.

P.S.: For reverse geocoding the list of stations, we are using https://nominatim.openstreetmap.org/. It might either happen that some API semantics changed or that the program started violating the Nominatim Usage Policy in any way.

1 Like

Running luftdatenpumpe locally on my workstation successfully yields two stations in Veglie, Italy:

LDI station 19604
  {
    "station": {
      "station_id": 19604,
      "position": {
        "latitude": 40.34,
        "longitude": 17.964,
        "altitude": 43.1,
        "country": "IT",
        "exact_location": 0,
        "indoor": 0,
        "geohash": "srhv2z8vx913"
      },
      "location": {
        "place_id": 136685134,
        "licence": "Data \u00a9 OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
        "osm_type": "way",
        "osm_id": 258406334,
        "lat": "40.3398711314178",
        "lon": "17.9640732926525",
        "display_name": "Veglie, Lecce, Puglia, 73015, Italia",
        "boundingbox": [
          "40.3397004",
          "40.3411381",
          "17.9637731",
          "17.9695617"
        ],
        "address": {
          "country_code": "IT",
          "country": "Italia",
          "state": "Puglia",
          "county": "Lecce",
          "postcode": "73015",
          "town": "Veglie",
          "city": "Veglie",
          "road": "unknown"
        }
      },
      "name": "Veglie, Lecce, Puglia, IT"
    }
LDI station 19575
  {
    "station": {
      "station_id": 19575,
      "position": {
        "latitude": 40.31988460214,
        "longitude": 17.96343773603,
        "altitude": 36.3,
        "country": "IT",
        "exact_location": 1,
        "indoor": 0,
        "geohash": "srhv2u0867mb"
      },
      "location": {
        "place_id": 142613583,
        "licence": "Data \u00a9 OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
        "osm_type": "way",
        "osm_id": 288569743,
        "lat": "40.3198817298648",
        "lon": "17.9634360846187",
        "display_name": "Veglie, Lecce, Puglia, 73045, Italia",
        "boundingbox": [
          "40.3196403",
          "40.3220242",
          "17.9597578",
          "17.963856"
        ],
        "address": {
          "country_code": "IT",
          "country": "Italia",
          "state": "Puglia",
          "county": "Lecce",
          "postcode": "73045",
          "town": "Veglie",
          "city": "Veglie",
          "road": "unknown"
        }
      },
      "name": "Veglie, Lecce, Puglia, IT"
    }

Update: Running luftdatenpumpe on our servers works again without further ado so we have been able to investigate further.

Most probably, we’ve identified the problem. The records about station_id=19575 in the PostGIS database are missing an appropriate sensor_type identifier.

weatherbase=> select station_id, name, sensor_type from ldi_network where station_id in (19604, 19575);
 station_id |           name            | sensor_type
------------+---------------------------+-------------
      19575 | Veglie, Lecce, Puglia, IT |
      19575 | Veglie, Lecce, Puglia, IT |
      19604 | Veglie, LE, PUG, IT       | BME280
      19604 | Veglie, LE, PUG, IT       | SDS011
(4 rows)

Now, we will have to investigate why this might have happened. Do you have any idea about this?

Running luftdatenpumpe stations --network=ldi --station=19575 --reverse-geocode does yield appropriate sensor type identifiers, see BME280 and SDS011.

[
  {
    "station_id": 19575,
    "name": "Veglie, Lecce, Puglia, IT",
    "position": {
      "latitude": 40.31988460214,
      "longitude": 17.96343773603,
      "altitude": 36.3,
      "country": "IT",
      "exact_location": 1,
      "indoor": 0,
      "geohash": "srhv2u0867mb"
    },
    "location": {
      "place_id": 142613583,
      "licence": "Data \u00a9 OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright",
      "osm_type": "way",
      "osm_id": 288569743,
      "lat": "40.3198817298648",
      "lon": "17.9634360846187",
      "display_name": "Veglie, Lecce, Puglia, 73045, Italia",
      "boundingbox": [
        "40.3196403",
        "40.3220242",
        "17.9597578",
        "17.963856"
      ],
      "address": {
        "country_code": "IT",
        "country": "Italia",
        "state": "Puglia",
        "county": "Lecce",
        "postcode": "73045",
        "town": "Veglie",
        "city": "Veglie",
        "road": "unknown"
      }
    },
    "sensors": [
      {
        "sensor_id": 33217,
        "sensor_type_name": "BME280",
        "sensor_type_id": 17
      },
      {
        "sensor_id": 33216,
        "sensor_type_name": "SDS011",
        "sensor_type_id": 14
      }
    ]
  }
]

Dear Andrea,

After upgrading luftdatenpumpe on our production server to its most recent version the other day, we missed to recreate the central database view ldi_network through luftdatenpumpe/rdbms.py at 0.18.2 · panodata/luftdatenpumpe · GitHub. In turn, the database field name sensor_type has been deprecated and replaced by sensor_type_id resp. sensor_type_name [1].

In order to account for this change, we will have to update our Grafana dashboards. On [2], you will be able to see both of your stations already. Enjoy!

With kind regards,
Andreas.

[1] luftdatenpumpe/CHANGES.rst at 0.18.2 · panodata/luftdatenpumpe · GitHub

[2] Luft: LDI / luftdaten.info Verlauf

1 Like

After adjusting things accordingly, station 19575 is now also on the map for Italy again.

image

https://weather.hiveeyes.org/grafana/d/ZKf6O2TWk/luftdaten-info-map-for-italy

1 Like