Impossibilité d'exporter quelque soit le format

[marineteneur]

Bonjour,

J'ai depuis peu refait une installation de GeoNature 2.12.3 à neuf et quand je veux faire un export (en n'importe quel format) j'ai une erreur 500 Internal Server Error, alors que de son coté l'api fonctionne.

J'ai fait la montée en version de la dépendance Utils-Flask-SQLAlchemy en v0.3.5 mais rien ne change.

Dans le log geonature j'ai cette erreur qui s'affiche :

`[2023-09-07 14:20:18 +0200] [2419573] [ERROR] Exception on /exports/2/json [POST]
Traceback (most recent call last):
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 611, in connect
sock = self.retry.call_with_retry(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 612, in
lambda: self._connect(), lambda error: self.disconnect(error)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 677, in _connect
raise err
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 665, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 119, in reconnect_on_error
yield
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 169, in _consume_from
self._pubsub.subscribe(key)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1592, in subscribe
ret_val = self.execute_command("SUBSCRIBE", *new_channels.keys())
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1433, in execute_command
self.connection = self.connection_pool.get_connection(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 1387, in get_connection
connection.connect()
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 617, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 611, in connect
sock = self.retry.call_with_retry(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 612, in
lambda: self._connect(), lambda error: self.disconnect(error)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 677, in _connect
raise err
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 665, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
yield
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/base.py", line 787, in send_task
self.backend.on_task_call(P, task_id)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 365, in on_task_call
self.result_consumer.consume_from(task_id)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 161, in consume_from
return self.start(task_id)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 139, in start
self._consume_from(initial_task_id)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 169, in _consume_from
self._pubsub.subscribe(key)
File "/usr/lib/python3.9/contextlib.py", line 135, in exit
self.gen.throw(type, value, traceback)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 122, in reconnect_on_error
self._ensure(self._reconnect_pubsub, ())
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 376, in ensure
return retry_over_time(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 106, in _reconnect_pubsub
metas = self.backend.client.mget(self.subscribed_to)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/commands/core.py", line 1893, in mget
return self.execute_command("MGET", *args, **options)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1235, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 1387, in get_connection
connection.connect()
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 617, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
resp = make_response(f(*args, **kwargs))
File "/home/geonatureadmin/geonature/backend/geonature/core/gn_permissions/decorators.py", line 56, in decorated_view
return view_func(*args, **kwargs)
File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/blueprint.py", line 173, in getOneExportThread
generate_export.delay(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay
return self.apply_async(args, kwargs)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/task.py", line 575, in apply_async
return app.send_task(
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/base.py", line 788, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/lib/python3.9/contextlib.py", line 135, in exit
self.gen.throw(type, value, traceback)
File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: Error 111 connecting to localhost:6379. Connection refused.
`

Et dans le log geonature-worker :

[2023-09-07 13:39:29,460: INFO/Beat] beat: Starting... [2023-09-07 13:39:30,358: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 111 connecting to localhost:6379. Connection refused.. Trying again in 2.00 seconds... (1/100)

[camillemonchicourt]

Quelle version du module Exports ?

[camillemonchicourt]

As-tu bien redémarré les services après l'installation du module ?

sudo systemctl restart geonature
sudo systemctl restart geonature-worker

[marineteneur]

J'ai la version 1.5.2.
Oui j'ai tout relancé avec la commande suivante :
sudo systemctl restart taxhub usershub postgresql geonature geonature-worker apache2

[TheoLechemia]

Celery n'a pas l'air de tourner ? Ou alors peut être pas sur le bon port ?
Que renvoie la commande : sudo systemctl status redis

[marineteneur]

Merci beaucoup ! en effet redis était inactif, après un sudo systemctl start redis le problème a été réglé :)

[maximetoma]

J'ai de nouveau eu un problème d'export des CSV dernièrement, j'ai vérifié le status de redis qui était actif
J'ai quand même effectué un restart sur le service de redis, et les exports CSV ont de nouveau fonctionnés

Je ne sais pas ce qui peut provoquer ça.... :/

Avant le restart

redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; preset: enabled)
Active: active (running) since Sat 2023-05-06 07:13:34 CEST; 5 months 19 days ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Main PID: 1434416 (redis-server)
Status: "Ready to accept connections"
Tasks: 5 (limit: 38353)
Memory: 4.0M
CPU: 9h 7min 8.751s
CGroup: /system.slice/redis-server.service
└─1434416 "/usr/bin/redis-server 127.0.0.1:6379"

Après le restart

redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; preset: enabled)
Active: active (running) since Tue 2023-10-24 15:26:45 CEST; 14s ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Main PID: 1901904 (redis-server)
Status: "Ready to accept connections"
Tasks: 5 (limit: 38353)
Memory: 8.0M
CPU: 57ms
CGroup: /system.slice/redis-server.service
└─1901904 "/usr/bin/redis-server 127.0.0.1:6379"

[maximetoma]

Après observation, c'est le worker qui pose problème
J'ai encore redémarrer aujourd'hui pour que ça marche

Avant le restart du worker

[2023-10-25 10:38:23,672: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 628, in start
    c.loop(*c.loop_args())
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 1326, in on_readable
    self.cycle.on_readable(fileno)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 562, in on_readable
    chan.handlers[type]()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 906, in _receive
    ret.append(self._receive_one(c))
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 916, in _receive_one
    response = c.parse_response()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 796, in parse_response
    response = self._execute(conn, try_read)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 772, in _execute
    return conn.retry.call_with_retry(
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 49, in call_with_retry
    fail(error)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 774, in <lambda>
    lambda error: self._disconnect_raise_connect(conn, error),
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 761, in _disconnect_raise_connect
    raise error
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 773, in <lambda>
    lambda: command(*args, **kwargs),
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 794, in try_read
    return conn.read_response(disconnect_on_error=False, push_request=True)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 493, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/resp2.py", line 15, in read_response
    result = self._read_response(disable_decoding=disable_decoding)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/resp2.py", line 25, in _read_response
    raw = self._buffer.readline()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/socket.py", line 115, in readline
    self._read_from_socket()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/socket.py", line 68, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2023-10-25 10:38:23,679: WARNING/MainProcess] /home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2023-10-25 10:38:23,683: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 111 connecting to localhost:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)

[2023-10-25 10:38:25,692: INFO/MainProcess] Connected to redis://localhost:6379/0
[2023-10-25 10:38:25,697: INFO/MainProcess] mingle: searching for neighbors
[2023-10-25 10:38:26,706: INFO/MainProcess] mingle: all alone
[2023-10-25 10:38:33,900: INFO/MainProcess] Task gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba] received
[2023-10-25 10:38:33,900: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba]: Generate export 189...
[2023-10-25 10:38:37,007: INFO/ForkPoolWorker-7] Send email notification to maxime.toma (maxime.toma@lpo.fr)
[2023-10-25 10:38:37,008: INFO/MainProcess] Task geonature.core.notifications.tasks.send_notification_mail[0a6fcd54-6361-494c-8d5a-07804cbfbb59] received
[2023-10-25 10:38:37,009: INFO/ForkPoolWorker-8] Launch mail.
[2023-10-25 10:38:37,010: INFO/ForkPoolWorker-7] Send database notification to maxime.toma
[2023-10-25 10:38:37,012: ERROR/ForkPoolWorker-7] Task gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba] raised unexpected: ValueError("Invalid fields for <Schema(many=False)>: {'geom_2154'}.")
Traceback (most recent call last):
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/geonature/celery_app.py", line 12, in __call__
    return self.run(*args, **kwargs)
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/tasks.py", line 62, in generate_export
    export_data_file(export_id, file_name, export_url, format, id_role, filters)
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils_export.py", line 153, in export_data_file
    raise exp
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils_export.py", line 140, in export_data_file
    export_as_file(
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils/export.py", line 29, in export_as_file
    _export_as_file(
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils/export.py", line 75, in _export_as_file
    func_dict[file_format](
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/export.py", line 43, in export_csv
    schema = schema_class(only=only or None)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla/schema.py", line 51, in __init__
    super().__init__(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/schema.py", line 175, in __init__
    super().__init__(*args, only=only, exclude=exclude, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow_sqlalchemy/load_instance_mixin.py", line 45, in __init__
    super().__init__(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 404, in __init__
    self._init_fields()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 979, in _init_fields
    raise ValueError(message)
ValueError: Invalid fields for <Schema(many=False)>: {'geom_2154'}.
[2023-10-25 10:38:37,768: INFO/ForkPoolWorker-8] Task geonature.core.notifications.tasks.send_notification_mail[0a6fcd54-6361-494c-8d5a-07804cbfbb59] succeeded in 0.7593830302357674s: None
[2023-10-25 10:39:25,456: INFO/MainProcess] beat: Shutting down...

Après le restart du worker

[2023-10-25 10:39:29,871: INFO/Beat] beat: Starting...
[2023-10-25 10:39:29,871: INFO/MainProcess] Connected to redis://localhost:6379/0
[2023-10-25 10:39:29,872: INFO/MainProcess] mingle: searching for neighbors
[2023-10-25 10:39:30,878: INFO/MainProcess] mingle: all alone
[2023-10-25 10:39:30,892: INFO/MainProcess] celery@ad337eea-02e7-4865-bd16-9a14049e635c.alx.host ready.
[2023-10-25 10:39:38,133: INFO/MainProcess] Task gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b] received
[2023-10-25 10:39:38,134: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b]: Generate export 189...
[2023-10-25 10:39:50,656: INFO/ForkPoolWorker-7] Send email notification to maxime.toma (maxime.toma@lpo.fr)
[2023-10-25 10:39:50,659: INFO/MainProcess] Task geonature.core.notifications.tasks.send_notification_mail[46d4d612-c651-4664-907c-103daabf7efd] received
[2023-10-25 10:39:50,660: INFO/ForkPoolWorker-8] Launch mail.
[2023-10-25 10:39:50,661: INFO/ForkPoolWorker-7] Send database notification to maxime.toma
[2023-10-25 10:39:50,664: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b]: Export 189 generated.
[2023-10-25 10:39:50,664: INFO/ForkPoolWorker-7] Task gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b] succeeded in 12.530796222388744s: None
[2023-10-25 10:39:51,246: INFO/ForkPoolWorker-8] Task geonature.core.notifications.tasks.send_notification_mail[46d4d612-c651-4664-907c-103daabf7efd] succeeded in 0.5856118053197861s: None