forked from ckan/ckanext-xloader
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathconfig_declaration.yaml
180 lines (177 loc) · 7.87 KB
/
config_declaration.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
version: 1
groups:
- annotation: ckanext-xloader settings
options:
- key: ckanext.xloader.site_url
example: http://ckan-dev:5000
description: |
Provide an alternate site URL for the xloader_submit action.
This is useful, for example, when the site is running within a docker network
validators: configured_default("ckan.site_url",None)
required: false
- key: ckanext.xloader.jobs_db.uri
default: sqlite:////tmp/xloader_jobs.db
description: |
The connection string for the jobs database used by XLoader. The
default of an sqlite file is fine for development. For production use a
Postgresql database.
validators: not_missing
required: true
- key: ckanext.xloader.api_token
example: eyJ0eXAiOiJKV1QiLCJh.eyJqdGkiOiJ0M2VNUFlQWFg0VU.8QgV8em4RA
description: |
Uses a specific API token for the xloader_submit action instead of the
apikey of the site_user. Will be mandatory when dropping support for
CKAN 2.9.
required: false
- key: ckanext.xloader.formats
example: csv application/csv xls application/vnd.ms-excel
description: |
The formats that are accepted. If the value of the resource.format is
anything else then it won't be 'xloadered' to DataStore (and will therefore
only be available to users in the form of the original download/link).
Case insensitive. Defaults are listed in plugin.py.
required: false
- key: ckanext.xloader.max_content_length
default: 1_000_000_000
example: 100000
description: |
The maximum file size that XLoader will attempt to load.
type: int
required: false
- key: ckanext.xloader.use_type_guessing
default: False
example: False
description: |
By default, xloader will first try to add tabular data to the DataStore
with a direct PostgreSQL COPY. This is relatively fast, but does not
guess column types. If this fails, xloader falls back to a method more
like DataPusher's behaviour. This has the advantage that the column types
are guessed. However it is more error prone and far slower.
To always skip the direct PostgreSQL COPY and use type guessing, set
this option to True.
type: bool
required: false
legacy_key: ckanext.xloader.just_load_with_messytables
- key: ckanext.xloader.strict_type_guessing
default: True
example: False
description: |
Use with ckanext.xloader.use_type_guessing to set strict true or false
for type guessing. If set to False, the types will always fallback to string type.
Strict means that a type will not be guessed if parsing fails for a single cell in the column.
type: bool
- key: ckanext.xloader.max_type_guessing_length
default: 0
example: 100000
description: |
The maximum file size that will be passed to Tabulator if the
use_type_guessing flag is enabled. Larger files will use COPY even if
the flag is set. Defaults to 1/10 of the maximum content length.
type: int
required: false
- key: ckanext.xloader.parse_dates_dayfirst
default: False
example: False
description: |
Whether ambiguous dates should be parsed day first. Defaults to False.
If set to True, dates like '01.02.2022' will be parsed as day = 01,
month = 02.
NB: isoformat dates like '2022-01-02' will be parsed as YYYY-MM-DD, and
this option will not override that.
See https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.parse
for more details.
type: bool
required: false
- key: ckanext.xloader.parse_dates_yearfirst
default: False
example: False
description: |
Whether ambiguous dates should be parsed year first. Defaults to False.
If set to True, dates like '01.02.03' will be parsed as year = 2001,
month = 02, day = 03. See https://dateutil.readthedocs.io/en/stable/parser.html#dateutil.parser.parse
for more details.
type: bool
required: false
- key: ckanext.xloader.job_timeout
default: 3600
example: 3600
description: |
The maximum time for the loading of a resource before it is aborted.
Give an amount in seconds. Default is 60 minutes
type: int
required: false
- key: ckanext.xloader.ignore_hash
default: False
example: False
description: |
Ignore the file hash when submitting to the DataStore, if set to True
resources are always submitted (if their format matches), if set to
False (default), resources are only submitted if their hash has changed.
type: bool
required: false
- key: ckanext.xloader.max_excerpt_lines
default: 0
example: 100
description: |
When loading a file that is bigger than `max_content_length`, xloader can
still try and load some of the file, which is useful to display a
preview. Set this option to the desired number of lines/rows that it
loads in this case.
If the file-type is supported (CSV, TSV) an excerpt with the number of
`max_excerpt_lines` lines will be submitted while the `max_content_length`
is not exceeded.
If set to 0 (default) files that exceed the `max_content_length` will
not be loaded into the datastore.
type: int
required: false
- key: ckanext.xloader.ssl_verify
default: True
example: True
description: |
Requests verifies SSL certificates for HTTPS requests. Setting verify to
False should only be enabled during local development or testing. Default
to True.
type: bool
required: false
- key: ckanext.xloader.validation.requires_successful_report
default: False
example: True
description: |
Resources are required to pass Validation from the ckanext-validation
plugin to be able to get XLoadered.
type: bool
required: false
- key: ckanext.xloader.validation.enforce_schema
default: True
example: False
description: |
Resources are expected to have a Validation Schema, or use the default ones if not.
If this option is set to `False`, Resources that do not have
a Validation Schema will be treated like they do not require Validation.
See https://github.com/frictionlessdata/ckanext-validation?tab=readme-ov-file#data-schema
for more details.
- key: ckanext.xloader.clean_datastore_tables
default: False
example: True
description: |
Enqueue jobs to remove Datastore tables from Resources that have a format
that is not in ckanext.xloader.formats after a Resource is updated.
type: bool
required: false
- key: ckanext.xloader.show_badges
default: True
example: False
description: |
Controls whether or not the status badges display in the front end.
type: bool
required: false
- key: ckanext.xloader.debug_badges
default: False
example: True
description: |
Controls whether or not the status badges display all of the statuses. By default,
the badges will display "pending", "running", and "error". With debug_badges enabled,
they will also display "complete", "active", "inactive", and "unknown".
type: bool
required: false