Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Read_SDMX method #159

Closed
wants to merge 28 commits into from
Closed

Add Read_SDMX method #159

wants to merge 28 commits into from

Conversation

javihern98
Copy link
Contributor

@javihern98 javihern98 commented Dec 18, 2024 •
edited
Loading

Summary of changes:

  • Added convenience method read_sdmx to use same method for all parsers.
  • Added SDMXFormat enumeration with the supported formats
  • Refactored message.py to avoid circular imports and place there only the Message and Header classes
  • Enhanced input_processor to check for xml, csv and json files
  • Fixed readers to use always Short_URN as keys of the result, to be placed on Message structure or data fileds
  • Changed data type on Component.concept to Union[Concept, ItemReference] to allow the reading of DataStructureDefinition files without references.
  • Added support for URL retrieving
  • Made httpx dependency as mandatory

@javihern98
Draft code for read_sdmx.
3664415 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98 javihern98 linked an issue Dec 18, 2024 that may be closed by this pull request
Add read_sdmx convenience method #156
Open
@javihern98
Merge branch 'develop' into 156-add-read_sdmx-convenience-method
bdf8145
@javihern98
Refactored code on Message class to move submission and ActionType to…
2d87af7 
… their classes. Added read_sdmx method to reader

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Linting and mypy changes
617aa4c 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Refactored code to ensure we use the Short URN as keys of the message…
2c2cab3 
… content. Use ItemReference as ConceptIdentity if necessary. Fixed tests accordingly.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added tests for read_sdmx with csv files.
ffb2a8e 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Linting and mypy changes.
01f5489 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Adapted structures tests to reach max code coverage
ab63f01 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Removed unique_id method and adapted code to use Reference.
8b960d2 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added utils methods and classes to all.
41e7a8d 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Adapted tests for input processor to max code coverage.
b140de0 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Adapted tests for read_sdmx to max code coverage.
0c03933 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added tag to concept and itemReference to avoid serialization issues.
f567c50 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Merge remote-tracking branch 'refs/remotes/origin/develop' into 156-a…
85076af 
…dd-read_sdmx-convenience-method

# Conflicts:
#	src/pysdmx/model/message.py
@javihern98
Fixed message imports to prevent circular imports. Ruff automatic fix…
fe1fbb4 
…es. Added get_datasets methods. Maxed code coverage in all code.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98 javihern98 requested review from sosna and stratosn December 19, 2024 16:21
@javihern98
Copy link
Contributor Author

@sosna @stratosn I would like to inclñude this as well on release 1.0.0, if we have still time to review this tomorrow. We will adapt the documentation as well if this is approved on time.

@javihern98 javihern98 marked this pull request as ready for review December 19, 2024 16:22
javihern98
javihern98 commented Dec 19, 2024
View reviewed changes
src/pysdmx/io/reader.py Outdated
"""Enumeration of supported SDMX read formats."""

SDMX_ML_2_1 = "SDMX-ML 2.1"
# SDMX_JSON_2 = "SDMX-JSON 2.0.0"
Copy link
Contributor Author

@javihern98 javihern98 Dec 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sosna This is a placeholder to include as well methods to read the JSON formats, still I have not found the way to implement them. Any ideas?

javihern98
javihern98 commented Dec 19, 2024
View reviewed changes
src/pysdmx/model/dataflow.py
@@ -108,7 +109,7 @@ class Component(Struct, frozen=True, omit_defaults=True):
id: str
required: bool
role: Role
concept: Concept
concept: Union[Concept, ItemReference]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sosna In order to add the compatibility to the "Concept Identity Reference", I added the tag attribute to true both here and in ItemReference. It passes all the tests, but pelase check if it may raise any future problems (As far as I could check, it should not)

sosna
sosna reviewed Dec 20, 2024
View reviewed changes
Copy link
Collaborator

@sosna sosna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @javihern98. There is a lot of work we need to do next year about formats serialization and deserialization. Is there any important reason to "rush" this now, a bit as a last minute addition, or is this something we can discuss when there is more capacity early next year?

@javihern98
Copy link
Contributor Author

Hi @javihern98. There is a lot of work we need to do next year about formats serialization and deserialization. Is there any important reason to "rush" this now, a bit as a last minute addition, or is this something we can discuss when there is more capacity early next year?

We do not need to rush it and discuss it further after Christmas😁

@javihern98
Adapted ReadSDMX to infer the format automatically.
6b6886f 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Linting changes.
b6e6200 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Renamed ReadFormat to SDMXFormat. Added Submission and Error formats …
03dd8ad 
…detection on SDMX-ML. Fixed tests.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Ignored mypy error.
bd27711 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added to_schema method to DataStructureDefinition. Draft code for get…
5818c40 
…_datasets method.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Replaced whole URN to Short URN in Dataset. Added tests to match cove…
6a4bac7 
…rage.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added get_datasets to io
d38ed7f 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added URL parsing support on read_sdmx. Made httpx a mandatory depend…
e2b2946 
…ency.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Merge branch 'develop' into 156-add-read_sdmx-convenience-method
4eca815
@javihern98
Updated poetry lock.
183fe4d 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Merge remote-tracking branch 'origin/156-add-read_sdmx-convenience-me…
d8fa929 
…thod' into 156-add-read_sdmx-convenience-method
@javihern98
Copy link
Contributor Author

Awaiting to finish #164 before merging this

@javihern98 javihern98 marked this pull request as draft January 10, 2025 12:32
@javihern98
Updated poetry lock and used correct version of httpx. Updated tests …
3b48f9e 
…with mocks.

Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Added validate flag to get_datasets.
ed6303a 
Signed-off-by: javier.hernandez <javier.hernandez@meaningfuldata.eu>
@javihern98
Copy link
Contributor Author

Closing as it is included in #172

@javihern98 javihern98 closed this Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sosna sosna sosna left review comments

@stratosn stratosn Awaiting requested review from stratosn

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add read_sdmx convenience method
2 participants
@javihern98 @sosna