Refactor to use polars #244

jcadam14 · 2024-08-08T19:24:02Z

Closes #243

Lots of changes, lots of comments added. Please ask questions.

github-actions · 2024-08-08T19:30:30Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/regtech_data_validator
check_functions.py					74
checks.py
cli.py					112-113
data_formatters.py					189-192, 195, 200-202, 211-214
global_data.py
phase_validations.py
schema_template.py
validation_results.py
validator.py					110, 115, 132, 206, 231-233, 243-246
Project Total

_{This report was generated by python-coverage-comment-action}

lchen-2101

LGTM

jcadam14 added 9 commits June 14, 2024 11:11

polars testing

267153f

More polars updates

a873e2e

More polars updates

e83221e

More polars testing

cfe754a

Rewrite of function for better readability.

ba37110

Updates to data formatters

92c32d5

Switch to polars batching and generators

06bd9a8

Final updates for the polars conversion. Comments, cleaned up code, etc

de78658

Merge branch 'main' into test_polars

eff3884

jcadam14 requested review from hkeeler, guffee23 and nargis-sultani August 8, 2024 19:24

jcadam14 self-assigned this Aug 8, 2024

Merge branch 'main' into test_polars

d4e066f

jcadam14 and others added 15 commits August 9, 2024 13:56

Testing new mac commits

1cdf338

Removed psutil

b76995d

Updated to latest polars

c2001b1

Added polars-lts-cpu for ARM support

a8c4f81

Removed polars in favor of polars-lts-cpu

2a73355

Updated report writer from sink_csv to write_csv

186a7e9

Removed memory resource testing code

7ce9068

Merge branch 'main' into test_polars

c179b60

Changed to pull severity value, not enum

4324951

Trying enum again

f427134

Using string instead of enum in dataframe

5dfe595

Trying out boto3 over fsspec

82a36a2

S3 bucket updated

a85667e

Added truncating errors for json processing

df6e246

Fixing row error

37b03d3

jcadam14 added 6 commits September 26, 2024 09:32

Updated to try and shrink down the memory footprint of the dataframes

452fbd9

Revert boto3 to 1.34

2f0493d

Added phase to dataframe

0089588

Fixed phase bug

1ced8ce

Added scope back to dataframe

d81ff0d

missing scope in final frame

e66a4e1

jcadam14 marked this pull request as draft September 26, 2024 16:23

jcadam14 closed this Sep 26, 2024

jcadam14 added 6 commits September 26, 2024 11:04

Trying to use dicts instead of dataframes

3efa1bf

switched back to returning a dataframe

515bca6

Reworked validator to allow for correct real and max errors

ca9b9c7

Added default max size per json group

1277923

Linting, and fixed pytests

98f1160

Had to add s3fs

bd7a2e6

jcadam14 reopened this Oct 4, 2024

Updated poetry.lock

f2c9412

jcadam14 marked this pull request as ready for review October 4, 2024 19:43

jcadam14 added 4 commits October 5, 2024 15:57

Fixed ordering of records

566e63c

Updated validator code

8881727

Removed psutiling

640d2ce

Removed unused code

95b9d05

jcadam14 requested a review from lchen-2101 November 6, 2024 14:44

lchen-2101 approved these changes Nov 6, 2024

View reviewed changes

jcadam14 merged commit 4398f7e into main Nov 6, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor to use polars #244

Refactor to use polars #244

jcadam14 commented Aug 8, 2024

github-actions bot commented Aug 8, 2024 •

edited

Loading

lchen-2101 left a comment

Refactor to use polars #244

Refactor to use polars #244

Conversation

jcadam14 commented Aug 8, 2024

github-actions bot commented Aug 8, 2024 • edited Loading

Coverage report

lchen-2101 left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 8, 2024 •

edited

Loading