ISSUE 393 : Decoding the content into utf-8 to avoid encoding issues #394
Conversation
…tats_bref function. The issue is because we are wrongly encoding a bytes object parsed to string what we need to do instead is, decode the bytes object directly.
…stats_bref function.
The issue is because we are wrongly encoding a bytes object parsed to string
what we need to do instead is, decode the bytes object directly.
|
Could you elaborate on how to replicate the old issue? @bdilday Just to validate this not introduce a regression |
|
Hi, @bdilday Replicating the old issue, there is no data truncated with my fix. I used the same example as the previous issue pybaseball-2023-12-04_20.36.11.mp4 |
it's tricky to replicate because it depends not only on the data but on the (randomly generated) links in the page header. That is, if the page contained a link to the german language or spanish language fbref pages, AND a player in the table had a non-ascii character in their name, then the table got truncated. but the problem seems to be resolved now. Here's an example that should reproduce the original issue but works fine |
|
So, my PR should also solve the issue reported on ISSUE 393. Also tested the snippet you've shared with my changes and is working as expected. FYI @bdilday |
Yes, I think this PR looks good to go! |
The issue is that we are wrongly encoding a bytes object parsed to a string
what we need to do instead is, decode the bytes object directly.
This commit changes that, to decoding the content directly from bytes into utf-8
See, the results after these changes:
#393 (comment)