Issue 386: LookupError: unknown encoding: utf8mb3

[Samira-El]

Issue: #386

For databases/tables that use utf8mb3, row events cannot be parse and fail with LookupError: unknown encoding: utf8mb3 because pymysql has no charset utf8mb3.

python-mysql-replication/pymysqlreplication/row_event.py

Line 255 in b306dd7

def __read_string(self, size, column):

Traceback (most recent call last):
  File "/home/github/python-mysql-replication/examples/mariadb_gtid/read_event.py", line 82, in <module>
    binlogevent.dump()
  File "/home/github/python-mysql-replication/pymysqlreplication/event.py", line 45, in dump
    self._dump()
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 526, in _dump
    super(WriteRowsEvent, self)._dump()
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 460, in _dump
    print("Changed rows: %d" % (len(self.rows)))
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 474, in rows
    self._fetch_rows()
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 469, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 522, in _fetch_one_row
    row["values"] = self._read_column_data(self.columns_present_bitmap)
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 157, in _read_column_data
    values[name] = self.__read_string(1, column)
  File "/home/github/python-mysql-replication/pymysqlreplication/row_event.py", line 265, in __read_string
    string = string.decode(encoding, decode_errors)
LookupError: unknown encoding: utf8mb3

I'm proposing this fix that intercepts the charset and changes it to utf8 if it's utf8mb3.
In Mariadb and Mysql utf8 is an alias for utf8mb3, and mysql has deprecated utf8mb3 and recommends using utf8mb4.

[Samira-El]

Hi @julien-duponchelle,

Can you take a look at this potential fix please?

[Samira-El]

Another solution would be to bump pymysql to v1.1.0 which now has support for utfmb3
https://github.com/PyMySQL/PyMySQL/blob/main/CHANGELOG.md#v110

[julien-duponchelle]

@Samira-El can you confirm that we can bump to v1.1.0 of PyMySQL ? It sound a more clean idea

Thanks a lot for the help

[Samira-El]

Hi, we're running PyMySQL v1.1.0 and mysql-replication v0.40 in production against Mariadb instances and we're no longer running into this issue.

[julien-duponchelle]

Thanks I did a new release with the version of PyMYSQL changed.