Emojis in course description cause failures after Olive -> Palm upgrade

[regisb]

Bug description

After upgrading from Olive to Palm, if a single course contains emojis in its short description (such as "📌" or "📣") then the LMS dashboard will fail to load.

How to reproduce

Launch an olive platform:

export TUTOR_ROOT=$(pwd)
pip install 'tutor[full]<16.0.0'
tutor local lauch
tutor local do createuser --superuser --staff --password=1234 admin admin@overhang.io

• Go to http://studio.local.overhang.io, login and create a new course.
• Go to "Schedule & details"
• In "Course Short Description" write "📌". Hit save.

Then, upgrade to Palm:

pip install --upgrade 'tutor[full]'
tutor local launch

Load the LMS dashboard: http://local.overhang.io. It will produce a 500 error with the following logs:

tutor_local-lms-1            | Traceback (most recent call last):                                                                                                                                           
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner                                                                      
tutor_local-lms-1            |     response = get_response(request)                                                                                                                                         
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response                                                                  
tutor_local-lms-1            |     response = wrapped_callback(request, *callback_args, **callback_kwargs)                                                                                                  
tutor_local-lms-1            |   File "/opt/pyenv/versions/3.8.15/lib/python3.8/contextlib.py", line 75, in inner                                                                                           
tutor_local-lms-1            |     return func(*args, **kwds)                                                                                                                                               
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view                                                              
tutor_local-lms-1            |     return view_func(request, *args, **kwargs)                                                                                                                               
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view                                                                    
tutor_local-lms-1            |     response = view_func(request, *args, **kwargs)                                                                                                                           
tutor_local-lms-1            |   File "/openedx/edx-platform/./openedx/core/djangoapps/util/maintenance_banner.py", line 42, in _decorated                                                                  
tutor_local-lms-1            |     return func(request, *args, **kwargs)                                                                                                                                    
tutor_local-lms-1            |   File "/openedx/edx-platform/common/djangoapps/student/views/dashboard.py", line 904, in student_dashboard                                                                  
tutor_local-lms-1            |     response = render_to_response(dashboard_template, context)                                                                                                               
tutor_local-lms-1            |   File "/openedx/edx-platform/common/djangoapps/edxmako/shortcuts.py", line 188, in render_to_response                                                                       
tutor_local-lms-1            |     return HttpResponse(render_to_string(template_name, dictionary, namespace, request), **kwargs)                                                                           
tutor_local-lms-1            |   File "/openedx/edx-platform/common/djangoapps/edxmako/shortcuts.py", line 178, in render_to_string                                                                         
tutor_local-lms-1            |     return template.render(dictionary, request)                                                                                                                              
tutor_local-lms-1            |   File "/openedx/edx-platform/common/djangoapps/edxmako/template.py", line 82, in render                                                                                     
tutor_local-lms-1            |     return self.mako_template.render_unicode(**context_dictionary)                                                                                                           
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/mako/template.py", line 444, in render_unicode                                                                             
tutor_local-lms-1            |     return runtime._render(                                                                                                                                                  
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/mako/runtime.py", line 874, in _render                                                                                     
tutor_local-lms-1            |     _render_context(                                                                                                                                                         
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/mako/runtime.py", line 916, in _render_context                                                                             
tutor_local-lms-1            |     _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)                                                                                                            
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/mako/runtime.py", line 943, in _exec_template                                                                              
tutor_local-lms-1            |     callable_(context, *args, **kwargs)                                                                                                                                      
tutor_local-lms-1            |   File "/tmp/mako_lms/e2555543ddce7402e2f4f062ddd48a94/main.html.py", line 378, in render_body                                                                               
tutor_local-lms-1            |     __M_writer(filters.html_escape(filters.decode.utf8(next.body())))                                                                                                        
tutor_local-lms-1            |   File "/tmp/mako_lms/e2555543ddce7402e2f4f062ddd48a94/dashboard.html.py", line 225, in render_body                                                                          
tutor_local-lms-1            |     course_overview = CourseOverview.get_from_id(enrollment.course_id)                                                                                                       
tutor_local-lms-1            |   File "/openedx/edx-platform/./openedx/core/lib/cache_utils.py", line 74, in decorator                                                                                      
tutor_local-lms-1            |     result = wrapped(*args, **kwargs)                                                                                                                                        
tutor_local-lms-1            |   File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 407, in get_from_id                                                        
tutor_local-lms-1            |     course_overview = cls.load_from_module_store(course_id)                                                                                                                  
tutor_local-lms-1            |   File "/openedx/edx-platform/./openedx/core/djangoapps/content/course_overviews/models.py", line 305, in load_from_module_store                                             
tutor_local-lms-1            |     course_overview.save()                                                                                                                                                   
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/model_utils/models.py", line 38, in save                                                                                   
tutor_local-lms-1            |     super().save(*args, **kwargs)                                                                                                                                            
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save                                                                               
tutor_local-lms-1            |     self.save_base(using=using, force_insert=force_insert,                                                                                                                   
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base                                                                          
tutor_local-lms-1            |     updated = self._save_table(                                                                                                                                              
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table                                                                        
tutor_local-lms-1            |     updated = self._do_update(base_qs, using, pk_val, values, update_fields,                                                                                                 
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update                                                                         
tutor_local-lms-1            |     return filtered._update(values) > 0                                                                                                                                      
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update                                                                           
tutor_local-lms-1            |     return query.get_compiler(self.db).execute_sql(CURSOR)                                                                                                                   
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql                                                               
tutor_local-lms-1            |     cursor = super().execute_sql(result_type)                                                                                                                                
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql                                          
tutor_local-lms-1            |     cursor.execute(sql, params)                                                                                                                                              
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute                                                                          
tutor_local-lms-1            |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)                                                                                      
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers                                                           
tutor_local-lms-1            |     return executor(sql, params, many, context)                                                                                                                              
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute                                                                         
tutor_local-lms-1            |     return self.cursor.execute(sql, params)                                                                                                                                  
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__                                                                                  
tutor_local-lms-1            |     raise dj_exc_value.with_traceback(traceback) from exc_value                                                                                                              
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute                                                                         
tutor_local-lms-1            |     return self.cursor.execute(sql, params)                                                                                                                                  
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute                                                                     
tutor_local-lms-1            |     return self.cursor.execute(query, args)                                                                                                                                  
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute                                                                                  
tutor_local-lms-1            |     res = self._query(query)                                                                                                                                                 
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query                                                                                   
tutor_local-lms-1            |     db.query(q)                                                                                                                                                              
tutor_local-lms-1            |   File "/openedx/venv/lib/python3.8/site-packages/MySQLdb/connections.py", line 254, in query                                                                                
tutor_local-lms-1            |     _mysql.connection.query(self, query)                                                                                                                                     
tutor_local-lms-1            | django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x93\\x8C' for column 'short_description' at row 1")                                           
tutor_local-caddy-1          | {"level":"error","ts":1696252737.2936494,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.24.0.1","remote_port":"47008","proto":"HTTP/1.1"
,"method":"GET","host":"local.overhang.io","uri":"/dashboard"},"user_id":"","duration":0.312947733,"size":3140,"status":500}                                                                                
tutor_local-lms-1            | [pid: 7|app: 0|req: 2/18] 172.24.0.3 () {46 vars in 3090 bytes} [Mon Oct  2 13:18:56 2023] GET /dashboard => generated 9591 bytes in 311 msecs (HTTP/1.1 500) 7 headers in 50
7 bytes (1 switches on core 0)                                                                                                                               

Additional context

This issue was initially raised here: https://discuss.openedx.org/t/mysql-error-after-upgrade-from-olive-to-palm/11304/

It is very likely that this issue is related to #887.

[Danyal-Faheem]

MySQL is also upgraded from 5.7 -> 8.0 when upgrading from Olive to Palm.

However, the character set is maintained across MySQL upgrades and Olive utilized the utf8mb3 character set as default and it stays the same in Palm after the upgrade process. This character set does not support emojis and therefore, this error occurs.

To resolve this, a new do command was added in v19.0.0 through #1079. You should utilize this command to upgrade the character set and collation of your database and this issue should be resolved.

For more details, please go through the docs regarding the convert-mysql-utf8mb4-charset do command.

Note that this issue should not occur for new installations that utilize Tutor v18.1.0 and later as the default character set and collation was upgraded to utf8mb4.

CC: @DawoudSheraz