Edit this page | Blame

Fix Broken UTF-8 characters in our Database

Tags

  • assigned: bonfacem, arthur
  • type: database
  • priority: high

Description

We have jumbled up text in our database and this has been the case for years. It's impractical for a user to do the fixes using the metadata editing form because there are too many cases. A script that fixes this should be created to fix this issue.

This thread has some really nice ideas

An example of a broken unicode character is: "??????". The character "??????" appears broken because it is not a valid Unicode character. This can happen for a number of reasons, such as a mistake when typing or pasting the character, corruption during transmission (most likely the case) or storage, or a lack of support for the character in the font or software being used to display the text.

To find the correct replacement for the character "??????", or any other character for the matter, you can look up its Unicode code point. In this case, the code point for "??????" is "U+2273", which corresponds to the character "???". You can then use this code point to search for and replace the broken character with the correct character in the text.

Tables I've had to convert:

  • Investigators
  • InfoFiles
  • closed
(made with skribilo)