MySQL 8.0.1: Japanese collation for utf8mb4

In MySQL 8.0.1, in addition to new as_cs collations (accent sensitive, case sensitive) for utf8mb4, we have also added a new collation for Japanese.

Introducing utf8mb4_ja_0900_as_cs

Collating rules for for Japanese are complex.  Japanese has multiple writing systems with katakana, hiragana, kanji, romaji. On top of that, for a single character, there are fullwidth and halfwidth symbols.  For example, how we are going to sort ‘あ’, ‘ア’, ‘a’, ‘ア’?

According to the reorder rule for Japanese defined by CLDR: [reorder Latn Kana Hani], ‘a’ should sort before all others because ‘a’ is Latin letter and others are all Kana (katakana and hiragana). Then how about the sorting order of ‘あ’, ‘ア’and ‘ア’? We sort them as the rule defines: ‘&あ<<<<ア=ア’

Why are ‘あ’, ‘ア’and ‘ア’are equal? The rule says ‘あ’ should sort before ‘ア’ on the quaternary level! Yes, that’s true. But CLDR defines the default collating strength, 3 for Japanese. This means the quaternary difference is ignored by default. This might not be what user wants. We will consider adding more collations for Japanese based on user input.

JIS X 0208 (http://www.jisc.go.jp/app/pager?id=94516) is a specification published by Japanese Industrial Standard. It defines a set of Japanese kanji (totally 6,355 kanji characters) and the sorting order of them. utf8mb4_ja_0900_as_cs sorts these kanji characters as defined by JIS X 0208. But there are still many kanji characters not in the specification, for they are not so common. For these kanji characters, utf8mb4_ja_0900_as_cs sorts them with their implicit weight (http://www.unicode.org/reports/tr10/#Implicit_Weights) defined by UCA.

For example, we have kanji characters: ‘王’, ‘人’, ‘兵’, ‘﨎’, ‘㐀’. The first three characters are in the character set defined by JIS X 0208, the latter two are not.

Conclusion

This post follows on from earlier posts where we have been describing our work on improving utf8 support as part of the switch to make it the default character set.  If you haven’t read our earlier posts, please do:

Please also try out this new collation and let us know your feedback!

3 thoughts on “MySQL 8.0.1: Japanese collation for utf8mb4

Leave a Reply

Your email address will not be published. Required fields are marked *

Please enter * Time limit is exhausted. Please reload CAPTCHA.