MySQL 8.0 Source Code Improvements

With this post, I want to bring your attention to source code improvements in MySQL 8.0. MySQL 8.0 modernizes the code base by using C++11 constructs, being warning-free on more compilers and platforms, being UBSan- and ASan- clean, improving header file dependencies, improving the coding style, and better developer documentation.

C++11

With MySQL 8.0, we have started to modernize our C++ usage by using C++11 constructs. Here are some examples of C++11 constructs in use by 8.0:

  • std::atomic — fixed lots of bugs for us over the old use of volatile, replaced home-grown compiler-specific atomic
  • thread_local — gave us nice speedups over our old code
  • Initializers in structs/classes — simplified construction/destruction
  • std::snprintf — replaced the old my_snprintf
  • std::unordered_map/std::unordered_set — much faster than the old HASH
  • std::call_once — replaced our home-grown my_thread_once
  • std::unique_ptr — makes cleanup significantly safer (RAII pattern)
  • constexpr functions — enabled some speedups in InnoDB

MySQL 8.0 moving to C++11 requires GCC version  >= 4.8 (Linux), Clang >= 3.4 (macOS, FreeBSD), Visual Studio >= 2015 (Windows) or Oracle Studio 12.5 (Solaris). For comparison, MySQL 5.7 compiled fine with GCC >= 4.4, Clang >= 3.3, VS 2013 or Oracle Studio 12.2.

Similarly, the client library now assumes a C99 or C++11 compiler, in order to be able to access now-standard language features such as the bool type. The set of shipped client headers has also been slimmed down significantly; you no longer need the complete set of server headers to build against the client library, just a much smaller, platform-independent set.

Header File Dependencies

We have started cleaning up header files dependencies, i.e.  work on “include what you use” and on reorganizing header files to remove build dependencies. We have fixed ambiguous include paths; almost all should now be from the root. Incrementality has increased a lot after e.g. my_global.h went away, and sql_class.h was also reduced a fair bit in weight. Shipped client headers are self-contained and much more sane. For example, client headers are now platform independent (no difference between 32- and 64-bit Linux).

Warning-Free

With MySQL 8.0, we have invested in getting warning-free compiles on a wider range of compilers and platforms. MySQL 8.0 is warning-free on GCC up to and including 8, and warning free on Clang on Mac for the first time. MySQL 8.0 is warning-free with GCC 8 and Clang 6 with these options:

  • C: -Wall -Wextra -Wformat-security -Wvla -Wundef -Wmissing-format-attribute -Wwrite-strings -Werror
  • CC+: -Wall -Wextra -Wformat-security -Wvla -Wundef -Wmissing-format-attribute -Woverloaded-virtual -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wlogical-op -Werror

Code Sanitizers

With MySQL 8.0,  we have had a strong focus on AddressSanitizer (Asan) and UndefinedBehaviorSanitizer (UBsan). We strongly believe in using  tools like Asan and UBsan to increase code quality. We are now Asan-clean with options -fsanitize=address -fsanitize-address-use-after-scope  under GCC 8 and Clang 6.  We are also UBsan-clean with option -fsanitize=undefined under GCC 8 and Clang 6. We’ve fixed a number of bugs detected by UBsan and Asan.

Coding Standard

With MySQL 8.0 we have abandoned the historical MySQL Server and InnoDB coding styles and moved on to the Google C++ Style Guide. The code base has been reformatted as of MySQL 8.0.11. The motivation for this change was to achieve one common coding style (earlier the Server and InnoDB layers had different coding styles), as well as getting tooling support to enforce the coding style (clang-format --style=google). For the time being, it is mostly the formatting that has changed, but we have also started the process of conforming to other aspects of the Google C++ Style Guide.

Benefits of a common coding standard include:

  • Easier to get started for new developers (only one style, and a
    style that’s used by others, so it’s not MySQL specific).
  • In some cases, makes the code easier to read (some parts were hard to
    read earlier).
  • Better tool support for auto-formatting, which again leads to easier
    detection of bugs hiding in incorrect indentation.
  • One style all over => easier to write code and focus on what matters.

Developer Documentation

With MySQL 8.0 we are moving away from the historical MySQL Internals Manual and on to the MySQL Source Code Documentation. As the name indicates, the MySQL source code documentation resides in the source code and is managed in the same way as any other source code. We have finished the work in some areas, such as documenting the MySQL Test Framework, and for other areas we will add developer documentation over time. The developer documentation is written in Doxygen.

That’s it for now, and thank you for using MySQL !

About Geir Hoydalsvik

Geir Høydalsvik has been working with MySQL Database team since 2008. He is currently employed by Oracle, based in Norway. He is Senior Software Development Director and responsible for the development and maintenance of MySQL Database. He has a background in the database industry, working for the database startup company Clustra Inc. on the Clustra database and for Sun Microsystems on Java DB. He has a Master degree in Computer Science and a PhD in Software Engineering from the Norwegian University of Science and Technology.

7 thoughts on “MySQL 8.0 Source Code Improvements

      1. Ack re. TSan false positives, we don’t have real experience with it on MySQL codebase (my attempts on early 8.0 are at https://bugs.mysql.com/bug.php?id=83034, I haven’t revisited it with recent releases) – been waiting for clear baseline from you. We do have some experience with it on the TokuDB library, where real bugs have been fixed and we see that RocksDB uses it regularly.

        It looks like TSan itself is being actively developed, perhaps the false positive situation is better with recent LLVM/GCC releases.

        Sooner or later, we look forward very much into introducing its use, as killing whole classes of bugs with an automated tool sounds very promising.

  1. Warning free, at least with a standard cmake build (no -W options changed), does not seem accurate. Clang 6 or 7, Centos 7.4 x74

    /git/mysql-server_dbg/extra/re2/re2/regexp.cc:479:12: warning: comparison of unsigned enum expression < 0 is always false
    [-Wtautological-unsigned-enum-zero-compare]
    if (code = arraysize(kErrorStrings))
    ~~~~ ^ ~

    /git/mysql-server_dbg/extra/protobuf/protobuf-2.6.1/src/google/protobuf/compiler/cpp/cpp_message.cc:512:10: warning: comparison of two
    values with different enumeration types in switch statement (‘FieldDescriptor::CppType’ and
    ‘google::protobuf::internal::WireFormatLite::CppType’) [-Wenum-compare-switch]
    case internal::WireFormatLite::CPPTYPE_BOOL:
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    (etc.)

    /git/mysql-server_dbg/extra/protobuf/protobuf-2.6.1/src/google/protobuf/extension_set_heavy.cc:430:14: warning: comparison of two values
    with different enumeration types in switch statement (‘FieldDescriptor::Type’ and
    ‘google::protobuf::internal::WireFormatLite::FieldType’) [-Wenum-compare-switch]
    case WireFormatLite::TYPE_MESSAGE:
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
    (etc.)

    /git/mysql-server_dbg/extra/icu/source/i18n/numfmt.cpp:1308:15: warning: comparison of unsigned enum expression < 0 is always false
    [-Wtautological-unsigned-enum-zero-compare]
    if (style = UNUM_FORMAT_STYLE_COUNT) {
    ~~~~~ ^ ~

    /git/mysql-server_dbg/extra/icu/source/i18n/reldatefmt.cpp:550:27: warning: comparison of unsigned enum expression >= 0 is always true
    [-Wtautological-unsigned-enum-zero-compare]
    if (style >= 0 && genericUnit != INVALID_UNIT) {
    ~~~~~ ^ ~

    And others. I hope it is not necessary to hack in -W compiler options as given?

    1. Hi Roel, Looks like you look at warnings found in external libraries. We use these as they are and exclude such warnings rom our count. I think this makes sense. But please file a bug if you see a warning which should not be there. Geir

Leave a Reply