Here is a summary of the improvements to MTR test tool and suite in 5.7.4
Added 69 new tests and enhanced several existing tests in the MTR suite.
Test Suite migration
Test suite migration activity is continuing and in 5.7.4 we completed migration of replication suites rpl/binlog and about 75% of main suite. Relevant WLs are:
- WL#6921 Migrate rpl suite to run with innodb engine
- WL#6922 Migrate binlog suite to run with innodb storage engine
- WL#7263 Migrate myisam specific tests
- WL#7405 Migrate partition tests in the main suite
- WL#7402 Migrate tests in main suite for federated, blackhole, merge and csv engine
- WL#7411 Migrate tests that do not show any result difference when run with innodb engine
- WL#7410 Migrate authentication tests in main suite
- WL#7404 Migrate ctype tests in the main suite
Test Suite for replication (rpl) made GTID agnostic
Global Transaction Identifiers (GTID) was introduced in 5.6 and MTR tests were added specifically for testing GTID. To improve test coverage we wanted to run all tests in rpl suite with GTID ON. This was not possible because many will have result differences and fail. We have addressed this in 5.7.4 and have added regular regression runs of rpl suite with GTID turned on. Here is an example command line that is run regularly:
perl mysql-test-run.pl –force –timer –debug-server –parallel=auto –experimental=collections/default.experimental –comment=rpl_gtid-debug –vardir=var-rpl_gtid-debug –suite=rpl –mysqld=–enforce-gtid-consistency –mysqld=–log-slave-updates –mysqld=–gtid-mode=on –skip-test-list=collections/disabled-gtid-on.list –big-test –testcase-timeout=60 –suite-timeout=360
Refer WL#7205 for more details.
Moved InnoDB compression tests to a separate suite
Tests for InnoDB compression were part of the InnoDB suite and hence it was not easy to run only these tests with different compression options. Compression tests are now moved to a new suite called innodb_zip. This change is done from 5.5 onwards and following command lines were added to regression runs.
- perl mysql-test-run.pl –vardir=var-compressed_log0 –force –big-test –comment=compressed_log0 –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=0 –suite=innodb_zip
- perl mysql-test-run.pl –vardir=var-compressed_log1 –force –big-test –comment=compressed_log1 –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=1 –suite=innodb_zip
- perl mysql-test-run.pl –vardir=var-compressed_log0_level1 –force –big-test –comment=compressed_log0_level1 –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=0 –mysqld=–innodb-compression-level=1 –suite=innodb_zip
- perl mysql-test-run.pl –vardir=var-compressed_log1_level9 –force –big-test –comment=compressed_log1_level9 –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=1 –mysqld=–innodb-compression-level=9 –suite=innodb_zip
- perl mysql-test-run.pl –vardir=var-compressed_log0_level9_4k –force –big-test –comment=compressed_log0_level9_4k –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=0 –mysqld=–innodb-compression-level=9 –mysqld=–innodb-page-size=4k –suite=innodb_zip
- perl mysql-test-run.pl –vardir=var-compressed_log1_level1_8k –force –big-test –comment=compressed_log1_level1_8k –testcase-timeout=60 –debug-server –parallel=auto –experimental=collections/default.experimental –mysqld=–innodb-log-compressed-pages=1 –mysqld=–innodb-compression-level=1 –mysqld=–innodb-page-size=8k –suite=innodb_zip
Minor enhancement to mysql-test-run.pl
A new option “–do-test-list was added to mysql-test-run.pl . This will take a file name as an argument and run the tests listed in the file.
eg: perl mysql-test-run.pl do-test-list=mytests.list
Above command will run all tests listed in mytests.list. This is useful in development when we are working on a feature that will have impact on tests spread across multiple suites. The relevant tests can now be grouped together and run with a simple command line.
This is a continuation of my post on migration-of-mtr-suites-to-use-innodb . To set the context here is a quick recap.
MySQL 5.5 had the following changes with respect to the default engine
- Default engine in the server changed from MyISAM to InnoDB
- MTR modified to start server with the old default MyISAM. (This was required because historically most test results were recorded with MyISAM and many of them would fail if tests were run with the new server default, InnoDB)
- Tests retained as is in 5.5 and planned to migrate them to run with default engine in a future release
In MySQL 5.7 release we have started the migration project. Right in the beginning we realized that the switch of default engine had an unexpected side effect. MTR tests that were developed post 5.5 were also getting recorded and run with MyISAM unless an engine was explicitly specified in the test. I.e. we were in a state where not only old tests, but also new tests were running with the old default engine MyISAM. As a result the backlog of tests that had to be migrated was growing over time. We decided to fix the problem with new tests first and old tests later.
The only way we could force new tests to run with InnoDB by default was to remove the engine switch in MTR. This however would bring back the problem of existing tests failing with result difference. We decided to tackle this problem with a different approach. Instead of switching the engine in the test tool itself, it was switched in the existing tests. This is where the force_myisam_default.inc makes its entry. If this file is included it will tell MTR to start server with MyISAM. It was added to all existing tests that did not have any explicit engine specification at the test level. To be more precise it was added to tests that did not have have_innodb.inc, have_archive.inc etc. We then removed the switch in MTR, so that by default it starts the server with its default engine which is InnoDB. With these 2 changes we got to the state where new tests by default were running with InnoDB.This was a big step forward and on hindsight I think we should have done this instead of the engine switch in 5.5 .
Having fixed the problem with new tests, we have now shifted our focus to migration of old tests. The migration project does not in any way reduce the test coverage on MyISAM; rather it just adds InnoDB coverage where it was missing. Post migration tests for features which are dependent on the engine will have MyISAM and InnoDB variants. Tests for features that are expected to have same behavior on all engines may run only with the default engine InnoDB.
We have outlined following strategy for the migration project:
- Maintain MyISAM and innodb variants for tests and subtests that are dependent on the engine
- Remove MyISAM dependencies from tests that are not meant for testing MyISAM features
- Retain tests for MyISAM only features in MyISAM
There is an element of risk in this project since any small mistake can lead to missing or even worse wrong tests. However be rest assured that this is being handled with utmost care by our strong QA team. To ensure correctness and stability all result differences are analyzed and discussed with the concerned people before making any changes. Some of the differences were due to bugs and these are fixed before test migration. Some others are expected and are accepted as they are. Listed below are a few examples of the changes made to the result files:
- Engine name in SHOW CREATE TABLE
- Adding “sort by” to SELECT queries
- Adding “analyze table” statement before EXPLAIN to get consistent test output
- Minor differences to EXPLAIN output
As of 5.7.3 release migration is complete for auth_sec, federated, funcs_1, funcs_2, jp, opt_trace, perfschema, stress and sys_vars suites. It is not all done yet and you will see more and more tests getting migrated in the upcoming milestone releases.
If you have any questions or feedback regarding this project please post them here. I would love to hear what the community thinks about this.
In MySQL 5.7.2 a new include file “–source include/force_myisam_default.inc“ made its appearance in most of the .test files in the MTR suite. If you were wondering about this read on.
I will explain this change in two blogs. The first will describe why we are doing this and the next will explain how it is being done.
In order to set the context let me delve a bit into history. Starting from MySQL 5.5, the default storage engine for new tables is InnoDB. This change was coupled with a reverse switch in mysql-test-run, which made MyISAM the default storage engine for the server started through MTR(mysql-test-run). As a result default storage engine in the server was innodb, but most tests were run with the old default MyISAM.
Let me explain why such a switch was required. The usual practice in MTR test development was to not specify an engine in create table statement, unless the test is specifically for a non-default engine like merge, archive, innodb etc. As a result tables in most tests were created with the default engine which is MyISAM. From a test coverage perspective this was a perfectly sound strategy. Tests for engine specific properties were run with the appropriate engines and tests for engine independent properties were run with the default engine, MyISAM. However this strategy became a problem when the default engine was changed. Tables in tests that do not specify any engine started getting created with the new default innodb leading to result differences and in turn test failures. There were hundreds of failing tests and fixing all this was an enormous task.
To overcome this situation following 2 options were considered:
- Option 1 : Migrate all tests to run with innodb engine. This would involve analyzing all test failures and modifying the test or result files as required.
- Option 2: Switch the default storage engine in MTR so that there is no need to change the test or result files.
Option 1 would have been the ideal, but it became quickly obvious that this will be a time and resource intensive solution. On the other hand option 2 was easy, but risky. A careful evaluation of the test suite showed that the risk with option 2 is not very high. The tests that did not have any engine specified were either tests for MyISAM only features or tests that were not considered to be sensitive to the engine. For everything else there were engine specific variants of the test. For example the partitioning test has 6 flavors – partition_archive, partition_federated, partition_csv, partition_innodb, partition_blackhole and partition_myisam . So the risk with option 2 was not very significant and was acceptable at least in the short term.
After evaluating the pros and cons of both options it was decided to go with option 2 for 5.5 release and to implement option 1 in a future release. MySQL 5.5 and 5.6 were released using MyISAM as the default in MTR suite. In 5.7 we have started implementing option 2 and this is is the reason for the force_myisam_default.inc.
Did I confuse you? I said migrate to innodb and the inc file says force_myisam_default. Well, if you are confused wait for my next blog
Early this year Oracle released MySQL 5.6 – Best MySQL Release Ever. This release delivered not only quality, but also quantity in terms of number of features. See a comprehensive list here . The blogs below also refer to the massive changes introduced in 5.6
It is no mean task to deliver so many features with high quality that too for a feature rich product like MySQL. This was made possible by the increased focus given to testing during 5.6 development . Testing was integrated far better than in the past with MySQL processes and test team was grown significantly. After the first few milestone releases developers and other stake holders started seeing benefits of the new processes and that resulted in better cooperation between development and test teams which in turn resulted in a bunch of happy testers :). In this post I will take you through the process that was established for new feature qualification.
Features are delivered through milestones which Tomas Ulin has explained very well at http://insidemysql.com/the-milestone-release-model-revisited/. MySQL Server QA has the following goals for new features:
- Complete functional and non functional test coverage of changed and new functionality
- No regressions
- More than 80% Code Coverage
QA involvement starts as soon as the requirements and specifications of the feature are finalized by the development team. QA reviews available documents and provides feedback on the design, usability, testability etc. A discussion follows with the developer and changes are made as needed to ensure that the feature can be tested.
Once the specifications and requirements are acceptable QA starts working on the test plan. First QA documents all scenarios that needs to be tested. This includes stand alone tests, integration tests, non functional tests etc. Next step is to identify tools for automating the scenarios. Most commonly used functional test tools in MySQL are:
MTR based tests are added by default for any new feature. RQG use started from 5.5 and it gained wide acceptance in 5.6. At least 90% of the features in 5.6 have gone through RQG testing and this helped weed out many bugs which would have been nearly impossible to find using MTR.
While the developers are working on the product code , QA engineers start working on the automated tests, test infrastructure improvements etc.Final round of testing starts after the feature has passed code reviews. This phase can last anywhere between a few days to months depending on the complexity of the feature, stability of the code etc.
Features gets signed off only when the following conditions are met:
- No open bugs in the new feature – This is very strictly enforced and even minor bugs are not allowed. We believe that bugs are easiest to fix when the code is new and hence this can help us deliver features that are of high quality.
- No regressions – A feature gets developed on a tree which gets tested regularly through a continuous integration testing tool. Any regressions are detected and fixed before sign off.
- Acceptable Code coverage numbers – Code coverage report is generated for the changed lines of code and the minimum expected coverage is 80%. Most features have coverage of more than 90%. Any uncovered lines of code are analyzed and wherever possible new tests are added to increase code coverage.
- All new tests are added to the automated regression suite.
- All planned testing has completed with satisfactory results.
This process worked really well in 5.6 and we will use our experiences to improve this further and deliver an even better 5.7 !!