########## The MySQL Test Framework ########## /** @page PAGE_MYSQL_TEST_RUN The MySQL Test Framework This manual describes the MySQL test framework, consisting of the test driver and the test script language. -----

Table of Contents

- @subpage PAGE_PREFACE - @subpage PAGE_INTRODUCTION - @subpage PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS - @subpage PAGE_RUNNING_TESTCASES - @subpage PAGE_WRITING_TESTCASES - @subpage PAGE_MYSQLTEST_PROGRAMS - @subpage PAGE_MYSQLTEST_LANGUAGE_REFERENCE - @subpage PAGE_UNIT_TESTS - @subpage PAGE_PLUGINS */ ############################################################################### ## Preface /** @page PAGE_PREFACE Preface MySQL distributions include a set of test cases and programs for running them. These tools constitute the MySQL test framework that provides a means for verifying that MySQL Server and its client programs operate according to expectations. The test cases consist mostly of SQL statements, but can also use test language constructs that control how to run tests and verify their results. This manual describes the MySQL test framework. It describes the programs used to run tests and the language used to write test cases. */ ############################################################################### ## Introduction to the MySQL Test Framework /** @page PAGE_INTRODUCTION Introduction to the MySQL Test Framework MySQL distributions include a test suite: a set of test cases and programs for running them. (If you find that the test suite is not included in your distribution, look for a similar distribution with -test in the name and install that as well.) These tools constitute the MySQL test framework that provides a means for verifying that MySQL Server and its client programs operate according to expectations. The test cases consist mostly of SQL statements, but can also use test language constructs that control how to run tests and verify their results. Distributions also provide facilities for running unit tests and creating new unit tests. This document describes the components of the MySQL test framework, how the test programs work, and the language used for writing test cases. It also provides a tutorial for developing test cases and executing them. The application that runs the test suite is named @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". Its location is the mysql-test directory, which is present both in source and binary MySQL Server distributions. On platforms other than Windows, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" is also available through the shortened name mtr in the same directory, as either a symbolic link or a copy. The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" application starts MySQL servers, restarts them as necessary when a specific test case needs different start arguments, and presents the test result. For each test case, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" invokes the @ref PAGE_MYSQLTEST "mysqltest" program (also referred to as the “test engine”) to read the test case file, intepret the test language constructs, and send SQL statements to the server. Input for each test case is stored in a file, and the expected result from running the test is stored in another file. The actual result is compared to the expected result after running the test. For a MySQL source distribution, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" is located in the mysql-test directory, and @ref PAGE_MYSQLTEST "mysqltest" is located in the client directory. The mysql-test and client directories are located in the root directory of the distribution. For a MySQL binary distribution, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" is located in the mysql-test directory, and @ref PAGE_MYSQLTEST "mysqltest" is located in the same directory where other client programs such as mysql or mysqladmin are installed. The locations of the mysql-test and other directories depend on the layout used for the distribution format. Within the mysql-test directory, test case input files and result files are stored in the t and r directories, respectively. The input and result files have the same basename, which is the test name, but have extensions of .test and .result, respectively. For example, for a test named “decimal”, the input and result files are mysql-test/t/decimal.test and mysql-test/r/decimal.result. Each test file is referred to as one test case, but usually consists of a sequence of related tests. An unexpected failure of a single statement in a test case makes the test case fail. There are several ways a test case can fail: This method of checking test results puts some restrictions on how test cases can be written. For example, the result cannot contain information that varies from run to run, such as the current time. However, if the information that varies is unimportant for test evaluation, there are ways to instruct the test engine to replace those fields in the output with fixed values. Because the test cases consist mostly of SQL statements in a text file, there is no direct support for test cases that are written in C, Java, or other languages. Such tests are not within the scope of this test framework. But the framework does support executing your own scripts and initiating them with your own data. Also, a test case can execute an external program, so in some respects the test framework can be extended for uses other than testing SQL statements. Finally, it is possible to embed small pieces of Perl code within the test. This can sometimes be used to perform actions or execute logic which is beyond the capabilities of the test language or SQL. */ ############################################################################### ## MySQL Test Framework Components /** @page PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS MySQL Test Framework Components

Table of Contents

- @subpage PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS - @subpage PAGE_REPORT_BUGS The MySQL test framework consists of programs that run tests, and directories and files used by those programs. Test Framework Programs ----------------------- The MySQL test framework uses several programs: Test suite programs can be found in these locations: Test Framework Directories and Files ------------------------------------- The test suite is located in the mysql-test directory, which contains the following components: Unit test-related files are located in the unittest directory. Additional files specific to storage engines and plugins may be present under the subdirectories of the storage or plugin directories. Test Execution and Evaluation ----------------------------- There are a number of targets in the top-level Makefile that can be used to run sets of tests. make test only runs unit tests. Other targets run subsets of the tests, or run tests with specific options for the test programs. Have a look at the Makefile to see what targets are available. A “test case” is a single file. The case might contain multiple individual test commands. If any individual command fails, the entire test case is considered to fail. Note that “fail” means “does not produce the expected result.” It does not necessarily mean “executes without error,” because some tests are written precisely to verify that an illegal statement does in fact produce an error. In such an instance, if the statement executes successfully without producing the expected error, that is considered failure of the test. Test case output (the test result) consists of: The disable_query_log and enable_query_log commands control logging of input SQL statements. The disable_result_log and enable_result_log commands control logging of SQL statement results, and warning or error messages resulting from those statements. @ref PAGE_MYSQLTEST "mysqltest" reads a test case file from its standard input by default. The `--test-file` or -x option can be given to name a test case file explicitly. @ref PAGE_MYSQLTEST "mysqltest" writes test case output to the standard output by default. The `--result-file` or -R option can be used to indicate the location of the result file. That option, together with the `--record` option, determine how @ref PAGE_MYSQLTEST "mysqltest" treats the test actual and expected results for a test case: @ref PAGE_MYSQLTEST "mysqltest" itself knows nothing of the t and r directories under the mysql-test directory. The use of files in those directories is a convention that is used by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", which invokes @ref PAGE_MYSQLTEST "mysqltest" with the appropriate options for each test case to tell @ref PAGE_MYSQLTEST "mysqltest" where to read input and write output. */ ############################################################################### ## Test Framework System Requirements /** @page PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS Test Framework System Requirements The @ref PAGE_MYSQLTEST "mysqltest" and @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" programs are written in C++ and are available on any system where MySQL itself can be compiled, or for which a binary MySQL distribution is available. Other parts of the test framework such as @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" are Perl scripts and should run on systems with Perl installed. @ref PAGE_MYSQLTEST "mysqltest" uses the diff program to compare expected and actual test results. If diff is not found, @ref PAGE_MYSQLTEST "mysqltest" writes an error message and dumps the entire contents of the .result and .reject files so that you can try to determine why a test did not succeed. If your system does not have diff, you may be able to obtain it from one of these sites: - http://www.gnu.org/software/diffutils/diffutils.html - http://gnuwin32.sourceforge.net/packages/diffutils.htm @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" cannot function properly if started from within a directory whose full path includes a space character, due to the complexities of handling this correctly in all the different contexts it will be used. */ ############################################################################### ## How to Report Bugs from Tests in the MySQL Test Suite /** @page PAGE_REPORT_BUGS How to Report Bugs from Tests in the MySQL Test Suite If test cases from the test suite fail, you should do the following: */ ############################################################################### ## Running Test Cases /** @page PAGE_RUNNING_TESTCASES Running Test Cases Typically, you run the test suite either from within a source tree (after MySQL has been built), or on a host where the MySQL server distribution has been installed. (If you find that the test suite is not included in your distribution, look for a similar distribution with -test in the name and install that as well.) To run tests, your current working directory should be the mysql-test directory of your source tree or installed distribution. In a source distribution, mysql-test is under the root of the source tree. In a binary distribution, the location of mysql-test depends on the distribution layout. The program that runs the test suite, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", will figure out whether you are in a source tree or an installed directory tree. To run the test suite, change location into your mysql-test directory and invoke the @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" script: @verbatim shell> cd mysql-test shell> ./mysql-test-run.pl @endverbatim @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" accepts options on the command line. For example: @verbatim shell> ./mysql-test-run.pl --force --suite=binlog @endverbatim By default, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" exits if a test case fails. `--force` causes execution to continue regardless of test case failure. For a full list of the supported options, see @ref PAGE_MYSQL_TEST_RUN_PL. To run one or more specific test cases, name them on the @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" command line. Test case files have names like t/test_name.test, where test_name is the name of the test case, but each name given on the command line should be the test case name, not the full test case file name. The following command runs the test case named rpl_abcd, which has a test file of t/rpl_abcd.test: @verbatim shell> ./mysql-test-run.pl rpl_abcd @endverbatim To run a family of test cases for which the names share a common prefix, use the `--do-test` option: @verbatim shell> ./mysql-test-run.pl --do-test=prefix @endverbatim For example, the following command runs the events tests (test cases that have names beginning with events): @verbatim shell> ./mysql-test-run.pl --do-test=events @endverbatim To run a specific named testsuite with all the test cases in it, use the `--suite` option: @verbatim shell> ./mysql-test-run.pl --suite=suite_name @endverbatim For example, the following command runs the replication tests located in the rpl suite: @verbatim shell> ./mysql-test-run.pl --suite=rpl @endverbatim @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" starts the MySQL server, sets up the environment for calling the @ref PAGE_MYSQLTEST "mysqltest" program, and invokes @ref PAGE_MYSQLTEST "mysqltest" to run the test case. For each test case to be run, @ref PAGE_MYSQLTEST "mysqltest" handles operations such as reading input from the test case file, creating server connections, and sending SQL statements to servers. The language used in test case files is a mix of commands that the @ref PAGE_MYSQLTEST "mysqltest" program understands and SQL statements. Input that @ref PAGE_MYSQLTEST "mysqltest" doesn't understand is assumed to consist of SQL statements to be sent to the database server. This makes the test case language familiar to those that know how to write SQL and powerful enough to add the control needed to write test cases. You need not start a MySQL server first before running tests. Instead, the @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" program will start the server or servers as needed. Any servers started for the test run use ports in the range from 13000 by default. @section SECTION_RUNNING_TESTS_IN_PARALLEL Running Tests in Parallel It is possible to run more than one instance of @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" simultaneously on the same machine. Both will by default use server ports from 13000 but will coordinate used port numbers as well as check for availibility, to avoid conflicts. Running several instances from the same mysql-test directory is possible but problematic. You must the use the `--vardir` to set different log directories for each instance. Even so, you can get into trouble becuse they will write .reject files to the same directories. It is also possible to have a single @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" run tests in several threads in parallel. Execution of the tests will be distributed among the threads. This is achieved using the `--parallel` option, with the number of threads as argument. The special value auto will ask @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to pick a value automatically, based on system information. The parallel option may also be given using the environment variable MTR_PARALLEL. @note Test cases which use not_parallel.inc are run at the end with a parallel value of 1 overriding the `--parallel` option value. Such test cases will be executed at the end of the test run one at a time. These tests also require `--non-parallel-test` option to be ON. */ ############################################################################### ## Writing Test Cases /** @page PAGE_WRITING_TESTCASES Writing Test Cases

Table of Contents

- @subpage PAGE_QUICK_START - @subpage PAGE_TESTCASE_CODING_GUIDELINES - @subpage PAGE_SAMPLE_TESTCASES - @subpage PAGE_CLEANUP_PREVIOUS_TEST_RUN - @subpage PAGE_GENERATE_TESTCASE_RESULT_FILE - @subpage PAGE_CHECK_EXPECTED_ERRORS - @subpage PAGE_CONTROL_INFORMATION_PRODUCED_TESTCASE - @subpage PAGE_DEALING_OUTPUT - @subpage PAGE_PASSING_OPTIONS - @subpage PAGE_TESTCASE_SPECIFIC_SERVER_OPTIONS - @subpage PAGE_TESTCASE_SPECIFIC_BOOTSTRAP_OPTIONS - @subpage PAGE_TESTCASE_SPECIFIC_CLIENT_OPTIONS - @subpage PAGE_INCLUDE_FILES - @subpage PAGE_BINARY_LOG_FORMAT - @subpage PAGE_WRITING_REPLICATION_TESTS - @subpage PAGE_THREAD_SYNCHRONIZATION - @subpage PAGE_SUPPRESSING_ERRORS_WARNINGS - @subpage PAGE_STOPPING_AND_RESTARTING_SERVER_DURING_TEST - @subpage PAGE_TIPS_WRITING_TESTCASES Normally, you run the test suite during the development process to ensure that your changes do not cause existing test cases to break. You can also write new test cases or add tests to existing cases. This happens when you fix a bug (so that the bug cannot reappear later without being detected) or when you add new capabilities to the server or other MySQL programs. This chapter provides guidelines for developing new test cases for the MySQL test framework. Some definitions: - One “test file” is one “test case”. - One “test case” might contain a “test sequence” (that is, a number of individual tests that are grouped together in the same test file). - A “command” is an input test that @ref PAGE_MYSQLTEST "mysqltest" recognizes and executes itself. A “statement” is an SQL statement or query that @ref PAGE_MYSQLTEST "mysqltest" sends to the MySQL server to be executed. */ ############################################################################### ## Writing a Test Case: Quick Start /** @page PAGE_QUICK_START Writing a Test Case: Quick Start The basic principle of test case evaluation is that output resulting from running a test case is compared to the expected result. This is just a diff comparison between the output and an expected-result file that the test writer provides. This simplistic method of comparison does not by itself provide any way to handle variation in the output that may occur when a test is run at different times. However, the test language provides commands for postprocessing result output before the comparison occurs. This enables you to manage certain forms of expected variation. Use the following procedure to write a new test case. In the examples, test_name represents the name of the test case. It is assumed here that you'll be using a development source tree, so that when you create a new test case, you can commit the files associated with it to the source repository for others to use.
  1. Change location to the test directory mysql-version/mysql-test: @verbatim shell> cd mysql-version/mysql-test @endverbatim mysql-version represents the root directory of your source tree.
  2. Create the test case in a file t/test_name.test. You can do this with any text editor. For details of the language used for writing @ref PAGE_MYSQLTEST "mysqltest" test cases, see @ref PAGE_MYSQLTEST_LANGUAGE_REFERENCE.
  3. Create an empty result file: @verbatim shell> touch r/test_name.result @endverbatim
  4. Run the test: @verbatim shell> ./mysql-test-run.pl test_name @endverbatim
  5. Assuming that the test case produces output, it should fail because the output does not match the result file (which is empty at this point). The failure results in creation of a reject file named mysql-test/var/log/test_name.reject. Examine this file. If the reject file appears to contain the output that you expect the test case to produce, copy its content to the result file: @verbatim shell> cp mysql-test/var/log/test_name.reject r/test_name.result @endverbatim Another way to create the result file is by invoking @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the `--record` option to record the test output in the result file: @verbatim shell> ./mysql-test-run.pl --record test_name @endverbatim
  6. Run the test again. This time it should succeed: @verbatim shell> ./mysql-test-run.pl test_name @endverbatim You can also run the newly created test case as part of the entire suite: @verbatim shell> ./mysql-test-run.pl @endverbatim
It is also possible to invoke the @ref PAGE_MYSQLTEST "mysqltest" program directly. If the test case file refers to environment variables, you will need to define those variables in your environment first. For more information about the @ref PAGE_MYSQLTEST "mysqltest" program, see @ref PAGE_MYSQLTEST. */ ############################################################################### ## Test Case Coding Guidelines /** @page PAGE_TESTCASE_CODING_GUIDELINES Test Case Coding Guidelines

Table of Contents

- @subpage PAGE_NAMING_ORGANIZATION_GUIDELINES - @subpage PAGE_CONTENT_FORMATTING_GUIDELINES - @subpage PAGE_NAMING_CONVENTIONS */ ############################################################################### ## File Naming and Organization Guidelines /** @page PAGE_NAMING_ORGANIZATION_GUIDELINES File Naming and Organization Guidelines Test case file names may use alphanumeric characters (A-Z, a-z, 0-9), underscore ('_') or dash ('-'), but should not start with underscore or dash. No other special characters are allowed, if used, then an error is thrown and the test run is aborted. Test names have traditionally used lowercase only, and we recommend continuing this for tests added to the common repository, though uppercase letters are also supported. Test cases are located in the mysql-test/t directory. Test case file names consist of the test name with a .test suffix. For example, a test named foo should be written in the file mysql-test/t/foo.test. In addition to this directory, tests are organized in test suites, located in subdirectories under the suite directory. For example, a test named bar under the replication suite rpl may be stored in the file mysql- test/suite/rpl/t/bar.test. In practice, the file would likely be called rpl_bar.test as tests in a suite usually also have the suite name as a prefix. This is just a convention from the time when suites were not supported, and not a requirement for test naming. One test case file can be a collection of individual tests that belong together. If one of the tests fails, the entire test case fails. Although it may be tempting to write each small test into a single file, that will be too inefficient and makes test runs unbearably slow. So make the test case files not too big, not too small. Each test case (that is, each test file) must be self contained and independent of other test cases. Do not create or populate a table in one test case and depend on the table in a later test case. If you have some common initialization that needs to be done for multiple test cases, create an include file. That is, create a file containing the initialization code in the mysql-test/include directory, and then put a source command in each test case that requires the code. For example, if several test cases need to have a given table created and filled with data, put the statements to do that in a file named mysql-test/include/create_my_table.inc. The .inc is a convention, not a requirement. Then put the following command in each test case file that needs the initialization code: @verbatim --source include/create_my_table.inc @endverbatim The file name in the source command is relative to the mysql-test directory. Remember to drop the table at the end of each test that creates it. */ ############################################################################### ## Test Case Content-Formatting Guidelines /** @page PAGE_CONTENT_FORMATTING_GUIDELINES Test Case Content-Formatting Guidelines When you write a test case, please keep in mind the following general guidelines. There are C/C++ coding guidelines in the MySQL Internals manual; please apply them when it makes sense to do so: @ref PAGE_CODING_GUIDELINES. Other guidelines may be found in this page, which discusses general principles of test-case writing: MySQL Internals: [How to Create Good Test Cases] (https://dev.mysql.com/doc/internals/en/good-tests.html) The following guidelines are particularly applicable to writing test cases: Example SQL statement, formatted onto multiple lines for readability: @verbatim SELECT f1 AS "my_column", f10 .... FROM mysqltest1.t5 WHERE (f2 BETWEEN 17 AND 25 OR f2 = 61) AND f3 IN (SELECT .... FROM mysqltest1.t4 WHERE .....) ORDER BY ... ; @endverbatim Example test file header: @verbatim ########### suite/funcs_1/t/a_processlist_val_no_prot.test ############# # # # Testing of values within INFORMATION_SCHEMA.PROCESSLIST # # # # The prepared statement variant of this test is # # suite/funcs_1/t/b_processlist_val_ps.test. # # # # There is important documentation within # # suite/funcs_1/datadict/processlist_val.inc # # # # Note(mleich): # # The name "a_process..." with the unusual prefix "a_" is # # caused by the fact that this test should run as second test, that # # means direct after server startup and a_processlist_priv_no_prot. # # Otherwise the connection IDs within the processlist would differ. # # # # Creation: # # 2007-08-09 mleich Implement this test as part of # # WL#3982 Test information_schema.processlist # # # ######################################################################## @endverbatim Example test reference to bug report: @verbatim # Bug #3671 Stored procedure crash if function has "set @variable=param" @endverbatim */ ############################################################################### ## Naming Conventions for Database Objects /** @page PAGE_NAMING_CONVENTIONS Naming Conventions for Database Objects It is possible to run test cases against a production server. This is very unlikely to happen by accident, as @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will start its own server unless you use the `--extern` Even so, try to write test cases in a way that reduces the risk that running tests will alter or destroy important tables, views, or other objects. (DROP DATABASE statements are particularly dangerous if written using names that could exist on a customer's machine.) To avoid such problems, we recommend the following naming conventions: For examples of how to name objects, examine the existing test cases. Of course, you can name columns and other objects inside tables as you wish. */ ############################################################################### ## Sample Test Case /** @page PAGE_SAMPLE_TESTCASES Sample Test Case Here is a small sample test case: @verbatim --disable_warnings DROP TABLE IF EXISTS t1; SET @@sql_mode='NO_ENGINE_SUBSTITUTION'; --enable_warnings SET SQL_WARNINGS=1; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES ("hej"); DROP TABLE t1; @endverbatim The first few lines try to clean up from possible earlier runs of the test case by dropping the t1 table. The test case uses disable_warnings to prevent warnings from being written to the output because it is not of any interest at this point during the test to know whether the table t1 was there. The value of sql_mode is changed to NO_ENGINE_SUBSTITUTION to avoid the error thrown by the second insert statement. After this, the test case uses enable_warnings so that subsequent warnings will be written to the output. The test case also enables verbose warnings in MySQL using the SET SQL_WARNINGS=1; statement. Next, the test case creates the table t1 and tries some operations. Creating the table and inserting the first row are operations that should not generate any warnings. The second insert should generate a warning because it inserts a nonnumeric string into a numeric column. At the end, it drops the table t1. The output that results from running the test looks like this: @verbatim DROP TABLE IF EXISTS t1; SET @@sql_mode='NO_ENGINE_SUBSTITUTION'; SET SQL_WARNINGS=1; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES ("hej"); Warnings: Warning 1366 Incorrect integer value: 'hej' for column 'a' at row 1 DROP TABLE t1; @endverbatim Note that the result includes not only the output from SQL statements, but the statements themselves. Statement logging can be disabled with the disable_query_log test language command. There are several options for controlling the amount of output from running the tests. If there was a test failure, it will be reported to the screen. You can see the actual output from the last unsuccessful run of the test case in the reject file mysql-test/var/log/test_name.reject. */ ############################################################################### ## Cleaning Up from a Previous Test Run /** @page PAGE_CLEANUP_PREVIOUS_TEST_RUN Cleaning Up from a Previous Test Run For efficiency, the @ref PAGE_MYSQLTEST "mysqltest" test engine does not start with a clean new database for running each test case, so a test case generally starts with a “cleaning up section”. Assume that a test case will use two tables named t1 and t2. The test case should begin by making sure that any old tables with those names do not exist: @verbatim --disable_warnings DROP TABLE IF EXISTS t1,t2; --enable_warnings @endverbatim The disable_warnings command instructs the test engine not to log any warnings until an enable_warnings command occurs or the test case is ended. (MySQL generates a warning if the table t1 or t2 does not exist.) Surrounding this part of the test case with commands to disable and enable warnings makes its output the same regardless of whether the tables exist before the test is started. After ensuring that the tables do not exist, we are free to put in any SQL statements that create and use the tables t1 and t2. The test case should also clean up at the end of the test by dropping any tables that it creates. Let's put in some SQL code into this test case: @verbatim --disable_warnings DROP TABLE IF EXISTS t1,t2; --enable_warnings CREATE TABLE t1 ( Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL, Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL ); CREATE TABLE t2 (Period SMALLINT); INSERT INTO t1 VALUES (9410,9412); INSERT INTO t2 VALUES (9410),(9411),(9412),(9413); SELECT PERIOD FROM t1; SELECT * FROM t1; SELECT t1.* FROM t1; SELECT * FROM t1 INNER JOIN t2 USING (Period); DROP TABLE t1, t2; @endverbatim If a test case creates other objects such as stored programs or user accounts, it should take care to also clean those up at the beginning and end of the test. Temporary files should also be removed, either at the end or just after they have been used. */ ############################################################################### ## Generating a Test Case Result File /** @page PAGE_GENERATE_TESTCASE_RESULT_FILE Generating a Test Case Result File The test code we just wrote contains no checks of the result. The test will report a failure for one of two reasons: - An individual SQL statement fails with an error - The overall test case result does not match what was expected Note that these are the reasons why @ref PAGE_MYSQLTEST "mysqltest" would fail; if the test is run from @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" the test may fail for additional reasons. In the first case, @ref PAGE_MYSQLTEST "mysqltest" aborts with an error. The second case requires that we have a record of the expected result so that it can be compared with the actual result. To generate a file that contains the test result, run the test with the `--record` option, like this: @verbatim shell> cd mysql-test shell> ./mysql-test-run.pl --record foo @endverbatim Running the test as shown creates a result file named mysql-test/r/foo.result that has this content: @verbatim DROP TABLE IF EXISTS t1,t2; CREATE TABLE t1 ( Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL, Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL ); CREATE TABLE t2 (Period SMALLINT); INSERT INTO t1 VALUES (9410,9412); INSERT INTO t2 VALUES (9410),(9411),(9412),(9413); SELECT period FROM t1; period 9410 SELECT * FROM t1; Period Varor_period 9410 9412 SELECT t1.* FROM t1; Period Varor_period 9410 9412 SELECT * FROM t1 INNER JOIN t2 USING (Period); Period Varor_period 9410 9412 DROP TABLE t1, t2; ok @endverbatim If we look at this result file, it contains the statements in the foo.test file together with the output from the SELECT statements. The output for each statement includes a row of column headings followed by data rows. Rows have columns separated by Tab characters. At this point, you should inspect the result file and determine whether its contents are as expected. If so, let it be part of your test case. If the result is not as expected, you have found a problem, either with the server or the test. Determine the cause of the problem and fix it. For example, the test might produce output that varies from run to run. To deal with this, you can postprocess the output before the comparison occurs. See @ref PAGE_DEALING_OUTPUT. */ ############################################################################### ## Checking for Expected Errors /** @page PAGE_CHECK_EXPECTED_ERRORS Checking for Expected Errors A good test suite checks not only that operations succeed as they ought, but also that they fail as they ought. For example, if a statement is illegal, the server should reject it with an error message. The test suite should verify that the statement fails and that it fails with the proper error message. The test engine enables you to specify “expected failures.” Let's say that after we create t1, we try to create it again without dropping it first: @verbatim --disable_warnings DROP TABLE IF EXISTS t1,t2; --enable_warnings CREATE TABLE t1 ( Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL, Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL ); CREATE TABLE t2 (Period SMALLINT); INSERT INTO t1 VALUES (9410,9412); INSERT INTO t2 VALUES (9410),(9411),(9412),(9413); SELECT period FROM t1; SELECT * FROM t1; SELECT t1.* FROM t1; SELECT * FROM t1 INNER JOIN t2 USING (Period); CREATE TABLE t1 (something SMALLINT(4)); @endverbatim The result is failure and an error: @verbatim At line 21: query 'CREATE TABLE t1 (something SMALLINT(4))' failed: 1050: Table 't1' already exists @endverbatim To handle this error and indicate that indeed we do expect it to occur, we can put an error command before the second create table statement. Either of the following commands test for this particular MySQL error: @verbatim --error 1050 --error ER_TABLE_EXISTS_ERROR @endverbatim 1050 is the numeric error code and ER_TABLE_EXISTS_ERROR is the symbolic name. Symbolic names are more stable than error numbers because the numbers sometimes change, particularly for those created during recent development. For such errors, use of numbers rather than the names in a test case will require test to be revised should the numbers change. After we make a change to add an error command before the CREATE TABLE statement and run the test again, the end of the result will look like this: @verbatim CREATE TABLE t1 (something SMALLINT(4)); ERROR 42S01: Table 't1' already exists @endverbatim In this case, the result shows the statement that causes the error, together with the resulting error message. The fact that @ref PAGE_MYSQLTEST "mysqltest" does not terminate and that the error message becomes part of the result indicates that the error was expected. You can also test for errors by specifying an SQLSTATE value. For MySQL error number 1050, the corresponding SQLSTATE value is 42S01. To specify an SQLSTATE value in an error command, use an S prefix: @verbatim --error S42S01 @endverbatim A disadvantage of SQLSTATE values is that sometimes they correspond to more than one MySQL error code. Using the SQLSTATE value in this case might not be specific enough (it could let through an error that you do not actually expect). If you want to test for multiple errors, the error command allows multiple arguments, separated by commas. For example: @verbatim --error ER_NO_SUCH_TABLE,ER_KEY_NOT_FOUND @endverbatim For a list of MySQL error codes, symbolic names, and SQLSTATE values, see https://dev.mysql.com/doc/refman/8.0/en/error-messages-server.html. You can also examine the mysqld_error.h and sql_state.h files in the include directory of a MySQL source distribution. It is also possible to use symbolic error names to refer to client errors: @verbatim --error CR_SERVER_GONE_ERROR @endverbatim For a list of MySQL client error codes, see https://dev.mysql.com/doc/refman/8.0/en/error-messages-client.html. You can also examine the errmsg.h file in the include directory of a MySQL source distribution. The built-in variable $mysql_errno contains the numeric error returned by the most recent SQL statement sent to the server, or 0 if the statement executed successfully. This may be useful after statements that may or may not fail, or fail in more than one way (more than one argument to the error command), in case you need to perform different actions. Note that this applies to SQL statements, not to other commands. There is also a variable $mysql_errname which contains the symbolic name of the last error. In some cases the symbolic name is not available; in those cases the variable will contain the string "". For new test development we recommend testing against the name rather than the number, since the number may change in future versions. If the last statement succeeded ($mysql_errno is 0), this variable is an empty string. */ ############################################################################### ## Controlling the Information Produced by a Test Case /** @page PAGE_CONTROL_INFORMATION_PRODUCED_TESTCASE Controlling the Information Produced by a Test Case By default, the @ref PAGE_MYSQLTEST "mysqltest" test engine produces output only from select, show, and other SQL statements that you expect to produce output (that is, statements that create a result set). It also produces output from certain commands such as echo and exec. @ref PAGE_MYSQLTEST "mysqltest" can be instructed to be more or less verbose. Suppose that we want to include in the result the number of rows affected by or returned by SQL statements. To do this, add the following line to the test case file preceding the first table-creation statement: @verbatim --enable_info @endverbatim After rerunning the test by invoking @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the `--record` option to record the new result, the result file will contain more information: @verbatim DROP TABLE IF EXISTS t1,t2; CREATE TABLE t1 ( Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL, Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL ); affected rows: 0 CREATE TABLE t2 (Period SMALLINT); affected rows: 0 INSERT INTO t1 VALUES (9410,9412); affected rows: 1 INSERT INTO t2 VALUES (9410),(9411),(9412),(9413); affected rows: 4 info: Records: 4 Duplicates: 0 Warnings: 0 SELECT period FROM t1; period 9410 affected rows: 1 SELECT * FROM t1; Period Varor_period 9410 9412 affected rows: 1 SELECT t1.* FROM t1; Period Varor_period 9410 9412 affected rows: 1 SELECT * FROM t1 INNER JOIN t2 USING (Period); Period Varor_period 9410 9412 affected rows: 1 DROP TABLE t1, t2; affected rows: 0 ok @endverbatim To turn off the affected-rows reporting, add this command to the test case file: @verbatim --disable_info @endverbatim In general, options can be enabled and disabled for different parts of the test file. Suppose that we are interested in the internals of the database as well. We could enable the display of query metadata using enable_metadata. With this option enabled, the test output is a bit verbose. However, as mentioned earlier, the option can be enabled and disabled selectively so that it is enabled only for those parts of the test case where it interests you to know more. If you perform an operation for which you have no interest in seeing the statements logged to the result, you can disable statement logging. For example, you might be initializing a table where you don't really expect a failure, and you are not interested in seeing the initialization statements in the test result. You can use the disable_query_log command to temporarily disable recording of input SQL statements, and enable recording again with enable_query_log. You can disable the recording of the output from executing commands using disable_result_log and enable recording again with enable_result_log. */ ############################################################################### ## Dealing with Output That Varies Per Test Run /** @page PAGE_DEALING_OUTPUT Dealing with Output That Varies Per Test Run It is best to write each test case so that the result it produces does not vary for each test run, or according to factors such as the time of day, differences in how program binaries are compiled, the operating system, and so forth. For example, if the result contains the current date and time, the test engine has no way to verify that the result is correct. However, sometimes a test result is inherently variable according to external factors, or perhaps there is a part of a result that you simply do not care about. @ref PAGE_MYSQLTEST "mysqltest" provides commands that enable you to postprocess test output into a more standard format so that output variation across test runs will not trigger a result mismatch. One such command is replace_column, which specifies that you want to replace whatever is in a given column with a string. This makes the output for that column the same for each test run. To see how this command works, add the following row after the first insert in the test case: @verbatim INSERT INTO t1 VALUES (DATE_FORMAT(NOW(), '%s'),9999); @endverbatim Then record the test result and run the test again: @verbatim shell> ./mysql-test-run.pl --record foo shell> ./mysql-test-run.pl foo @endverbatim Most likely, a failure will occur and @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will display the difference between the expected result and what we actually got, like this (the header has been simplified): @verbatim CURRENT_TEST: main.foo --- r/foo.result 2009-11-17 16:22:38 +++ var/log/foo.reject 2009-11-17 16:22:47 @@ -10,15 +10,15 @@ SELECT period FROM t1; period 9410 -0038 +0047 SELECT * FROM t1; Period Varor_period 9410 9412 -0038 9999 +0047 9999 SELECT t1.* FROM t1; Period Varor_period 9410 9412 -0038 9999 +0047 9999 SELECT * FROM t1 INNER JOIN t2 USING (Period); Period Varor_period 9410 9412 mysqltest: Result content mismatch @endverbatim The actual numbers will likely be different for your case, and the format of the diff may also vary. If we are not really interested in the first column, one way to eliminate this mismatch is by using the replace_column command. The duration of the effect of this command is the next SQL statement, so we need one before each select statement: @verbatim --replace_column 1 SECONDS SELECT period FROM t1; --replace_column 1 SECONDS SELECT * FROM t1; --replace_column 1 SECONDS SELECT t1.* FROM t1; @endverbatim In the replace_column commands, SECONDS could be any string. Its only purpose is to map variable output onto a constant value. If we record the test result again, we will succeed each time we run the test after that. The result file will look like this: @verbatim DROP TABLE IF EXISTS t1,t2; CREATE TABLE t1 ( Period SMALLINT(4) UNSIGNED ZEROFILL DEFAULT '0000' NOT NULL, Varor_period SMALLINT(4) UNSIGNED DEFAULT '0' NOT NULL ); affected rows: 0 CREATE TABLE t2 (Period SMALLINT); affected rows: 0 INSERT INTO t1 VALUES (9410,9412); affected rows: 1 INSERT INTO t1 VALUES (DATE_FORMAT(NOW(), '%s'),9999); affected rows: 1 INSERT INTO t2 VALUES (9410),(9411),(9412),(9413); affected rows: 4 info: Records: 4 Duplicates: 0 Warnings: 0 SELECT period FROM t1; period SECONDS SECONDS affected rows: 2 SELECT * FROM t1; Period Varor_period SECONDS 9412 SECONDS 9999 affected rows: 2 SELECT t1.* FROM t1; Period Varor_period SECONDS 9412 SECONDS 9999 affected rows: 2 SELECT * FROM t1 INNER JOIN t2 USING (Period); Period Varor_period 9410 9412 affected rows: 1 DROP TABLE t1, t2; affected rows: 0 ok @endverbatim */ ############################################################################### ## Passing Options from mysql-test-run.pl to mysqld or mysqltest /** @page PAGE_PASSING_OPTIONS Passing Options from mysql-test-run.pl to mysqld or mysqltest @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" supports several options that enable you to pass options to other programs. Each of these options takes a value consisting of one or more comma-separated options: */ ############################################################################### ## Specifying Test Case-Specific Server Options /** @page PAGE_TESTCASE_SPECIFIC_SERVER_OPTIONS Specifying Test Case-Specific Server Options Within a test case, many system variables can be set by using statements such as these: @verbatim SET sql_warnings= 1; SET sql_mode= 'NO_AUTO_VALUE_ON_ZERO'; @endverbatim But sometimes you need to restart the server to use command-line options that are specific to a given test case. You can specify these options in a file named mysql-test/t/test_name-master.opt. When a file named t/test_name-master.opt exists, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for extra options that the server needs to be run with when executing the test_name test case. If no server has yet been started or the current server is running with different options, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" restarts the server with the new options. You may refer to environment variables in the option file, using the usual $VAR_NAME syntax. It is also possible to refer to optional variables using the syntax $?VAR_NAME. This will be replaced with an empty string if the variable is not set. As a special case, the option `--skip-core-file` will be interpreted by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", which will then block the server from producing any core files or crash dumps during this test. This may be useful for tests that intentionally crash the server. Another special case is `--testcase-timeout`=minutes which can be used to set a different, longer timeout for a particular test case. The given timeout in minutes will be used for this test if it's longer than the default. Files in the mysql-test/t directory with names ending in -slave.opt are similar, but they are used for slave servers in replication tests. Sometimes it's also necessary to execute some external commands before starting the server, such as creating a directory. If you add a file named t/test_name-master.sh, it will be executed by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" before it starts the server; a similar file may be created for the slave in replication tests. Because the .sh file is executed through /bin/sh, it cannot be used on Windows, and any tests using such a file will automatically be skipped if you run on Windows. For this reason, this mechanism may be replaced with a more portable one in some future release of MySQL. */ ############################################################################### ## Specifying Test Case-Specific Bootstrap Options /** @page PAGE_TESTCASE_SPECIFIC_BOOTSTRAP_OPTIONS Specifying Test Case-Specific Bootstrap Options If a test has to run with a particular value of a bootstrap variable such as `--innodb-page-size` or `--innodb-data-file-path`, the option can be passed on the command line while running @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". For example, consider a test that can only run with `--innodb-page-size`=8k. The test can be run like this: @verbatim shell> ./mysql-test-run.pl test_name_8k --initialize=--innodb-page-size=8k @endverbatim This will initialize the data directory with 8k page size, and start the server with that value. It is also possible to pass bootstrap options in the master.opt file of the test, so that the test can run with the specified value of the bootstrap options without using any command line arguments. The usage is: @verbatim --initialize --innodb-page-size=8k @endverbatim or @verbatim --initialize=--innodb-page-size=8k @endverbatim Specifying bootstrap variables in the opt file is the preferred method. Each bootstrap variable must be specified as the value of a `--initialize` option in the opt file to ensure @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" recognizes that the variable must be used during server initialization. If there are bootstrap options in the file, the current data directory is deleted, and initialized again with the options set in the file. The server is also started with the bootstrap options passed along with the other options in the opt file. Similarly, bootstrap options passed in the slave.opt will be used to reinitialize the slave server in replication scenarios. Bootstrap options set in the opt file are passed to the server first, followed by the server options. So, server start up options have precedence over bootstrap options, regardless of the order in which they are set in the opt file. Note that since the bootstrap options are passed in the opt files, they have precedence over the command line bootstrap options. So, if a test has a master.opt file containing `--innodb-page-size`=8k, and while it is being run, `--innodb-page-size`=4k is passed on the command line, the test will run with 8k page size. Some tests require the server to be restarted multiple times with different server options, as well different bootstrap options. In such cases, the existing data directory is deleted inside the test and a new data directory is initialized with the bootstrap options. When the server is started with the bootstrap options after this, the SQL queries will run with the specified options. @verbatim --exec $MYSQLD --no-defaults --initialize-insecure --lc_messages_dir=$MYSQL_SHAREDIR --innodb_page_size=8K --basedir=$MYSQLD_BASEDIR --datadir=$MYSQL_TMP_DIR/datadir1/data --init-file=$BOOTSTRAP_SQL --secure-file-priv="" @endverbatim */ ############################################################################### ## Specifying Test Case-Specific Client Options /** @page PAGE_TESTCASE_SPECIFIC_CLIENT_OPTIONS Specifying Test Case-Specific Client Options If a test case needs a specific client option, you can specify such options in a file named testname-client.opt. When a test case has its own -client.opt file, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for any additional options that the @ref PAGE_MYSQLTEST "mysqltest" should run with while executing the test case. For example, consider a test case example.test needs latin1 as default character set for client and connection, then that option can be specified in example-client.opt file. @verbatim --default-character-set=latin1 @endverbatim You may refer to environment variables in the option file, using the usual $VAR_NAME syntax. It is also possible to refer to optional variables using the syntax $?VAR_NAME. This will be replaced with an empty string if the variable is not set. */ ############################################################################### ## Using Include Files to Simplify Test Cases /** @page PAGE_INCLUDE_FILES Using Include Files to Simplify Test Cases The include directory contains many files intended for inclusion into test case files. These include files serve many purposes, but in general, they encapsulate operations of varying complexity into single files so that you can perform each operation in a single step. Include files are available for operations such as these: You can think of an include file as a rudimentary form of subroutine that is “called” at the point of inclusion. You can “pass parameters” by setting variables before including the file and referring to them within the file. You can “return” values by setting variables within the file and referring them following inclusion of the file. */ ############################################################################### ## Controlling the Binary Log Format Used for Tests /** @page PAGE_BINARY_LOG_FORMAT Controlling the Binary Log Format Used for Tests

Table of Contents

- @subpage PAGE_CONTROLLING_BINARY_LOG_FORMAT - @subpage PAGE_SPECIFY_BINARY_LOG_FORMAT The server can do binary logging using statement-based logging (SBL), which logs events as statements that produce data changes, or row-based logging (RBL), which logs events as changes to individual rows. It also supports mixed logging, which switches between SBL and RBL automatically as necessary. The server's global binlog_format system variable indicates which log format is in effect. It has possible values of STATEMENT, ROW, and MIXED (not case sensitive). This system variable can be set at server startup by specifying `--binlog_format`=value on the command line or in an option file. A user who has the SUPER privilege can change the log format at runtime. For example: @verbatim SET GLOBAL binlog_format = STATEMENT; @endverbatim Some tests require a particular binary log format. You can exercise control over the binary log format in two ways: - To control the log format that the server uses for an entire test run, you can pass options to @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" that tell it which format mysqld should use. - To verify that a particular log format is in effect for a specific test case, you can use an appropriate include file that checks the current format and exits if the format is other than what is required. The following sections describe how to use these techniques. */ ############################################################################### ## Controlling the Binary Log Format Used for an Entire Test Run /** @page PAGE_CONTROLLING_BINARY_LOG_FORMAT Controlling the Binary Log Format Used for an Entire Test Run To specify the binary log format for a test run, you can use the `--mysqld` or `--combination` option to tell @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to pass a logging option to mysqld. For example, the following command runs the tests from the rpl suite that have names that begin with rpl_row. The tests are run once with the binary log format set to STATEMENT: @verbatim shell> ./mysql-test-run.pl --suite=rpl --do-test=rpl_row --mysqld=--binlog_format=statement @endverbatim To run tests under multiple log formats, use two or more instances of the `--combination` option. The following command runs the same tests as the preceding command, but runs them once with the binary log format set to ROW and a second time with the format set to MIXED: @verbatim shell> ./mysql-test-run.pl --suite=rpl --do-test=rpl_row --combination=--binlog_format=row --combination=--binlog_format=mixed @endverbatim The `--combination` option must be given at least twice or it has no effect. As an alternative to using the `--combination` option, you can create a file named combinations in the test suite directory and list the options that you would specify using `--combination`, one line per option. For the preceding @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" command, the suite name is rpl, so you would create a file named suite/rpl/combinations with these contents: @verbatim [row] --binlog_format=row [mixed] --binlog_format=mixed @endverbatim Then invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" like this: @verbatim shell> ./mysql-test-run.pl --suite=rpl --do-test=row @endverbatim Similar to combinations file in the test suite directory, you can also create a file named test_name.combinations in the test directory and list the options that you would specify using `--combination`, one line per option. If there exist both suite level combinations file and a test_name.combinations file, then the test_name.combinations file will take the priority i.e combinations specified in suite level combinations file will be ignored. The format of the combinations file is similar to that of my.cnf files (section names followed by options for each section). @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" displays the section name following the test name when it reports the test result. Any `--combination` options specified on the command line override those found in a combinations file. The `--combination` option and the combinations file have different scope. The `--combination` option applies globally to all tests run by a given invocation of @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". The combinations file is placed in a test suite directory and applies only to tests in that suite. The test_name.combinations file is placed in a test directory and applies only to test_name. */ ############################################################################### ## Specifying the Required Binary Log Format for Individual Test Cases /** @page PAGE_SPECIFY_BINARY_LOG_FORMAT Specifying the Required Binary Log Format for Individual Test Cases To specify within a test case that a particular binary log format is required, include one of the following lines to indicate the format: @verbatim --source include/have_binlog_format_row.inc --source include/have_binlog_format_statement.inc --source include/have_binlog_format_mixed.inc @endverbatim The following files can be used for tests that support two binary log formats: @verbatim --source include/have_binlog_format_mixed_or_row.inc --source include/have_binlog_format_mixed_or_statement.inc --source include/have_binlog_format_row_or_statement.inc @endverbatim Before @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs the test case, it checks whether the value that it is using for the binlog_format system variable matches what the test requires, based on whether the test refers to one of the preceding include files. If binlog_format does not have an appropriate value, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" skips the test. If a test supports all binary log formats, none of the have_binlog_format_*.inc include files should be used in the test file. A test that includes no such file is assumed to support all formats. */ ############################################################################### ## Writing Replication Tests /** @page PAGE_WRITING_REPLICATION_TESTS Writing Replication Tests If you are writing a replication test case, the test case file should begin with this command: @verbatim --source include/master-slave.inc @endverbatim To switch between the master and slave, use these commands: @verbatim connection master; connection slave; @endverbatim If you need to do something on an alternative connection, you can use connection master1; for the master and connection slave1; for the slave. To run the master with additional options for your test case, put them in command-line format in t/test_name-master.opt. When a file named t/test_name-master.opt exists, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" examines it for extra options that the server needs to be run with when executing the test_name test case. If no server has yet been started or the current server is running with different options, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" restarts the server with the new options. For the slave, similar principles apply, but you should list additional options in t/test_name-slave.opt. Several include files are available for use in tests that enable better control over the behavior of slave server I/O and SQL threads. The files are located in the include directory and have names of the form wait_for_slave_*.inc. By using these files, you can help make replication tests more stable because it will be more likely that test failures are due to replication failures, not due to problems with the tests themselves. The slave-control include files address the issue that it is not always sufficient to use a START SLAVE or STOP SLAVE statement by itself: When the statement returns, the slave may not have reached the desired operational state. For example, with START SLAVE, the following considerations apply: To verify that a slave has reached the desired state, combine the use of START SLAVE or STOP SLAVE with an appropriate “wait” include file. The file contains code that waits until the state has been reached or a timeout occurs. For example, to verify that both slave threads have started, do this: @verbatim START SLAVE --source include/wait_for_slave_to_start.inc @endverbatim Similarly, to stop both slave threads, do this: @verbatim STOP SLAVE; --source include/wait_for_slave_to_stop.inc @endverbatim The following list describes the include files that are available for slave control: */ ############################################################################### ## Thread Synchronization in Test Cases /** @page PAGE_THREAD_SYNCHRONIZATION Thread Synchronization in Test Cases The Debug Sync facility allows placement of synchronization points in the code. They can be activated by statements that set the debug_sync system variable. An active synchronization point can emit a signal or wait for a signal to be emitted by another thread. This waiting times out after 300 seconds by default. The `--debug-sync-timeout`=N option for @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" changes that timeout to N seconds. A timeout of zero disables the facility altogether, so that synchronization points will not emit or wait for signals, even if activated. The purpose of the timeout is to avoid a complete lockup in test cases. If for some reason the expected signal is not emitted by any thread, the execution of the affected statement will not block forever. A warning shows up when the timeout happens. That makes a difference in the test result so that it will not go undetected. For test cases that require the Debug Sync facility, include the following line in the test case file: @verbatim --source include/have_debug_sync.inc @endverbatim For a description of the Debug Sync facility and how to use synchronization points, see [MySQL Internals: Test Synchronization] (https://dev.mysql.com/doc/internals/en/test-synchronization.html). */ ############################################################################### ## Suppressing Errors and Warning /** @page PAGE_SUPPRESSING_ERRORS_WARNINGS Suppressing Errors and Warning After a test is finished, and if it didn't fail for some other reason, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will check the log written by the server(s) during the test for any warnings or errors. If it finds any, it will print them out and the test will be reported as failed. However, many warnings and also a few errors may occur normally without signifying a real problem. Also, many tests purposefully perform actions that will provoke warnings. For these not to result in a test failure, it is possible to specify that certain messages are to be suppressed. There is a list of global suppressions; warnings that will always be suppressed. These are listed in the file include/mtr_warnings.sql. Any error or warning that contains one of the listed strings will be ignored. It is not necessary to include the whole text, and regular expressions can be used, such as .* to match any substring or [0-9]* to match any number. You will rarely need to change or add to this global list, but you may need to suppress a particular error or warning for a new test case. A typical case is a test that performs some illegal action on purpose to test the respone; if this also results in a warning in the server log, this should not cause this particular test to fail. To add a suppresion for a warning text like for example The table 't23456' is full where the number in the table name can vary, add this line anywhere in the test case file: @verbatim call mtr.add_suppression("The table 't[0-9]*' is full"); @endverbatim This may be put anythere in the test, but we recommend putting it either near the top, or just after the action that may result in the warning. If you're adding this line to an existing test, keep in mind that it will be echoed in the output, so the exact same line also needs to be added to the corresponding place in the result file. Alternatively, you may turn off logging like this, and will then not have to edit the result file: @verbatim --disable_query_log call mtr.add_suppression("The table 't[0-9]*' is full") --enable_query_log @endverbatim It is also possible to intruct @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to skip the check for errors and warnings completely, by use of the `--nowarnings` command line option. */ ############################################################################### ## Stopping and restarting a Server During a Test /** @page PAGE_STOPPING_AND_RESTARTING_SERVER_DURING_TEST Stopping and Restarting a Server During a Test If a server dies during execution of a test case, this will be interpreted as a failure. However, there may be cases where you actually want to stop and possibly restart a server intentionally. It is possible to let the system know you expect a server to terminate, and to either wait or have it restarted immediately: Before you initiate the action that will stop the server, the test case should write either restart or wait to the file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect. The easiest way to do this is using the exec echo construct like in the following example. If you write restart into this file, the server will be immediately restarted. If you write wait instead, the server will remain down, but can be restarted at a later time by writing restart to the file. @verbatim --exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect --shutdown_server 10 --source include/wait_until_disconnected.inc # Do something while server is down --enable_reconnect --exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect --source include/wait_until_connected_again.inc @endverbatim The file name to write the command to will be mysqld.2.expect for the slave server in replication tests. Note that you have to use $MYSQLTEST_VARDIR/tmp for the directory here; if you use $MYSQL_TMP_DIR instead, it will not work when running tests in parallel. For your convenience, there are files which can be used to do what's shown in the example. */ ############################################################################### ## Other Tips for Writing Test Cases /** @page PAGE_TIPS_WRITING_TESTCASES Other Tips for Writing Test Cases */ ############################################################################### ## MySQL Test Programs /** @page PAGE_MYSQLTEST_PROGRAMS MySQL Test Programs

Table of Contents

- @subpage PAGE_MYSQLTEST - @subpage PAGE_MYSQL_CLIENT_TEST - @subpage PAGE_MYSQL_TEST_RUN_PL - @subpage PAGE_MYSQL_STRESS_TEST_PL This chapter describes the test programs that run test cases. For information about the language used for writing test cases, see @ref PAGE_MYSQLTEST_LANGUAGE_REFERENCE. The test suite uses the following programs: */ ############################################################################### ## mysqltest — Program to Run Test Cases /** @page PAGE_MYSQLTEST mysqltest — Program to Run Test Cases The @ref PAGE_MYSQLTEST "mysqltest" program runs a test case against a MySQL server and optionally compares the output with a result file. This program reads input written in a special test language. Typically, you invoke @ref PAGE_MYSQLTEST "mysqltest" using @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" rather than invoking it directly. Features of @ref PAGE_MYSQLTEST "mysqltest": - Can send SQL statements to MySQL servers for execution - Can execute external shell commands - Can test whether the result from an SQL statement or shell command is as expected - Can connect to one or more standalone mysqld servers and switch between connections By default, @ref PAGE_MYSQLTEST "mysqltest" reads the test case on the standard input. To run @ref PAGE_MYSQLTEST "mysqltest" this way, you normally invoke it like this: @verbatim shell> mysqltest [options] [db_name] < test_file @endverbatim You can also name the test case file with a `--test- file`=file_name option. The exit value from @ref PAGE_MYSQLTEST "mysqltest" is 0 for success, 1 for failure, and 62 if it skips the test case (for example, if after checking some preconditions it decides not to run the test). @ref PAGE_MYSQLTEST "mysqltest" supports the following options: */ ############################################################################### ## mysql_client_test — Test Client API /** @page PAGE_MYSQL_CLIENT_TEST mysql_client_test — Test Client API The @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" program is used for testing aspects of the MySQL client API that cannot be tested using @ref PAGE_MYSQLTEST "mysqltest" and its test language. @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" is run as part of the test suite. The source code for the programs can be found in in tests/mysql_client_test.c in a source distribution. The program serves as a good source of examples illustrating how to use various features of the client API. @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" is used in a test by the same name in the main tests suite of @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" but may also be run directly. Unlike the other programs listed here, it does not read an external description of what tests to run. Instead, all tests are coded into the program, which is written to cover all aspects of the C language API. @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" supports the following options: */ ############################################################################### ## mysql-test-run.pl — Run MySQL Test Suite /** @page PAGE_MYSQL_TEST_RUN_PL mysql-test-run.pl — Run MySQL Test Suite The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" Perl script is the main application used to run the MySQL test suite. It invokes @ref PAGE_MYSQLTEST "mysqltest" to run individual test cases. Invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" in the mysql-test directory like this: @verbatim shell> mysql-test-run.pl [options] [test_name] ... @endverbatim Each test_name argument names a test case. The test case file that corresponds to the test name is t/test_name.test. For each test_name argument, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs the named test case. With no test_name arguments, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" runs all .test files in the t subdirectory. If no suffix is given for the test name, a suffix of .test is assumed. Any leading path name is ignored. These commands are equivalent: @verbatim shell> mysql-test-run.pl mytest shell> mysql-test-run.pl mytest.test shell> mysql-test-run.pl t/mytest.test @endverbatim A suite name can be given as part of the test name. That is, the syntax for naming a test is: @verbatim [suite_name.]test_name[.suffix] @endverbatim If a suite name is given, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" looks in that suite for the test. The test file corresponding to a test named suite_name.test_name is found in suite/suite_name/t/test_name.test. There is also an implicit suite name main for the tests in the top t directory. With no suite name, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" looks in the default list of suites for a match and runs the test in any suites where it finds the test. Suppose that the default suite list is main, binlog, rpl, and that a test mytest.test exists in the main and rpl suites. With an argument of mytest or mytest.test, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will run mytest.test from the main and rpl suites. To run a family of test cases for which the names share a common prefix, use the `--do-test`=prefix option. For example, `--do-test`=rpl runs the replication tests (test cases that have names beginning with rpl). `--skip-test` has the opposite effect of skipping test cases for which the names share a common prefix. The argument for the `--do-test` and `--skip-test` options also allows more flexible specification of which tests to perform or skip. If the argument contains a pattern metacharacter other than a lone period, it is interpreted as a Perl regular expression and applies to test names that match the pattern. If the argument contains a lone period or does not contain any pattern metacharacters, it is interpreted the same way as previously and matches test names that begin with the argument value. For example, `--do-test`=testa matches tests that begin with testa, `--do-test`=main.testa matches tests in the main test suite that begin with testa, and `--do-test`=main.*testa matches test names that contain main followed by testa with anything in between. In the latter case, the pattern match is not anchored to the beginning of the test name, so it also matches names such as xmainytesta. It is also possible to put a list of test names in a file and have @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" run those tests, using the option `--do-test-list`=file. The tests should be listed one per line in the file. A line beginning with # indicates a comment and is ignored. Similary an empty line in the file is also ignored. @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" also supports a `--do-suite` option, which is similar to `--do-test` but permits specifying entire suites of tests to run. To perform setup prior to running tests, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" needs to invoke mysqld with the `--initialize` and `--skip-grant-tables` options. If MySQL was built with the compiler flag -DDISABLE_GRANT_OPTIONS, then `--initialize`, `--skip-grant-tables`, and `--init-file` will be disabled. To handle this, set the MYSQLD_BOOTSTRAP environment variable to the full path name of a server that has all options enabled. @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will use that server to perform setup; it is not used to run the tests. The init_file test will fail if `--init-file` is disabled. This is an expected failure in this case. To run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" on Windows, you'll need either Cygwin or ActiveState Perl to run it. You may also need to install the modules required by the script. To run the test script, change location into the mysql-test directory, set the MTR_VS_CONFIG environment variable to the configuration you selected earlier (or use the `--vs-config` option), and invoke @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". For example (using Cygwin and the bash shell): @verbatim shell> cd mysql-test shell> export MTR_VS_CONFIG=debug shell> ./mysqltest-run.pl --force --timer shell> ./mysqltest-run.pl --force --timer --ps-protocol @endverbatim @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" uses several environment variables. Some of them are listed in the following table. Some of these are set from the outside and used by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", others are set by @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" instead, and may be referred to in tests.
Variable Description
MTR_BUILD_THREAD If set, defines which port number range is used for the server
MTR_MAX_PARALLEL If set, defines maximum number of parallel threads if `--parallel`=auto is given
MTR_MEM If set to anything, will run tests with files in "memory" using tmpfs or ramdisk. Not available on Windows. Same as `--mem` option
MTR_NAME_TIMEOUT Setting of a timeout in minutes or seconds, corresponding to command line option `--name-timeout`. Avaliable timeout names are TESTCASE, SUITE (both in minutes) and START, SHUTDOWN, CTEST (all in seconds). MTR_CTEST_TIMEOUT is for ctest unit tests.
MTR_PARALLEL If set, defines number of parallel threads executing tests. Same as `--parallel` option
MTR_PORT_BASE If set, defines which port number range is used for the server
MTR_RECORD Set to 1 if a MTR run is started with `--record` option, 0 otherwise.
MTR_UNIQUE_IDS_DIR The method in which the free ports are allocated is by maintaining a list under the unique ids directory. In case there are multiple chroot environments on the same host, then the same set of ports may get allocated for all environments, because there will be multiple unique ids directories in different physical locations, but MTR assumes it is the same directory. This will lead to a conflict while reserving ports for the different MTR runs. To avoid this problem, when using chroot environments, MTR_UNIQUE_IDS_DIR environment variable can be set to a common location, so that all the MTR processes will use the same unique ids directory. Users have to make sure this variable is set to the same path on all environments, and that this common path is mounted on all the environments.
MYSQL_CONFIG_EDITOR Path name to mysql_config_editor binary.
MYSQL_TEST Path name to @ref PAGE_MYSQLTEST "mysqltest" binary
MYSQL_TEST_DIR Full path to the mysql-test directory where tests are being run from
MYSQL_TEST_LOGIN_FILE Path name to login file used by mysql_config_editor. If not set, the default is $HOME/.mylogin.cnf, or \%APPDATA%\\MySQL\\\.mylogin.cnf on Windows.
MYSQL_TMP_DIR Path to temp directory used for temporary files during tests
MYSQLD Full path to server executable used in tests.
MYSQLD_BOOTSTRAP Full path name to mysqld that has all options enabled
MYSQLD_BOOTSTRAP_CMD Full command line used for initial database setup for this test batch
MYSQLD_CMD Command line for starting server as used in tests, with the minimum set of required arguments.
MYSQLTEST_VARDIR Path name to the var directory that is used for logs, temporary files, and so forth
NUMBER_OF_CPUS Defines number of processors.
TSAN_OPTIONS Path name to a file containing ThreadSanitizer suppressions.
The variable MTR_PORT_BASE is a more logical replacement for the original variable MTR_BUILD_THREAD. It gives the actual port number directly (will be rounded down to a multiple of 10). If you use MTR_BUILD_THREAD, the port number is found by multiplying this by 10 and adding 10000. Tests sometimes rely on certain environment variables being defined. For example, certain tests assume that MYSQL_TEST is defined so that @ref PAGE_MYSQLTEST "mysqltest" can invoke itself with exec $MYSQL_TEST. Other tests may refer to the last three variables listed in the preceding table, to locate files to read or write. For example, tests that need to create files will typically put them in $MYSQL_TMP_DIR/file_name. The variable $MYSQLD_CMD will include any server options added with the `--mysqld` option to @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", but will not include server options added specifically for the currently running test. @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" supports the options in the following list. An argument of -- tells @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" not to process any following arguments as options. @note The hostname resolves to 127.0.0.1 and not to the actual IP address. */ ############################################################################### ## mysql-stress-test.pl — Server Stress Test Program /** @page PAGE_MYSQL_STRESS_TEST_PL mysql-stress-test.pl — Server Stress Test Program The @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" Perl script performs stress-testing of the MySQL server. @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" requires version of Perl that has been built with threads support. Invoke @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" like this: @verbatim shell> mysql-stress-test.pl [options] @endverbatim @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" supports the following options: */ ############################################################################### ## mysqltest Language Reference /** @page PAGE_MYSQLTEST_LANGUAGE_REFERENCE mysqltest Language Reference

Table of Contents

- @subpage PAGE_MYSQL_TEST_INPUT_CONVENTIONS - @subpage PAGE_MYSQL_TEST_COMMANDS - @subpage PAGE_MYSQL_TEST_VARIABLES - @subpage PAGE_MYSQL_TEST_FLOW_CONTROL - @subpage PAGE_ERROR_HANDLING This chapter describes the test language implemented by @ref PAGE_MYSQLTEST "mysqltest". The language allows input to contain a mix of comments, commands executed by @ref PAGE_MYSQLTEST "mysqltest" itself, and SQL statements that @ref PAGE_MYSQLTEST "mysqltest" sends to a MySQL server for execution. Terminology notes: ------------------ */ ############################################################################### ## mysqltest Input Conventions /** @page PAGE_MYSQL_TEST_INPUT_CONVENTIONS mysqltest Input Conventions @ref PAGE_MYSQLTEST "mysqltest" reads input lines and processes them as follows: After @ref PAGE_MYSQLTEST "mysqltest" reads a command up to a delimiter and executes it, input reading restarts following the delimiter and any remaining input on the line that contains the delimiter is treated as though it begins on a new line. Consider the following two input lines: @verbatim echo issue a select statement; select 1; echo done issuing the select statement; @endverbatim That input contains two commands and one SQL statement: @verbatim echo issue a SELECT statement SELECT 1; echo done issuing the SELECT statement @endverbatim Similarly, "#" comments can begin on a command line following a delimiter: @verbatim SELECT 'hello'; # select a string value @endverbatim On a multiple-line command, "#" or “`--`” at the beginning of the second or following lines is not special. Thus, the second and third lines of the following variable-assignment command are not taken as comments. Instead, the variable $a is set to a value that contains two linefeed characters: @verbatim let $a= This is a variable # assignment that sets a variable -- to a multiple-line value; @endverbatim `--` commands and normal commands have complementary properties with regard to how @ref PAGE_MYSQLTEST "mysqltest" reads them: - A “`--`” command is terminated by a newline, regardless of how many delimiters it contains. - A normal command (without “`--`”) is terminated by the delimiter (semicolon), no matter how many newlines it contains. @ref PAGE_MYSQLTEST "mysqltest" commands can be written either with a leading “`--`”) or as normal command input (no leading “`--`”). Use the command delimiter only in the latter case. Thus, these two lines are equivalent: @verbatim --sleep 2 sleep 2; @endverbatim The equivalence is true even for the delimiter command. For example, to set the delimiter to “//”, either of these commands work: @verbatim --delimiter // delimiter //; @endverbatim To set the delimiter back to “;”, use either of these commands: @verbatim --delimiter ; delimiter ;// @endverbatim A potential ambiguity occurs because a command line can contain either a @ref PAGE_MYSQLTEST "mysqltest" command or an SQL statement. This has a couple of implications: - No @ref PAGE_MYSQLTEST "mysqltest" command should be the same as any keyword that can begin an SQL statement. - Should extensions to SQL be implemented in the future, it is possible that a new SQL keyword could be impossible for @ref PAGE_MYSQLTEST "mysqltest" to recognize as such if that keyword is already used as a @ref PAGE_MYSQLTEST "mysqltest" command. Any ambiguity can be resolved by using the “`--`” syntax to force interpetation as a @ref PAGE_MYSQLTEST "mysqltest" command, or the query command to force interpretation as SQL. All file paths used in test commands should use forward slash \"/\" as the directory separator as in Unix. They will be automatically converted when needed if the test is run on Windows. We also recommend putting all temporary or auxiliary files made during the test under the directory referred to by $MYSQL_TMP_DIR. Do not put them under fixed full paths like /tmp. This will help ensuring portability of the test, and avoiding conflicts with other programs. $MYSQL_TMP_DIR is equivalent to $MYSQLTEST_VARDIR/tmp if you are not running with parallel test threads, but if you run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with `--parallel`, they will be different. It is therefore best to be consistent and use $MYSQL_TMP_DIR. Commands named disable_X or enable_X, except parsing, reconnect and rpl_parse, can take an optional modifier ONCE. If this is added, the relevant setting is enabled or disabled only for the next command or statement, after which it is reverted to whatever it was before. Note that the word ONCE must be in upper case; this was chosen in order to make it more visible when reading the test script. For example, `--disable_query_log` ONCE will ensure query log is disabled for the next statement, but will not affect whether or not query log is enabled for statements following the next. It is possible to enable/disable more that one property (e.g. both query log and result log) for a single statement using the ONCE modifier. */ ############################################################################### ## mysqltest Commands /** @page PAGE_MYSQL_TEST_COMMANDS mysqltest Commands @ref PAGE_MYSQLTEST "mysqltest" supports the commands described in this section. Command names are not case sensitive. Some examples of command use are given, but you can find many more by searching the test case files in the mysql-test/t directory. */ ############################################################################### ## mysqltest Variables /** @page PAGE_MYSQL_TEST_VARIABLES mysqltest Variables You can define variables and refer to their values. You can also refer to environment variables, and there is a built-in variable that contains the result of the most recent SQL statement. To define a variable, use the let command. Examples: @verbatim let $a= 14; let $b= this is a string; --let $a= 14 --let $b= this is a string @endverbatim The variable name cannot contain whitespace or the “=” character. If a variable has a numeric value, you can increment or decrement the value: @verbatim inc $a; dec $a; --inc $a --dec $a @endverbatim inc and dec are commonly used in while loops to modify the value of a counter variable that controls loop execution. The result from executing a query can be assigned to a variable by enclosing the query within backtick (“`”) characters: @verbatim let $q= `select version()`; @endverbatim References to variables can occur in the echo, eval, exec, and system commands. Variable references are replaced by their values. A nonquery value assigned to a variable in a let command can also refer to variables. Variable references that occur within \`query\` are expanded before the query is sent to the server for execution. You can refer to environment variables. For example, this command displays the value of the $PATH variable from the environment: @verbatim --echo $PATH @endverbatim $mysql_errno is a built-in variable that contains the numeric error returned by the most recent SQL statement sent to the server, or 0 if the statement executed successfully. $mysql_errno has a value of −1 if no statement has yet been sent. $mysql_errname similarly contains the symbolic name of the last error, or an empty string if there was no error. @ref PAGE_MYSQLTEST "mysqltest" first checks @ref PAGE_MYSQLTEST "mysqltest" variables and then environment variables. @ref PAGE_MYSQLTEST "mysqltest" variable names are not case sensitive. Environment variable names are case sensitive. */ ############################################################################### ## mysqltest Flow Control Constructs /** @page PAGE_MYSQL_TEST_FLOW_CONTROL mysqltest Flow Control Constructs The syntax for if block looks like this: @verbatim if (expr) { command list } @endverbatim The syntax for while block looks like this: @verbatim while (expr) { command list } @endverbatim An expression result is true if nonzero, false if zero. If the expression begins with !, the sense of the test is reversed. If the expression is a string that does not begin with a numeric digit (possibly preceeded by a plus or minus sign), it evaluates as true if non-empty. Any white space is ignored in this case, so a string consisting of only white space is false. There is no provision for else with if. For a while loop, make sure that the loop includes some exit condition that eventually occurs. This can be done by writing expr so that it becomes false at some point. The allowable syntax for expr is $var_name, !$var_name, a string or integer, or \`query\`. The expression can also be a simple comparison, where the left hand side must be a variable, and the right hand side can be any type valid for the single expression except the negated variable. The supported operators are ==, !=, <, <=, > and >=. Only the first two may be used if the right hand side does not evaluate to an integer. With ==, strings must match exactly. If you use a string on the right hand side of the comparison, it does not have to be quoted even if it contains spaces. It may optionally be enclosed in single or double quotation marks which will then be stripped off before comparison. This is in contrast to let statements, where quoting is not stripped. Examples of the expression syntax with comparisons (only the header shown): @verbatim while ($counter<5) ... if ($value == 'No such row') ... if ($slave_count != $master_count) ... @endverbatim The opening { (curly brace) must be separated from the preceding ) (right parenthesis) by whitespace, such as a space or a line break. Variable references that occur within \`query\` are expanded before the query is sent to the server for execution. */ ############################################################################### ## Error Handling /** @page PAGE_ERROR_HANDLING Error Handling If an expected error is specified and that error occurs, @ref PAGE_MYSQLTEST "mysqltest" continues reading input. If the command is successful or a different error occurs, @ref PAGE_MYSQLTEST "mysqltest" aborts. If no expected error is specified, @ref PAGE_MYSQLTEST "mysqltest" aborts unless the command is successful. (It is implicit that you expect $mysql_errno to be 0.) By default, @ref PAGE_MYSQLTEST "mysqltest" aborts for certain conditions: You can disable the abort for errors of the first type by using the disable_abort_on_error command. In this case, when errors occur for statements that should succeed, @ref PAGE_MYSQLTEST "mysqltest" continues processing intput. disable_abort_on_error does not cause @ref PAGE_MYSQLTEST "mysqltest" to ignore errors for the other two types, where you explicitly state which error you expect. This behavior is intentional. The rationale is that if you use the error command to specify an expected error, it is assumed that the test is sufficiently well characterized that only the specified error is accceptable. If you do not use the error command, it is assumed that you might not know which error to expect or that it might be difficult to characterize all possible errors that could occur. In this case, disable_abort_on_error is useful for causing @ref PAGE_MYSQLTEST mysqltest to continue processing input. This can be helpful in the following circumstances: */ ############################################################################### ## Creating and Executing Unit Tests /** @page PAGE_UNIT_TESTS Creating and Executing Unit Tests

Table of Contents

- @subpage PAGE_UNIT_TESTS_TAP - @subpage PAGE_UNIT_TESTS_GOOGLE - @subpage PAGE_UNIT_TESTS_MAIN_TEST_RUNS Storage engines and plugins can have unit tests to test their components. The top-level Makefile target test-unit runs all unit tests: It scans the storage engine and plugin directories, recursively, and executes all executable files having a name that ends with -t. The following sections describe MySQL unit testing using TAP and the Google Test framework. */ ############################################################################### ## Unit Testing Using TAP /** @page PAGE_UNIT_TESTS_TAP Unit Testing Using TAP The unit-testing facility based on the Test Anything Protocol (TAP) is mainly used when developing Perl and PHP modules. To write unit tests for C/C++ code, MySQL has developed a library for generating TAP output from C/C++ files. Each unit test is written as a separate source file that is compiled to produce an executable. For the unit test to be recognized as a unit test, the executable file has to be of the format mytext-t. For example, you can create a source file named mytest-t.c that compiles to produce an executable mytest-t. The executable will be found and run when you execute make test or make test-unit in the distribution top-level directory. Example unit tests can be found in the unittest/examples directory of a MySQL source distribution. The code for the MyTAP protocol is located in the unittest/mytap directory. Each unit test file should be stored in a storage engine or plugin directory (storage/engine_name or plugin/plugin_name), or one of its subdirectories. A reasonable convention is to create a unittest subdirectory under the storage engine or plugin directory and create unit test files in unittest. */ ############################################################################### ## Unit Testing Using the Google Test Framework /** @page PAGE_UNIT_TESTS_GOOGLE Unit Testing Using the Google Test Framework The Google Test unit-testing framework is available in MySQL source trees and distributions. Google Test, like MyTAP, provides a unit- testing framework, but Google Test provides richer functionality, such as: - A rich set of predicates - User-defined predicates and assertions - Automatic test registration - Nice error reporting when a predicate fails (with line number, expected and actual values, and additional comments) - Test fixtures, and setup/teardown logic - Death tests - Disabled tests - Test filtering and shuffling Google Test runs on many platforms. Some functionality is missing on some platforms (such as death tests and parameterized tests), so those features should not be used. This section provides notes about using Google Test within the context of MySQL development. For general-purpose information about Google Test, see these resources: - Main Google Test page: http://code.google.com/p/googletest - Primer: http://code.google.com/p/googletest/wiki/GoogleTestPrimer - Advanced guide: http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide - Google presentation: http://docs.google.com/present/view?id=dfsbxvm5_0f5s4pvf9 @section SECTION_INSTALL_GOOGLE_TEST Installing Google Test and Running Unit Tests MySQL sources do not include Google Test. To install it so that you can use it, use one of these approaches: Installing Google Test in individual source trees is the recommended method. The single-instance installation method can be used only if all MySQL builds on a machine take place in the same environment (same operating system, same compiler), for reasons discussed at https://groups.google.com/group/googletestframework/browse_thread/thread/668eff1cebf5309d?pli=1. At configuration time, CMake looks for gtest.h. The build process compiles all Google Test-based unit tests or ignores them, depending on whether gtest.h is found. After the build has completed, to run the tests, change location into the unittest/gunit directory and execute the ctest command. It need not be invoked with any options, but you can run it with the `--help` option to see what options are available. Another way to run the unit tests is with this command: @verbatim make test-unit @endverbatim For internal MySQL testing, PushBuild has been extended to install Google Test as a separate “package”. All trees named mysql-trunk.* are set up to depend on this package. @section SECTION_WRITING_UNIT_TESTS Writing Unit Tests for Google Test The Google Test unit test files are located in the unittest/gunit directory. You can look at these files to see how tests are written. Here are some examples: - sql_list-t.cc: A simple test of some list classes - mdl-t.cc: Some tests of metadata locking (MDL), including testing of lock acquisition from multiple threads - mdl_mytap-t.cc: The same tests as mdl-t.cc, but written for MyTAP, to illustrate some features of Google Test Most MyTAP-based tests are likely easily converted to Google Test. However, there might be low-level tests that absolutely must be run on every platform, and thus require MyTAP. As currently implemented, the Google Test unit-test programs produce TAP output rather than the “plain” alternative. This can be disabled by using the `--disable-tap-output` command-line option when running a test executable. To see what options are available, run a test executable with the `--help` option. */ ############################################################################### ## Unit Tests Added to Main Test Runs /** @page PAGE_UNIT_TESTS_MAIN_TEST_RUNS Unit Tests Added to Main Test Runs @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will also run unit tests at the end of full test runs, when being run from within a build directory. It depends on the unit tests having been built and defined in a file CTestTestfile.cmake in the top level build directory. Those will normally be there after a build using CMake, but will not be in a binary package. The unit tests are run simply by executing ctest with no arguments from the top level build directory. The result will be shown as a single test at the end, named unit_tests which passes if and only if all unit tests pass. A summary of the result will be printed, including the name of any failed unit tests. The set of unit tests will be counted as one test (either passed or failed) in the overall test summary. Unit tests will by default be run only if you have not specified any specific tests or suites on the command line for @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". This can be overridden by setting the environment variable MTR_UNIT_TESTS to 0 or 1. This in turn can be overriden by a command line argument `--unit-tests` or `--nounit-tests`. If the file CTestTestfile.cmake and the ctest command are not both available, unit tests will be silently skipped, unless you have used the command line option `--unit-tests`. */ ############################################################################### ## Plugins for Testing Plugin Services /** @page PAGE_PLUGINS Plugins for Testing Plugin Services MySQL server plugins have access to server “services”, as described in [MySQL Services for Plugins] (https://dev.mysql.com/doc/refman/8.0/en/plugin-services.html). MySQL distributions include plugins that demonstrate how to test plugin service APIs: The source code for the plugins is located in the plugin/test_services directory of MySQL source distributions. The README file in that directory contains instructions for running the test cases available for the test_services and test_services_threaded plugins. @note The test plugins in plugin/test_services are daemon plugins (see [Daemon Plugins] (https://dev.mysql.com/doc/refman/8.0/en/plugin-types.html#daemon-plugin-type)). For an example of a nondaemon service-testing plugin plugin, see the test_security_context.cc file in the plugin/audit_null directory. This file creates an audit plugin for testing the security_context service. Use the following procedure to create a new service-testing plugin based on one of those provided in the plugin/test_services directory. Assume that you want to create a new plugin named test_myservice (or test_myservice_threaded to test in threaded context).
  1. Select a source file to use as a basis for the new plugin:
    • To begin with a bare bones plugin, copy test_framework.cc to test_myservice.cc.
    • To begin with a plugin that already includes code for running tests in unthreaded context, copy test_services.cc to test_myservice.cc.
    • To begin with a plugin that already includes code for running tests in threaded context, copy test_services_threaded.cc to test_myservice_threaded.cc.
  2. There is a plugin descriptor near the end of the new source file. Modify this descriptor appropriately for your plugin. Change the name, author, and descr members that indicate the plugin name and author and provide a description. For example, if you copied test_framework.cc, those members look like this: @verbatim "test_framework", "Horst Hunger", "Test framework", @endverbatim Change them to something like this: @verbatim "test_myservice", "Your Name Here", "Test My Service", @endverbatim
  3. Modify your source file appropriately for the service to be tested:
    • If you copied test_framework.cc, your file has no tests initially and is set up for unthreaded context. In this case, add code to the test_services_plugin_init() function. This code should invoke the service to be tested.
    • If you copied test_services.cc or test_services_threaded.cc, the file contains tests for the my_plugin_log_service service in unthreaded or threaded contexts. Replace or modify those tests with code for your own tests. those tests with code for your own tests.
Compiling your plugin creates a plugin library file, which you should install in the directory named by the [plugin_dir] https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_plugin_dir) system variable. The file base name is the same as that of the source file. The file name suffix differs per platform (for example, .so for Unix and Unix-like systems, .dll for Windows). To install or unintall your plugin at server startup, use the [`--plugin-load`] (https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_plugin-load) or [`--plugin-load-add`] (https://dev.mysql.com/doc/refman/8.0/en/server-options.html#option_mysqld_plugin-load-add) option. For example, you can use these lines in an option file (adjust the file name as necessary): @verbatim [mysqld] plugin-load-add=test_myservice.so @endverbatim To install or uninstall the plugin at runtime, use these statements (adjust the plugin name and file name as necessary): @verbatim INSTALL PLUGIN test_myservice SONAME 'test_myservice.so'; UNINSTALL PLUGIN test_myservice; @endverbatim For addition information about plugin loading, see [Installing and Uninstalling Plugins] (https://dev.mysql.com/doc/refman/8.0/en/server-plugin-loading.html). For information about creating and running test cases for your new plugin, adapt the instructions in the README file in the plugin/test_services directory. Test cases for the test_services and test_services_threaded plugins are located in mysql-test/suite/test_services. */ ###############################################################################