MySQL :: MySQL 8.0 Reference Manual :: 8.4.4 Internal Temporary Table Use in MySQL

MySQL 8.0 Reference Manual / ... / Internal Temporary Table Use in MySQL

8.4.4 Internal Temporary Table Use in MySQL

In some cases, the server creates internal temporary tables while processing statements. Users have no direct control over when this occurs.

The server creates temporary tables under conditions such as these:

Evaluation of UNION statements, with some exceptions described later.
Evaluation of some views, such those that use the TEMPTABLE algorithm, UNION, or aggregation.
Evaluation of derived tables (see Section 13.2.11.8, “Derived Tables”).
Evaluation of common table expressions (see Section 13.2.13, “WITH Syntax (Common Table Expressions)”).
Tables created for subquery or semi-join materialization (see Section 8.2.2, “Optimizing Subqueries, Derived Tables, View References, and Common Table Expressions”).
Evaluation of statements that contain an ORDER BY clause and a different GROUP BY clause, or for which the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue.
Evaluation of DISTINCT combined with ORDER BY may require a temporary table.
For queries that use the SQL_SMALL_RESULT modifier, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.
To evaluate INSERT ... SELECT statements that select from and insert into the same table, MySQL creates an internal temporary table to hold the rows from the SELECT, then inserts those rows into the target table. See Section 13.2.6.1, “INSERT ... SELECT Syntax”.
Evaluation of multiple-table UPDATE statements.
Evaluation of GROUP_CONCAT() or COUNT(DISTINCT) expressions.
Evaluation of window functions (see Section 12.21, “Window Functions”) uses temporary tables as necessary.

To determine whether a statement requires a temporary table, use EXPLAIN and check the Extra column to see whether it says Using temporary (see Section 8.8.1, “Optimizing Queries with EXPLAIN”). EXPLAIN will not necessarily say Using temporary for derived or materialized temporary tables. For statements that use window functions, EXPLAIN with FORMAT=JSON always provides information about the windowing steps. If the windowing functions use temporary tables, it is indicated for each step.

When the server creates an internal temporary table (either in memory or on disk), it increments the Created_tmp_tables status variable. If the server creates the table on disk (either initially or by converting an in-memory table) it increments the Created_tmp_disk_tables status variable.

Some query conditions prevent the use of an in-memory temporary table, in which case the server uses an on-disk table instead:

Presence of a BLOB or TEXT column in the table. However, the TempTable storage engine, which is the default storage engine for in-memory internal temporary tables in MySQL 8.0, supports binary large object types as of MySQL 8.0.13. See Internal Temporary Table Storage Engine.
Presence of any string column with a maximum length larger than 512 (bytes for binary strings, characters for nonbinary strings) in the SELECT list, if UNION or UNION ALL is used.
The SHOW COLUMNS and DESCRIBE statements use BLOB as the type for some columns, thus the temporary table used for the results is an on-disk table.

The server does not use a temporary table for UNION statements that meet certain qualifications. Instead, it retains from temporary table creation only the data structures necessary to perform result column typecasting. The table is not fully instantiated and no rows are written to or read from it; rows are sent directly to the client. The result is reduced memory and disk requirements, and smaller delay before the first row is sent to the client because the server need not wait until the last query block is executed. EXPLAIN and optimizer trace output reflects this execution strategy: The UNION RESULT query block is not present because that block corresponds to the part that reads from the temporary table.

These conditions qualify a UNION for evaluation without a temporary table:

The union is UNION ALL, not UNION or UNION DISTINCT.
There is no global ORDER BY clause.
The union is not the top-level query block of an {INSERT | REPLACE} ... SELECT ... statement.

Internal Temporary Table Storage Engine

An internal temporary table can be held in memory and processed by the TempTable or MEMORY storage engine, or stored on disk by the InnoDB storage engine.

Storage Engine for In-Memory Internal Temporary Tables

The internal_tmp_mem_storage_engine session variable defines the storage engine for in-memory internal temporary tables. Permitted values are TempTable (the default) and MEMORY.

The TempTable storage engine provides efficient storage for VARCHAR and VARBINARY columns. Storage of other binary large object types is supported as of MySQL 8.0.13. The temptable_max_ram configuration option defines the maximum amount of RAM that can be occupied by the TempTable storage engine before it starts allocating space from disk in the form memory-mapped temporary files (the default) or InnoDB on-disk internal temporary tables. The default temptable_max_ram setting is 1GiB. The temptable_use_mmap variable (introduced in MySQL 8.0.16) controls whether the TempTable storage engine uses memory-mapped files or InnoDB on-disk internal temporary tables when the temptable_max_ram limit is exceeded. The default setting is temptable_use_mmap=ON.

Use of memory-mapped temporary files by the TempTable storage engine as an overflow mechanism for in-memory temporary tables is governed by these rules:

Temporary files are created in the directory defined by the tmpdir variable.
Temporary files are deleted immediately after they are created and opened, and therefore do not remain visible in the tmpdir directory. The space occupied by temporary files is held by the operating system while temporary files are open. The space is reclaimed when temporary files are closed by the TempTable storage engine, or when the mysqld process is shut down.
Data is never moved between RAM and temporary files, within RAM, or between temporary files.
New data is stored in RAM if space becomes available within the limit defined by temptable_max_ram. Otherwise, new data is stored in temporary files.
If space becomes available in RAM after some of the data for a table is written to temporary files, it is possible for the remaining table data to be stored in RAM.

If the TempTable storage engine is configured to use InnoDB on-disk internal temporary tables as the overflow mechanism (temptable_use_mmap=OFF), an in-memory table that exceeds the temptable_max_ram limit is converted to an InnoDB on-disk internal temporary table, and any rows belonging to that table are moved from memory to the InnoDB on-disk internal temporary table. The internal_tmp_disk_storage_engine (removed in MySQL 8.0.16) variable setting has no affect on the TempTable storage engine overflow mechanism.

Consider using InnoDB on-disk internal temporary tables as the TempTable overflow mechanism if the TempTable storage engine often exceeds the memory limit defined by the temptable_max_ram variable and uses excessive space in the temporary directory for memory-mapped files. This may occur due to use of large internal temporary tables or extensive use of internal temporary tables. InnoDB on-disk internal temporary tables are created in session temporary tablespaces, which reside in the data directory by default. For more information, see Section 15.6.3.5, “Temporary Tablespaces”.

The memory/temptable/physical_ram and memory/temptable/physical_disk Performance Schema instruments can be used to monitor TempTable space allocation from memory and disk. memory/temptable/physical_ram reports the amount of allocated RAM. memory/temptable/physical_disk reports the amount of space allocated from disk when memory-mapped files are used as the TempTable overflow mechanism (temptable_use_mmap=ON). If the physical_disk instrument reports a value other than 0 and memory-mapped files are used as the TempTable overflow mechanism, the temptable_max_ram threshold was reached at some point. Data can be queried in Performance Schema memory summary tables such as memory_summary_global_by_event_name. See Section 26.12.16.10, “Memory Summary Tables”.

When using the MEMORY storage engine for in-memory temporary tables, MySQL automatically converts an in-memory temporary table to an on-disk table if it becomes too large. The maximum size for in-memory temporary tables is defined by the tmp_table_size or max_heap_table_size value, whichever is smaller. This differs from MEMORY tables explicitly created with CREATE TABLE. For such tables, only the max_heap_table_size variable determines how large a table can grow, and there is no conversion to on-disk format.

Storage Engine for On-Disk Internal Temporary Tables

Starting with MySQL 8.0.16, the server always uses the InnoDB storage engine for managing internal temporary tables on disk.

In MySQL 8.0.15 and earlier, the internal_tmp_disk_storage_engine variable was used to define the storage engine used for on-disk internal temporary tables. This variable was removed in MySQL 8.0.16, and the storage engine used for this purpose is no longer user-configurable.

In MySQL 8.0.15 and earlier: For common table expressions (CTEs), the storage engine used for on-disk internal temporary tables cannot be MyISAM. If internal_tmp_disk_storage_engine=MYISAM, an error occurs for any attempt to materialize a CTE using an on-disk temporary table.

In MySQL 8.0.15 and earlier: When using internal_tmp_disk_storage_engine=INNODB, queries that generate on-disk internal temporary tables that exceed InnoDB row or column limits return Row size too large or Too many columns errors. The workaround is to set internal_tmp_disk_storage_engine to MYISAM.

Internal Temporary Table Storage Format

When in-memory internal temporary tables are managed by the TempTable storage engine, rows that include VARCHAR columns, VARBINARY columns, or other binary large object type columns (supported as of MySQL 8.0.13) are represented in memory by an array of cells, with each cell containing a NULL flag, the data length, and a data pointer. Column values are placed in consecutive order after the array, in a single region of memory, without padding. Each cell in the array uses 16 bytes of storage. The same storage format applies when the TempTable storage engine exceeds the temptable_max_ram limit and starts allocating space from disk as memory-mapped files or InnoDB on-disk internal temporary tables.

When in-memory internal temporary tables are managed by the MEMORY storage engine, fixed-length row format is used. VARCHAR and VARBINARY column values are padded to the maximum column length, in effect storing them as CHAR and BINARY columns.

Previous to MySQL 8.0.16, on-disk internal temporary tables were managed by the InnoDB or MyISAM storage engine (depending on the internal_tmp_disk_storage_engine setting). Both engines store internal temporary tables using dynamic-width row format. Columns take only as much storage as needed, which reduces disk I/O, space requirements, and processing time compared to on-disk tables that use fixed-length rows. Beginning with MySQL 8.0.16, internal_tmp_disk_storage_engine is not supported, and internal temporary tables on disk are always handled by InnoDB.

When using the MEMORY storage engine, statements can initially create an in-memory internal temporary table and then convert it to an on-disk table if the table becomes too large. In such cases, better performance might be achieved by skipping the conversion and creating the internal temporary table on disk to begin with. The big_tables variable can be used to force disk storage of internal temporary tables.

PREV HOME UP NEXT