FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Extend the EET DML oracle to INSERT support by tlmorgan24 · Pull Request #1358 · sqlancer/sqlancer · GitHub

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .java  (4) All 1 file type selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
82 changes: 75 additions & 7 deletions src/sqlancer/common/gen/EETDMLGenerator.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@
* <p>
* Adapted from the DQE oracle, state is observed with an auxiliary column ({@link EETDMLGenerator#ROW_ID_COLUMN}) which
* uniquely identifies each row. The rows are stamped with identifiers once, before both executions of the statement run
* (each in a rolled-back transaction), so both executions observe the same identifiers regardless of how they are
* produced. The resulting state is compared as a full post-image (each surviving row's identifier and content column
* values), which covers every DML statement: a DELETE removes rows from it, an UPDATE changes values in it.
* (each in a rolled-back transaction), so both executions observe the same identifiers. The resulting state is compared
* as a full post-image (each surviving row's identifier and content column values), which covers any of the three DML
* statements (DELETE, UPDATE, INSERT).
*
* <p>
* Most of these statements are standard SQL, likely common to most DBMSs, so are provided as {@code default} methods.
Expand Down Expand Up @@ -66,6 +66,15 @@ public interface EETDMLGenerator<E extends Expression<C>, T extends AbstractTabl
*/
List<Map.Entry<C, E>> generateSetAssignments();

/**
* Generates a fresh value expression for each content column of the current table, used as an INSERT statement's
* inserted values. The returned expressions are positionally aligned with {@link AbstractTable#getColumns()}, and
* each is transformed by the oracle.
*
* @return one fresh random value expression per content column, in {@link AbstractTable#getColumns()} order
*/
List<E> generateInsertValues();

/**
* Creates a DBMS-specific {@link EETTransformer} backed by this generator, used to rewrite the statement's
* expressions into semantically equivalent ones.
Expand Down Expand Up @@ -106,6 +115,17 @@ public interface EETDMLGenerator<E extends Expression<C>, T extends AbstractTabl
*/
String rowIdColumnType();

/**
* A SQL expression, evaluated once per source row of an {@code INSERT ... SELECT}, that derives the inserted row's
* {@link #ROW_ID_COLUMN} value from the source row's identifier. It must be deterministic (so both the original and
* transformed statements assign the same identifiers), unique per source row, and distinct from every existing
* identifier (so an inserted row never collides with the source row it was derived from in the post-image). DBMS-
* specific because it names a suitable derivation function (e.g. a hash of the source identifier).
*
* @return the SQL expression deriving an inserted row's identifier from the source row's {@link #ROW_ID_COLUMN}
*/
String insertedRowIdExpression();

// --- Standard-SQL statements (override only where the DBMS's dialect differs) ---

/**
Expand Down Expand Up @@ -139,8 +159,9 @@ default String dropRowIdColumnStatement(T table) {
*
* <p>
* This single value-level snapshot is the comparison surface for all DML statements: a DELETE removes rows from it,
* an UPDATE changes column values in it. Row identity alone (which the identifier already captures) would suffice
* for DELETE, but not for UPDATE, where the two runs could touch the same rows yet write different values.
* an UPDATE changes column values in it, an INSERT adds rows to it. Row identity alone (which the identifier
* already captures) would suffice for DELETE, but not for UPDATE, where the two runs could touch the same rows yet
* write different values.
*
* @param table
* the table to snapshot
Expand Down Expand Up @@ -223,8 +244,55 @@ default String updateStatement(T table, List<Map.Entry<C, E>> assignments, E pre
}

/**
* Renders the trailing {@code ORDER BY ... LIMIT n} clause shared by {@link #deleteStatement} and
* {@link #updateStatement}, or the empty string when {@code limit} is null.
* SQL that inserts a new row into {@code table} for each source row (optionally filtered by {@code predicate}),
* setting each content column to its corresponding value in {@code values}, optionally limited to the first
* {@code limit} source rows (see {@link #orderByLimitClause}).
*
* <p>
* The {@code INSERT ... SELECT} form is used rather than {@code INSERT ... VALUES} because the transformed value
* expressions reference the table's columns (the transformer injects column references into its equivalent
* sub-expressions), which are legal in a {@code SELECT} but not in a {@code VALUES} clause. Each inserted row's
* {@link #ROW_ID_COLUMN} is derived from its source row via {@link #insertedRowIdExpression()}, giving it a
* deterministic identifier that is unique and distinct from every existing one, so the two statements' post-images
* align (and inserted rows never collide with their source rows).
*
* @param table
* the table to insert into
* @param values
* one value expression per content column, positionally aligned with {@link AbstractTable#getColumns()};
* each is rendered via {@link #asString}
* @param predicate
* the WHERE predicate filtering the source rows, or {@code null} to insert from every source row;
* rendered via {@link #asString}
* @param orderByColumns
* the columns to order the source rows by before the row-id tiebreaker (may be empty); only used when
* {@code limit} is non-null
* @param limit
* the maximum number of source rows to insert from, or {@code null} for no limit
*
* @return the SQL statement
*/
default String insertStatement(T table, List<E> values, E predicate, List<C> orderByColumns, Integer limit) {
List<String> columnNames = new ArrayList<>();
columnNames.add(ROW_ID_COLUMN);
List<String> selectItems = new ArrayList<>();
selectItems.add(insertedRowIdExpression());
List<C> columns = table.getColumns();
for (int i = 0; i < columns.size(); i++) {
columnNames.add(columns.get(i).getName());
selectItems.add(asString(values.get(i)));
}
String statement = "INSERT INTO " + table.getName() + " (" + String.join(", ", columnNames) + ") SELECT "
+ String.join(", ", selectItems) + " FROM " + table.getName();
if (predicate != null) {
statement += " WHERE " + asString(predicate);
}
return statement + orderByLimitClause(orderByColumns, limit);
}

/**
* Renders the trailing {@code ORDER BY ... LIMIT n} clause shared by {@link #deleteStatement},
* {@link #updateStatement} and {@link #insertStatement}, or the empty string when {@code limit} is null.
*
* <p>
* The rows are ordered by {@code orderByColumns} followed by {@link #ROW_ID_COLUMN} as a tiebreaker. Because the
Expand Down
70 changes: 62 additions & 8 deletions src/sqlancer/common/oracle/EETDMLOracle.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,15 @@
* statements can be compared against the same starting state without permanently modifying the database. The state is
* captured as a full post-image: each surviving row's identifier together with its content column values, ordered by
* the identifier. This single value-level surface covers every DML statement — a DELETE removes rows from it, an UPDATE
* changes values in it (row identity alone would suffice for DELETE, but not for UPDATE, which also transforms the
* written values). Because rolling back a statement requires a transactional storage engine, the DBMS-specific setup
* must ensure only such engines are used while this oracle is active.
* changes values in it, an INSERT adds rows to it (row identity alone would suffice for DELETE, but not for UPDATE,
* which also transforms the written values). Because rolling back a statement requires a transactional storage engine,
* the DBMS-specific setup must ensure only such engines are used while this oracle is active.
*
* <p>
* DELETE and UPDATE are currently supported (one is chosen at random per check). Statement reduction is not yet
* implemented (there is no {@link sqlancer.Reproducer Reproducer}), so the finding is reported without database
* DELETE, UPDATE and INSERT are currently supported (one is chosen at random per check). INSERT uses the
* {@code INSERT ... SELECT} form so its transformed value expressions may reference columns; each inserted row is given
* a deterministic identifier derived from its source row so the two runs' post-images align. Statement reduction is not
* yet implemented (there is no {@link sqlancer.Reproducer Reproducer}), so the finding is reported without database
* reduction.
*
* @param <E>
Expand Down Expand Up @@ -104,9 +106,11 @@ public void check() throws SQLException {
orderByColumns = Randomly.subset(table.getColumns());
}

StatementPair statements = Randomly.getBoolean()
? generateUpdateStatements(table, predicate, transformedPredicate, orderByColumns, limit)
: generateDeleteStatements(table, predicate, transformedPredicate, orderByColumns, limit);
// Generators for the different kinds of statement this oracle supports. One is chosen at random per check
List<DMLStatementGenerator<E, T, C>> statementGenerators = List.of(this::generateDeleteStatements,
this::generateUpdateStatements, this::generateInsertStatements);
StatementPair statements = Randomly.fromList(statementGenerators).generate(table, predicate,
transformedPredicate, orderByColumns, limit);
String originalStatement = statements.original;
String transformedStatement = statements.transformed;
generatedQueryString = originalStatement;
Expand Down Expand Up @@ -137,6 +141,22 @@ public void check() throws SQLException {
}
}

/**
* Generates a DML statement of one kind together with its transformed counterpart. The kinds share this signature
* so the oracle can pick one of them at random per check.
*
* @param <E>
* the DBMS-specific expression class
* @param <T>
* the DBMS-specific table class
* @param <C>
* the DBMS-specific column class
*/
@FunctionalInterface
private interface DMLStatementGenerator<E, T, C> {
StatementPair generate(T table, E predicate, E transformedPredicate, List<C> orderByColumns, Integer limit);
}

/**
* A DML statement and its transformed counterpart, which must leave the database in the same state.
*/
Expand Down Expand Up @@ -201,6 +221,40 @@ private StatementPair generateDeleteStatements(T table, E predicate, E transform
gen.deleteStatement(table, transformedPredicate, orderByColumns, limit));
}

/**
* Generates an {@code INSERT ... SELECT} and its transformed counterpart. Besides the WHERE predicate, which
* filters the source rows and is optional here, INSERT also transforms each inserted value in a scalar context.
*
* <p>
* The ordering and limit cap the source rows the statement reads, so it inserts one row per source row kept.
*
* @param table
* the table being modified
* @param predicate
* the WHERE predicate of the original statement
* @param transformedPredicate
* the transformed WHERE predicate, used by the transformed statement
* @param orderByColumns
* the columns ordering the source rows, empty if the statement is not capped by a limit
* @param limit
* the maximum number of source rows to insert from, or {@code null} for no limit
*
* @return the original statement together with its transformed counterpart
*/
private StatementPair generateInsertStatements(T table, E predicate, E transformedPredicate, List<C> orderByColumns,
Integer limit) {
List<E> values = gen.generateInsertValues();
List<E> transformedValues = new ArrayList<>();
for (E value : values) {
transformedValues.add(transformer.transform(value, false));
}
boolean withPredicate = Randomly.getBoolean();
return new StatementPair(
gen.insertStatement(table, values, withPredicate ? predicate : null, orderByColumns, limit),
gen.insertStatement(table, transformedValues, withPredicate ? transformedPredicate : null,
orderByColumns, limit));
}

/**
* Executes {@code statement} inside a transaction that is always rolled back, and returns the resulting post-image:
* the surviving rows' identifier and content column values, ordered by identifier (the resulting database state). A
Expand Down
16 changes: 16 additions & 0 deletions src/sqlancer/mysql/gen/MySQLExpressionGenerator.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,14 @@ public List<Map.Entry<MySQLColumn, MySQLExpression>> generateSetAssignments() {
return assignments;
}

@Override
public List<MySQLExpression> generateInsertValues() {
// One value per content column, in schema order (aligned with the INSERT column list). As with the normal
// INSERT workload, each value is an arbitrary expression (not type-matched to the column); any resulting

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Ah, I wasn't aware or forgot that the INSERT generator is untyped. Probably, it should be typed, as it would make it much more likely to generate meaningful databases. But, I guess that's for another PR.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Yes, FYI this isn't MySQL-specific, it is a similar story for most other weakly typed DBMSs

// type/range/constraint error is on the oracle's expected-error allow-list.
return columns.stream().map(c -> generateExpression()).collect(Collectors.toList());
}

@Override
public MySQLSelect generateSelect() {
return new MySQLSelect();
Expand Down Expand Up @@ -408,4 +416,12 @@ public String rowIdColumnType() {
// Holds a 36-character UUID string produced by stampRowIdsStatement.
return "VARCHAR(36)";
}

@Override
public String insertedRowIdExpression() {
// The source row's identifier with its dashes removed: deterministic (identical across both runs) and unique
// per
// source row. Fits the identifier column's VARCHAR(36).
return String.format("REPLACE(%s, '-', '')", ROW_ID_COLUMN);
}
}
10 changes: 9 additions & 1 deletion src/sqlancer/mysql/gen/MySQLTableGenerator.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,15 @@ public static List<TableOptions> getRandomTableOptions() {
}

private void appendTableOptions() {
List<TableOptions> tableOptions = TableOptions.getRandomTableOptions();
List<TableOptions> tableOptions = new ArrayList<>(TableOptions.getRandomTableOptions());
// The EET DML oracle rolls back each statement to compare database states, which requires a transactional
// engine. The ENGINE option already forces InnoDB when the oracle is active (see the ENGINE case below), but it
// is only emitted when randomly chosen; otherwise the table would inherit the server's default engine, which is
// not guaranteed transactional. Force the option to always be present so the engine is never left to the
// server default.
if (globalState.usesEETDML() && !tableOptions.contains(TableOptions.ENGINE)) {
tableOptions.add(TableOptions.ENGINE);
}
int i = 0;
for (TableOptions o : tableOptions) {
if (i++ != 0) {
Expand Down
Loading

Back | FazBrowse Home | New Git URL