@shollyman I would like to hear your thoughts on this.

The current cell magic line parser is not aware of Python structures (dicts, lists ...) and can thus incorrectly parse the --params option (example issues: #166 and #108). The default parser breaks these structures on whitespace, and while we stitch these parts back together and operate on the result, this only happens after the parsing has already taken place and bugs have sneaked in.

I took a stab at it and came up with a POC custom parser that successfully parses non-trivial input lines such as the following:

%%bigquery    target_var --use_legacy_sql --project my.project.name --params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2,)}, "answer": -42, 'array': ['aaa bbb', ["nested", 'list'], 'ccc ddd'], '--params': 'tricky --inner-option string literal',} --verbose  

The following test snippets demonstrates that the parser correctly recognized the options:

from google.cloud.bigquery.ipython_magics import line_arg_parser as lap

example_line = (
    "  target_var "
    "--use_legacy_sql "
    "--project my.project.name "
    "--params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2,)}, "
    "\"answer\": -42, 'array': ['aaa bbb', [\"nested\", 'list'], 'ccc ddd'], "
    "'--params': 'tricky --inner-option string literal',} "
    "--verbose  "
)

lexer = lap.lexer.Lexer(line)
parser = lap.parser.Parser(lexer)
parse_tree = parser.input_line()

printer = lap.visitors.TreePrinter()
printer.visit(parse_tree)

Output:

Input:
    destination_var:    target_var
    Command options:
        --use_legacy_sql 
        --project my.project.name
        --params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2)}, "answer": -42, 'array': ['aaa bbb', ["nested", 'list'], 'ccc ddd'], '--params': 'tricky --inner-option string literal'}
        --verbose 

The custom parser could be used to extract out the --params <...> part, evaluate it separately, and only feed the remainder of the input to the default parser.

Do you think we should pursue this further?

The grammar used is of course a simplification that is aimed to be "good enough" to reliably extract the --params part in typical uses cases, but it does not try to be bullet-proof. For instance, it currently does not support non-string dict keys, sets, floats in exponential notation, etc., although that can all be added depending on how much we want to complicate. :)

cc: @tswast Do customers complain about the default parser's limitations often? Do you think there even exist reasonable use cases where --params`dictionary would be incredibly complex?

Do customers complain about the default parser's limitations often?

I wouldn't say often, but it has come up more than once