Execute Dynamically Generated Code – Real Python

Attending to Know Python’s exec()

Python’s built-in exec() perform means that you can execute any piece of Python code. With this perform, you’ll be able to execute dynamically generated code. That’s the code that you simply learn, auto-generate, or get hold of throughout your program’s execution. Usually, it’s a string.

The exec() perform takes a bit of code and executes it as your Python interpreter would. Python’s exec() is like eval() however much more highly effective and susceptible to safety points. Whereas eval() can solely consider expressions, exec() can execute sequences of statements, in addition to imports, perform calls and definitions, class definitions and instantiations, and extra. Basically, exec() can execute a whole absolutely featured Python program.

The signature of exec() has the next kind:

exec(code [, globals [, locals]])

The perform executes code, which will be both a string containing legitimate Python code or a compiled code object.

If code is a string, then it’s parsed as a set of Python statements, which is then internally compiled into bytecode, and eventually executed, except a syntax error happens through the parsing or compilation step. If code holds a compiled code object, then it’s executed straight, making the method a bit extra environment friendly.

The globals and locals arguments permit you to present dictionaries representing the global and local namespaces during which exec() will run the goal code.

The exec() perform’s return worth is None, most likely as a result of not each piece of code has a closing, distinctive, and concrete outcome. It might simply have some side effects. This habits notably differs from eval(), which returns the results of the evaluated expression.

To get an preliminary feeling of how exec() works, you’ll be able to create a rudimentary Python interpreter with two strains of code:

>>> whereas True:
...     exec(enter("->> "))
...

->> print("Hiya, World!")
Hiya, World!

->> import this
The Zen of Python, by Tim Peters

Lovely is healthier than ugly.
Express is healthier than implicit.
Easy is healthier than advanced.
    ...

->> x = 10
->> if 1 <= x <= 10: print(f"x is between 1 and 10")
10 is between 1 and 10

On this instance, you employ an infinite while loop to imitate the habits of a Python interpreter or REPL. Contained in the loop, you employ input() to get the consumer’s enter on the command line. Then you definitely use exec() to course of and run the enter.

This instance showcases what’s arguably the principle use case of exec(): executing code that involves you as a string.

You’ll generally use exec() when you might want to dynamically run code that comes as a string. For instance, you’ll be able to write a program that generates strings containing legitimate Python code. You possibly can construct these strings from components that you simply get hold of at totally different moments in your program’s execution. You may also use the consumer’s enter or another enter supply to assemble these strings.

When you’ve constructed the goal code as strings, then you should utilize exec() to execute them as you’d execute any Python code.

On this state of affairs, you’ll be able to not often make certain of what your strings will include. That’s one purpose why exec() implies critical safety dangers. That is notably true in case you’re utilizing untrusted enter sources, like a consumer’s direct enter, in constructing your code.

In programming, a perform like exec() is an extremely highly effective instrument as a result of it means that you can write packages that generate and execute new code dynamically. To generate this new code, your packages will use data obtainable at runtime solely. To run the code, your packages will use exec().

Nonetheless, with nice energy comes nice accountability. The exec() perform implies critical security risks, as you’ll study quickly. So, you must keep away from utilizing exec() more often than not.

Within the following sections, you’ll learn the way exec() works and how you can use this perform for executing code that comes as strings or as compiled code objects.

Working Code From a String Enter

The commonest solution to name exec() is with code that comes from a string-based enter. To construct this string-based enter, you should utilize:

• Single strains of code or one-liner code snippets
• A number of strains of code separated by semicolons
• A number of strains of code separated by newline characters
• A number of strains of code inside triple-quoted strings and with correct indentation

A one-liner program consists of a single line of code that performs a number of actions directly. Say that you’ve got a sequence of numbers, and also you need to construct a brand new sequence containing the sum of squares of all of the even numbers in an enter sequence.

To unravel this drawback, you should utilize the next one-liner code:

>>> numbers = [2, 3, 7, 4, 8]

>>> sum(quantity**2 for quantity in numbers if quantity % 2 == 0)
84

Within the highlighted line, you employ a generator expression to compute the sq. worth of all of the even numbers within the enter sequence of values. Then you definitely use sum() to compute the overall sum of squares.

To run this code with exec(), you simply want to rework your one-liner code right into a single-line string:

>>> exec("outcome = sum(quantity**2 for quantity in numbers if quantity % 2 == 0)")
>>> outcome
84

On this instance, you categorical the one-liner code as a string. Then you definitely feed this string into exec() for execution. The one distinction between your unique code and the string is that the latter shops the computation lead to a variable for later entry. Keep in mind that exec() returns None fairly than a concrete execution outcome. Why? As a result of not each piece of code has a closing distinctive outcome.

Python means that you can write a number of statements in a single line of code, utilizing semicolons to separate them. Regardless that this apply is discouraged, nothing will cease you from doing one thing like this:

>>> title = enter("Your title: "); print(f"Hiya, title!")
Your title: Leodanis
Hiya, Leodanis!

You should utilize semicolons to separate a number of statements and construct a single-line string that serves as an argument to exec(). Right here’s how:

>>> exec("title = enter('Your title: '); print(f'Hiya, title!')")
Your title: Leodanis
Hiya, Leodanis!

The thought of this instance is that you may mix a number of Python statements right into a single-line string by utilizing semicolons to separate them. Within the instance, the primary assertion takes the consumer’s enter, whereas the second assertion prints a greeting message to the display screen.

You may also mixture a number of statements in a single-line string utilizing the newline character, n:

>>> exec("title = enter('Your title: ')nprint(f'Hiya, title!')")
Your title: Leodanis
Hiya, Leodanis!

The newline character makes exec() perceive your single-line string as a multiline set of Python statements. Then exec() runs the aggregated statements in a row, which works like a multiline code file.

The ultimate method to constructing a string-based enter for feeding exec() is to make use of triple-quoted strings. This method is arguably extra versatile and means that you can generate string-based enter that appears and works like regular Python code.

It’s vital to notice that this method requires you to make use of correct indentation and code formatting. Think about the next instance:

>>> code = """
... numbers = [2, 3, 7, 4, 8]
...
... def is_even(quantity):
...     return quantity % 2 == 0
...
... even_numbers = [number for number in numbers if is_even(number)]
...
... squares = [number**2 for number in even_numbers]
...
... outcome = sum(squares)
...
... print("Unique knowledge:", numbers)
... print("Even numbers:", even_numbers)
... print("Sq. values:", squares)
... print("Sum of squares:", outcome)
... """

>>> exec(code)
Unique knowledge: [2, 3, 7, 4, 8]
Even numbers: [2, 4, 8]
Sq. values: [4, 16, 64]
Sum of squares: 84

On this instance, you employ a triple-quoted string to offer the enter to exec(). Be aware that this string seems to be like every common piece of Python code. It makes use of applicable indentation, naming fashion, and formatting. The exec() perform will perceive and execute this string as an everyday Python code file.

It’s best to word that once you move a string with code to exec(), the perform will parse and compile the goal code into Python bytecode. In all instances, the enter string ought to include legitimate Python code.

If exec() finds any invalid syntax through the parsing and compilation steps, then the enter code received’t run:

>>> exec("print('Hiya, World!)")
Traceback (most up-to-date name final):
  File "<stdin>", line 1, in <module>
    File "<string>", line 1
      print('Hiya, World!)
            ^
SyntaxError: unterminated string literal (detected at line 1)

On this instance, the goal code incorporates a name to print() that takes a string as an argument. This string isn’t correctly ended with a closing single citation mark, so exec() raises a SyntaxError mentioning the difficulty and doesn’t run the enter code. Be aware that Python pinpoints the error initially of the string fairly than on the finish, the place the closing single citation mark ought to go.

Working code that comes as a string, such as you did within the instance above, is arguably the pure approach of utilizing exec(). Nonetheless, if you might want to run the enter code many occasions, then utilizing a string as an argument will make the perform run the parsing and compilation steps each time. This habits could make your code inefficient when it comes to execution velocity.

On this state of affairs, probably the most handy method is to compile the goal code beforehand after which run the ensuing bytecode with exec() as many occasions as wanted. Within the following part, you’ll learn to use exec() with compiled code objects.

Executing Compiled Code

In apply, exec() will be fairly gradual once you use it to course of strings containing code. When you ever have to dynamically run a given piece of code greater than as soon as, then compiling it beforehand would be the most performant and really useful method. Why? Since you’ll be operating the parsing and compilation steps solely as soon as after which reusing the compiled code.

To compile a bit of Python code, you should utilize compile(). This built-in perform takes a string as an argument and runs a one-time bytecode compilation on it, producing a code object that you may then move to exec() for execution.

The signature of compile() has the next kind:

compile(supply, filename, mode, flags=0, dont_inherit=False, optimize=-1)

On this tutorial, you’ll solely use the three first arguments to compile(). The supply argument holds the code that you might want to compile into bytecode. The filename argument will maintain the file from which the code was learn. To learn from a string object, you’ll need to set filename to the "<string>" worth.

Lastly, compile() can generate code objects that you may execute utilizing both exec() or eval(), relying on the mode argument’s worth. This argument must be set to "exec" or "eval", relying on the goal execution perform:

>>> string_input = """
... def sum_of_even_squares(numbers):
...     return sum(quantity**2 for quantity in numbers if quantity % 2 == 0)
...
... print(sum_of_even_squares(numbers))
... """

>>> compiled_code = compile(string_input, "<string>", "exec")
>>> exec(compiled_code)

>>> numbers = [2, 3, 7, 4, 8]
>>> exec(compiled_code)
84

>>> numbers = [5, 3, 9, 6, 1]
>>> exec(compiled_code)
36

Compiling often-repeated code up entrance with compile() can assist you barely enhance your code’s efficiency by skipping the parsing and bytecode compilation steps on every name to exec().

Working Code From Python Supply Information

You may also use exec() to run code that you simply’ve learn from a dependable .py file in your file system or someplace else. To do that, you should utilize the built-in open() perform to learn the file’s content material as a string, which you’ll be able to then move as an argument to exec().

For instance, say that you’ve got a Python file named hey.py containing the next code:

# hey.py

print("Hiya, Pythonista!")
print("Welcome to Actual Python!")

def greet(title="World"):
    print(f"Hiya, title!")

This pattern script prints a greeting and a welcome message to the display screen. It additionally defines a pattern greet() perform for testing functions. The perform takes a reputation as an argument and prints a personalized greeting to the display screen.

Now get again to your Python interactive session and run the next code:

>>> with open("hey.py", mode="r", encoding="utf-8") as hey:
...     code = hey.learn()
...

>>> exec(code)
Hiya, Pythonista!
Welcome to Actual Python!

>>> greet()
Hiya, World!

>>> greet("Pythonista")
Hiya, Pythonista!

On this instance, you first open the goal .py file as an everyday textual content file utilizing the built-in open() perform in a with statement. Then you definitely name .read() on the file object to learn the file’s content material into the code variable. This name to .learn() returns the file’s content material as a string. The ultimate step is to name exec() with this string as an argument.

This instance runs the code and makes the greet() perform and objects that stay in hey.py obtainable in your present namespace. That’s why you should utilize greet() straight. The key behind this habits has to do with the globals and locals arguments, which you’ll study within the subsequent part.

Utilizing the approach within the above instance, you’ll be able to open, learn, and execute any file containing Python code. This system may fit once you don’t know up entrance which supply information you’ll be operating. So, you’ll be able to’t write import module, since you don’t know the module’s title once you’re writing the code.

When you ever select to make use of this system, then just be sure you solely execute code from trusted supply information. Ideally, probably the most dependable supply information are people who you have consciously created to run dynamically. You need to by no means run code information that come from exterior sources, together with your customers, with out inspecting the code first.

Utilizing the globals and locals Arguments

You possibly can move an execution context to exec() utilizing the globals and locals arguments. These arguments can settle for dictionary objects that’ll work as the worldwide and native namespaces that exec() will use to run the goal code.

These arguments are non-compulsory. When you omit them, then exec() will execute the enter code within the present scope, and all of the names and objects on this scope can be obtainable to exec(). Likewise, all of the names and objects that you simply outline within the enter code can be obtainable within the present scope after the decision to exec().

Think about the next instance:

>>> code = """
... z = x + y
... """

>>> # World names are accessible from exec()
>>> x = 42
>>> y = 21

>>> z
Traceback (most up-to-date name final):
    ...
NameError: title 'z' will not be outlined

>>> exec(code)
>>> # Names in code can be found within the present scope
>>> z
63

This instance reveals that in case you name exec() with out offering particular values to the globals and locals arguments, then the perform runs the enter code within the present scope. On this case, the present scope is the worldwide one.

Be aware that after you name exec(), the names outlined within the enter code are additionally obtainable within the present scope. That’s why you’ll be able to entry z within the closing line of code.

When you solely present a worth to globals, then that worth have to be a dictionary. The exec() perform will use this dictionary for each world and native names. This habits will prohibit entry to most names within the present scope:

>>> code = """
... z = x + y
... """

>>> x = 42
>>> y = 21

>>> exec(code, "x": x)
Traceback (most up-to-date name final):
    ...
NameError: title 'y' will not be outlined

>>> exec(code, "x": x, "y": y)

>>> z
Traceback (most up-to-date name final):
    ...
NameError: title 'z' will not be outlined

Within the first name to exec(), you employ a dictionary because the globals argument. As a result of your dictionary doesn’t present a key holding the y title, the decision to exec() doesn’t have entry to this title and raises a NameError exception.

Within the second name to exec(), you present a distinct dictionary to globals. On this case, the dictionary incorporates each variables, x and y, which permits the perform to work accurately. Nonetheless, this time you don’t have entry to z after the decision to exec(). Why? Since you’re utilizing a customized dictionary to offer an execution scope to exec() fairly than falling again to your present scope.

When you name exec() with a globals dictionary that doesn’t include the __builtins__ key explicitly, then Python will robotically insert a reference to the built-in scope or namespace underneath that key. So, all of the built-in objects can be accessible out of your goal code:

>>> code = """
... print(__builtins__)
... """

>>> exec(code, )
'__name__': 'builtins', '__doc__': "Constructed-in features, ...

On this instance, you’ve supplied an empty dictionary to the globals argument. Be aware that exec() nonetheless has entry to the built-in namespace as a result of this namespace is robotically inserted into the supplied dictionary underneath the __builtins__ key.

When you present a worth for the locals argument, then it may be any mapping object. This mapping object will maintain the native namespace when exec() is operating your goal code:

>>> code = """
... z = x + y
... print(f"z=")
... """

>>> x = 42  # World title

>>> def func():
...     y = 21  # Native title
...     exec(code, "x": x, "y": y)
...

>>> func()
z=63

>>> z
Traceback (most up-to-date name final):
    ...
NameError: title 'z' will not be outlined

The decision to exec() is embedded in a perform on this instance. Due to this fact, you will have a world (module-level) scope and a neighborhood (function-level) scope. The globals argument offers the x title from the worldwide scope, and the locals argument offers the y title from the native scope.

Be aware that after operating func(), you don’t have entry to z as a result of this title was created underneath the native scope of exec(), which isn’t obtainable from the surface.

With the globals and locals arguments, you’ll be able to tweak the context during which exec() runs your code. These arguments are fairly useful with regards to minimizing the safety dangers related to exec(), however you must nonetheless just be sure you’re operating code from trusted sources solely. Within the following part, you’ll study these safety dangers and how you can cope with them.

Uncovering and Minimizing the Safety Dangers Behind exec()

As you’ve discovered to this point, exec() is a robust instrument that means that you can execute arbitrary code that involves you as strings. It’s best to use exec() with excessive care and warning due to its capacity to run any piece of code.

Sometimes, the code that feeds exec() is dynamically generated at runtime. This code could have many sources of enter, which may embrace your program consumer, different packages, a database, a stream of knowledge, and a community connection, amongst others.

On this state of affairs, you’ll be able to’t be fully positive what the enter string will include. So, the likelihood of going through an untrusted and malicious supply of enter code is fairly excessive.

The safety points related to exec() are the most typical purpose why many Python builders advocate avoiding this perform altogether. Discovering a greater, sooner, extra strong, and safer resolution is sort of at all times attainable.

Nonetheless, in case you should use exec() in your code, then the widely really useful method is to make use of it with specific globals and locals dictionaries.

One other vital subject with exec() is that it breaks a elementary assumption in programming: the code that you simply’re at present studying or writing is the code that you simply’ll be executing once you fireplace up your program. How does exec() break this assumption? It makes your packages run new and unknown code that’s dynamically generated. This new code will be laborious to observe, keep, and even management.

Within the following sections, you’ll dive into a couple of suggestions, methods, and practices that you must apply in case you ever want to make use of exec() in your code.

>>> whereas True:
...     exec(enter("->> "))
...
->> print("Hiya, World!")
Hiya, World!

>>> exec("import os", "__builtins__": , )
Traceback (most up-to-date name final):
    ...
ImportError: __import__ not discovered

>>> x = 42
>>> y = 21

>>> exec("print(x + y)", , )
Traceback (most up-to-date name final):
    ...
NameError: title 'x' will not be outlined

>>> exec("print(min([2, 3, 7, 4, 8]))", , )
2

>>> exec("print(len([2, 3, 7, 4, 8]))", , )
5

>>> exec("print(min([2, 3, 7, 4, 8]))", "__builtins__": , )
Traceback (most up-to-date name final):
    ...
NameError: title 'print' will not be outlined

>>> allowed_builtins = "__builtins__": "min": min, "print": print
>>> exec("print(min([2, 3, 7, 4, 8]))", allowed_builtins, )
2

>>> exec("print(len([2, 3, 7, 4, 8]))", allowed_builtins, )
Traceback (most up-to-date name final):
    ...
NameError: title 'len' will not be outlined

Placing exec() Into Motion

Up up to now, you’ve discovered how the built-in exec() perform works. You recognize that you should utilize exec() to run both string-based or compiled-code enter. You additionally discovered that this perform can take two non-compulsory arguments, globals and locals, which let you tweak the execution namespace for exec().

Moreover, you’ve discovered that utilizing exec() implies some critical safety points, together with permitting customers to run arbitrary Python code in your pc. You studied some really useful coding practices that assist decrease the safety dangers related to exec() in your code.

Within the following sections, you’ll code a couple of sensible examples that’ll assist you to spot these use instances during which utilizing exec() will be applicable.

>>> from timeit import timeit

>>> timeit("'-'.be a part of(str(n) for n in vary(100))", quantity=10000)
0.1282792080000945

# timeit.py

# ...

class Timer:
    """Class for timing execution velocity of small code snippets."""

    def __init__(
        self,
        stmt="move",
        setup="move",
        timer=default_timer,
        globals=None
    ):
        """Constructor.  See class doc string."""
        self.timer = timer
        local_ns = 
        global_ns = _globals() if globals is None else globals
        # ...
        src = template.format(stmt=stmt, setup=setup, init=init)
        self.src = src  # Save for traceback show
        code = compile(src, dummy_src_name, "exec")
        exec(code, global_ns, local_ns)
        self.inside = local_ns["inner"]

    # ...

# calculations.py

def add(a, b):
    """Return the sum of two numbers.

    Assessments:
        >>> add(5, 6)
        11
        >>> add(2.3, 5.4)
        7.7
        >>> add("2", 3)
        Traceback (most up-to-date name final):
        TypeError: numeric kind anticipated for "a" and "b"
    """
    if not (isinstance(a, (int, float)) and isinstance(b, (int, float))):
        increase TypeError('numeric kind anticipated for "a" and "b"')
    return a + b

# ...

$ python -m doctest calculations.py

# doctest.py

class DocTestRunner:
    # ...

    def __run(self, take a look at, compileflags, out):
        # ...
        strive:
            # Do not blink!  That is the place the consumer's code will get run.
            exec(
                compile(instance.supply, filename, "single", compileflags, True),
                take a look at.globs
            )
            self.debugger.set_continue() # ==== Instance Completed ====
            exception = None
        besides KeyboardInterrupt:
        # ...

# calculations.py

def add(a, b):
    """Return the sum of two numbers.

    Assessments:
        >>> import os; os.system("ls -l")
        0
    """
    if not (isinstance(a, (int, float)) and isinstance(b, (int, float))):
        increase TypeError('numeric kind anticipated for "a" and "b"')
    return a + b

# settings.conf

font_face = ""
font_size = 10
line_numbers = True
tab_size = 4
auto_indent = True

>>> from pathlib import Path

>>> def load_config(config_file):
...     config_file = Path(config_file)
...     code = compile(config_file.read_text(), config_file.title, "exec")
...     config_dict = 
...     exec(code, "__builtins__": , config_dict)
...     return config_dict
...

>>> load_config("settings.conf")

    'font_face': '',
    'font_size': 10,
    'line_numbers': True,
    'tab_size': 4,
    'auto_indent': True

Conclusion

You’ve discovered how you can use the built-in exec() perform to execute Python code from a string or bytecode enter. This perform offers a fast instrument for executing dynamically generated Python code. You additionally discovered how you can decrease the safety dangers related to exec() and when it’s okay to make use of the perform in your code.

On this tutorial, you’ve discovered how you can:

• Work with Python’s built-in exec() perform
• Use Python’s exec() to run string-based and compiled-code enter
• Assess and decrease the safety dangers related to utilizing exec()

Moreover, you’ve coded a couple of sensible examples that helped you higher perceive when and how you can use exec() in your Python code.