header_md("""Python object primer for Python3 / meta classes""" )
header_md("""Introduction""", nesting=2)
print_md("""
Python is good at creating the illusion of being a simple programming language. Sometimes this illusion fails, like when you have to deal with the import/module system [my attempts to get it](https://github.com/MoserMichael/pythonimportplayground). Another area of complexity is the object system, last week I tried to understand [python enums](https://docs.python.org/3/library/enum.html), it turns that they are built on top of [meta classes](https://github.com/python/cpython/blob/2c56c97f015a7ea81719615ddcf3c745fba5b4f3/Lib/enum.py#L511), So now I have come to realize, that I really don't know much about python and its object system. The purpose of this text is to figure out, how the python object system ticks.
Wait, but where does the __dict__ attribute come from?
The [built-in getattr](https://docs.python.org/3/library/functions.html#getattr) function can return this built-in __dict__ attribute!
Interesting: the python notation object.member_name can mean different things:
1) for built-in attributes it means a call to getattr
2) for object instances (assigned in the __init__ method of the class) it means a call to retrieve the __dict__ attribute, and then a lookup of the variable name in that dictionary.
""")
print_md( """foo_obj.__dict__ and getattr(foo_obj,'__dict__',None) is the same thing! """)
The getattr builtin function has a good part, its return value can be checked for None. This can be used, in order to check if the argument is an object with a __dict__ attribute.
""")
eval_and_quote("""base_obj = object()""")
print_md("An object of built-in type ", type(base_obj), " doesn't have a __dict__ member")
eval_and_quote("""assert getattr(base_obj, '__dict__', None) is None""")
eval_and_quote("""int_obj = 42""")
print_md("An object of built-in type ", type(int_obj), " doesn't have a __dict__ member")
eval_and_quote("""assert getattr(int_obj, '__dict__', None) is None""")
print_md("""
The [dir builtin](https://docs.python.org/3/library/functions.html#dir) function does different things, depending on the argument,
for regular objects it returns a "list that contains the object’s attributes’ names, the names of its class’s attributes, and recursively of the attributes of its class’s base classes.",
header_md("""How classes are represented""", nesting=3)
print_md("""The built-in function [type](https://docs.python.org/3/library/functions.html#type), is returning the class of an object, when applied to a variable (to be more exact: type is a built-in class, and not a built-in function, more on that later)""")
eval_and_quote("""
# Make a new object instance of type Foo class.
foo_obj=Foo()
print("class of object foo_obj - type(foo_obj): ", type(foo_obj))
# That's the same as showing the __class__ member of the variable (in Python3)
print("foo_obj.__class__ :", foo_obj.__class__)
""")
print_md("""
The class is an object, it's purpose is to hold the static data that is shared between all object instances.
Each object has a built-in __class__ attribute, that refers to this class object.
Note that the name of the class includes the module name, __main__ if the class is defined in the file given as argument to the python interpreter.
Also note that the type built-in of type(foo_obj) is really the same as: str(foo_obj.__class__) (for Python3)
""")
print_md("""
Again, the built in attribute __class__ can also be accessed with the getattr built-in function.
""")
eval_and_quote( """
print("foo_obj.__class__ and getattr(foo_obj,'__class__',None) is the same thing!")
The __mro__ member is a list of types that stands for 'method resoultion order', when searching for an instance method, this list is searched in order to resolve the method name.
The Python runtime creates this lists by enumerating all of its base classes recursively, in depth first traversal order. For each class it follows the base classes, from the left ot the right
This list is used to resolve a member function 'member_function' of an object, when you call it via: obj_ref.member_function()
The class object has a __dict__ too - here you will see all the class variables (for Foo these are class_var and class_var2) and class methods (defined with @staticmethod), but also the object methods (with the self parameter)
The object instance holds the __dict__ attribute of the object instance, it's value is a dictionary that holds the object instance members.
The class is an object that is shared between all object instances, and it holds the static data (class variables, class methods)
What happens upon: foo = Foo() ?
Take the type of Foo - the metaclass of Foo, the metaclass both knows how to create an instance of the class Foo, and the object instances.
A metaclass is derived from built-in class 'type', The 'type' constructor with three argument creates a new class object. [see reference](https://docs.python.org/3/library/functions.html#type)
class_obj = Foo
The metaclass is used as a 'callable' - it has a __call__ method, and can therefore be called as if it were a function (see more about callables in the course on [decorators](https://github.com/MoserMichael/python-obj-system/blob/master/decorator.md))
Now this __call__ method creates and initialises the object instance.
The implementation of __call__ now does two steps:
- Class creation is done in the [__new__](https://docs.python.org/3/reference/datamodel.html#object.__new__) method of the metaclass. The __new__ method creates the Foo class, it is called exactly once, upon class declaration (you will see this shortly, in the section on custom meta classes)
- It uses the Foo class and calls its to create and initialise the object (call the __new__ method of the Foo class, in order to create an instance of Foo, then calls the __init__ instance method of the Foo class, on order to initialise it). This all done by the __call__ method of the metaclass.
header_md("""Metaclasses for implementing singleton objects""", nesting=3)
print_md("""
An object can define a different way of creating itself, it can define a custom metaclass, which will do exactly the same object creation steps described in the last section.
Let's examine a custom metaclass for creating singleton objects.
""")
eval_and_quote("""
# metaclass are always derived from the type class.
# the type class has functions to create class objects
# the type class has also a default implementation of the __call__ method, for creating object instances.
class Singleton_metaclass(type):
# invoked to create the class object instance (for holding static data)
# this function is called exactly once, in order to create the class instance!
# all singleton objects of the same class are referring to the same object
assert id(sqrt_root_two_a) == id(sqrt_root_two_b)
""")
header_md("""Passing arguments to metaclasses""", nesting=3)
print_md(""""
Lets extend the previous singleton creating metaclass, so that it can pass parameters to the __init__ method of the object, these parameters are defined together with the metaclass specifier.
""")
eval_and_quote("""
# metaclass are always derived from the type class.
# The type class has functions to create class objects
# The type class has also a default implementation of the __call__ method, for creating object instances.
class Singleton_metaclass_with_args(type):
# invoked to create the class object instance (for holding static data)
# this function is called exactly once, in order to create the class instance!
header_md("""Metaclasses in the Python3 standard library""", nesting=2)
print_md("""
This section lists examples of meta-classes in the python standard library. Looking at the standard library of a language is often quite useful, when learning about the intricacies of a programming language.
""")
header_md("""ABCMeta class""", nesting=3)
print_md("""The purpose of this metaclass is to define abstract base classes (also known as ABC), as defined in [PEP 3119](https://www.python.org/dev/peps/pep-3119/), the documentation for the metaclass [ABCMeta class](https://docs.python.org/3/library/abc.html).
You can define an abstract method in a base class, which must be implemented in a derived class, so that the base class defines a contract that must be implemented by any derived class.""")
eval_and_quote("""
import abc
from six import add_metaclass
@add_metaclass(abc.ABCMeta)
class Shape(object):
@abc.abstractmethod
def log_me(self):
pass
class Line(Shape):
def log_me(self):
print("this is a Line")
class AnotherShape(Shape):
pass
a=Line()
a.log_me()
try:
t=AnotherShape()
except TypeError as err:
print("Failure to provide an implementation is checked upon instantiation! This is a dynamic programming language!!!")
""")
print_md("""The requirement for providing an implementation for a required baseclass method is checked upon object creation.""")
print_md("""A python metaclass imposes a different behavior for builtin function [isinstance](https://docs.python.org/3/library/functions.html#isinstance) and [issubclass](https://docs.python.org/3/library/functions.html#issubclass) Only classes that are [registered](https://docs.python.org/3/library/abc.html#abc.ABCMeta.register) with the metaclass, are reported as being subclasses of the given metaclass. The referenced PEP explains, why this is needed, i didn't quite understand the explanation. Would be helpful if the reader can clarify this issue.""")
header_md("""Enum classes""", nesting=3)
print_md("""Python has support for [enum classes](https://docs.python.org/3/library/enum.html). An enum class lists a set of integer class variables, these variables can then be accessed both by their name, and by their integer value.
An example usage: Note that the class doesn't have a constructor, everything is being taken care of by the baseclass [enum.Enum](https://docs.python.org/3/library/enum.html#enum.Enum) which is making use of a meta-class in he definition of the Enum class [here](https://docs.python.org/3/library/enum.html), this metaclass [EnumMeta source code](https://github.com/python/cpython/blob/f6648e229edf07a1e4897244d7d34989dd9ea647/Lib/enum.py#L161) then creates a behind the scene dictionary, that maps the integer values to their constant names.
The advantage is, that you get an exception, when accessing an undefined constant, or name. There are also more things there, please refer to the linked [documentation](https://docs.python.org/3/library/enum.html)
print("Access by name: Rainbow['GREEN']:", Rainbow['GREEN'])
print("Access by value: Rainbow(4):", Rainbow(4))
# which is the same thing
assert id(Rainbow['GREEN']) == id(Rainbow(4))
""")
header_md("""Conclusion""", nesting=2)
print_md("""
Python meta-classes and decorators are very similar in their capabilities.
Both are tools for [metaprogramming](https://en.wikipedia.org/wiki/Metaprogramming), tools for modifying the program text, and treating and modifying code, as if it were data.
I would argue, that decorators are most often the easiest way of achieving the same goal.
However some things, like hooking the classification of classes and objects (implementing class methods [__instancecheck__ and __subclasscheck__](https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks), can only be done with meta-classes.
I hope, that this course has given you a better understanding, of what is happening under the hood, which would be a good thing.