In Python every class can have instance attributes. By default Pythonuses a dict to store an object's instance attributes. This is reallyhelpful as it allows setting arbitrary new attributes at runtime.
However, for small classes with known attributes it might be abottleneck. The dict wastes a lot of RAM. Python can't just allocatea static amount of memory at object creation to store all theattributes. Therefore it sucks a lot of RAM if you create a lot ofobjects (I am talking in thousands and millions). Still there is a wayto circumvent this issue. It involves the usage of __slots__ totell Python not to use a dict, and only allocate space for a fixed setof attributes. Here is an example with and without __slots__:
Without__slots__:
You need to allow unlimited read access to a resource when it is not being modified while keeping write access exclusive.
- There are different way to prevent setting attributes and make attributes read only on object in python. We can use any one of the following way to make attributes readonly Property Descriptor Using descriptor methods get and set Using slots (only restricts setting arbitary attributes) Property Descriptor Python ships with built in function called property.
- In Python every class can have instance attributes. By default Python uses a dict to store an object's instance attributes. This is really helpful as it allows setting arbitrary new attributes at runtime. However, for small classes with known attributes it might be a bottleneck. The dict wastes a lot of RAM. Python can't just allocate a.
With__slots__:
The second piece of code will reduce the burden on your RAM. Some peoplehave seen almost 40 to 50% reduction in RAM usage by using thistechnique.
On a sidenote, you might want to give PyPy a try. It does all of theseoptimizations by default.
Below you can see an example showing exact memory usage with and without __slots__ done in IPython thanks to https://github.com/ianozsvald/ipython_memory_usage
Python Metaprogramming - Properties on Steroids
Metaprogramming is an advanced topic, it requires a good understanding of the language itself. I assume you already know Python well enough even though I include references to complementary resources. If you are interested only in the final result, there is a link to the code at the end of the article.
Python is a very dynamic language, everything is an object, objects are created by classes (usually) but classes can also be created by otherclases or functions (this is amazing). Objects, once created, can be modified (add/remove/replace properties, methods, attributes, etc…) and thismeans that you can do a lot of metaprogramming in python. Metaprogramming is a key that can open the doors to heaven or hell.
Python Slots Read Only Version
Ok but what is metaprogramming? here is a definition from wikipedia:
Metaprogramming is a programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running.
-- Harald Sondergaard. 'Course on Program Analysis and Transformation'. Retrieved 18 September 2014.
-- Czarnecki, Krzysztof; Eisenecker, Ulrich W. (2000). Generative Programming. ISBN 0-201-30977-7.
In this article, I will use metaprogramming to change how properties are defined in a class, how they can be documented, initialized, how to set a default value, how to make them read-only and observable, and as a bonus, I will improve memory usage of the objects created by the class. And as a second bonus, I will seal the object against attribute injections. I call this 'Properties on Steroids'.
Requirements
- A property must be defined in a succinct and pythonic way.
- A property definition can setup a default value.
- A property must support docstring (
__doc__). - A Property can have a type hint.
- A Property can be read-only or read-and-write.
- A Property can be observable.
- A Property must not use more memory than traditional python property (@property)
- Bonus: Field storage can be optimized
- Bonus2: Objects created from the class must be protected from field injection.
- DO NOT USE A METACLASS
Important concepts used in the solution
Classes are created at runtime
In python, classes are created at runtime, so the code inside the class scope can modify the resulting class by adding/removing/replacing things in the class scope (local).
For more info about classes in Python: https://docs.python.org/3/tutorial/classes.html
Scopes
There are two scopes in python: local and global. Scopes are symbol tables of what is reachable from the current point in code. you can access the symbol table using the builtin functions locals() and globals(). The important part here is that you can modify the scope just adding/replacing/removing things in the symbol table.
For example, if you want to define a variable in the local scope dynamically:
For more info about scopes in python: https://realpython.com/python-scope-legb-rule/
Decorators
In python, functions are objects and can be passed to other functions like any other object. The idea of a decorator is a function that receives another function and returns a new function based on the original. There is a special syntax in python to call a decorator function just on function definition and effectively replacing the original function with the one returned by the decorator.
Example:
The above code will print:
For more info about decorators: https://realpython.com/primer-on-python-decorators/
Properties
Python has a special class called properties, it allows us to create getter/setter/deleter for a field. In combination with decorators you can define functional properties.
We will change this pattern to add more features like observability, and automatic usage of slots.
For more info about properties: https://docs.python.org/3/howto/descriptor.html#properties
Slots
Objects in Python do store attributes in an internal dictionary called __dict__, it allows dynamic creation of attributes in any object but uses additional memory for the dict object itself. If you want that your object do not support dynamic attribute creation, you can remove the __dict__ mechanism and use object slots with the additional benefit of memory savings.
I will not explain slots here, but you can find detailed info in the following resources:
Context Managers
Context managers are objects that can execute code at the beginning and at the end of a code block. they are used with the with statement.
For more info about context managers: https://docs.python.org/3/library/contextlib.html
Proposed Solution
Ok, now with all the tools in the bag, we can create our own monster.
The final goal:
Result:
The traditional equivalent code:
The differences
initially it appears to have no major advantajes, but the classes Car and CarTraditional and the objects mycar and mycar2 are very different.
Field injection
With Metaprog:
With traditional code:
Default values
With Metaprog:
Default values are specified at property definition.
With traditional code:
Default values are defined by assignment in constructor
Observability
With Metaprog:
Multiple attributes can be observed with the same listener, listener specified on property definition, listener is called only if new value is different from current.
With traditional code:You must implement observability by your own.
Read-Only / Read-Write
With Metaprog:
One single definition will create readonly or read-write property.
With traditional code:
In Python every class can have instance attributes. By default Pythonuses a dict to store an object's instance attributes. This is reallyhelpful as it allows setting arbitrary new attributes at runtime.
However, for small classes with known attributes it might be abottleneck. The dict wastes a lot of RAM. Python can't just allocatea static amount of memory at object creation to store all theattributes. Therefore it sucks a lot of RAM if you create a lot ofobjects (I am talking in thousands and millions). Still there is a wayto circumvent this issue. It involves the usage of __slots__ totell Python not to use a dict, and only allocate space for a fixed setof attributes. Here is an example with and without __slots__:
Without__slots__:
You need to allow unlimited read access to a resource when it is not being modified while keeping write access exclusive.
- There are different way to prevent setting attributes and make attributes read only on object in python. We can use any one of the following way to make attributes readonly Property Descriptor Using descriptor methods get and set Using slots (only restricts setting arbitary attributes) Property Descriptor Python ships with built in function called property.
- In Python every class can have instance attributes. By default Python uses a dict to store an object's instance attributes. This is really helpful as it allows setting arbitrary new attributes at runtime. However, for small classes with known attributes it might be a bottleneck. The dict wastes a lot of RAM. Python can't just allocate a.
With__slots__:
The second piece of code will reduce the burden on your RAM. Some peoplehave seen almost 40 to 50% reduction in RAM usage by using thistechnique.
On a sidenote, you might want to give PyPy a try. It does all of theseoptimizations by default.
Below you can see an example showing exact memory usage with and without __slots__ done in IPython thanks to https://github.com/ianozsvald/ipython_memory_usage
Python Metaprogramming - Properties on Steroids
Metaprogramming is an advanced topic, it requires a good understanding of the language itself. I assume you already know Python well enough even though I include references to complementary resources. If you are interested only in the final result, there is a link to the code at the end of the article.
Python is a very dynamic language, everything is an object, objects are created by classes (usually) but classes can also be created by otherclases or functions (this is amazing). Objects, once created, can be modified (add/remove/replace properties, methods, attributes, etc…) and thismeans that you can do a lot of metaprogramming in python. Metaprogramming is a key that can open the doors to heaven or hell.
Python Slots Read Only Version
Ok but what is metaprogramming? here is a definition from wikipedia:
Metaprogramming is a programming technique in which computer programs have the ability to treat other programs as their data. It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running.
-- Harald Sondergaard. 'Course on Program Analysis and Transformation'. Retrieved 18 September 2014.
-- Czarnecki, Krzysztof; Eisenecker, Ulrich W. (2000). Generative Programming. ISBN 0-201-30977-7.
In this article, I will use metaprogramming to change how properties are defined in a class, how they can be documented, initialized, how to set a default value, how to make them read-only and observable, and as a bonus, I will improve memory usage of the objects created by the class. And as a second bonus, I will seal the object against attribute injections. I call this 'Properties on Steroids'.
Requirements
- A property must be defined in a succinct and pythonic way.
- A property definition can setup a default value.
- A property must support docstring (
__doc__). - A Property can have a type hint.
- A Property can be read-only or read-and-write.
- A Property can be observable.
- A Property must not use more memory than traditional python property (@property)
- Bonus: Field storage can be optimized
- Bonus2: Objects created from the class must be protected from field injection.
- DO NOT USE A METACLASS
Important concepts used in the solution
Classes are created at runtime
In python, classes are created at runtime, so the code inside the class scope can modify the resulting class by adding/removing/replacing things in the class scope (local).
For more info about classes in Python: https://docs.python.org/3/tutorial/classes.html
Scopes
There are two scopes in python: local and global. Scopes are symbol tables of what is reachable from the current point in code. you can access the symbol table using the builtin functions locals() and globals(). The important part here is that you can modify the scope just adding/replacing/removing things in the symbol table.
For example, if you want to define a variable in the local scope dynamically:
For more info about scopes in python: https://realpython.com/python-scope-legb-rule/
Decorators
In python, functions are objects and can be passed to other functions like any other object. The idea of a decorator is a function that receives another function and returns a new function based on the original. There is a special syntax in python to call a decorator function just on function definition and effectively replacing the original function with the one returned by the decorator.
Example:
The above code will print:
For more info about decorators: https://realpython.com/primer-on-python-decorators/
Properties
Python has a special class called properties, it allows us to create getter/setter/deleter for a field. In combination with decorators you can define functional properties.
We will change this pattern to add more features like observability, and automatic usage of slots.
For more info about properties: https://docs.python.org/3/howto/descriptor.html#properties
Slots
Objects in Python do store attributes in an internal dictionary called __dict__, it allows dynamic creation of attributes in any object but uses additional memory for the dict object itself. If you want that your object do not support dynamic attribute creation, you can remove the __dict__ mechanism and use object slots with the additional benefit of memory savings.
I will not explain slots here, but you can find detailed info in the following resources:
Context Managers
Context managers are objects that can execute code at the beginning and at the end of a code block. they are used with the with statement.
For more info about context managers: https://docs.python.org/3/library/contextlib.html
Proposed Solution
Ok, now with all the tools in the bag, we can create our own monster.
The final goal:
Result:
The traditional equivalent code:
The differences
initially it appears to have no major advantajes, but the classes Car and CarTraditional and the objects mycar and mycar2 are very different.
Field injection
With Metaprog:
With traditional code:
Default values
With Metaprog:
Default values are specified at property definition.
With traditional code:
Default values are defined by assignment in constructor
Observability
With Metaprog:
Multiple attributes can be observed with the same listener, listener specified on property definition, listener is called only if new value is different from current.
With traditional code:You must implement observability by your own.
Read-Only / Read-Write
With Metaprog:
One single definition will create readonly or read-write property.
With traditional code:
You must define a getter and a setter if the property is writable.
Constructor arguments to field assigment
With Metaprog:
Constructor arguments can set defined properties automatically.
With traditional code:
You must set properties by hand.
Implementation
Ok, in a small sample class like Car there is no much advantage but in a large system with many classes and classes with many mutable and 'immutable' attributes and complex state changes it will add lot of productivity in a very pythonic way.
Lets see the magic:
API
self_properties(self, scope, exclude=(), save_args=False)
This utility function copy all the symbols in scope to self as properties. If you call it at the beginning of the constructor and pass the local scope, it will just copy the function arguments.
Slot Machine In Python
if you want to exlude something from the copy, just do this:
if you want to save all arguments as a tuple (additionally):
properties
This is where the magic happens. properties is a context manager that do the following:
- Define a prop decorator to create properties.
- Manage
__slots__automatically for the class. - Clean itself from the class scope.
@meta.prop
prop is the decorator, it transforms functions into properties.
Arguments:
read_only: bool: Will create a read only propertylistener: Union[str,bool]: specify the method to call on change if str. if bool, it will default to ‘_changed'auto_dirty: bool: Will set a field_is_dirtyto true if the property change.
Monty Python Slots
Thanks for reading.
Gist on Github: https://gist.github.com/mnesarco/e9440a196824af4bae439e4aeb4b6dcc
Python Slots Read Only One
pythonmetaprogrammingadvanced
Python Slot Machine Code
Please enable JavaScript to view the comments powered by Disqus.