module
– a theano object system¶
Note
Module addresses similar needs to shared. New code is encouraged to use shared variables.
Now that we’re familiar with the basics, we introduce Theano’s more advanced interface, Module. This interface allows you to define Theano “files” which can have variables and methods sharing those variables. The Module system simplifies the way to define complex systems such as a neural network. It also lets you load and save these complex systems using Python’s pickle mechanism.
Remake of the “state” example¶
Let’s use Module to re-implement the example using state.
>>> m = Module()
>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')
>>> m.new_state = m.state + m.inc
>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})
>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})
>>> acc = m.make(state = 0)
>>> acc.state, acc.inc
(array(0.0), None)
>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)
>>> acc.state = 39.99
>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)
This deserves to be broken up a bit...
>>> m = Module()
Here we instantiate an empty Module.
If you can imagine that Theano is a way of generating code (expression
graphs),
then a Module()
is like a fresh blank file.
>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')
Then we declare Variables for use in our Module. Since we assign these input Variables as attributes of the Module, they will be member Variables of the Module. Member Variables are special in a few ways, which we will see shortly.
Note
There is no need to name the Variable explicitly here. m.state
will
be given the name 'state'
automatically, because it is being assigned
to the attribute named 'state'
.
Note
Since we made it a member of m
, the acc
object will have an
attribute called inc
. This attribute will keep its default value of
None throughout the example.
>>> m.new_state = m.state + m.inc
This line creates a Variable corresponding to some symbolic computation.
Although this line also assigns a Variable to a Module attribute, it
does not become a member Variable like state
and inc
because it
represents an expression result.
>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})
Here we declare a Method. The three arguments are as follow:
- inputs: a list of input Variables
- outputs: a list of output Variables, or
None
.None
is equivalent to returning an empty list of outputs. - updates: a dictionary mapping member Variables to Variables. When we call the function that this Method compiles to, it will replace (update) the values associated with the member Variables.
>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})
We declare another Method, that has no outputs.
>>> acc = m.make(state = 0)
This line is what does the magic (well, compilation).
The m
object contains symbolic things such as Variables and Methods.
Calling make
on m
creates an object that can do real
computation and whose attributes contain values such as numbers and numpy
ndarrays.
At this point something special happens for our member Variables too.
In the acc
object, make allocates room to store numbers for m
‘s
member Variables. By using the string 'state'
as a keyword
argument, we tell Theano to store the number 0
for the member
Variable called state
. By not mentioning the inc
variable, we
associate None
to the inc
Variable.
>>> acc.state, acc.inc
(array(0.0), None)
Since state
was declared as a member Variable of m
, we can
access it’s value in the acc
object by the same attribute.
Ditto for inc
.
Note
Members can also be accessed using a dictionary-like notation.
The syntax acc['state']
is equivalent to acc.state
.
>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)
When we call the acc.add
method, the value 2
is used for the
symbolic m.inc
. The first line evaluates the output and all the
updates given in the updates
argument of the call to Method that
declared acc
. We only had one update which mapped state
to
new_state
and you can see that it works as intended, adding the
argument to the internal state.
Note also that acc.inc
is still None
after our call. Since
m.inc
was listed as an input in the call to Method that created
m.add
, when acc.add
was created by the call to m.make
, a
private storage container was allocated to hold the first parameter.
If we had left m.inc
out of the Method input list, then acc.add
would have used acc.inc
instead.
>>> acc.state = 39.99
The state can also be set. When we manually set the value of a member attribute like this, then subsequent calls to the methods of our module will use the new value.
>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)
Here, note that acc.add
and acc.sub
share access to the same state
value but update it in different ways.
Using Inheritance¶
A friendlier way to use Module is to implement your functionality as a subclass of Module:
from theano.compile import Module, Method
import theano.tensor as T
class Accumulator(Module):
def __init__(self):
super(Accumulator, self).__init__() # don't forget this
self.inc = T.dscalar()
self.state = T.dscalar()
self.new_state = self.inc + self.state
self.add = Method(inputs = self.inc,
outputs = self.new_state,
updates = {self.state: self.new_state})
self.sub = Method(inputs = self.inc,
outputs = None,
updates = {self.state: self.state - self.inc})
if __name__ == '__main__':
m = Accumulator()
acc = m.make(state = 0)
This is just like the previous example except slightly fancier.
Warning
Do not forget to call the constructor of the parent class!
(That’s the call to super().__init__
in the previous code block.)
If you forget it, you’ll get strange behavior :(
Extending your Module with Python methods¶
Let’s say we want to add a method to our accumulator to print out the
state and we want to call it print_state
. There are two mechanisms to do
this: let’s call them _instance_method and InstanceType.
Mechanism 1: _instance_method¶
This is the preferred way of adding a few instance methods with a minimum of boilerplate code.
All we need to do to use this mechanism is to give a method called
_instance_print_state
to our Module class.
Any method called like _instance_XXX
will cause the object
obtained through a call to make
to have a method called XXX
.
Note that when we define _instance_print_state
there are two “self”
arguments: self
which is symbolic and obj
which is the compiled
object (the one that contains values).
Hint: self.state
is the symbolic state variable and
prints out as “state”, whereas obj.state
is the state’s actual
value in the accumulator and prints out as “0.0”.
Mechanism 2: InstanceType¶
If a number of instance methods are going to be defined, and especially if you
will want to inherit from the kind of class that gets instantiated by make
,
you might prefer to consider using the InstanceType mechanism.
Adding custom initialization¶
As was said in the previous section, you can add functionality with
_instance_XXX
methods. One of these methods is actually special:
_instance_initialize
will be called with whatever arguments you
give to make
. There is a default behavior which we have used,
where we give the states’ initial values with keyword arguments
(acc.make(state = 0)
). If you want more personalized behavior, you
can override the default with your own method, which has to be called
_instance_initialize
.
Nesting Modules¶
Probably the most powerful feature of Theano’s Modules is that one can be included as an attribute to another so that the storage of each is available to both.
As you read through examples of Theano code, you will probably see many instances of Modules being nested in this way.
module
– API documentation¶
Classes implementing Theano’s Module system.
For design notes, see doc/advanced/module.txt
-
exception
theano.compile.module.
AllocationError
¶ Exception raised when a Variable has no associated storage.
-
class
theano.compile.module.
Component
(no_warn=False)¶ Base class for the various kinds of components which are not structural but may be meaningfully used in structures (Member, Method, etc.)
-
allocate
(memo)¶ Populates the memo dictionary with gof.Variable -> io.In pairings. The value field of the In instance should contain a gof.Container instance. The memo dictionary is meant to tell the build method of Components where the values associated to certain variables are stored and how they should behave if they are implicit inputs to a Method (needed to compute its output(s) but not in the inputs or updates lists).
-
build
(mode, memo)¶ Makes an instance of this Component using the mode provided and taking the containers in the memo dictionary.
A Component which builds nothing, such as External, may return None.
The return value of this function will show up in the Module graph produced by make().
-
make
(*args, **kwargs)¶ Allocates the necessary containers using allocate() and uses build() to make an instance which will be returned. The initialize() method of the instance will be called with the arguments and the keyword arguments. If ‘mode’ is in the keyword arguments it will be passed to build().
-
make_no_init
(mode=None)¶ Allocates the necessary containers using allocate() and uses build() with the provided mode to make an instance which will be returned. The initialize() method of the instance will not be called.
-
pretty
(**kwargs)¶ Returns a pretty representation of this Component, suitable for reading.
-
-
class
theano.compile.module.
ComponentDictInstance
(component, __items__)¶ ComponentDictInstance is meant to be instantiated by ComponentDict.
-
class
theano.compile.module.
ComponentDictInstanceNoInit
(component, __items__)¶ Component Instance that allows new items to be added
-
class
theano.compile.module.
ComponentList
(*_components)¶ ComponentList represents a sequence of Component. It builds a ComponentListInstance.
-
class
theano.compile.module.
Composite
(no_warn=False)¶ Composite represents a structure that contains Components.
-
allocate
(memo)¶ Does allocation for each component in the composite.
-
components
()¶ Returns all components.
-
components_map
()¶ Returns (key, value) pairs corresponding to each component.
-
flat_components
(include_self=False)¶ Generator that yields each component in a flattened hierarchy of composites and components. If include_self is True, the list will include the Composite instances, else it will only yield the list of leaves.
-
flat_components_map
(include_self=False, path=None)¶ Generator that yields (path, component) pairs in a flattened hierarchy of composites and components, where path is a sequence of keys such that:
component is self[path[0]][path[1]]...
If include_self is True, the list will include the Composite instances, else it will only yield the list of leaves.
-
get
(item)¶ Get the Component associated to the key.
-
set
(item, value)¶ Set the Component associated to the key.
-
-
class
theano.compile.module.
CompositeInstance
(component, __items__)¶ Generic type which various Composite subclasses are intended to build.
-
class
theano.compile.module.
External
(r)¶ External represents a Variable which comes from somewhere else (another module) or is a temporary calculation.
-
build
(mode, memo)¶ Builds nothing.
-
-
theano.compile.module.
FancyModuleInstance
¶ alias of
ModuleInstance
-
class
theano.compile.module.
Member
(r)¶ Member represents a Variable which is a state of a Composite. That Variable will be accessible from a built Composite and it is possible to do updates on Members.
Member builds a gof.Container.
-
allocate
(memo)¶ If the memo does not have a Container associated to this Member’s Variable, instantiates one and sets it in the memo.
-
build
(mode, memo)¶ Returns the Container associated to this Member’s Variable.
-
-
class
theano.compile.module.
Method
(inputs, outputs, updates=None, mode=None)¶ Method is a declaration of a function. It contains inputs, outputs and updates. If the Method is part of a Composite which holds references to Members, the Method may use them without declaring them in the inputs, outputs or updates list.
inputs, outputs or updates may be strings. In that case, they will be resolved in the Composite which is the parent of this Method.
Method builds a Function (same structure as a call to theano.function)
-
allocate
(memo)¶ Method allocates nothing.
-
build
(mode, memo, allocate_all=False)¶ Compile a function for this Method.
Parameters: allocate_all – if True, storage will be allocated for all needed Variables even if there is no associated storage for them in the memo. If allocate_all is False, storage will only be allocated for Variables that are reachable from the inputs list. Returns: a function that implements this method Return type: Function instance
-
inputs
= []¶ function inputs (see compile.function)
If Module members are named explicitly in this list, then they will not use shared storage. Storage must be provided either via an io.In value argument, or at the point of the function call.
-
mode
= None¶ This will override the Module compilation mode for this Method
-
outputs
= None¶ function outputs (see compile.function)
-
resolve_all
()¶ Convert all inputs, outputs, and updates specified as strings to Variables.
This works by searching the attribute list of the Module to which this Method is bound.
-
updates
= {}¶ update expressions for module members
If this method should update the shared storage value for a Module member, then the update expression must be given in this dictionary.
Keys in this dictionary must be members of the module graph–variables for which this Method will use the shared storage.
The value associated with each key should be a Variable (or a string that can be resolved to a Variable) representing the computation of a new value for this shared storage after each function call.
-
-
class
theano.compile.module.
Module
(*args, **kw)¶ WRITEME
You should inherit from Module with the members will be other Modules or Components. To make more specialized elements of a Module graph, consider inheriting from Component directly.
-
InstanceType
¶ alias of
ModuleInstance
-
make
(*args, **kwargs)¶ Allocates the necessary containers using allocate() and uses build() to make an instance which will be returned. The initialize() method of the instance will be called with the arguments and the keyword arguments. If ‘mode’ is in the keyword arguments it will be passed to build().
-
make_module_instance
(*args, **kwargs)¶ Module’s __setattr__ method hides all members under local_attr. This method iterates over those elements and wraps them so they can be used in a computation graph. The “wrapped” members are then set as object attributes accessible through the dotted notation syntax (<module_name> <dot> <member_name>). Submodules are handled recursively.
-
old__setattr__
(attr, value)¶
-
-
class
theano.compile.module.
ModuleInstance
(component, __items__)¶ WRITEME
Note: ModuleInstance is meant to be instantiated by Module. This differs from ComponentDictInstance on a key point, which is that getattr does a similar thing to getitem. Note: ModuleInstance is compatible for use as ComponentDict.InstanceType.
-
theano.compile.module.
func_to_mod
(f)¶ Creates a dummy module, with external member variables for the input parameters required by the function f, and a member output defined as:
output <= f(**kwinit)
-
theano.compile.module.
register_wrapper
(condition, wrapper, no_warn=False)¶ Parameters: - condition (function x -> bool) – this function should return True iff wrapper can sensibly turn x into a Component.
- wrapper (function x -> Component) – this function should convert x into an instance of a Component subclass.
-
theano.compile.module.
wrap
(x)¶ Wraps x in a Component. Wrappers can be registered using register_wrapper to allow wrapping more types.
It is necessary for Module attributes to be wrappable. A Module with an attribute that is not wrappable as a Component, will cause Component.make to fail.
-
theano.compile.module.
wrapper
(x)¶ Returns a wrapper function appropriate for x Returns None if not appropriate wrapper is found