oneflow.nn.LayerNorm¶
-
class
oneflow.nn.
LayerNorm
(normalized_shape: Union[int, Tuple[int], oneflow.Size], eps: float = 1e-05, elementwise_affine: bool = True)¶ Applies Layer Normalization over a mini-batch of inputs as described in the paper Layer Normalization
\[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by
normalized_shape
. \(\gamma\) and \(\beta\) are learnable affine transform parameters ofnormalized_shape
ifelementwise_affine
isTrue
. The standard-deviation is calculated via the biased estimator.Note
Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the
affine
option, Layer Normalization applies per-element scale and bias withelementwise_affine
.This layer uses statistics computed from input data in both training and evaluation modes.
- Parameters
normalized_shape (int or list or oneflow.Size) –
input shape from an expected input of size
\[[* \times \text{normalized_shape}[0] \times \text{normalized_shape}[1] \times \ldots \times \text{normalized_shape}[-1]]\]If a single integer is used, it is treated as a singleton list, and this module will
normalize over the last dimension which is expected to be of that specific size.
eps – a value added to the denominator for numerical stability. Default: 1e-5
elementwise_affine – a boolean value that when set to
True
, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default:True
.
- Shape:
Input: \((N, *)\)
Output: \((N, *)\) (same shape as input)
For example:
>>> import numpy as np >>> import oneflow as flow >>> input_arr = np.array( ... [ ... [ ... [[-0.16046895, -1.03667831], [-0.34974465, 0.26505867]], ... [[-1.24111986, -0.53806001], [1.72426331, 0.43572459]], ... ], ... [ ... [[-0.77390957, -0.42610624], [0.16398858, -1.35760343]], ... [[1.07541728, 0.11008703], [0.26361224, -0.48663723]], ... ], ... ], ... dtype=np.float32, ... ) >>> x = flow.Tensor(input_arr) >>> m = flow.nn.LayerNorm(2) >>> y = m(x).numpy() >>> y array([[[[ 0.99997395, -0.99997395], [-0.999947 , 0.999947 ]], [[-0.99995965, 0.9999595 ], [ 0.99998784, -0.99998784]]], [[[-0.9998348 , 0.99983466], [ 0.9999914 , -0.9999914 ]], [[ 0.9999785 , -0.9999785 ], [ 0.9999646 , -0.9999646 ]]]], dtype=float32)
-
__init__
(normalized_shape: Union[int, Tuple[int], oneflow.Size], eps: float = 1e-05, elementwise_affine: bool = True) → None¶ Initialize self. See help(type(self)) for accurate signature.
Methods
__call__
(*args, **kwargs)Call self as a function.
__delattr__
(name, /)Implement delattr(self, name).
__dir__
()Default dir() implementation.
__eq__
(value, /)Return self==value.
__format__
(format_spec, /)Default object formatter.
__ge__
(value, /)Return self>=value.
__getattr__
(name)__getattribute__
(name, /)Return getattr(self, name).
__gt__
(value, /)Return self>value.
__hash__
()Return hash(self).
__init__
(normalized_shape[, eps, …])Initialize self.
__init_subclass__
This method is called when a class is subclassed.
__le__
(value, /)Return self<=value.
__lt__
(value, /)Return self<value.
__ne__
(value, /)Return self!=value.
__new__
(**kwargs)Create and return a new object.
__reduce__
()Helper for pickle.
__reduce_ex__
(protocol, /)Helper for pickle.
__repr__
()Return repr(self).
__setattr__
(name, value)Implement setattr(self, name, value).
__sizeof__
()Size of object in memory, in bytes.
__str__
()Return str(self).
__subclasshook__
Abstract classes can override this to customize issubclass().
_apply
(fn[, applied_dict])_get_name
()_load_from_state_dict
(state_dict, prefix, …)_named_members
(get_members_fn[, prefix, recurse])_save_to_state_dict
(destination, prefix, …)_shallow_repr
()add_module
(name, module)Adds a child module to the current module.
apply
(fn)Applies
fn
recursively to every submodule (as returned by.children()
) as well as self.buffers
([recurse])Returns an iterator over module buffers.
children
()Returns an iterator over immediate children modules.
cpu
()Moves all model parameters and buffers to the CPU.
cuda
([device])Moves all model parameters and buffers to the GPU.
double
()Casts all floating point parameters and buffers to
double
datatype.eval
()Sets the module in evaluation mode.
extra_repr
()Set the extra representation of the module
float
()Casts all floating point parameters and buffers to
float
datatype.forward
(x)half
()Casts all floating point parameters and buffers to
half
datatype.load_state_dict
(state_dict[, strict])Copies parameters and buffers from
state_dict
into this module and its descendants.modules
()Returns an iterator over all modules in the network.
named_buffers
([prefix, recurse])Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
named_children
()Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules
([memo, prefix])Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters
([prefix, recurse])Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
parameters
([recurse])Returns an iterator over module parameters.
register_buffer
(name, tensor[, persistent])Adds a buffer to the module.
register_forward_hook
(hook)Registers a forward hook on the module.
register_forward_pre_hook
(hook)Registers a forward pre-hook on the module.
register_parameter
(name, param)Adds a parameter to the module.
reset_parameters
()state_dict
([destination, prefix, keep_vars])Returns a dictionary containing a whole state of the module.
to
([device])Moves the parameters and buffers.
to_consistent
(*args, **kwargs)This interface is no longer available, please use
oneflow.nn.Module.to_global()
instead.to_global
([placement, sbp])Convert the parameters and buffers to global.
train
([mode])Sets the module in training mode.
zero_grad
([set_to_none])Sets gradients of all model parameters to zero.
Attributes
__annotations__
__constants__