# oneflow.nn.init¶

## Operators for initialization¶

oneflow.nn.init.xavier_uniform_(tensor, gain=1.0, *, data_format='NCHW')

The interface is consistent with PyTorch. The documentation is referenced from: https://pytorch.org/docs/1.10/nn.init.html.

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from $$\mathcal{U}(-a, a)$$ where

$a = \text{gain} \times \sqrt{\frac{6}{\text{fan_in} + \text{fan_out}}}$

Also known as Glorot initialization.

Parameters
• tensor – an n-dimensional flow.Tensor

• gain – an optional scaling factor

Examples

>>> w = flow.empty(3, 5)
>>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))

oneflow.nn.init.xavier_normal_(tensor, gain=1.0, *, data_format='NCHW')

The interface is consistent with PyTorch. The documentation is referenced from: https://pytorch.org/docs/1.10/nn.init.html.

Fills the input Tensor with values according to the method described in Understanding the difficulty of training deep feedforward neural networks - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from $$\mathcal{N}(0, \text{std}^2)$$ where

$\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan_in} + \text{fan_out}}}$

Also known as Glorot initialization.

Parameters
• tensor – an n-dimensional flow.Tensor

• gain – an optional scaling factor

Examples

>>> w = flow.empty(3, 5)
>>> nn.init.xavier_normal_(w)

oneflow.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu', *, data_format='NCHW')

The interface is consistent with PyTorch. The documentation is referenced from: https://pytorch.org/docs/1.10/nn.init.html.

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from $$\mathcal{U}(-\text{bound}, \text{bound})$$ where

$\text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan_mode}}}$

Also known as He initialization.

Parameters
• tensor – an n-dimensional flow.Tensor

• a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

• mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

Examples

>>> w = flow.empty(3, 5)
>>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')

oneflow.nn.init.kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu', *, data_format='NCHW')

The interface is consistent with PyTorch. The documentation is referenced from: https://pytorch.org/docs/1.10/nn.init.html.

Fills the input Tensor with values according to the method described in Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from $$\mathcal{N}(0, \text{std}^2)$$ where

$\text{std} = \frac{\text{gain}}{\sqrt{\text{fan_mode}}}$

Also known as He initialization.

Parameters
• tensor – an n-dimensional flow.Tensor

• a – the negative slope of the rectifier used after this layer (only used with 'leaky_relu')

• mode – either 'fan_in' (default) or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with 'relu' or 'leaky_relu' (default).

Examples

>>> w = flow.empty(3, 5)
>>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')

oneflow.nn.init.orthogonal_(tensor, gain=1.0)

The interface is consistent with PyTorch. The documentation is referenced from: https://pytorch.org/docs/stable/nn.init.html.

Fills the input Tensor with a (semi) orthogonal matrix, as described in Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened.

Parameters
• tensor – an n-dimensional torch.Tensor, where $$n \geq 2$$

• gain – optional scaling factor

Examples

>>> w = flow.empty(3, 5)
>>> nn.init.orthogonal_(w)