oneflow.nn.SELU

class oneflow.nn.SELU(inplace: bool = False)

Applies the element-wise function:

The formula is:

\[\text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))\]

with \(\alpha = 1.6732632423543772848170429916717\) and

\(\text{scale} = 1.0507009873554804934193349852946\).

Warning

When using kaiming_normal or kaiming_normal_ for initialisation, nonlinearity='linear' should be used instead of nonlinearity='selu' in order to get Self-Normalizing Neural Networks. See torch.nn.init.calculate_gain() for more information.

More details can be found in the paper Self-Normalizing Neural Networks.

Shape:
  • Input: \((N, *)\) where * means, any number of additional dimensions

  • Output: \((N, *)\), same shape as the input

For example:

>>> import numpy as np
>>> import oneflow as flow
>>> x = np.array([1, 2, 3]).astype(np.float32)
>>> input = flow.Tensor(x)
>>> selu = flow.nn.SELU()
>>> out = selu(input)
>>> out
tensor([1.0507, 2.1014, 3.1521], dtype=oneflow.float32)