oneflow.nn.SELU¶
-
class
oneflow.nn.
SELU
(inplace: bool = False)¶ Applies the element-wise function:
The formula is:
\[\text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))\]with \(\alpha = 1.6732632423543772848170429916717\) and
\(\text{scale} = 1.0507009873554804934193349852946\).
Warning
When using
kaiming_normal
orkaiming_normal_
for initialisation,nonlinearity='linear'
should be used instead ofnonlinearity='selu'
in order to get Self-Normalizing Neural Networks. Seetorch.nn.init.calculate_gain()
for more information.More details can be found in the paper Self-Normalizing Neural Networks.
- Shape:
Input: \((N, *)\) where * means, any number of additional dimensions
Output: \((N, *)\), same shape as the input
For example:
>>> import numpy as np >>> import oneflow as flow >>> x = np.array([1, 2, 3]).astype(np.float32) >>> input = flow.Tensor(x) >>> selu = flow.nn.SELU() >>> out = selu(input) >>> out tensor([1.0507, 2.1014, 3.1521], dtype=oneflow.float32)