oneflow.nn.QuickGELU

class oneflow.nn.QuickGELUTensor

Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs

\[\text{QuickGELU}(x) = x * \sigma(1.702x) = x * \frac{1}{1 + \exp(-1.702x)}\]
Parameters

input (oneflow.Tensor) – Input Tensor

Returns

A Tensor has same shape as the input.

Return type

oneflow.Tensor

For example:

>>> import oneflow as flow

>>> input = flow.Tensor([-0.5, 0, 0.5])
>>> gelu = flow.nn.QuickGELU()

>>> out = gelu(input)
>>> out
tensor([-0.1496,  0.0000,  0.3504], dtype=oneflow.float32)