oneflow.layers

Operators with variables

oneflow.layers.batch_normalization(inputs: oneflow_api.BlobDesc, axis: int = -1, momentum: float = 0.99, epsilon: float = 0.001, center: bool = True, scale: bool = True, beta_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, gamma_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, beta_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, gamma_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, moving_mean_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, moving_variance_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, trainable: bool = True, training: bool = True, name: str = 'BatchNorm') → oneflow_api.BlobDesc

The BatchNormalization Layer.

This layer can be used in conv or dense layer.

The input data will be normalized by the mean and variance of the current batch data

Parameters
  • inputs (oneflow_api.BlobDesc) – Input Blob.

  • axis (int, optional) – An int specifies the axis that should be normalized . Default is -1, which normalizes the last axis.

  • momentum (float, optional) – A float specifies the momentum for the moving average. Defaults to 0.99.

  • epsilon (float, optional) – A small float added to avoid division by zero. Defaults to 0.001.

  • center (bool, optional) – A boolean specifies whether to add offset to normalized Blob. Defaults to True.

  • scale (bool, optional) – A boolean specifies whether to multiply normalized Blob by gamma. Defaults to True.

  • beta_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for beta. Defaults to None.

  • gamma_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for gamma. Defaults to None.

  • beta_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for beta. Defaults to None.

  • gamma_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for gamma. Defaults to None.

  • moving_mean_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving mean. Defaults to None.

  • moving_variance_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving variance. Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • training (bool, optional) – A boolean specifies whether now is training the model. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Returns

A Blob with same shape of input.

Return type

oneflow_api.BlobDesc

Raises

ValueError – If axis is out of dimension of input.

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def batch_norm_Job(x: tp.Numpy.Placeholder((1, 64, 128, 128))
) -> tp.Numpy:
    initializer = flow.truncated_normal(0.1)
    conv2d = flow.layers.conv2d(
        x,
        filters=128,
        kernel_size=3,
        strides=2,
        padding='SAME',
        kernel_initializer=initializer,
        name="Conv2d"
    )
    batch_norm = flow.layers.batch_normalization(
        conv2d,
        axis=1
    )
    return batch_norm


x = np.random.randn(1, 64, 128, 128).astype(np.float32)
out = batch_norm_Job(x)

# out.shape (1, 128, 64, 64)
oneflow.layers.batch_normalization_add_relu(inputs: oneflow_api.BlobDesc, addend: Optional[oneflow_api.BlobDesc] = None, axis: int = -1, momentum: float = 0.99, epsilon: float = 0.001, center: bool = True, scale: bool = True, beta_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, gamma_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, beta_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, gamma_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, moving_mean_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, moving_variance_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, trainable: bool = True, training: bool = True, name: str = 'BatchNorm') → oneflow_api.BlobDesc

Fused flow.layers.batch_normalization + flow.math.add + flow.math.relu

Parameters
  • inputs (oneflow_api.BlobDesc) – Input Blob.

  • addend (oneflow_api.BlobDesc) – Blob add to batch_normalization output.

  • axis (int, optional) – An int specifies the axis that should be normalized . Default is -1, which normalizes the last axis.

  • momentum (float, optional) – A float specifies the momentum for the moving average. Defaults to 0.99.

  • epsilon (float, optional) – A small float added to avoid division by zero. Defaults to 0.001.

  • center (bool, optional) – A boolean specifies whether to add offset to normalized Blob. Defaults to True.

  • scale (bool, optional) – A boolean specifies whether to multiply normalized Blob by gamma. Defaults to True.

  • beta_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for beta. Defaults to None.

  • gamma_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for gamma. Defaults to None.

  • beta_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for beta. Defaults to None.

  • gamma_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for gamma. Defaults to None.

  • moving_mean_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving mean. Defaults to None.

  • moving_variance_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving variance. Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • training (bool, optional) – A boolean specifies whether now is training the model. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Returns

A Blob with same shape of input.

Return type

oneflow_api.BlobDesc

Raises

ValueError – If axis is out of dimension of input.

oneflow.layers.batch_normalization_relu(inputs: oneflow_api.BlobDesc, axis: int = -1, momentum: float = 0.99, epsilon: float = 0.001, center: bool = True, scale: bool = True, beta_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, gamma_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, beta_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, gamma_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, moving_mean_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, moving_variance_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, trainable: bool = True, training: bool = True, name: str = 'BatchNorm') → oneflow_api.BlobDesc

Fused flow.layers.batch_normalization + flow.math.relu

Parameters
  • inputs (oneflow_api.BlobDesc) – Input Blob.

  • axis (int, optional) – An int specifies the axis that should be normalized . Default is -1, which normalizes the last axis.

  • momentum (float, optional) – A float specifies the momentum for the moving average. Defaults to 0.99.

  • epsilon (float, optional) – A small float added to avoid division by zero. Defaults to 0.001.

  • center (bool, optional) – A boolean specifies whether to add offset to normalized Blob. Defaults to True.

  • scale (bool, optional) – A boolean specifies whether to multiply normalized Blob by gamma. Defaults to True.

  • beta_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for beta. Defaults to None.

  • gamma_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for gamma. Defaults to None.

  • beta_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for beta. Defaults to None.

  • gamma_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for gamma. Defaults to None.

  • moving_mean_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving mean. Defaults to None.

  • moving_variance_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for moving variance. Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • training (bool, optional) – A boolean specifies whether now is training the model. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Returns

A Blob with same shape of input.

Return type

oneflow_api.BlobDesc

Raises

ValueError – If axis is out of dimension of input.

oneflow.layers.categorical_ordinal_encoder(input_tensor: oneflow_api.BlobDesc, capacity: int, hash_precomputed: bool = True, name: str = 'CategoricalOrdinalEncoder') → oneflow_api.BlobDesc

This operator uses oneflow.categorical_ordinal_encode to encapsulate a categorical_ordinal_encoder. More details please refer to oneflow.categorical_ordinal_encode

Parameters
  • input_tensor (oneflow_api.BlobDesc) – The input Blob.

  • capacity (int) – The capacity of hash table.

  • hash_precomputed (bool, optional) – We currently only support the ‘True’ mode. The internal hash value will no longer be computed. Defaults to True.

  • name (str, optional) – The name for the operation. Defaults to “CategoricalOrdinalEncoder”.

Returns

The result Blob.

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp

@flow.global_function()
def categorical_ordinal_encoder_Job(x: tp.Numpy.Placeholder((3, 3), dtype=flow.int32)
) -> tp.Numpy:
    return flow.layers.categorical_ordinal_encoder(x, 16)

x = np.array([[7, 0, 2],
            [1, 7, 2],
            [0, 1, 7]]).astype(np.int32)

out = categorical_ordinal_encoder_Job(x)

# out [[1 0 2]
#      [3 1 2]
#      [0 3 1]]
oneflow.layers.conv1d(inputs: oneflow_api.BlobDesc, filters: int, kernel_size: Union[int, Tuple[int]] = 1, strides: Union[int, Tuple[int]] = 1, padding: Union[str, Tuple[Tuple[int, int], Tuple[int, int], Tuple[int, int]]] = 'VALID', data_format: str = 'NCW', dilation_rate: Union[int, Tuple[int], None] = None, groups: int = 1, activation: Optional[Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc]] = None, use_bias: bool = True, kernel_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, bias_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, kernel_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, bias_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, trainable: bool = True, name: str = 'Conv1d', weight_name: Optional[str] = None, bias_name: Optional[str] = None) → oneflow_api.BlobDesc

1D convolution layer.

This layer computes a 1-D convolution with 3D input Blob and filters.

Parameters
  • inputs (oneflow_api.BlobDesc) – A 3D input Blob.

  • filters (int) – An integer specifies the dimensionality of the output space.

  • kernel_size (Union[int, List[int], Tuple[int]], optional) – An integer or tuple/list specifies the height and width of the convolution window. When it is an integer, a square window is applied to the input. Defaults to 1.

  • strides (Union[int, List[int], Tuple[int]], optional) – An integer or tuple/list specifies the strides of the convolution window along the height and width. When it is an integer, the same value for the all spatial dimesions is applied. Defaults to 1.

  • padding (str, Tuple[IntPair, IntPair, IntPair], optional) – padding: string “SAME” or “SAME_LOWER” or “SAME_UPPER” or “VALID” or Tuple[IntPair, IntPair, IntPair] indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. Defaults to “VALID”.

  • data_format (str, optional) – A string specifies the format of the input Blob, one of “NCW” or “NWC” (default: “NCW”). “NCW” cooresponds to channels_first, i.e. the input Blob with shape (batch_size, channels, width). “NWC” cooresponds to channels_last, i.e. the input Blob with shape (batch_size, channels, width). Defaults to “NCW”.

  • dilation_rate (Optional[Union[int, Tuple[int]]], optional) – An integer or tuple/list specifies the dilation rate for the dilated convolution. When it is an integer, the same dilation rate is applied for the all dimensions. Defaults to 1.

  • groups (int, optional) – A positive integer specifies number of groups for the Group conv. Defaults to 1.

  • activation (Optional[ Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc] ], optional) – Activation function. Defaults to None.

  • use_bias (bool, optional) – A boolean specifies whether to use a bias vector. Defaults to True.

  • kernel_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the kernel weights matrix. Defaults to None.

  • bias_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the bias vector. Defaults to None.

  • kernel_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the kernel weights matrix. Defaults to None.

  • bias_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the bias vector . Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Raises
  • ValueError – If the type of kernel_size is not one of integer, list, tuple.

  • ValueError – The number of groups must be positive and number of filters must be divisible by it.

  • ValueError – If data_format is not one of ‘NCW’, ‘NWC’.

  • ValueError – If number of input channels is not divisible by number of groups or less than number of groups.

  • ValueError – Number of group must be one when data_format is ‘NWC’.

Returns

A 3D Blob with the shape of (batch_size, filters, new_width).

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def conv1d_Job(x: tp.Numpy.Placeholder((1, 64, 32))
) -> tp.Numpy:
    initializer = flow.truncated_normal(0.1)
    conv1d = flow.layers.conv1d(
        x,
        filters=128,
        kernel_size=3,
        strides=1,
        padding='SAME',
        kernel_initializer=initializer,
        name="Conv1d"
    )
    return conv1d


x = np.random.randn(1, 64, 32).astype(np.float32)
out = conv1d_Job(x)

# out.shape (1, 128, 32)
oneflow.layers.conv2d(inputs: oneflow_api.BlobDesc, filters: int, kernel_size: Union[int, Tuple[int, int]] = 1, strides: Union[int, Tuple[int, int]] = 1, padding: Union[str, Tuple[Tuple[int, int], Tuple[int, int], Tuple[int, int], Tuple[int, int]]] = 'VALID', data_format: str = 'NCHW', dilation_rate: Union[int, Tuple[int, int], None] = None, groups: int = 1, activation: Optional[Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc]] = None, use_bias: bool = True, kernel_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, bias_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, kernel_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, bias_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, trainable: bool = True, name: str = 'Conv2d', weight_name: Optional[str] = None, bias_name: Optional[str] = None) → oneflow_api.BlobDesc

2D convolution layer.

This layer computes a 2D convolution with 4D input Blob and filters.

Parameters
  • inputs (oneflow_api.BlobDesc) – A 4D input Blob.

  • filters (int) – An integer specifies the dimensionality of the output space.

  • kernel_size (Union[int, List[int], Tuple[int]], optional) – An integer or tuple/list specifies the height and width of the convolution window. When it is an integer, a square window is applied to the input. Defaults to 1.

  • strides (Union[int, List[int], Tuple[int]], optional) – An integer or tuple/list specifies the strides of the convolution window along the height and width. When it is an integer, the same value for the all spatial dimesions is applied. Defaults to 1.

  • padding (str, Tuple[IntPair, IntPair, IntPair, IntPair], optional) – padding: string “SAME” or “SAME_LOWER” or “SAME_UPPER” or “VALID” or Tuple[IntPair, IntPair, IntPair] indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. Defaults to “VALID”.

  • data_format (str, optional) – A string specifies the format of the input Blob, one of “NCHW” or “NHWC” (default: “NCHW”). “NCHW” cooresponds to channels_first, i.e. the input Blob with shape (batch_size, channels, height, width). “NHWC” cooresponds to channels_last, i.e. the input Blob with shape (batch_size, height, width, channels). Defaults to “NCHW”.

  • dilation_rate (int, optional) – An integer or tuple/list specifies the dilation rate for the dilated convolution. When it is an integer, the same dilation rate is applied for the all dimensions. Defaults to 1.

  • groups (int, optional) – A positive integer specifies number of groups for the Group conv. Defaults to 1.

  • activation (Optional[ Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc] ], optional) – Activation function. Defaults to None.

  • use_bias (bool, optional) – A boolean specifies whether to use a bias vector. Defaults to True.

  • kernel_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the kernel weights matrix. Defaults to None.

  • bias_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the bias vector. Defaults to None.

  • kernel_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the kernel weights matrix. Defaults to None.

  • bias_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the bias vector . Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

  • weight_name (Optional[str], optional) – This weight’s name. Defaults to None.

  • bias_name (Optional[str], optional) – This bias’s name. Defaults to None.

Raises
  • ValueError – If the type of kernel_size is not one of integer, list, tuple.

  • ValueError – The number of groups must be positive and number of filters must be divisible by it.

  • ValueError – If data_format is not one of ‘NCHW’, ‘NHWC’.

  • ValueError – If number of input channels is not divisible by number of groups or less than number of groups.

  • ValueError – Number of group must be one when data_format is ‘NHWC’.

Returns

A 4D Blob with the shape of (batch_size, filters, new_height, new_width).

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def conv2d_Job(x: tp.Numpy.Placeholder((1, 256, 32, 32))
) -> tp.Numpy:
    initializer = flow.truncated_normal(0.1)
    conv2d = flow.layers.conv2d(
        x,
        filters=128,
        kernel_size=3,
        strides=1,
        padding='SAME',
        kernel_initializer=initializer,
        name="Conv2d"
    )
    return conv2d


x = np.random.randn(1, 256, 32, 32).astype(np.float32)
out = conv2d_Job(x)

# out.shape (1, 128, 32, 32)
oneflow.layers.conv3d(inputs: oneflow_api.BlobDesc, filters: int, kernel_size: Union[int, Sequence[int]] = 1, strides: Union[int, Sequence[int]] = 1, padding: Union[str, Tuple[Tuple[int, int], Tuple[int, int], Tuple[int, int], Tuple[int, int], Tuple[int, int]]] = 'VALID', data_format: str = 'NCDHW', dilation_rate: Union[int, Tuple[int, int], None] = None, groups: int = 1, activation: Optional[Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc]] = None, use_bias: bool = True, kernel_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, bias_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, kernel_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, bias_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, trainable: bool = True, name: str = 'Conv3d', weight_name: Optional[str] = None, bias_name: Optional[str] = None) → oneflow_api.BlobDesc

3D convolution layer.

This layer computes 3D convolution with 5D input Blob and filters

Parameters
  • inputs (oneflow_api.BlobDesc) – A 5D input Blob.

  • filters (int) – An integer specifies the dimensionality of the output space.

  • kernel_size (Union[int, List[int], Sequence[int]], optional) – An integer or tuple/list specifies the height and width of the convolution window. When it is an integer, a square window is applied to the input. Defaults to 1.

  • strides (Union[int, List[int], Sequence[int]], optional) – An integer or tuple/list specifies the strides of the convolution window along the height and width. When it is an integer, the same value for the all spatial dimesions is applied. Defaults to 1.

  • padding (str, Tuple[IntPair, IntPair, IntPair, IntPair, IntPair], optional) – padding: string “SAME” or “SAME_LOWER” or “SAME_UPPER” or “VALID” or Tuple[IntPair, IntPair, IntPair, IntPair, IntPair] indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. Defaults to “VALID”.

  • data_format (str, optional) – A string specifies the format of the input Blob, one of “NCDHW” or “NDHWC” (default: “NCDHW”). “NCDHW” cooresponds to channels_first, i.e. the input Blob with shape (batch_size, channels, depth, height, width). “NDHWC” cooresponds to channels_last, i.e. the input Blob with shape (batch_size, channels, depth, height, width). Defaults to “NCDHW”.

  • dilation_rate (int, optional) – An integer or tuple/list specifies the dilation rate for the dilated convolution. When it is an integer, the same dilation rate is applied for the all dimensions. Defaults to 1.

  • groups (int, optional) – A positive integer specifies number of groups for the Group conv. Defaults to 1.

  • activation (Optional[ Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc] ], optional) – Activation function. Defaults to None.

  • use_bias (bool, optional) – A boolean specifies whether to use a bias vector. Defaults to True.

  • kernel_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the kernel weights matrix. Defaults to None.

  • bias_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the bias vector. Defaults to None.

  • kernel_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the kernel weights matrix. Defaults to None.

  • bias_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the bias vector . Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

  • weight_name (Optional[str], optional) – This weight’s name. Defaults to None.

  • bias_name (Optional[str], optional) – This bias’s name. Defaults to None.

Raises
  • ValueError – If the type of kernel_size is not one of integer, list, tuple.

  • ValueError – The number of groups must be positive and number of filters must be divisible by it.

  • ValueError – If data_format is not one of ‘NCDHW’, ‘NDHWC’.

  • ValueError – If number of input channels is not divisible by number of groups or less than number of groups.

  • ValueError – Number of group must be one when data_format is ‘NDHWC’.

Returns

A 5D Blob with the shape of (batch_size, filters, new_height, new_width).

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def conv3d_Job(x: tp.Numpy.Placeholder((1, 64, 16, 16, 16))
) -> tp.Numpy:
    initializer = flow.truncated_normal(0.1)
    conv3d = flow.layers.conv3d(
        x,
        filters=128,
        kernel_size=3,
        strides=1,
        padding='SAME',
        kernel_initializer=initializer,
        name="Conv3d"
    )
    return conv3d


x = np.random.randn(1, 64, 16, 16, 16).astype(np.float32)
out = conv3d_Job(x)

# out.shape (1, 128, 16, 16, 16)
oneflow.layers.dense(inputs: oneflow_api.BlobDesc, units: int, activation: Optional[Callable[[oneflow_api.BlobDesc, str], oneflow_api.BlobDesc]] = None, use_bias: bool = True, kernel_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, bias_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, kernel_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, bias_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, trainable: bool = True, name: str = 'Dense', model_distribute: oneflow_api.distribute.Distribute = <oneflow_api.distribute.BroadcastDistribute object>) → oneflow_api.BlobDesc

Fully-connected layer.

The fully-connected layer multiplies input Blob with weight matrix and produces an Output Blob.

Parameters
  • inputs (oneflow_api.BlobDesc) – A 2D input Blob.

  • units (int) – A positive integer for the dimensionality of the output space.

  • activation (Optional[oneflow_api.BlobDesc], optional) – Activation function. Defaults to None.

  • use_bias (bool, optional) – A boolean specifies whether to use a bias vector. Defaults to True.

  • kernel_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the kernel weights matrix. Defaults to None.

  • bias_initializer (Optional[initializer_conf_util.InitializerConf], optional) – Initializer for the bias vector. Defaults to None.

  • kernel_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer function applied to the kernel weights matrix. Defaults to None.

  • bias_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – Regularizer for the bias vector. Defaults to None.

  • trainable (bool, optional) – A boolean specifies whether to train the variables. Defaults to True.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

  • model_distribute (oneflow_api.distribute.Distribute, optional) – Define the way to ditribute the model. Defaults to oneflow_api.distribute.broadcast().

Returns

A N-D Blob with the shape of (batch_size, units).

Return type

oneflow_api.BlobDesc

Raises
  • ValueError – The dimension of input Blob must be less than 2.

  • VauleError – Model distribute must be in auto, broadcast, split.

  • ValueError – The input must be a 2D Blob when the model distribute is split.

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def dense_Job(x: tp.Numpy.Placeholder((1, 256))
) -> tp.Numpy:
    initializer = flow.truncated_normal(0.1)
    hidden = flow.layers.dense(
        x,
        512,
        activation=flow.nn.relu,
        kernel_initializer=initializer,
        name="dense1",
    )
    return hidden


x = np.random.randn(1, 256).astype(np.float32)
out = dense_Job(x)

# out.shape (1, 512)
oneflow.layers.layer_norm(inputs: oneflow_api.BlobDesc, center: bool = True, scale: bool = True, trainable: bool = True, begin_norm_axis: int = 1, begin_params_axis: int = -1, epsilon: float = 1e-05, name: str = 'LayerNorm') → oneflow_api.BlobDesc

Layer Normalization.

Parameters
  • inputs (oneflow_api.BlobDesc) – Input Blob.

  • center (bool, optional) – A boolean specifies whether to shift input Blob. Defaults to True.

  • scale (bool, optional) – A boolean specifies whether to scale input Blob. Defaults to True.

  • trainable (bool, optional) – A boolean specifies whether to train variables. Defaults to True.

  • begin_norm_axis (int, optional) – An integer specifies which axis to normalize at first. Defaults to 1.

  • begin_params_axis (int, optional) – An integer specifies which axis params at . Defaults to -1.

  • epsilon (float, optional) – A small float is added to avoid division by zero. Defaults to 1e-5.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Returns

A normalized Blob with same shape of input.

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def layer_norm_Job(x: tp.Numpy.Placeholder((1, 64, 128, 128))
) -> tp.Numpy:
    layer_norm = flow.layers.layer_norm(
        x,
        name="LayerNorm1"
    )
    return layer_norm


x = np.random.randn(1, 64, 128, 128).astype(np.float32)
out = layer_norm_Job(x)

# out.shape (1, 64, 128, 128)
oneflow.layers.layer_norm_grad(dy: oneflow_api.BlobDesc, x: oneflow_api.BlobDesc, mean: oneflow_api.BlobDesc, inv_variance: oneflow_api.BlobDesc, begin_norm_axis: int = 1, name: str = 'LayerNormGrad') → oneflow_api.BlobDesc

Layer normalization

Parameters
  • dy (oneflow_api.BlobDesc) – Upstream derivstives.

  • x (oneflow_api.BlobDesc) – Input Blob.

  • mean (oneflow_api.BlobDesc) – Mean over neurons.

  • inv_variance (oneflow_api.BlobDesc) – Variance over neurons.

  • begin_norm_axis (int, optional) – An integer specifies which axis to normalize at first. Defaults to 1.

  • name (Optional[str], optional) – This layer’s name. Defaults to None.

Returns

Gradient with respect to input Blob.

Return type

oneflow_api.BlobDesc

oneflow.layers.layer_norm_param_grad(dy: oneflow_api.BlobDesc, norm: oneflow_api.BlobDesc, gamma: oneflow_api.BlobDesc, begin_params_axis: int = -1, name: str = 'LayerNormParamGrad') → Tuple[oneflow_api.BlobDesc, oneflow_api.BlobDesc, oneflow_api.BlobDesc]

Backward pass for layer normalization

Parameters
  • dy (oneflow_api.BlobDesc) – Upstream derivstives.

  • norm (oneflow_api.BlobDesc) – Normalized output.

  • gamma (oneflow_api.BlobDesc) – Scale parameter.

  • begin_params_axis (int, optional) – From which parameters to begin with. Defaults to -1.

  • name (Optional[str], optional) – This layer’s name. Defaults to ‘LayerNormParamGrad’.

Returns

normalized_diff: Gradient with respect to input Blob. beta_diff: Gradient with respect to shift parameter beta. gamma_diff: Gradient with respect to scale parameter gamma.

Return type

Tuple[oneflow_api.BlobDesc]

oneflow.layers.prelu(inputs: oneflow_api.BlobDesc, alpha_initializer: Optional[oneflow.core.job.initializer_conf_pb2.InitializerConf] = None, alpha_regularizer: Optional[oneflow.core.job.regularizer_conf_pb2.RegularizerConf] = None, shared_axes: Optional[Sequence[int]] = None, trainable: bool = True, name: str = 'PRelu', model_distribute: oneflow_api.distribute.Distribute = <oneflow_api.distribute.BroadcastDistribute object>) → oneflow_api.BlobDesc

The Prelu(Parametric Rectified Linear Unit) activation.

The \(\alpha\) is a parameter that can be trained in network

The equation is

\[out = max(0, x) + \alpha*min(0, x)\]
Parameters
  • inputs (oneflow_api.BlobDesc) – The input Blob.

  • alpha_initializer (Optional[initializer_conf_util.InitializerConf], optional) – The initializer of alpha. Defaults to None.

  • alpha_regularizer (Optional[regularizer_conf_util.RegularizerConf], optional) – The regularizer of alpha. Defaults to None.

  • shared_axes (Optional[Sequence[int]], optional) – The axis along which to share learnable parameters for the prelu activation function. Defaults to None.

  • trainable (bool, optional) – Whether to train the parameter \(\alpha\). Defaults to True.

  • name (str, optional) – The name for the operation. Defaults to “PRelu”.

  • model_distribute (oneflow_api.distribute.Distribute, optional) – Define the way to ditribute the model. Defaults to oneflow_api.distribute.broadcast().

Returns

The activated Blob

Return type

oneflow_api.BlobDesc

For example:

import oneflow as flow
import oneflow.typing as tp

BATCH_SIZE = 100


def lenet(data, train=False):
    initializer = flow.truncated_normal(0.1)
    conv1 = flow.layers.conv2d(
        data,
        32,
        5,
        padding="SAME",
        name="conv1",
        kernel_initializer=initializer,
    )
    prelu1 = flow.layers.prelu(conv1,
                            alpha_initializer=initializer,
                            shared_axes=[2, 3],
                            name="Prelu1")
    pool1 = flow.nn.max_pool2d(
        prelu1, ksize=2, strides=2, padding="SAME", name="pool1", data_format="NCHW"
    )
    conv2 = flow.layers.conv2d(
        pool1,
        64,
        5,
        padding="SAME",
        name="conv2",
        kernel_initializer=initializer,
    )
    prelu2 = flow.layers.prelu(conv2,
                            alpha_initializer=initializer,
                            shared_axes=[2, 3],
                            name="Prelu2")
    pool2 = flow.nn.max_pool2d(
        prelu2, ksize=2, strides=2, padding="SAME", name="pool2", data_format="NCHW"
    )
    reshape = flow.reshape(pool2, [pool2.shape[0], -1])
    hidden = flow.layers.dense(
        reshape,
        512,
        activation=flow.nn.relu,
        kernel_initializer=initializer,
        name="dense1",
    )
    if train:
        hidden = flow.nn.dropout(hidden, rate=0.5, name="dropout")
    return flow.layers.dense(hidden, 10, kernel_initializer=initializer, name="dense2")


@flow.global_function(type="train")
def train_job(
        images: tp.Numpy.Placeholder((BATCH_SIZE, 1, 28, 28), dtype=flow.float),
        labels: tp.Numpy.Placeholder((BATCH_SIZE,), dtype=flow.int32),
) -> tp.Numpy:
    with flow.scope.placement("gpu", "0:0"):
        logits = lenet(images, train=True)
        loss = flow.nn.sparse_softmax_cross_entropy_with_logits(
            labels, logits, name="softmax_loss"
        )

    lr_scheduler = flow.optimizer.PiecewiseConstantScheduler([], [0.1])
    flow.optimizer.SGD(lr_scheduler, momentum=0.9).minimize(loss)
    return loss
oneflow.layers.upsample_2d(x: oneflow_api.BlobDesc, size: Sequence[int] = (2, 2), data_format: str = 'NCHW', interpolation: str = 'nearest', name: str = 'Upsample2D')

The Upsample Layer, this layer can upsample the feature map to a specified scale.

Parameters
  • x ([type]) – Input Blob.

  • size (tuple, optional) – (height_scale, width_scale) Defaults to (2, 2).

  • data_format (str, optional) – A string specifies the format of the input Blob, one of “NCHW” or “NHWC” (default: “NCHW”). “NCHW” cooresponds to channels_first, i.e. the input Blob with shape (batch_size, channels, height, width). “NHWC” cooresponds to channels_last, i.e. the input Blob with shape (batch_size, height, width, channels).. Defaults to “NCHW”.

  • interpolation (str, optional) – Image interpolation algorithm to enlarge the image size. Defaults to “nearest”. “nearest” and “bilinear” are available now.

  • name ([type], optional) – This layer’s name. Defaults to None.

Raises
  • ValueError – interpolation must be “nearest” or “bilinear”.

  • ValueError – data_format must be “NHWC” or “NCHW”

Returns

oneflow_api.BlobDesc: A Blob which is the upsampled x. If size is (2, 2), the shape of return value is [N, C, 2H, 2W].

Return type

[type]

For example:

import oneflow as flow
import numpy as np
import oneflow.typing as tp


@flow.global_function()
def upsample_Job(x: tp.Numpy.Placeholder((1, 32, 32, 32))
) -> tp.Numpy:
    upsample = flow.layers.upsample_2d(
        x,
        size=(2, 2),
        name="Upsample1"
    )
    return upsample


x = np.random.randn(1, 32, 32, 32).astype(np.float32)
out = upsample_Job(x)

# out.shape (1, 32, 64, 64)