Tensor Attributes¶
Each local oneflow.Tensor
has a oneflow.dtype
, oneflow.device
, and global oneflow.Tensor
has a oneflow.dtype
, oneflow.placement
, oneflow.sbp
.
oneflow
oneflow.dtype¶
-
class
oneflow.
dtype
¶
A oneflow.dtype
is an object that represents the data type of a
oneflow.Tensor
. Oneflow has eight different data types:
Data type |
dtype |
CPU tensor |
GPU tensor |
---|---|---|---|
Boolean |
|
||
8-bit integer (unsigned) |
|
||
8-bit integer (signed) |
|
||
64-bit floating point |
|
||
32-bit floating point |
|
||
16-bit floating point |
|
||
32-bit integer (signed) |
|
||
64-bit integer (signed) |
|
To find out if a oneflow.dtype
is a floating point data type, the property is_floating_point
can be used, which returns True
if the data type is a floating point data type.
When the dtypes of inputs to an arithmetic operation (add, sub, div, mul) differ, we promote by finding the minimum dtype that satisfies the following rules:
If the type of a scalar operand is of a higher category than tensor operands (where complex > floating > integral > boolean), we promote to a type with sufficient size to hold all scalar operands of that category.
If a zero-dimension tensor operand has a higher category than dimensioned operands, we promote to a type with sufficient size and category to hold all zero-dim tensor operands of that category.
If there are no higher-category zero-dim operands, we promote to a type with sufficient size and category to hold all dimensioned operands.
A floating point scalar operand has dtype oneflow.get_default_dtype() and an integral non-boolean scalar operand has dtype oneflow.int64. Unlike numpy, we do not inspect values when determining the minimum dtypes of an operand. Quantized and complex types are not yet supported.
Promotion Examples:
>>> float_tensor = oneflow.ones(1, dtype=oneflow.float)
>>> double_tensor = oneflow.ones(1, dtype=oneflow.double)
>>> int_tensor = oneflow.ones(1, dtype=oneflow.int)
>>> long_tensor = oneflow.ones(1, dtype=oneflow.long)
>>> uint_tensor = oneflow.ones(1, dtype=oneflow.uint8)
>>> double_tensor = oneflow.ones(1, dtype=oneflow.double)
>>> bool_tensor = oneflow.ones(1, dtype=oneflow.bool)
# zero-dim tensors
>>> long_zerodim = oneflow.tensor(1, dtype=oneflow.long)
>>> int_zerodim = oneflow.tensor(1, dtype=oneflow.int)
>>> a,b=oneflow.tensor(5),oneflow.tensor(5)
>>> oneflow.add(a, b).dtype
oneflow.int64
# 5 is an int64, but does not have higher category than int_tensor so is not considered.
>>> (int_tensor + 5).dtype
oneflow.int32
>>> (int_tensor + long_zerodim).dtype
oneflow.int64
>>> (long_tensor + int_tensor).dtype
oneflow.int64
>>> (bool_tensor + long_tensor).dtype
oneflow.int64
>>> (bool_tensor + uint_tensor).dtype
oneflow.uint8
>>> (float_tensor + double_tensor).dtype
oneflow.float64
>>> (bool_tensor + int_tensor).dtype
oneflow.int32
# Since long is a different kind than float, result dtype only needs to be large enough
# to hold the float.
>>> oneflow.add(long_tensor, float_tensor).dtype
oneflow.float32
- When the output tensor of an arithmetic operation is specified, we allow casting to its dtype except that:
An integral output tensor cannot accept a floating point tensor.
A boolean output tensor cannot accept a non-boolean tensor.
A non-complex output tensor cannot accept a complex tensor
Casting Examples:
# allowed:
>>> float_tensor *= float_tensor
>>> float_tensor *= int_tensor
>>> float_tensor *= uint_tensor
>>> float_tensor *= bool_tensor
>>> int_tensor *= uint_tensor
# disallowed (RuntimeError: result type can't be cast to the desired output type):
>>> float_tensor *= double_tensor
>>> int_tensor *= float_tensor
>>> int_tensor *= long_tensor
>>> uint_tensor *= int_tensor
>>> bool_tensor *= int_tensor
>>> bool_tensor *= uint_tensor
oneflow.device¶
-
class
oneflow.
device
¶
A oneflow.device
is an object representing the device on which a oneflow.Tensor
is
or will be allocated.
The oneflow.device
contains a device type ('cpu'
or 'cuda'
) and optional device
ordinal for the device type. If the device ordinal is not present, this object will always represent
the current device for the device type, even after oneflow.cuda.set_device()
is called; e.g.,
a oneflow.Tensor
constructed with device 'cuda'
is equivalent to 'cuda:X'
where X is
the result of oneflow.cuda.current_device()
.
A oneflow.Tensor
’s device can be accessed via the Tensor.device
property.
A oneflow.device
can be constructed via a string or via a string and device ordinal
Via a string:
>>> oneflow.device('cuda:0')
device(type='cuda', index=0)
>>> oneflow.device('cpu')
device(type='cpu', index=0)
>>> oneflow.device('cuda') # current cuda device
device(type='cuda', index=0)
Via a string and device ordinal:
>>> oneflow.device('cuda', 0)
device(type='cuda', index=0)
>>> oneflow.device('cpu', 0)
device(type='cpu', index=0)
Note
The oneflow.device
argument in functions can generally be substituted with a string.
This allows for fast prototyping of code.
>>> # Example of a function that takes in a oneflow.device
>>> cuda1 = oneflow.device('cuda:1')
>>> oneflow.randn((2,3), device=cuda1)
>>> # You can substitute the oneflow.device with a string
>>> oneflow.randn((2,3), device='cuda:1')
Note
For legacy reasons, a device can be constructed via a single device ordinal, which is treated
as a cuda device. This matches Tensor.get_device()
, which returns an ordinal for cuda
tensors and is not supported for cpu tensors.
>>> oneflow.device(1)
device(type='cuda', index=1)
Note
Methods which take a device will generally accept a (properly formatted) string or (legacy) integer device ordinal, i.e. the following are all equivalent:
>>> oneflow.randn((2,3), device=oneflow.device('cuda:1'))
>>> oneflow.randn((2,3), device='cuda:1')
>>> oneflow.randn((2,3), device=1) # legacy
oneflow.placement¶
-
class
oneflow.
placement
¶ A
oneflow.placement
is an object representing the device group on which aoneflow.Tensor
is or will be allocated. Theoneflow.placement
contains a device type (‘cpu’ or ‘cuda’) and corresponding device sequence.A
oneflow.Tensor
’s placement can be accessed via the Tensor.placement property.A oneflow.placement can be constructed in several ways:
>>> import oneflow as flow >>> p = flow.placement(type="cuda", ranks=[0, 1, 2, 3]) >>> p oneflow.placement(type="cuda", ranks=[0, 1, 2, 3]) >>> p = flow.placement(type="cuda", ranks=[[0, 1], [2, 3]]) >>> p oneflow.placement(type="cuda", ranks=[[0, 1], [2, 3]])
oneflow.placement.all¶
-
oneflow.placement.
all
(device_type) → oneflow.placement¶ Returns a placement that contains all available devices.
- Parameters
device_type (str) – cuda or cpu
For examples:
# Runs on 4 ranks import oneflow as flow p = flow.placement.all("cuda") # oneflow.placement(type="cuda", ranks=[0, 1, 2, 3]) p = flow.placement.all("cpu") # oneflow.placement(type="cpu", ranks=[0, 1, 2, 3])
oneflow.env.all_device_placement¶
-
oneflow.env.
all_device_placement
(device_type) → oneflow.placement¶ Returns a placement that contains all available devices.
Note
It is recommended to use oneflow.placement.all instead of this function.
- Parameters
device_type (str) – cuda or cpu
For examples:
# Runs on 4 ranks import oneflow as flow p = flow.env.all_device_placement("cuda") # oneflow.placement(type="cuda", ranks=[0, 1, 2, 3]) p = flow.env.all_device_placement("cpu") # oneflow.placement(type="cpu", ranks=[0, 1, 2, 3])
oneflow.sbp.sbp¶
-
class
oneflow.sbp.
sbp
¶ A
oneflow.sbp
is an object representing that how the data of the global tensor is distributed across the ranks of theTensor
placement.oneflow.sbp
includes three types:oneflow.sbp.split(dim)
Indicates that the global tensor is evenly divided according to the dimension dim and distributed on each rank.
oneflow.sbp.broadcast()
Indicates that the global tensor is replicated on each rank.
oneflow.sbp.partial_sum()
Indicates that the value of the global tensor is element-wise sum of the local tensors distributed in each rank.
A
oneflow.Tensor
’s sbp can be accessed via the Tensor.sbp property.A
oneflow.sbp
can be constructed in several ways:>>> import oneflow as flow >>> s = flow.sbp.split(0) >>> s oneflow.sbp.split(dim=0) >>> b = flow.sbp.broadcast() >>> b oneflow.sbp.broadcast >>> p = flow.sbp.partial_sum() >>> p oneflow.sbp.partial_sum