oneflow.Tensor.to_global¶
-
Tensor.
to_global
(placement=None, sbp=None, **kwargs) → Tensor¶ Creates a global tensor if this tensor is a local tensor, otherwise performs Tensor placement and/or sbp conversion.
Note
This tensor can be local tensor or global tensor.
For local tensor
Both placement and sbp are required.
The returned global tensor takes this tensor as its local component in the current rank.
There is no data communication usually, but when sbp is
oneflow.sbp.broadcast
, the data on rank 0 will be broadcast to other ranks.For global tensor
At least one of placement and sbp is required.
If placement and sbp are all the same as this tensor’s own placement and sbp, then returns this tensor own.
Warning
When the input tensor is a local tensor and sbp is
oneflow.sbp.broadcast
, the data on the non-0 rank will be modified. If you want to keep the input local tensor unchanged, please set the arg copy to True.- Parameters
placement (flow.placement, optional) – the desired placement of returned global tensor. Default: None
sbp (flow.sbp.sbp or tuple of flow.sbp.sbp, optional) – the desired sbp of returned global tensor. Default: None
- Keyword Arguments
grad_sbp (flow.sbp.sbp or tuple of flow.sbp.sbp, optional) – manually specify the sbp of this tensor’s grad tensor in the backward pass. If None, the grad tensor sbp will be infered automatically. It is only used if this tensor is a global tensor. Default: None
check_meta (bool, optional) – indicates whether to check meta information. If set to True, check the input meta information on each rank. Default: True if this tensor is a local tensor, False if this tensor is a global tensor
copy (bool, optional) – When copy is set, copy occurres in this operation. For local tensor, the returned global tensor takes the replication of this tensor as its local component in the current rank. For global tensor, a new Tensor is created even when the Tensor already matches the desired conversion. Default: False
For local tensor:
>>> # Run on 2 ranks respectively >>> import oneflow as flow >>> input = flow.tensor([0., 1.], dtype=flow.float32) >>> output = input.to_global(placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.split(0)], check_meta=False) >>> print(output.size()) >>> print(output)
>>> # results on rank 0 oneflow.Size([4]) tensor([0., 1., 0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)
>>> # results on rank 1 oneflow.Size([4]) tensor([0., 1., 0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)
For global tensor:
>>> # Run on 2 ranks respectively >>> import oneflow as flow >>> input = flow.tensor([0., 1.], dtype=flow.float32, placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.broadcast]) >>> output = input.to_global(placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.split(0)]) >>> print(output.size()) >>> print(output)
>>> # results on rank 0 oneflow.Size([2]) tensor([0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)
>>> # results on rank 1 oneflow.Size([2]) tensor([0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)