oneflow.Tensor.to_global

Tensor.to_global(placement=None, sbp=None, **kwargs)Tensor

Creates a global tensor if this tensor is a local tensor, otherwise performs Tensor placement and/or sbp conversion.

Note

This tensor can be local tensor or global tensor.

  • For local tensor

    Both placement and sbp are required.

    The returned global tensor takes this tensor as its local component in the current rank.

    There is no data communication usually, but when sbp is oneflow.sbp.broadcast, the data on rank 0 will be broadcast to other ranks.

  • For global tensor

    At least one of placement and sbp is required.

    If placement and sbp are all the same as this tensor’s own placement and sbp, then returns this tensor own.

Warning

When the input tensor is a local tensor and sbp is oneflow.sbp.broadcast, the data on the non-0 rank will be modified. If you want to keep the input local tensor unchanged, please set the arg copy to True.

Parameters
  • placement (flow.placement, optional) – the desired placement of returned global tensor. Default: None

  • sbp (flow.sbp.sbp or tuple of flow.sbp.sbp, optional) – the desired sbp of returned global tensor. Default: None

Keyword Arguments
  • grad_sbp (flow.sbp.sbp or tuple of flow.sbp.sbp, optional) – manually specify the sbp of this tensor’s grad tensor in the backward pass. If None, the grad tensor sbp will be infered automatically. It is only used if this tensor is a global tensor. Default: None

  • check_meta (bool, optional) – indicates whether to check meta information. If set to True, check the input meta information on each rank. Default: True if this tensor is a local tensor, False if this tensor is a global tensor

  • copy (bool, optional) – When copy is set, copy occurres in this operation. For local tensor, the returned global tensor takes the replication of this tensor as its local component in the current rank. For global tensor, a new Tensor is created even when the Tensor already matches the desired conversion. Default: False

For local tensor:

>>> # Run on 2 ranks respectively
>>> import oneflow as flow
>>> input = flow.tensor([0., 1.], dtype=flow.float32) 
>>> output = input.to_global(placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.split(0)], check_meta=False) 
>>> print(output.size()) 
>>> print(output) 
>>> # results on rank 0
oneflow.Size([4])
tensor([0., 1., 0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)
>>> # results on rank 1
oneflow.Size([4])
tensor([0., 1., 0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)

For global tensor:

>>> # Run on 2 ranks respectively
>>> import oneflow as flow
>>> input = flow.tensor([0., 1.], dtype=flow.float32, placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.broadcast]) 
>>> output = input.to_global(placement=flow.placement("cpu", ranks=[0, 1]), sbp=[flow.sbp.split(0)]) 
>>> print(output.size()) 
>>> print(output) 
>>> # results on rank 0
oneflow.Size([2])
tensor([0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)
>>> # results on rank 1
oneflow.Size([2])
tensor([0., 1.], placement=oneflow.placement(type="cpu", ranks=[0, 1]), sbp=(oneflow.sbp.split(dim=0),), dtype=oneflow.float32)