How to use PyTorch Context Parallel for long-context transformer training
PyTorch Context Parallel shards long sequences across devices so each rank only holds a context slice for attention and KV handling — this makes 1M-token training feasible in the PyTorch/Torchtitan workflow — but it is still a distributed training feature that depends on correct process-group setup, NCCL communication, and long-context-aware model partitioning.