PyTorch, one of the most widely used machine learning (ML) techniques, has announced the first experimental release of its next-generation series, PyTorch 2.0, a major leap forward. This new release aims to assist in speed machine learning training and development while being backwards compatible with the current PyTorch application code.
PyTorch was originally a machine learning framework developed by Meta, but its development was transferred to the newly established PyTorch Foundation as part of the Linux Foundation in September 2022. This 2.0 release is the first major update after the transfer.
PyTorch 2.0 maintains the same Eager Mode development and user experience but radically changes and powers up how PyTorch works at the compiler level.
The latest publicly available release of PyTorch is version 1.13, which was released at the end of October. With code contributions from IBM, this release makes significant changes that will allow machine learning frameworks to work more effectively with general-purpose ethernet-based networking for large-scale workloads.
Pytorch 2.0 introduces a new paradigm called torch.compile that will give users solid speedups, which was impossible with PyTorch 1.0’s default Eager Mode.
About 160 open-source models tested by the PyTorch project on early builds of 2.0 reported a 43% speedup, with one line addition to the codebase working reliably.
With PyTorch 2.0, developers will start experimenting with Eager Mode, and once they are done training models for a long time, they will turn on Compile Mode to further improve performance.
By now, you probably know that PyTorch got its name from Python, an open-source programming language widely used in data science. However, recent PyTorch releases are not entirely written in Python, and part of the framework is written in the C++ programming language. Performance improvements have been achieved by moving many parts of torch.nn from Python to C++.
However, the PyTorch project will move the torch.nn related code back to Python in the later updates. C++ is generally faster than Python, but the new compiler (torch.compile) is faster than running equivalent code in C++.
Development of Pytorch 2.0 will continue over the next few months, with general availability scheduled for March 2023.
