Summary:
Replace our current custom transformer implementation with the inbuilt transformer available in PyTorch 2.0 for better maintainability and potential performance improvements.
Detailed Description
We currently utilize a custom-built transformer model for various tasks within our project. With the release of PyTorch 2.0, there's an opportunity to replace this custom implementation with the inbuilt transformer provided by PyTorch.
Benefits
Maintainability: Leveraging PyTorch's inbuilt functionalities would reduce the maintenance overhead.
Performance: PyTorch's implementation is highly optimized and could offer performance benefits.
Community Support: Using a widely-adopted library can make it easier to find solutions to potential issues and enable smoother collaboration.
Proposed Changes
Identify all instances in the codebase where the custom transformer is used. (see src/rydberggpt/models/rydberg_encoder_decoder/)
Evaluate the feasibility of replacing each instance with PyTorch's transformer.
Update the code and perform rigorous testing to ensure that the new implementation meets or exceeds the current performance metrics.
Additional Resources
PyTorch 2.0 Transformer DocumentationInstead of using our custom transformer implementation, replace custom transformer with pytorch 2.0 inbuild transformer.
Summary:
Replace our current custom transformer implementation with the inbuilt transformer available in PyTorch 2.0 for better maintainability and potential performance improvements.
Detailed Description
We currently utilize a custom-built transformer model for various tasks within our project. With the release of PyTorch 2.0, there's an opportunity to replace this custom implementation with the inbuilt transformer provided by PyTorch.
Benefits
Maintainability: Leveraging PyTorch's inbuilt functionalities would reduce the maintenance overhead.
Performance: PyTorch's implementation is highly optimized and could offer performance benefits.
Community Support: Using a widely-adopted library can make it easier to find solutions to potential issues and enable smoother collaboration.
Proposed Changes
Identify all instances in the codebase where the custom transformer is used. (see src/rydberggpt/models/rydberg_encoder_decoder/)
Evaluate the feasibility of replacing each instance with PyTorch's transformer.
Update the code and perform rigorous testing to ensure that the new implementation meets or exceeds the current performance metrics.
Additional Resources
PyTorch 2.0 Transformer DocumentationInstead of using our custom transformer implementation, replace custom transformer with pytorch 2.0 inbuild transformer.