yandex/AliceAI-T5-35B-A0.6B
35B • Updated • 767 • 112
Ermakov Petr
ermakovpetr
AI & ML interests
LLM, Search, Diffusion
Recent Activity
liked a model 5 days ago
yandex/AliceAI-T5-35B-A0.6B published a model 6 days ago
yandex/AliceAI-T5-35B-A0.6B updated a model 6 days ago
yandex/AliceAI-T5-35B-A0.6BOrganizations
liked a
model 5 days ago
published a
model 6 days ago
updated a
model 6 days ago
upvoted a collection 6 months ago
published a
dataset 10 months ago
liked 2
models 12 months ago
liked 2
datasets 12 months ago
upvoted a paper about 1 year ago
liked a
dataset over 1 year ago
Огромное вам спасибо. Продолжение?
10
#5 opened over 1 year ago
by 
Makar7
liked a
model over 1 year ago
reacted to artnitolog's post with 🤝🤗🚀🔥❤️👍 over 1 year ago
Post
2665
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
upvoted a paper almost 2 years ago