• English
    • Tiếng Việt
  • English 
    • English
    • Tiếng Việt
  • Login
View Item 
  •   VinSpace Home
  • The College of Health Sciences
  • Nguyen Thi Tuyet, MD., PhD.
  • View Item
  •   VinSpace Home
  • The College of Health Sciences
  • Nguyen Thi Tuyet, MD., PhD.
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Inhibition of mitochondrial phosphate carrier prevents high phosphate-induced superoxide generation and vascular calcification

Thumbnail
View/Open
Inhibition of mitochondrial phosphate carrier prevents high phosphate-induced superoxide generation and vascular calcification.pdf (3.098Mb)
Date
2023
Author
Nguyen, Thi Nhung
Nguyen, Thi Tuyet
Nguyen, Thu Ha
Lee, Ji-Min
Kim, Min-Ji
Qi, Xu-Feng
Cha, Seung-Kuy
Lee, In-Kyu
Park, Kyu-Sang
Metadata
Show full item record
Abstract
Multi-head attention is a driving force behind state-of-the-art transformers, which achieve remarkable performance across a variety of natural language processing (NLP) and computer vision tasks. It has been observed that for many applications, those attention heads learn redundant embedding, and most of them can be removed without degrading the performance of the model. Inspired by this observation, we propose Transformer with a Mixture of Gaussian Keys (Transformer-MGK), a novel transformer architecture that replaces redundant heads in transformers with a mixture of keys at each head. These mixtures of keys follow a Gaussian mixture model and allow each attention head to focus on different parts of the input sequence efficiently. Compared to its conventional transformer counterpart, Transformer-MGK accelerates training and inference, has fewer parameters, and requires fewer FLOPs to compute while achieving comparable or better accuracy across tasks. Transformer-MGK can also be easily extended to use with linear attention. We empirically demonstrate the advantage of Transformer-MGK in a range of practical applications, including language modeling and tasks that involve very long sequences. On the Wikitext-103 and Long Range Arena benchmark, Transformer-MGKs with 4 heads attain comparable or better performance to the baseline transformers with 8 heads.
URI
https://vinspace.edu.vn/handle/VIN/298
Collections
  • Nguyen Thi Tuyet, MD., PhD. [5]

Contact Us | Send Feedback
 

 

Browse

All of VinSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

LoginRegister

Contact Us | Send Feedback