Publications

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
We propose a smart quantized initialization for LoRA fine-tuning.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, Tuo Zhao
May 7, 2024
Cite PDF Code Slides Poster Models
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximatione
We propose a compression method that we approximate a weight matrix by a low-rank matrix and a sparse matrix.
Yixiao Li, Yifan Yu, Qiangru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao
Jun 26, 2023
PDF Cite Code Poster
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximatione