First Paper Friday: Haobo Zhang

University of Michigan School of Information doctoral student Haobo Zhang has published his first paper as a UMSI student. The paper examines a key problem in the development of large language models (LLMs), and addresses how to merge multiple task specific models into one efficient multi-task model without sacrificing performance. 

“Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging” will be presented at the 63rd Annual Meeting of the Association for Computational Linguistics. The paper is also authored by UMSI associate professor Jiayu Zhou, Zhang’s adviser. 

The publication of a PhD student’s first paper is a big milestone in their career, initiating them into the scholarly community as producers of knowledge. UMSI supports their work as part of our mission to share knowledge.  

“Our paper proposed how to merge multiple task-specific LoRA modules into a single LLM model so that the model can be equipped with skills for different domains,” Zhang says. “It showed that the critical point is the interference between model parameters and input data across different domains. We address this issue by constraining the LoRA modules in orthogonal spaces to reduce the interference.”

By managing how each LoRA (Low-Rank Adaption) module adapts to its task, Zhang’s approach allows a single model to seamlessly learn and switch between multiple specialized skills without losing accuracy or efficiency.

Zhang, a first year PhD student, previously worked in trustworthy AI. He received his master’s degree from Michigan State University before joining UMSI for his PhD. 

“Currently, I’m interested in LLM’s reasoning ability and its application in multi-agent systems,” he says. “I aim to understand, enhance and apply LLM’s reasoning skills to build a multi-agent system for health or finance.” 

Zhang is expected to graduate in December 2029. In the meantime, he says he’s enjoying “the diversity of thought as well as the friendly and passionate environment” at UMSI. 

Read “Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging” on arXiv, and read the abstract below: 

Fine-tuning large language models (LMs) for individual tasks yields strong performance but is expensive for deployment and storage. Recent works explore model merging to combine multiple task-specific models into a single multi-task model without additional training. However, existing merging methods often fail for models fine-tuned with low-rank adaptation (LoRA), due to significant performance degradation. In this paper, we show that this issue arises from a previously overlooked interplay between model parameters and data distributions. We propose Orthogonal Subspaces for Robust model Merging (OSRM) to constrain the LoRA subspace prior to fine-tuning, ensuring that updates relevant to one task do not adversely shift outputs for others. Our approach can seamlessly integrate with most existing merging algorithms, reducing the unintended interference among tasks. Extensive experiments on eight datasets, tested with three widely used LMs and two large LMs, demonstrate that our method not only boosts merging performance but also preserves single-task accuracy. Furthermore, our approach exhibits greater robustness to the hyperparameters of merging. These results highlight the importance of data-parameter interaction in model merging and offer a plug-and-play solution for merging LoRA models. 

RELATED

Follow Haobo Zhang’s PhD journey at UMSI by visiting his profile. 

Learn more information about UMSI’s PhD in Information program.