2024-06-16 · conference-paper · Multimodal Machine Learning Applications · en
highly citedfrom the source record
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this paper, we present the first systematic study to investigate the design choices of LMMs in a controlled setting under the LLaVA framework. We show that the fully-connected vision-language connector in LLaVA is surprisingly power-ful and data-efficient. With simple modifications to LLa VA, namely, using CLIP- ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~ 1 day on a single 8-AI00 node. Furthermore, we present some early exploration of open problems in LMMs, including scaling to higher resolution inputs, compositional capabilities, and model hallucination, etc. We hope this makes state-of-the-art LMM research more accessible. Code and model will be publicly available.
Claims drawn from cited facts, not live model generation.
This work concerns multimodal machine learning applications and appears as a conference paper, which implies it has passed peer review and should be given the credibility of a formally published conference contribution. Its conference-paper status means readers can treat its findings as having cleared peer review rather than as an unreviewed preprint.
3 cited facts
This work was produced by a small team of 3 researchers.
1 cited fact
The work has received 1,708 citations, placing it among highly cited works and marking it as an established reference point in its field. It references 53 works, indicating the breadth of prior literature it draws upon.
3 cited facts
Improved Baselines with Visual Instruction Tuning is a conference-paper, focusing on Multimodal Machine Learning Applications, published in 2024.
Improved Baselines with Visual Instruction Tuning has 1,708 citations in our record, placing it among the highly cited works we track.
Our record lists 3 authors for Improved Baselines with Visual Instruction Tuning and references 53 other works. Author names aren't in our data yet — the source link on the page lists the full byline.
Source facts, citations, and refresh stamp for this record.
Sources: openalex_works