ros/LlamaFactory
1
0
Fork 0
mirror of https://github.com/hiyouga/LlamaFactory.git synced 2026-02-03 08:53:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

[data] Fix bugs of use_audio_in_video in Qwen2.5 Omni (#7638)

Browse Source 
* cache _mm_inputs

* nit

* support for use_audio_in_video

* remove cache

* fix data

* Update mllm_video_audio_demo.json
This commit is contained in:
Kingsley
2025-04-08 18:40:10 +08:00
committed by GitHub
parent acb09fa3a3
commit 349c56c51c
5 changed files with 11 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/llamafactory/data/collator.py
View File

@@ -184,6 +184,7 @@ class MultiModalDataCollatorForSeq2Seq(DataCollatorForSeq2Seq):
rope_index_kwargs["second_per_grids"] = mm_inputs.get("video_second_per_grid")
if getattr(self.model.config, "model_type", None) == "qwen2_5_omni_thinker": # for qwen2omni
rope_index_kwargs["use_audio_in_video"] = getattr(self.processor, "use_audio_in_video", False)
feature_attention_mask = mm_inputs.get("feature_attention_mask", None)
if feature_attention_mask is not None:
audio_feature_lengths = torch.sum(