Fix more integration tests for important models#45254

ydshieh wants to merge 1 commit intomainhuggingface/transformers:mainfrom

fix_level_2huggingface/transformers:fix_level_2Copy head branch name to clipboard