Description
Describe the bug
Has anyone successfully trained HiDream-I1 using the diffusers repository? I trained the lora model of HiDream-I1 with the default learning rate. The output images were extremely prone to crash and even turned into noisy images. However, this problem did not occur when using the ai-toolkit repository with the same learning rate. Which Settings should I adjust?
Note:
I modified the data set loading method, but the same data set loading method does not cause this problem when used for the training of the Flux, so it should not be caused by this part of the modification.
Please help me, Thanks!
Reproduction
export MODEL_NAME="./HiDream/HiDream-E1-Full"
export INSTANCE_DIR="./train_tag.json"
export OUTPUT_DIR="./HiDream-I1/trained-hidream-lora-8"
CUDA_VISIBLE_DEVICES=3 python ./HiDream-I1/train_dreambooth_lora_hidream.py
--pretrained_model_name_or_path=$MODEL_NAME
--pretrained_tokenizer_4_name_or_path="./meta-llama/Llama-3.1-8B-Instruct"
--pretrained_text_encoder_4_name_or_path="./meta-llama/Llama-3.1-8B-Instruct"
--dataset_name=$INSTANCE_DIR
--output_dir=$OUTPUT_DIR
--image_column="image"
--caption_column="text"
--mixed_precision="bf16"
--resolution=1024
--train_batch_size=1
--gradient_accumulation_steps=4
--use_8bit_adam
--rank=8
--learning_rate=2e-4
--report_to="tensorboard"
--lr_scheduler="constant_with_warmup"
--lr_warmup_steps=100
--max_train_steps=50000
--checkpointing_steps=1000
--seed="0"
--offload \
Logs
System Info
Package Version Editable project location
accelerate 1.3.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiosignal 1.3.2
annotated-types 0.7.0
async-timeout 5.0.1
attrs 25.1.0
bitsandbytes 0.45.4
blessed 1.20.0
certifi 2024.12.14
charset-normalizer 3.4.1
contourpy 1.3.2
cycler 0.12.1
datasets 3.2.0
deepspeed 0.16.3
diffusers 0.33.0.dev0 /code/diffusers
dill 0.3.8
einops 0.8.0
filelock 3.17.0
fonttools 4.58.4
frozenlist 1.5.0
fsspec 2024.9.0
gpustat 1.1.1
hjson 3.1.0
huggingface-hub 0.27.1
idna 3.10
importlib_metadata 8.6.1
Jinja2 3.1.5
kiwisolver 1.4.8
MarkupSafe 3.0.2
matplotlib 3.10.3
modelscope 1.22.3
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
networkx 3.4.2
ninja 1.11.1.3
numpy 2.2.2
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-ml-py 12.560.30
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
packaging 24.2
pandas 2.2.3
peft 0.15.0
pillow 11.1.0
pip 24.2
prodigyopt 1.1.2
propcache 0.2.1
protobuf 5.29.3
psutil 6.1.1
py-cpuinfo 9.0.0
pyarrow 19.0.0
pydantic 2.10.6
pydantic_core 2.27.2
pyparsing 3.2.3
python-dateutil 2.9.0.post0
pytz 2025.1
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
safetensors 0.5.2
sentencepiece 0.2.0
setuptools 75.1.0
six 1.17.0
sympy 1.13.1
tokenizers 0.21.0
torch 2.5.1
torchvision 0.20.1
tqdm 4.67.1
transformers 4.48.1
triton 3.1.0
typing_extensions 4.12.2
tzdata 2025.1
urllib3 2.3.0
wcwidth 0.2.13
wheel 0.44.0
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
Who can help?
No response