How to train or finetune in caffe?

hello, I want to train ICNet with pascal voc dataset in caffe. I have changed the number of class in .prototxt, and written the solver.pototxt according to the paper. But the loss also around 0.5, and the accuracy is too low.
Because my GPU is 8G memory, I can only set mini_batchsize to 8, but by setting iter_size to 2. So the batch size is 16. Can you help me.
Looking forward to your reply！