8000 Lama2 70B-chat is not supported? · Issue #22 · FasterDecoding/REST · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Lama2 70B-chat is not supported? #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Chlience opened this issue Oct 17, 2024 · 2 comments
Open

Lama2 70B-chat is not supported? #22

Chlience opened this issue Oct 17, 2024 · 2 comments

Comments

@Chlience
Copy link

When I run:

RAYON_NUM_THREADS=6 CUDA_VISIBLE_DEVICES=0,1,2,3 python3 gen_model_answer_rest.py --model-path /models/LLAMA-2-series/llama-2-70b-chat --model-id llama-2-70b-chat --datastore-path ../datastore/datastore_chat_small.idx

I get:
(delete try and except)

Traceback (most recent call last):
  File "/rest/llm_judge/gen_model_answer_rest.py", line 497, in <module>
    run_eval(
  File "/rest/llm_judge/gen_model_answer_rest.py", line 154, in run_eval
    get_answers_func(
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/rest/llm_judge/gen_model_answer_rest.py", line 231, in get_model_answers
    output_ids, new_token, idx, _, start_time = rest_forward(
  File "/rest/llm_judge/gen_model_answer_rest.py", line 57, in rest_forward
    logits = initialize_logits(
  File "/rest/llm_judge/../rest/model/utils.py", line 49, in initialize_logits
    outputs, logits = model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/rest/llm_judge/../rest/model/rest_model.py", line 92, in forward
    outputs = self.base_model.model(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/rest/llm_judge/../rest/model/modeling_llama_kv.py", line 1044, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/rest/llm_judge/../rest/model/modeling_llama_kv.py", line 720, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/rest/llm_judge/../rest/model/modeling_llama_kv.py", line 594, in forward
    key_states = past_key_value[0].cat(key_states, dim=2)
  File "/rest/llm_judge/../rest/model/kv_cache.py", line 66, in cat
    dst.copy_(tensor)
RuntimeError: The size of tensor a (64) must match the size of tensor b (8) at non-singleton dimension 1
@zhenyuhe00
Copy link
Collaborator

Hi, It's caused by the group-query attention. You could consider using the llama3 branch with group-query attention fixed in this pr.

@scandukuri
Copy link
scandukuri commented Nov 22, 2024 •
edited
Loading

Yes re: llama3 branch suggestion - although note that only single GPU inference seems to be supported for REST model if I'm not mistaken so 70B model will have to fit on a single device. This is because the KV caching approach is nontrivial to implement parallelism for (?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Chlience @zhenyuhe00 @scandukuri
0