8000 Why use layer norm instead of RMS norm? · Issue #61 · mit-han-lab/omniserve · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
Why use layer norm instead of RMS norm? #61
Open
@YudiZh

Description

@YudiZh

In the implementation of w4a8 and w8a8, it seems that you have used layer normalization instead of RMS normalization. Why does this direct substitution of the normalization method work? Additionally, after testing and comparing the results of RMS_norm and layer_norm on w4a8_per_chn model, it was found that layer_norm seems to perform better on GSM8K. Could you explain this in more detail?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0