feat(trainer_builder): refactor trainer_builder and preserve optional callable for custom model dispatch function in isp mode

[season0528]

Motivation

1. Improve readability of trainer_builder, a.k.a, make the code self-documenting.
2. preserve optional callable for custom model_dispatch_func, might be useful when you try to integrate huggingface models with ISP.

Modification

internlm/core/trainer_builder.py
internlm/core/parallel/comm/isp.py
internlm/model/builder.py
internlm/model/ops/attention.py
internlm/train/pipeline.py

BC-breaking (Optional)

None

Use cases (Optional)

None

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[season0528]

Experimental results for acc/loss alignment:

Code base:
InternEvo-HFModels 6bfd957
InternEvo 7696aac

1. Huggingface InternLM2 pack mode, w/wo ISP. (800 steps, perfectly aligned, diff is e-3~e-2)

2. Huggingface InternLM2 unpack mode, w/wo ISP. (700 steps, perfectly aligned, diff is e-3~e-2)

3. Huggingface InternLM1 pack mode, w/wo ISP. (500 steps, perfectly aligned, diff is e-3~e-2)

4. Huggingface InternLM1 unpack mode, w/wo ISP. (700steps, might not perfectly aligned, diff is e-2~e-1)

[season0528]

ISP adaption code: https://github.com/InternLM/InternEvo-HFModels/commit/6bfd9576005817e74302000ffa35567aca8260b4
ISP adaption docs: https://github.com/InternLM/InternEvo-HFModels/blob/main/huggingface_model/README.md

For other huggingface models, just refer to the adaption code/docs of huggingface InternLM1 and InternLM2. (Just need to modify few lines of code)

[season0528]

Status (Done.):

• ISP adaption code.
• ISP adaption docs.

[season0528]

weight parallel also enabled and acc/loss aligned https://github.com/InternLM/InternEvo-HFModels/tree/enable_wp

[season0528]

Close this PR, since we choose to drop the repo https://github.com/InternLM/InternEvo-HFModels