8000 add example files for submission round v5.1 by anandhu-eng · Pull Request #2240 · mlcommons/inference · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

add example files for submission round v5.1 #2240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 7, 2025
Merged

add example files for submission round v5.1 #2240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Binary file added submission_round_v5.1/.DS_Store
View file
Open in desktop
Binary file not shown.
Binary file added submission_round_v5.1/1233270a89e8-reference-gpu-vllm-vdefault-default_config/.DS_Store
View file
Open in desktop
Binary file not shown.
Binary file added ...vdefault-default_config/1233270a89e8-reference-gpu-vllm-vdefault-default_config/.DS_Store
View file
Open in desktop
Binary file not shown.
Binary file added ...ault_config/1233270a89e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/.DS_Store
10000
View file
Open in desktop
Binary file not shown.
Binary file added ...fig/1233270a89e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/.DS_Store
View file
Open in desktop
Binary file not shown.
31 changes: 31 additions & 0 deletions ...ference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*

## Host platform

* OS version: Linux-5.10.0-34-cloud-amd64-x86_64-with-glibc2.35
* CPU version: x86_64
* Python version: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0]
* MLC version: unknown

## CM Run Command

See [CM installation guide](https://docs.mlcommons.org/inference/install/).

```bash
pip install -U mlcflow

mlc rm cache -f

mlc pull repo anandhu-eng@mlperf-automations --checkout=b994f10dc088cbd2d826163027c864facd6a6cfa


```
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf,
you should simply reload anandhu-eng@mlperf-automations without checkout and clean MLC cache as follows:*

```bash
mlc rm repo anandhu-eng@mlperf-automations
mlc pull repo anandhu-eng@mlperf-automations
mlc rm cache -f

```
4 changes: 4 additions & 0 deletions ...89e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/accuracy.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

Results

{'rouge1': '38.8286', 'rouge2': '15.964', 'rougeL': '24.5261', 'rougeLsum': '35.8828', 'gen_len': np.int64(8183389), 'gen_num': 13368}
120,588 changes: 120,588 additions & 0 deletions ...a89e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/console.out
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 29 additions & 0 deletions ...9e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/cpu_info.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"MLC_HOST_CPU_WRITE_PROTECT_SUPPORT": "yes",
"MLC_HOST_CPU_MICROCODE": "0xffffffff",
"MLC_HOST_CPU_FPU_SUPPORT": "yes",
"MLC_HOST_CPU_FPU_EXCEPTION_SUPPORT": "yes",
"MLC_HOST_CPU_BUGS": "spectre_v1 spectre_v2 spec_store_bypass swapgs taa mmio_stale_data retbleed eibrs_pbrsb",
"MLC_HOST_CPU_TLB_SIZE": "Not Found",
"MLC_HOST_CPU_CFLUSH_SIZE": "64",
"MLC_HOST_CPU_ARCHITECTURE": "x86_64",
"MLC_HOST_CPU_TOTAL_CORES": "12",
"MLC_HOST_CPU_ON_LINE_CPUS_LIST": "0-11",
"MLC_HOST_CPU_VENDOR_ID": "GenuineIntel",
"MLC_HOST_CPU_MODEL_NAME": "Intel(R) Xeon(R) CPU @ 2.20GHz",
"MLC_HOST_CPU_FAMILY": "6",
"MLC_HOST_CPU_THREADS_PER_CORE": "2",
"MLC_HOST_CPU_PHYSICAL_CORES_PER_SOCKET": "6",
"MLC_HOST_CPU_SOCKETS": "1",
"MLC_HOST_CPU_L1D_CACHE_SIZE": "192 KiB (6 instances)",
"MLC_HOST_CPU_L1I_CACHE_SIZE": "192 KiB (6 instances)",
"MLC_HOST_CPU_L2_CACHE_SIZE": "6 MiB (6 instances)",
"MLC_HOST_CPU_L3_CACHE_SIZE": "38.5 MiB (1 instance)",
"MLC_HOST_CPU_NUMA_NODES": "1",
"MLC_HOST_CPU_START_CORES": "0",
"MLC_HOST_CPU_TOTAL_LOGICAL_CORES": "12",
"MLC_HOST_CPU_TOTAL_PHYSICAL_CORES": "6",
"MLC_HOST_CPU_PHYSICAL_CORES_LIST": "0-5",
"MLC_HOST_MEMORY_CAPACITY": "85G",
"MLC_HOST_DISK_CAPACITY": "1.1T"
}
111 changes: 111 additions & 0 deletions ...9e8-reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/graph.graphml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
<?xml version='1.0' encoding='utf-8'?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
<graph edgedefault="directed">
<node id="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" />
<node id="detect,os" />
<node id="get,sys-utils-cm" />
<node id="get,python" />
<node id="get,mlcommons,inference,src,_branch.whisper-fix,_branch.master,_repo.https://github.com/anandhu-eng/inference" />
<node id="get-mlperf-inference-utils,e341e5f86d8342e5" />
<node id="get,mlperf,inference,src,_branch.whisper-fix,_repo.https://github.com/anandhu-eng/inference" />
<node id="get,mlperf,inference,utils" />
<node id="get-cuda-devices,7a3ede4d3558427a ( with-pycuda )" />
<node id="get,cuda,_toolkit" />
<node id="get,python3" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.pycuda )" />
<node id="get,generic-python-lib,_package.pycuda" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.numpy )" />
<node id="get,generic-python-lib,_package.numpy" />
<node id="get,cuda-devices,_with-pycuda" />
<node id="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" />
<node id="detect-cpu,586c8a43320142f7" />
<node id="detect,cpu" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.vllm )" />
<node id="get,generic-python-lib,_package.vllm" />
<node id="get,ml-model,llama3,_hf,_meta-llama/Llama-3.1-8B-Instruct" />
<node id="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" />
<node id="get-mlperf-inference-sut-configs,c2fbf72009e2445b" />
<node id="get,cache,dir,_name.mlperf-inference-sut-configs" />
<node id="get,sut,configs" />
<node id="get,mlcommons,inference,src,_branch.whisper-fix,_repo.https://github.com/anandhu-eng/inference" />
<node id="generate,user-conf,mlperf,inference,_wg-inference" />
<node id="get,loadgen,_wg-inference" />
<node id="get,mlcommons,inference,src" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.psutil )" />
<node id="get,generic-python-lib,_package.psutil" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.transformers )" />
<node id="get,generic-python-lib,_package.transformers" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.sentencepiece )" />
<node id="get,generic-python-lib,_package.sentencepiece" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.accelerate )" />
<node id="get,generic-python-lib,_package.accelerate" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.pybind11 )" />
<node id="get,generic-python-lib,_package.pybind11" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.nltk )" />
<node id="get,generic-python-lib,_package.nltk" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.evaluate )" />
<node id="get,generic-python-lib,_package.evaluate" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.absl-py )" />
<node id="get,generic-python-lib,_package.absl-py" />
<node id="get-generic-python-lib,94b62a682bc44791 ( package.rouge-score )" />
<node id="get,generic-python-lib,_package.rouge-score" />
<node id="get,dataset,cnndm,_validation,_datacenter,_llama3,_mlc,_rclone" />
<node id="benchmark-program,19f369ef47084895" />
<node id="benchmark-program-mlperf,cfff0132a8aa4018" />
<node id="benchmark-program,program" />
<node id="benchmark-mlperf" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="detect,os" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="get,sys-utils-cm" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="get,python" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="get,mlcommons,inference,src,_branch.whisper-fix,_branch.master,_repo.https://github.com/anandhu-eng/inference" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="get,mlperf,inference,utils" />
<edge source="app-mlperf-inference,d775cac873ee4231 ( reference,_llama3_1-8b,_vllm,_cuda,_valid,_r5.1-dev_default,_float16,_offline )" target="get,cuda-devices,_with-pycuda" />
<edge source="get-mlperf-inference-utils,e341e5f86d8342e5" target="get,mlperf,inference,src,_branch.whisper-fix,_repo.https://github.com/anandhu-eng/inference" />
<edge source="get-cuda-devices,7a3ede4d3558427a ( with-pycuda )" target="get,cuda,_toolkit" />
<edge source="get-cuda-devices,7a3ede4d3558427a ( with-pycuda )" target="get,python3" />
<edge source="get-cuda-devices,7a3ede4d3558427a ( with-pycuda )" target="get,generic-python-lib,_package.pycuda" />
<edge source="get-cuda-devices,7a3ede4d3558427a ( with-pycuda )" target="get,generic-python-lib,_package.numpy" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.pycuda )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.numpy )" target="get,python3" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="detect,os" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="detect,cpu" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,sys-utils-cm" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,python" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.vllm" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,ml-model,llama3,_hf,_meta-llama/Llama-3.1-8B-Instruct" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="generate,user-conf,mlperf,inference,_wg-inference" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,loadgen,_wg-inference" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,mlcommons,inference,src,_branch.whisper-fix,_repo.https://github.com/anandhu-eng/inference" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,mlcommons,inference,src" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.psutil" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.transformers" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.sentencepiece" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.accelerate" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.pybind11" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.nltk" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.evaluate" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.absl-py" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,generic-python-lib,_package.rouge-score" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="get,dataset,cnndm,_validation,_datacenter,_llama3,_mlc,_rclone" />
<edge source="app-mlperf-inference-mlcommons-python,ff149e9781fc4b65 ( cuda,_vllm,_offline,_llama3_1-8b,_float16 )" target="benchmark-mlperf" />
<edge source="detect-cpu,586c8a43320142f7" target="detect,os" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.vllm )" target="get,python3" />
<edge source="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" target="detect,os" />
<edge source="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" target="detect,cpu" />
<edge source="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" target="get,python" />
<edge source="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" target="get,sut,configs" />
<edge source="generate-mlperf-inference-user-conf,3af4475745964b93 ( wg-inference )" target="get,mlcommons,inference,src,_branch.whisper-fix,_repo.https://github.com/anandhu-eng/inference" />
<edge source="get-mlperf-inference-sut-configs,c2fbf72009e2445b" target="get,cache,dir,_name.mlperf-inference-sut-configs" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.psutil )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.transformers )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.sentencepiece )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.accelerate )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.pybind11 )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.nltk )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.evaluate )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.absl-py )" target="get,python3" />
<edge source="get-generic-python-lib,94b62a682bc44791 ( package.rouge-score )" target="get,python3" />
<edge source="benchmark-program,19f369ef47084895" target="detect,cpu" />
<edge source="benchmark-program-mlperf,cfff0132a8aa4018" target="benchmark-program,program" />
</graph>
</graphml>
7 changes: 7 additions & 0 deletions ...reference-gpu-vllm-vdefault-default_config/llama3_1-8b/offline/accuracy/measurements.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"starting_weights_filename": "TBD",
"retraining": "no",
"input_data_types": "fp32",
"weight_data_types": "fp32",
"weight_transformations": "none"
}
Loading
0