8000 [Benchmark] Adding MVTamperBench by amitbcp · Pull Request #5 · srikant86panda/VLMEvalKit · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Benchmark] Adding MVTamperBench #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account r 8000 elated emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 108 commits into
base: main
Choose a base branch
from
Open

[Benchmark] Adding MVTamperBench #5

wants to merge 108 commits into from

Conversation

amitbcp
Copy link
Collaborator
@amitbcp amitbcp commented Jan 8, 2025

srikant86panda and others added 30 commits November 11, 2024 15:43
@srikant86panda
MVBench data prep fixes
be173fc
@srikant86panda
handle missing video file with mvbench
6aaddc5
@srikant86panda
MVTamperBench base
101a33c
@FangXinyu-0913
merge main and add exception information in generate tsv
d5918e0
@FangXinyu-0913
add nturgb-d information to generate tsv function
9471a46
@srikant86panda
sample dataset
d0d0dd6
@srikant86panda
updated tamperbench
04ec81f
@srikant86panda
updated md5 and dataset name for complete dataset
1e127ec
@amitbcp
Merge branch 'open-compass:main' into main
618c003
resolve conflict and update with VLM
dd3bf5e
@srikant86panda
fix on moving perception file to correct folder
f7f7a11
@srikant86panda
Merge branch 'MVTamperBench' of https://github.com/srikant86panda/VLM…
efa3824 
…EvalKit into MVTamperBench
@srikant86panda
adding correct md5 for total ~19k sample
675987e
Local changes
b513f83
Merge remote-tracking branch 'origin/MVTamperBench' into MVTamperBench
fd850ed
@amitbcp
Merge branch 'open-compass:main' into main
b568447
Local Changes to run
297039f
resolve merge conflict
b5063ae
@srikant86panda
updated evaluation
869ed13
@srikant86panda
updated to latest change
f54ac3a
@srikant86panda
incroporated metrics and pre processing change
1454281
@srikant86panda
sample file support
ce511b7
@srikant86panda
reporting fix
0e6fb2c
@srikant86panda
fix with md5 check
4863148
@srikant86panda
updated md5 and dataset support
8b6c816
@srikant86panda
ntu data path fix
fb36aa5
@srikant86panda
dataset support added on run.py
9f0d451
@srikant86panda
ntu files removed
a6f2c2d
@srikant86panda
md5 updated
b18808f
@srikant86panda
run file updated
46dfe35
kennymckormick and others added 30 commits February 1, 2025 19:49
@kennymckormick
[Improvement] Refine reuse strategy & implement sympy timeout (open-c…
a810188 
…ompass#764)

* [Fix] Better Import

* [Improvement] Also reuse auxiliary files by default

* [Fix] Fix fetch_aux_files

* update

* implement timeout for sympy usages
@kennymckormick
[Fix] Fix minor issues for llava 1.5
02ff9db
@PhoenixZ810
add mathverse_visiononly_cot (open-compass#765)
8b823bd
@MMoshtaghi
Updated the generate in LLaVA-OneVision-HF (open-compass#761)
2f1f6c8 
* Fixed the cache_path in pixtral.py

* Updated response in llava.py

only return the model's generated response instead of the whole output

* Removed the unnecessary prompt print in LLaVA-OneVision-HF
@kennymckormick
[Minor] Support cmd vlmutil scan
701f9c0
@kennymckormick
[Benchmark] Support MMMU Pro (open-compass#768)
87f2dbe 
* support MMMU_Pro

* support MMMU_Pro as concat dataset

* [Fix] Fix Bug

* Fix Bug

* [Fix] Fix MMMU_Pro_V judge

* Fix vlmeval tools: EVAL

* Bug Fix

* [Fix] Fix MMMUPro COT evaluation

* fix

* Fix

* Fix
@dongyh20
[Model] Add Ola Model (open-compass#752)
fa829b9 
* add ola model

* Update ola_model.py

* Update builder.py
@kennymckormick
[Minor] Update SiliconFlowAPI
61ef196
@kennymckormick
[Minor] Fix SiliconFlowAPI
9a4136b
@kennymckormick
[Minor] Adopt CircularEval for all datasets with 'circular' in names
6e94c1e
@kennymckormick
[Improvement] Better Implementation for CircularEval (open-compass#770)
90a80d2 
* update .gitignore

* update CIRCULAR implementation

* [Refine] Update CircularEval
@PhoenixZ810
Claude Timeout (open-compass#771)
b1d59b7 
* add ClaudeWrapper Timeout, add Claude3-5V_Sonnet_20241022_tem07
@FangXinyu-0913
[Fix] Fix build dataset from config for image dataset
6b81b5e
@PhoenixZ810
Add Dynamath_noprompt (open-compass#776)
aaa1420 
* add ClaudeWrapper Timeout, add Claude3-5V_Sonnet_20241022_tem07

* add dynamath_noprompt
Removed debug prints
67e3a37
@marvinzh
[Fix] Move barrier to the starting point of the for-loop (open-compas…
30589e3 
…s#772)

current implement will not work if user use infer only mode under multiple processes
@runninglsy
add Ovis2
7f97843
@runninglsy
remove redundant parentheses
5d5693c
@Mor-Li
Merge pull request open-compass#782 from runninglsy/main
c577277 
[Model] Add Ovis2
@kennymckormick
Update SEEDBench2_Plus MD5
889822d
@zenjieli
[Minor] Fix bug in extracting dataset_name from eval file name (open-…
fa26094 
…compass#778)

Co-authored-by: zli <zli@mail.com>
@LingyiHongfd @FangXinyu-0913
update worlsense (open-compass#775)
bf73bae 
Co-authored-by: FangXinyu-0913 <fangxinyutju202009@126.com>
@kq-chen
[Fix] qwen2.5vl evaluation do not use custom prompt (open-compass#786)
a2a82bd
@kennymckormick
[Improvement] Support ImageSimpleQA data class and CreationBench data…
43af13e 
… class (open-compass#790)

* support mmbench_c

* Fix BUG

* update mmbench_c

* update SenseChat MaxTokens

* Use Dual Eval for MMBench_C by default

* [Fix] Fix MMBench-C

* [Fix] Update Max_Token Args

* [Fix] Fix Bug

* [Fix] Fix Circular

* update

* update

* update config

* update dlist

* Update Config

* Update dataset name

* add group live

* [Minor] Set max_tokens to 2048

* [Minor] Change the max_tokens to 2048 for models that will be evaluated on LiveMMBench

* remove print

* patch error

* fix md5

* [Minor] Update mmqa_display

* Fix

* minor
@PhoenixZ810
Update README and QUICKSTART (open-compass#795)
1a0335b 
* change readme and quickstart

* refine readme and quickstart.

* refine readme_cn.md
@PhoenixZ810
New document (open-compass#796)
2d8e44e 
* change readme and quickstart

* refine readme and quickstart.

* refine readme_cn.md

* modify date
@PhoenixZ810
Mmalignbench (open-compass#805)
4b94a54 
* add mmalignbench

* add judge model gpt4o
@kennymckormick
[Improvement] Refine Prompts for ShortQA Evaluation (open-compass#808)
88bfb0c
@yaogang2060
[Improvemnet] Remove Unnecessary vlmeval/vlm/internvl_chat.py (open-c…
e90a2c1 
…ompass#792)

open-compass#657 has deleted this file, while open-compass#729 added it back, which is not needed.😢
@FangXinyu-0913
Merge branch 'main' into MVTamperBench_updated
f5a59fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ppattnayak ppattnayak ppattnayak left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
0