Dataset + Benchmark + Reference Model

EgoTools

Tool-Centric Reasoning in Real-World Egocentric Videos

Shulin Tian^1,*, Junsu Kim^1,2,*, Shuai Liu^1,3, Hao Li¹, Yujiao Shen¹, Sihan Li¹, Zhe Yang¹, Yeongon Kim⁴, Runmao Yao¹, Yuhao Dong¹, Fangzhou Hong^1,★, Antonino Furnari⁵, Jingkang Yang³, Hongyuan Zhu⁶, Ziwei Liu^1,†,★

¹S-Lab, Nanyang Technological University ²KAIST ³Synvo AI ⁴Chung-Ang University ⁵University of Catania ⁶A*STAR

^*Equal contribution ^†Corresponding author ^★Ropedia Author

Paper PDF arXiv Coming soon Code Coming soon HuggingFace Coming soon

Tool-use montage

EgoTools evaluates whether VLMs can reason about tool choice, physical state changes, and long-horizon activity dependencies in real-world first-person video.

TL;DR

From real tool use to measurable reasoning

EgoTools pairs dense real-world egocentric supervision with a diagnostic benchmark and a reference model that demonstrates the value of training on tool-centric data.

100hEgocentric video

1,000QA pairs

7Tool-use domains

+11.0 ppOver the backbone

What makes EgoTools different?

Why EgoTools?

Tool-use reasoning goes beyond recognizing objects in a frame: models must infer why a tool is selected, what it changes, and how earlier actions shape later outcomes.

Tool-choice reasoning

EgoTools asks models to understand why a person uses one tool rather than another, grounded in object properties, task goals, and the surrounding context.

Physical state and outcome reasoning

The benchmark requires tracking how objects change after tool interactions, such as cutting, mixing, fastening, cleaning, measuring, or assembling.

Long-horizon dependency

Many questions require reasoning over minutes-long activities, where the relevant evidence may appear before or after the visible tool interaction.

Egocentric ambiguity

First-person videos include occlusion, hand-object interaction, motion, and viewpoint uncertainty, making simple frame lookup insufficient.

Explore EgoTools data

Enter the EgoTools World

Click a domain to browse scenario-style examples with representative clips, tool-use actions, narration, and benchmark-style QA.

Benchmark examples

Questions that depend on tools, time, and outcomes

EgoTools questions are multiple-choice, visually grounded, and tied to evidence moments spread across a first-person clip rather than a single object label. Select an answer to reveal the timestamped evidence and reasoning behind the benchmark label.

Demo progress 0 / 3 examples explored

Dataset & benchmark at a glance

Scale, domains, and reasoning taxonomy

A 40.34-hour benchmark subset covers seven real-world domains and four complementary reasoning tracks, with 900 human-crafted and 100 spatial questions.

Dataset coverage

Seven real-world domains with synchronized video, narration, and QA.

Kitchen Classroom Research Lab Repair Workshop Craft Office Household

40.34h quality-controlled benchmark subset

QA construction 1,000 total

Human-crafted QAs 900

Spatial QAs 100

Benchmark subset 40.34h

Benchmark tracks 4 tracks

Affordance & Causality 363 Perception & Grounding 236 Procedural Dynamics 222 Spatial Reasoning 179

Main result · EgoTools-Bench

EgoTools training raises overall accuracy from 50.0% to 61.0%

EgoTools supervision substantially improves tool-centric reasoning under strict source-video separation on the full quality-controlled 1,000-question EgoTools-Bench.

Per-track comparison

Gains across complementary reasoning skills

Exact values

Reference model trained with EgoTools-Data

Model	Overall	AC	PG	PD	SR
Qwen3-VL-8B-Instruct	50.0	46.1	45.9	57.1	55.4
EgoTools-8B (Full SFT)	58.2	49.6	58.8	66.8	68.5, best SR result
ReferenceEgoTools-8B (Full SFT + Refinement)	61.0, best Overall result	53.4, best AC result	60.8, best PG result	71.3, best PD result	64.1

AC: Affordance & Causality
PG: Perception & Grounding
PD: Procedural Dynamics
SR: Spatial Reasoning

Full supervised fine-tuning improves all four reasoning tracks. Format-aligned refinement further improves Overall, AC, PG, and PD; SR remains substantially above the backbone but reaches its highest score after Full SFT.

Training recipe and data details

Stage 1

Full supervised fine-tuning

220,334 EgoTools-derived instruction examples
Caption-, narration-, and video-QA-based supervision
Visual encoder and multimodal aligner frozen

Stage 2

Format-aligned refinement

LoRA refinement
4,722 quality-filtered EgoSchema-style synthetic MCQs
Generated from 318 EgoTools training videos and caption timelines
“EgoSchema-style” refers only to the multiple-choice format
No original EgoSchema videos or QA annotations are used

Analysis · Not the primary claim

Cross-benchmark Generalization Analysis

Transfer to general egocentric benchmarks is reported transparently as a secondary analysis alongside the primary EgoTools-Bench result.

Model	EgoSchema	EgoPlan	EgoThink
Qwen3-VL-8B-Instruct	68.8	42.3	61.0
EgoTools-8B (Full SFT)	51.7	33.9	60.1
EgoTools-8B (Full SFT + Refinement)	67.0	34.0	59.4

Format-aligned refinement substantially recovers EgoSchema performance without using original EgoSchema training data. Transfer remains benchmark-dependent: EgoTools’ primary claim is improved tool-centric reasoning on EgoTools-Bench, not uniform improvement across all egocentric benchmarks.

Construction pipeline

From first-person collection to a reference model

A reproducible six-stage path connects raw collection, rectification, annotation, benchmark construction, evaluation, and source-video-disjoint training.

1Egocentric Collection
Collect real tool-use activities across seven domains.
2Rectification
Prepare synchronized rectified clips for consistent inspection.
3Tool-Centric Narration
Describe tools, objects, actions, and task progress.
4Benchmark QA
Build grounded multiple-choice reasoning questions.
5VLM Evaluation
Evaluate tool choice, outcomes, time, and grounding.
6Reference Model
Train using source-video-disjoint EgoTools supervision.

Resources & release status

What is available now and what is planned

Unreleased items are shown as status information rather than active links.

Available now

Paper and approved project-page samples

The current paper PDF and the representative clips on this page can be viewed now.

Paper PDFAvailable

Coming with the official release

Release package

Dataset accessComing soon
Benchmark filesComing soon
Official splitsComing soon
Evaluation codeComing soon
Reference modelComing soon
License informationPlanned

FAQ

Common questions

Is EgoTools just object recognition?

No. It targets tool choice, physical state changes, spatial coordination, and temporally extended actions.

Are all videos publicly downloadable now?

No. Release-approved samples are shown here; full access details will accompany the official release package.

What should models output?

EgoTools-Bench uses multiple-choice QA grounded in timestamped visual evidence and reasoning explanations.

Citation

BibTeX

@article{egotools2026,
  title   = {EgoTools: Tool-Centric Reasoning in Real-World Egocentric Videos},
  author  = {Tian, Shulin and Kim, Junsu and Liu, Shuai and Li, Hao and Shen, Yujiao and Li, Sihan and Yang, Zhe and Kim, Yeongon and Yao, Runmao and Dong, Yuhao and Hong, Fangzhou and Furnari, Antonino and Yang, Jingkang and Zhu, Hongyuan and Liu, Ziwei},
  journal = {arXiv preprint},
  year    = {2026}
}

Acknowledgements

Thanks

We thank the data collection, annotation, engineering, and evaluation contributors.