Task understanding
Uses image and language context to infer the worker's intended assistance when the request is vague or effect-oriented.
Project
AffordRAG-Factory Edge AI Framework
A hierarchical edge VLM and knowledge graph framework that turns ambiguous shop-floor requests into explainable, executable multi-robot actions.
Overview
Edge-side embodied AI for high-mix manufacturing.
AffordRAG-Factory addresses manufacturing settings where workers describe problems by effect, not by exact robot command. The system combines local vision-language understanding with structured knowledge graph constraints so decisions remain interpretable and executable.
The framework targets low-compute, privacy-sensitive shop floors by using edge VLMs, retrieval-augmented affordance reasoning, robot capability checks, and a unified Skill-Code execution interface.
Citation
ASME IDETC/CIE 2026 framework paper
Meng, Y., Yang, X., Frericks, J. B., Xie, J., De Austria, T., Singh, A., & Morkos, B. (2026). AffordRAG-Factory: A hierarchical edge VLM and knowledge graph framework for ambiguous multi-robot task execution in manufacturing. In Proceedings of the ASME 2026 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. IDETC2026-192148.
Framework
How ambiguous worker requests become robot actions
Affordance reasoning
Retrieves feasible tools, actions, constraints, and robot capabilities from knowledge graph embeddings instead of relying on unconstrained generation.
Cross-platform execution
Maps grounded decisions into Skill-Code requests that can be routed to heterogeneous robot platforms through a common interface.
Results
Validated across benchmark tasks and real robots.
Ambiguous instruction understanding
Evaluated on 35 ambiguous manufacturing instructions, with edge VLM backbones showing high accuracy and much lower latency than a cloud model baseline.
Heterogeneous robot deployment
Validated with a Dobot E6 robotic arm, Unitree G1 humanoid, and Unitree Go2 quadruped across inspection, grasp-and-delivery, and navigation scenarios.
Executable and explainable decisions
Knowledge graph constraints reduce infeasible action generation while keeping the reasoning path inspectable before robot execution.
Links