← All projects

AffordRAG-Factory Edge AI Framework

A hierarchical edge VLM and knowledge graph framework that turns ambiguous shop-floor requests into explainable, executable multi-robot actions.

Edge-side embodied AI for high-mix manufacturing.

AffordRAG-Factory addresses manufacturing settings where workers describe problems by effect, not by exact robot command. The system combines local vision-language understanding with structured knowledge graph constraints so decisions remain interpretable and executable.

The framework targets low-compute, privacy-sensitive shop floors by using edge VLMs, retrieval-augmented affordance reasoning, robot capability checks, and a unified Skill-Code execution interface.

Submitted to ASME JCISE

The JCISE submission is an expanded version of our ASME IDETC/CIE 2026 conference paper.

Meng, Y., Yang, X., Frericks, J. B., Xie, J., De Austria, T., Singh, A., Favin, D., & Morkos, B. (2026). AffordRAG-Factory: A Hierarchical Edge VLM and Knowledge Graph Framework for Ambiguous Heterogeneous-Robot Task Execution in Manufacturing. Manuscript submitted to the ASME Journal of Computing and Information Science in Engineering (JCISE).

From worker requests to robot actions

Four-layer framework connecting task intention analysis, tool affordance reasoning, action reasoning, and robot allocation through knowledge graphs.
Images and language pass through four reasoning layers. Tool, action, and robot knowledge graphs constrain the decisions before a unified Skill-Code command is sent to a robot.

Humanoid tool handoff

Four views of a Unitree G1 recognizing tools, approaching a selected object, grasping it, and handing it to an engineer.
The Unitree G1 EDU demonstration follows a tool from visual recognition through grasping and handoff to the worker.

Results reported in the submission

Benchmark

Ambiguous instruction understanding

Evaluated on 119 ambiguous manufacturing instructions. Qwen3-VL-2B achieved 94.1% accuracy with an average response time of 897 ms. A pass required both the intended tool or semantic target and the expected Skill-Code output.

Robots

Heterogeneous robot deployment

Demonstrated on a Dobot E6 robotic arm for visual inspection, a Unitree G1 EDU humanoid for tool grasping and handoff, and a Unitree Go2 quadruped for navigation.

System

Local model deployment

Six local models from the SmolVLM and Qwen families were evaluated on an NVIDIA RTX 4070. All used less than 6 GB of GPU memory and exceeded 85% accuracy on the instruction benchmark.