2025.09 - 2025.11 Team Project ROS 2 / Python / C++

Shopee: Robot Shopping System

An autonomous in-store shopping system featuring "Pickee" and "Packee" robots. Integrates Nav2, MoveIt, and LLM-based voice interactions for a seamless retail automation experience.

Repository Back to Projects

Project Overview

Shopee is an automated robotic shopping system designed to streamline the picking and packing process in retail environments. Customers can order items via a mobile app, and the system coordinates two types of robots:

Pickee: Navigates to shelves, identifies items, and picks them using a robotic arm.
Packee: Receives items at a packing station and prepares them for delivery.

The system also features a "Night/Stocking Mode" where robots can follow staff members and respond to natural language voice commands to assist with inventory management.

Key Engineering Contributions

Central Control Architecture

Designed the complete multi-robot control architecture bridging ROS 2 nodes, LLM services, and the user app. Implemented a hybrid TCP/IP & UDP protocol to ensure reliable command delivery while maintaining real-time performance for robot state monitoring.

Main Server Implementation

Built the central backend service that orchestrates the entire shopping workflow. It manages the bi-directional communication between the "Pickee" and "Packee" robots, handles task allocation, and prevents deadlock situations during collaborative tasks.

FSM-Based Robot Control

Developed the core Finite State Machine (FSM) logic for the robots. This ensures robust failure recovery (e.g., retrying docking on failure) and smooth transitions between various operational modes like "Shopping," "Packing," and "Night/Stocking" modes.

Admin Debugging Dashboard

Created a modular GUI dashboard for engineers to monitor raw TCP data packets, check detailed ROS 2 node health, and visualize real-time robot battery/location status, significantly reducing on-site debugging time.

Core Technology

Autonomous Driving: Nav2-based path planning with custom velocity modifiers for dynamic environments.
Precision Parking: ArUco marker recognition with specialized preprocessing (grayscale/binarization) for high-precision alignment.
Arm Control: Visual servoing using Two-Stream Networks and PD control with Gaussian velocity profiles to reduce vibration.
Person Tracking: Custom YOLO models trained on staff uniforms for reliable "Follow Me" functionality.

Tech Stack

ROS 2 (Humble) Nav2 MoveIt YOLOv11 PoseCNN Whisper STT Qwen (LLM) Python & C++ PyQt6 TCP/IP & UDP

Technical Challenges & Solutions

Unstable Marker Recognition

Problem: Standard ArUco detection failed under varying lighting conditions.
Solution: Implemented a preprocessing pipeline with grayscale conversion and adaptive binarization to maximize marker contrast.

Picking Misalignment

Problem: Mobile robot stopping position variance caused arm reach failures.
Solution: Developed a real-time calibration routine that adjusts the arm's target coordinates based on the actual stopped position relative to the shelf marker.

LLM Hallucinations

Problem: General LLMs sometimes generated invalid navigation targets.
Solution: Conducted QLoRA SFT (Supervised Fine-Tuning) on a specific dataset of 500+ location-based queries to ensure the model output strictly adhered to the facility map.