Minecraft with ROS 2 & SLAM
A unique robotics experiment bridging the gap between gaming and robotics control. Uses micro-ROS to control a Minecraft character physically and perform real-time SLAM mapping inside the game world.
Project Overview
This project demonstrates the flexibility of ROS 2 by treating Minecraft as a robot simulation environment. It connects a physical ESP32-based joystick via micro-ROS to control the in-game player and extracts sensor data (simulated LiDAR) to generate 2D maps using Cartographer.
Key Engineering Contributions
Java-ROS 2 Bridge
Developed a custom Minecraft Forge Mod (Java) that acts as a
middleware.
It hooks into the game engine to extract simulated ray-tracing data and exposes it as
ROS 2 topics (/scan, /odom), effectively turning the game into
a high-fidelity simulator.
Embedded Control Interface
Implemented the physical hardware interface using an ESP32 and micro-ROS. Wrote the firmware to sample analog joystick data and stream it via UDP to the ROS 2 agent with minimal latency for smooth character control.
SLAM Pipeline Configuration
Configured and tuned the Google Cartographer Lua configuration files to accept the noisy, simulated laser scans from Minecraft. Optimized trajectory builder parameters to maintain map consistency despite utilizing a non-standard simulation environment.
System Configuration
- Input: Physical Joystick connected to ESP32.
- Transport: micro-ROS (Serial/UDP) -> ROS 2 Agent.
- Processing: ROS 2 Nodes convert joystick inputs to
/cmd_veland process/player/pointcloudfor SLAM. - Output: Minecraft character movement and RViz2 map visualization.
Tech Stack
Experiment Notes
Due to the latency in extracting point clouds from the Java-based game engine, rapid rotations can cause map drift. Best mapping results are achieved with slow, deliberate linear movements.