Tactile ball environment
Tactile Ball Robot
The Tactile Ball Robot is an open-source robotics platform designed to investigate tactile perception, adaptive locomotion, and embodied artificial intelligence using a spherical robotic body. Unlike conventional wheeled or legged robots, the platform explores how physical interaction with the environment can be used for both perception and control through tactile sensing.
Combining mechanical design, embedded electronics, and machine learning, the project provides an affordable and modular research platform suitable for researchers, students, and hobbyists interested in robotics, AI, and autonomous systems.
Motivation
Most mobile robots depend heavily on cameras or distance sensors to understand their surroundings. This project investigates an alternative approach by using touch as the primary sensing modality. Through direct interaction with the environment, the robot can collect tactile information to identify terrain, detect obstacles, and eventually adapt its movement.
The spherical design introduces a unique locomotion challenge, requiring coordinated internal actuation rather than conventional wheel or leg control. This makes the platform well suited for studying embodied intelligence, where a robot's physical structure contributes directly to its behaviour.
Features
- Open-source mechanical design with 3D printable components.
- Low-cost embedded electronics and microcontroller-based control.
- Modular architecture for rapid experimentation.
- Integrated tactile sensing for environmental interaction.
- Support for machine learning and reinforcement learning experiments.
- Expandable hardware and software framework.
- Designed for both research and education.
Research Applications
- Tactile perception and terrain classification.
- Adaptive locomotion across different surfaces.
- Reinforcement learning for autonomous control.
- Sensor fusion using tactile, inertial, and visual data.
- Embodied intelligence and morphology-driven behaviour.
- Generation of tactile datasets for machine learning.
Simulation gym environment
The gym environment found on the GitHub provides several terrain types for the ball to be placed in. The ball takes in two input images from cameras either side, this shows optical markers like a TacTip would have but on a rotating base. The outside is made of a soft approximation of tendons. There is a generation script under the generation folder where you can change the stiffness and damping, which should be calibrated to the silicone. You will notice that the ball has a skeletal outside, and yet the images it renders have a black background. To begin with we rendered the skin, but this increased the time of processing and only added an aesthetic value. We can still render the markers without needing a skin backdrop.
We have a keyboard tutorial for controlling the robot in real time. Space for jump, WASD keys for direction. The keyboard code can be found under the examples on the GitHub. You can change the gym environment within the keyboard example using the
gym.make("name of environment")
Check out an example of the marker data from one sensor.
Image to marker modelling
We can grab the markers using theenv.unwrapped.get_markers() which will return an array of N markers by 3 coordinates. Of course, for simplicity we could use this in the processing instead of using the image, but in reality we will have an iamge rather than 3D array.
We will need a model that takes in the input image, resize it for processing reasons and then map it to the 3D coordinates. The label data has been automatically gathered by presssing the soft sensor across the 3D texture dataset. The target pose defines the position and orientation of the object relative to the tactile sensor. The first three parameters correspond to the object's Cartesian position (x,y,z), while the remaining three parameters represent rotations about the x-, y-, and z-axes
(θx,θy,θz). Lowering the z-coordinate increases the indentation of the sensor into the object, while varying the rotational components changes the local orientation of the contacted surface. The object pose was systematically varied to generate a diverse set of tactile observations. The initial object position was defined as (x,y,z)=(0.3,0.3,0.22) m. The object was translated along the x- and y-axes, while the z-axis position was varied between 0.18 m and 0.22 m to simulate different contact depths and levels of sensor indentation. The orientation of the object was independently rotated around each axis, with roll, pitch, and yaw angles sampled between −π/2 and π/2 radians, covering a range of possible surface orientations. Additionally, multiple surface textures were evaluated, and the contact speed was varied between 0 and 0.3 m/s.
AS rotations varied, we standardized the rotations to the same origin using the following rotation. This converts the global coordinate system to a local one, so the model can be invariant of angular position.
\[ \mathbf{p}_{i}^{local} = R(\theta_x,\theta_y,\theta_z)^{T} \left( \mathbf{p}_{i}^{global} - \mathbf{t} \right) \]
We tested various regression models with 20 trials each, the dataset randomised with a different seed eachtime. We used a Ridge regression model with alpha as 1, Linear regression model, Random forest regression, multi-layer perceptron The multi-layer perceptron used layer sizes 4096, 512, 447 (149 markers with a 3D coordinate per marker). We imported the class from SciKit learn for simplicity. The model used ReLU activation, 100 epochs and a batch size of 64.
| Model | Av Train MSE | Av Test MSE | Std | Av training time (minutes)These times are for training on a computer with specs:
|
||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Linear Regression | 0.00455 | 0.00211 | 0.00212 | 13.3667 | ||||||||||||||
| Ridge Regression | 0.00066 | 0.00071 | 1.2739 x10^-5 | 0.4942 | ||||||||||||||
| Random Forest | - | - | - | - | ||||||||||||||
| MLP | 5.64 x10^-6 | 4.82 x10^-6 | 2.57 x10^-6 | 39.4825 | ||||||||||||||
| CNN | 2.9406 x10^-7 | 2.658 x10^-7 | 2.658 x10^-7 | 71.931 |