Contact Gap Survey: Physical AI for Dexterous Manipulation

The Evolution Toward Physical AI

Robotic manipulation has historically relied on a vision-centric paradigm where systems follow a perception-planning-control pipeline. This approach succeeds in structured environments where object geometries and contact conditions are predictable. However, for dexterous manipulation—tasks involving complex, multi-fingered interactions in unstructured environments—vision-only systems encounter a “contact gap.” Cameras can observe a scene, but they cannot directly perceive the local physical interactions occurring at the point of contact, such as force, shear, slip, or material compliance.

Figure 1: Work by Parag Chourasia at IIT, Jodhpur

This research identifies a necessary transition toward Physical Artificial Intelligence (AI). Unlike traditional robotics, Physical AI models the robot as an agent that continuously interacts with and adapts to the physical world. This requires a multimodal approach where tactile sensing serves as the primary interface for mechanical signals, bridging the gap between global visual observation and local physical execution. The survey categorizes the shift from asking “Can a robot detect contact?” to “Can a robot learn a general representation of physical contact?” and ultimately, “Can a Physical AI policy use contact representations to perform general dexterous manipulation?”

Physical AI for Dexterous Manipulation Architecture

Figure 2: Comparison of high-resolution vision-based tactile sensors (DIGIT, DigiTac, TacTip) that transform mechanical deformation into visual data for AI processing.

The Multimodal Interaction Framework

To achieve human-level dexterity, the authors propose a comprehensive observation model. In this framework, the state of the world is not just a set of visual pixels but a multisensory vector. The observation at time $t$, denoted as $O_t$, is defined as:

O_t = \{O_t^v, O_t^l, O_t^{tac}, O_t^f, O_t^p\}

Where $O_t^v$ represents visual data, $O_t^l$ is language-based intent, $O_t^{tac}$ represents high-resolution tactile information, $O_t^f$ accounts for 6-axis force/torque measurements, and $O_t^p$ is the robot’s proprioceptive state (joint angles and velocities). The objective is to learn a policy $\pi$ that maps these multimodal observations to a sequence of actions $a_t$:

\pi(a_t | O_t, O_{t-1}, \dots, O_{t-k}) \to a_t \in \mathbb{R}^d

This formulation moves away from static planning toward reactive, interaction-aware control. The authors suggest a hierarchical stack for Physical AI that decomposes the manipulation problem into discrete levels of abstraction:

\text{Physical AI} \approx \text{Semantic} \to \text{Motion} \to \text{Contact} \to \text{Interaction} \to \text{Manipulation}

At the higher levels, Vision-Language-Action (VLA) models handle semantic understanding and coarse motion planning. At the lower levels, contact state estimation and interaction control leverage high-frequency tactile and force feedback to adjust the robot’s behavior in real-time, ensuring stable grasps and precise movements.

Tactile Sensing Technologies for Dexterity

A core contribution of the survey is the evaluation of tactile sensing technologies based on their suitability for dexterous manipulation. While many transduction mechanisms exist, they vary significantly in their AI compatibility and spatial resolution.

Optical Tactile Sensors (e.g., GelSight, DIGIT): These sensors use a camera to track the deformation of a flexible elastomer. They provide high-resolution “tactile images,” allowing robots to infer contact geometry, texture, and force distribution. The relationship between the raw image $I_{raw}$ and the physical properties is described by:
$$I_{raw} \to f(\Delta \text{elastomer}) \to {\text{Force}, \text{Pressure}, \text{Geometry}}$$
Capacitive and Piezoresistive Skins: These are often used for large-area “electronic skins.” They are thin and flexible, making them ideal for covering entire robot fingers, though they typically offer lower spatial resolution than optical sensors.
Piezoelectric and Magnetic Sensors: These are highly sensitive to dynamic events like vibrations and slip. Magnetic sensors (e.g., ReSkin) are particularly notable for their durability and lack of internal wiring requirements.

The researchers argue that a hybrid approach is most effective. For example, combining an optical elastomer for high-resolution geometry with piezoelectric sensors for high-frequency vibration detection allows the robot to understand both the shape of the object and the moment it begins to slip.

Critical Contact Information

Dexterous manipulation requires more than just binary contact detection. The paper identifies 13 types of critical contact information that a Physical AI must process:

Normal and Shear Forces: Essential for maintaining grasp stability without crushing the object.
Pressure Distribution: Provides the exact shape of the contact area, which is vital for manipulating small or complex parts.
Slip and Micro-slip: The ability to detect incipient slip allows a robot to tighten its grip before an object falls.
Texture and Roughness: Enables the robot to distinguish between materials (e.g., silk vs. sandpaper) and adjust friction models accordingly.
Compliance/Stiffness: Allows the robot to understand if it is squeezing a soft fruit or a rigid metal tool.

By representing these properties, the robot can move beyond “blind” manipulation to a state where it “reasons” about objects based on their physical behavior during interaction.

Tactile Representation Learning

A major hurdle in tactile robotics is the sensor-specific nature of the data. To address this, the survey highlights the emergence of tactile foundation models. These models aim to learn general, reusable representations of touch that can be applied across different sensors and tasks.

The researchers detail the Sparsh framework, which utilizes Self-Supervised Learning (SSL) to pretrain tactile encoders. The process involves two stages. First, a tactile encoder $E_\phi$ transforms raw tactile data into a latent embedding $z_t$:

z_t = E_\phi(O_t^{tac})

During pretraining, a masked autoencoder approach is often used, where the model must reconstruct missing parts of a tactile image. The loss function $\mathcal{L}_{SSL}$ measures the reconstruction error:

\mathcal{L}_{SSL} = \|D_\theta(E_\phi(O_t^{tac} \odot M)) – O_t^{tac}\|_2^2

Where $M$ is a mask and $D_\theta$ is a decoder. Once pretrained, the frozen encoder $E_\phi$ provides features $z_t$ that a downstream policy $\pi_\psi$ can use for specific tasks:

a_t = \pi_\psi(z_t, O_t^v, O_t^l, O_t^p)

The evolution from Sparsh to Sparsh-Skin and Sparsh-X represents a move toward full-hand tactile skins and multisensory touch. Sparsh-X, for instance, treats touch as a multisensory phenomenon, combining tactile images with audio and pressure signals to create a richer representation of physical interaction.

Datasets and the Scaling Problem

Unlike computer vision, which benefits from billions of internet images, tactile data must be collected through physical interaction. This creates a data bottleneck. The paper reviews several emerging datasets designed to scale Physical AI:

FreeTacMan: Uses robot-free visuo-tactile data collection to gather large-scale human interaction data.
RoboTacDex: A visual-tactile-action dataset specifically for humanoid dexterous manipulation.
DEX-X: Leverages human videos and physical simulation to generate synthetic tactile supervision, addressing the difficulty of manual annotation.

The researchers emphasize that future datasets must be outcome-aware, meaning they must record not just the sensor readings, but also whether the resulting action (e.g., a grasp) was successful.

Check our discussion on LinkedIn https://www.linkedin.com/feed/update/urn:li:activity:7504761389954252800.

Challenges and Future Directions

The survey concludes by identifying several open challenges that remain in the field:

Cross-Sensor Generalization: A model trained on one type of sensor (e.g., DIGIT) often fails when applied to another (e.g., GelSight) due to differences in lighting, elastomer thickness, and camera resolution.
Temporal and Event-Based Sensing: Tactile interaction is inherently temporal. Current models often process single frames, missing the rich information contained in the sequence of contact events.
Sim-to-Real Gap: Simulating realistic tactile feedback is computationally expensive and physically complex, making it difficult to train robust policies in simulation for deployment in the real world.
Tactile-Language Grounding: There is a need to connect tactile signals to linguistic concepts. For example, a robot should be able to ground the word “slippery” in specific patterns of shear force and vibration.

To overcome these, the authors propose a “contact intelligence stack” that integrates high-resolution optical tactile skins, piezoelectric vibration sensors, and 6-axis force/torque sensors. By fusing these inputs through a multimodal transformer, Physical AI systems can achieve the level of interaction required for truly general-purpose dexterous manipulation.

Read our article Contact Gap Survey: Physical AI for Dexterous Manipulation, Bothe and Chourasia, 2026.

@misc{bothe2026contact,
title = {Contact Gap Survey: Physical AI for Dexterous Manipulation},
author = {Bothe, Chandrakant and Chourasia, Parag},
year = {2026},
publisher = {alphaXiv},
url = {https://www.alphaxiv.org/abs/2608.contact-gap-survey-physical-ai},
keywords = {Robotics (cs.RO), Artificial Intelligence (cs.AI), Machine Learning (cs.LG), Tactile Sensing, Dexterous Manipulation, Physical AI, Vision-Language-Action (VLA), FOS: Computer and information sciences}
}