Work done as final project for 6.4212 (Graduate Robotic Manipulation) at MIT.

We develop SplatGrasp, a stack for language-guided 3D grasp affordance generation and robotic manipulation directly from casual smartphone captures. Leveraging recent advances in 2D Gaussian Splatting (2DGS) for radiance field reconstruction, we generate high-fidelity object surfaces and stable depth maps. Next, we extract object-level and query-specific (e.g., “vase handle”) masks, project them into 3D via these depth maps, and fuse the language-information into truncated signed distance field (TSDF) voxel volumes to produce full-object and query-specific mesh assets that inform grasp generation. We show improved grasp specificity and success rates compared to geometry-only baselines, highlighting the promise of fusing geometric precision with open-vocabulary language understanding for robotic manipulation.

Modern robotic manipulation tasks demand not only reliable geometric reasoning but also semantic understanding. While geometry-focused approaches excel at finding stable grasps, they fail to incorporate user intent or contextual cues. Consider the instruction “grasp the vase by the handle”: a purely geometric planner may choose a stable but semantically irrelevant grasp. By pairing high-quality 2DGS geometric reconstruction with open-vocabulary segmentation aproaches, we produce language-informed affordance maps to reconstruct full-object and query-specific mesh assets via TSDF fusion. This yields a semantically rich 3D representation, enabling language-driven grasp planning. Compared to geometry-only baselines, our approach enhances grasp relevance and reliability, demonstrated through multi-stage pick-and-place experiments.

We begin with a set of $N$ images capturing the target object from multiple viewpoints. Using COLMAP, we estimate camera parameters $(\mathbf{K}_n, \mathbf{R}_n, \mathbf{t}_n)$ for each view $n=1,\dots,N$, which are the intrinsic calibration matrix and rotation and translation from world to camera coordinates respectively. This provides a consistent reference frame.

The 2DGS paradigm represents the scene with a set of $M$ planar Gaussian primitives defined by two tangent vectors $(\mathbf{t}_u, \mathbf{t}_v)$, scaling factors, and appearance parameters (e.g. opacity). Therefore, the constructed local coordinate system for each primitive is $\mathbf{R} = \left[ \mathbf{t}_u, \mathbf{t}_v, (\mathbf{t}_u \times \mathbf{t}_v) \right]$. The primitives in $uv$-space are of the form: $${ \mathcal{N}(u, v) = \exp\left( -\frac{u^2+v^2}{2} \right) }$$ We render novel views by projecting and alpha-compositing the splats guided by depth distortion and normal consistency losses, ensuring that the splats align with the true object surfaces. Once optimized, 2DGS provides stable depth maps $\{ Z_n \}$ for each viewpoint $n$. For a pixel $(u,v)$ in image $n$, its 3D coordinate is: $$ { \mathbf{X}_n(u,v) = \mathbf{R}_n^{-1} \left( Z_n(u,v)\mathbf{K}_n^{-1}\hat{\mathbf{x}} - \mathbf{t}_n \right) } $$ where $\hat{\mathbf{x}}=[u,v,1]^{\intercal}$ represents homogeneous image coordinates. 2DGS ensures high-fidelity depth data for subsequent steps.

Once we have the depth data from 2DGS, we use LangSAM to produce query and sub-region segmentation masks $M_{\text{obj}, n}(u,v), M_{\text{aff}, n}(u,v) \in [0,1]$ for each image $n$ from textual queries (e.g. "grasp the vase handle"). Then, we unproject each mask pixel into 3D by taking: $$ { \mathbf{X}_{\text{obj}, n}(u, v) := \mathbf{X}_n(u,v) \text{ for } \mathbf{M}_{\text{obj}, n}(u, v)=1 }$$ $$ { \mathbf{X}_{\text{aff}, n}(u, v) := \mathbf{X}_n(u,v) \text{ for } \mathbf{M}_{\text{aff}, n}(u, v)=1 }$$ We fuse these points into a TSDF volume. The TSDF at a voxel $\mathbf{x}$ is: $$ { D(\mathbf{x}) = \begin{cases} \frac{z_{\text{hit}} - z_{\text{surf}}}{\tau} & \text{if } |z_{\text{hit}} - z_{\text{surf}}| < \tau \\ \text{sgn}(z_{\text{hit}} - z_{\text{surf}}) & \text{otherwise} \end{cases} } $$ where $z_{\text{surf}}$ is the measured surface depth and $z_{\text{hit}}$ the voxel's projected depth. $\tau$ is a truncation parameter. Multiple scans are integrated by a weighted average of TSDF values: $$ { D'(\mathbf{x}) = \frac{w(\mathbf{x})D(\mathbf{x}) + w'(\tilde{D}(\mathbf{x}))}{w(\mathbf{x})+w'(\mathbf{x})} } $$ where $w(\mathbf{x})$ is a weight capturing confidence. Extracting a mesh $\mathcal{M}$ with Marching Cubes, we subsequently construct a sub-region point cloud $\{ \mathbf{X}_{\text{aff},n}\}$ aligned with $\mathcal{M}$, so we can assign affordance scores to each vertex $\mathbf{v}$ on $\mathcal{M}$: $$ { a(\mathbf{v}) := \exp \left( -\frac{\min || \mathbf{v} - \mathbf{a}_j ||^2 }{ 2\sigma^2_a } \right) } $$ where $\mathbf{a}_j$ are affordance points and $\sigma_a$ is a scaling parameter. Then, vertices near the affordance sub-region have $a(\mathbf{v})\approx 1$ while distant vertices have $a(\mathbf{v})\approx 0$. This yields an affordance field over $\mathcal{M}$.

Although agnostic to the method of grasp generation, we adopt the GraspNet baseline for general-purpose grasp generation. During inference, we construct a point cloud directly from the 2DGS-reconstructed vertices to pass into GraspNet, which exposes the top candidate poses. We use non-maximum suppression to eliminate redundant candidates and retain a diverse subset of $K$ near-optimal poses. Finally, we utilize TSDF assets to build a 1-to-1 mapping between the point cloud over $\mathcal{M}$ and the discretized affordances. We take advantage of the fact that $\mathcal{M}$ and $\mathbf{a}_j$ are jointly reconstructed. Therefore, it suffices to compute a 1-nearest-neighbor-search where queries are the $\mathbf{a}_j$ and search points are vertices of $\mathcal{M}$. To incorporate semantic affordances, we adopt a composite scoring metric for the top $k=1,\dots,K$ grasp candidate defined by: $$ { \text{score}_k := \text{softmax}(s_k) \cdot a_k } $$ where $s_k$ is the generated GraspNet score and $a_k$ is the affordance score. Good candidates have both high GraspNet score (geometric precision and stability) and high affordance score (semantic relevance).

We simulate the generated grasp poses in Drake by solving for the desired joint configuration $\mathbf{q} \in \mathbb{R}^7$, which should be close to $\mathbf{q}_{\text{nominal}}$. on the KUKA LBR iiwa such that the end-effector gripper pose $\mathbf{X}_{WG}=(\mathbf{R}_{WG}, \mathbf{p}_{WG})$ aligns with post-processed GraspNet target pose $\mathbf{X}_{\text{target}}=(\mathbf{R}_{\text{target}}, \mathbf{p}_{\text{target}})$. The inverse kinematics formulation is: $$ { \arg \min_{\mathbf{q}} \, (\mathbf{q} - \mathbf{q}_{\text{nominal}})^{\intercal} (\mathbf{q}-\mathbf{q}_{\text{nominal}}) } $$ $$ { \text{s.t. } \mathbf{p}_{\text{target}} - \epsilon \leq \mathbf{p}_{WG} \leq \mathbf{p}_{\text{target}} + \epsilon, \arccos \left( \frac{\text{trace}(\mathbf{R}^{\intercal}_{WG} \mathbf{R}_{\text{target}}) - 1}{2} \right) \leq \theta_{\text{bound}} } $$

SplatGrasp has wide-ranging applications in robotics and computer vision, particularly in scenarios requiring semantic understanding and precise object manipulation. In household robotics, our system works toward personalized assistance, such as retrieving objects or handling fragile items based on natural language instructions, as we expose finer-grained control by integrating human interaction. Likewise, the ability to combine geometric accuracy with language-driven segmentation makes it valuable in healthcare robotics for tasks like surgical tool handling where geometry-only approaches fall short in alignment with complex higher-level goals.