Abstract: This paper systematically advocates an interactive volumetric image manipulation framework, which can enable the rapid deployment and instant utility of patient-specific medical images in virtual surgery simulation while requiring little user involvement. We seamlessly integrate multiple technical elements to synchronously accommodate physics-plausible simulation and high-fidelity anatomical structures visualization. Given a volumetric image, in a user-transparent way, we build a proxy to represent the geometrical structure and encode its physical state without the need of explicit 3-D reconstruction. On the basis of the dynamic update of the proxy, we simulate large-scale deformation, arbitrary cutting, and accompanying collision response driven by a non-linear finite element method. By resorting to the upsampling of the sparse displacement field resulted from non-linear finite element simulation, the cut/deformed volumetric image can evolve naturally and serves as a time-varying 3-D texture to expedite direct volume rendering. Moreover, our entire framework is built upon CUDA (Beihang University, Beijing, China) and thus can achieve interactive performance even on a commodity laptop. The implementation details, timing statistics, and physical behavior measurements have shown its practicality, efficiency, and robustness.