Abstract: High-dimensional Gaussian filters, most notably the bilateral filter, are important tools for many computer graphics and vision tasks. In recent years, a number of techniques for accelerating their evaluation have been developed by exploiting the separability of these Gaussians. However, these techniques do not apply to the more general class of spatially varying Gaussian filters, as they cannot be expressed as convolutions. These filters are useful because the underlying data---e.g. images, range data, meshes or light fields---often exhibit strong local anisotropy and scale. We propose an acceleration method for approximating spatially varying Gaussian filters using a set of spatially invariant Gaussian filters each of which is applied to a segment of some non-disjoint partitioning of the dataset. We then demonstrate that the resulting ability to locally tilt, rotate or scale the kernel improves filtering performance in various applications over traditional spatially invariant Gaussian filters, without incurring a significant penalty in computational expense.