Abstract: Photo-realistic reproduction of material appearance from images has widespread use in applications ranging from movies over advertising to virtual prototyping. A common approach to this task is to reconstruct the small scale geometry of the sample and to capture the reflectance properties using spatially varying BRDFs. For this, multi-view and photometric stereo reconstruction can be used, both of which are limited regarding the amount of either view or light directions and suffer from either low- or high-frequency artifacts, respectively. In this paper, we propose a new algorithm combining both techniques to recover heightfields and spatially varying BRDFs while at the same time overcoming the above mentioned drawbacks. Our main contribution is a novel objective function which allows for the reconstruction of a heightfield and high quality SVBRDF including view dependent effects. Thereby, our method also avoids both low and high frequency artifacts. Additionally, our algorithm takes inter-reflections into account allowing for the reconstruction of undisturbed representations of the underlying material. In our experiments, including synthetic and real-world data, we show that our approach is superior to state-of-the-art methods regarding reconstruction error as well as visual impression. Both the reconstructed geometry and the recovered SVBRDF are highly accurate, resulting in a faithful reproduction of the materials characteristic appearance, which is of paramount importance in the context of material rendering.