Abstract: This paper describes a framework for 3D selection and manipulation in virtual environments. The framework provides some basic selection and manipulation techniques to achieve the interaction task. It also provides more than one input channel to interact with the system. The gist of the proposed framework is the semantic integration of multimodal input using spatial ontology and user context to integrate the interpretation results from the inputs into a single one. The spatial ontology, describing the spatial relationships between objects, is used together with the current user context to solve ambiguities coming from the user's commands. We discuss how the spatial ontology is defined and used to assist the user to perform object placements in the virtual environment.