Quantum effects in macroscopic systems have long been a fascination for researchers. Over the past decade mechanical oscillators have emerged as a leading system of choice for many such experiments. The work reported in this thesis investigates the effects of the radiation-pressure force of light on macroscopic mechanical structures. The basic system studied is a mechanical oscillator that is highly reflective and part of an optical resonator. It interacts with the optical cavity mode via the radiation-pressure force. Both the dynamics of the mechanical oscillation and the properties of the light field are modified through this interaction. The experiments use quantum optical tools (such as homodyning and down-conversion) with the goal of ultimately showing quantum behavior of the mechanical center of mass motion. Of particular value are the detailed descriptions of several novel experiments that pave the way towards this goal and are already shaping the field of quantum optomechanics, in particular optomechanical laser cooling and strong optomechanical coupling.