Gravity generated by large masses has been observed using a variety of probes from atomic interferometers to torsional balances. However, gravitational coupling between small masses has never been observed so far. Here, we demonstrate sensitive displacement sensing of the Brownian motion of an optically trapped 7 mg pendulum motion whose natural quality factor is increased to 10^{8} through dissipation dilution. The sensitivity for an integration time of one second corresponds to the displacement generated in a millimeter-scale gravitational experiment between the probe and a 100 mg source mass, whose position is modulated at the pendulum mechanical resonant frequency. Development of such a sensitive displacement sensor using a milligram-scale device will pave the way for a new class of experiments where gravitational coupling between small masses in quantum regimes can be achieved.