To determine the strength of noncovalent interactions that stabilize a membrane protein complex, we have developed an in vitro method for quantifying the dissociation of the bacteriorhodopsin (BR) lattice, a naturally occurring two-dimensional crystal. A lattice suspension was titrated with a short- and long-chain phosphatidylcholine mixture to dilute BR within the lipid bilayer. The fraction of BR in the lattice form as a function of added lipid was determined by visible circular dichroism spectroscopy and fit with a cooperative self-assembly model to obtain a critical concentration for lattice assembly. Critical concentration values of wild-type and mutant proteins were used to calculate the change in lattice stability upon mutation (DeltaDeltaG). By using this method, a series of mutant proteins was examined in which residues at the BR-BR interface were replaced with smaller amino acids, either Ala or Gly. Most of the mutant lattices were destabilized, with DeltaDeltaG values of 0.2-1.1 kcal/mol at 30 degrees C, consistent with favorable packing of apolar residues in the membrane. One mutant, I45A, was stabilized by approximately 1.0 kcal/mol, possibly due to increased lipid entropy. The DeltaDeltaG values agreed well with previous in vivo measurements, except in the case of I45A. The ability to measure the change in stability of mutant protein complexes in a lipid bilayer may provide a means of determining the contributions of specific protein-protein and protein-lipid interactions to membrane protein structure.