The molecular basis of recognition between plant cells is incompletely understood. Some principles established for recognition between animal cells may well apply to plant cell recognition, although, in contrast to animal cells, plant cells are encased by cell walls that play an active role in plant cell-cell recognition. The interaction that controls fertilization in flowering plants involves recognition between pollen or pollen tubes and the female sexual tissues. In many flowering plant families, self-incompatibility (S) genes operate to prevent inbreeding. In plants that have gametophytically controlled self-incompatibility, recognition of common S alleles in pollen tube and style results in arrest of pollen tube growth within the style. Self-incompatibility therefore provides a model cell-cell recognition system that is genetically defined. We have taken two approaches to defining cell recognition involved in gametophytic self-incompatibility in Nicotianas alata. Firstly, we have established the major features of the pollen tube wall and the matrix of the style transmitting tissue that are in contact with the growing pollen tube. Secondly, we have established the nature of style glycoproteins that are associated with the S genotype and have initiated a program to clone the genes coding for the protein component of these glycoproteins. Analyses of the pollen tube are consistent with the major polymers being a (1----3)-beta-D-glucan (callose) and a (1----5)-alpha-L-arabinan. The pollen tube has two distinct layers: gold immunocytochemistry using a monoclonal antibody directed to terminal alpha-L-arabinosyl residues shows the binding is confined to the outer layers. The major component of the extracellular matrix of the style transmitting tissue is a family of proteoglycans, the arabinogalactan-proteins. A major glycoprotein that segregates with the S2 allele is present in extracts of mature styles. This component has a high pI (greater than 9.5) and an apparent molecular weight of 32 X 10(3). It is not present in extracts of immature styles of N. alata genotypes bearing the S2 allele, or in extracts from other organs of N. alata or styles of other members of the Solanaceae. The isolated glycoprotein is an effective inhibitor of in vitro pollen tube growth. This evidence suggests that the S2-associated glycoprotein is either the product of the S2 allele, or a gene closely associated with the S gene. We have prepared a cDNA library from styles of one genotype and are screening this library with mRNA from mature and immature styles.(ABSTRACT TRUNCATED AT 400 WORDS)