Verifying participant comprehension continues to be a difficult ethical and regulatory challenge for clinical research. An increasing number of articles assessing methods to improve comprehension have been published, but they use a wide range of outcome measures including open-ended, closed-ended, and self-perceived measures of comprehension. Systematic comparisons of different measures have rarely been reported. This study evaluated the likely direction of bias observed when using open-ended, closed-ended, and perceived ease of comprehension measures among women administered a mock informed consent process in Mwanza, Tanzania. Participants were randomized to either a closed-ended or an open-ended assessment of comprehension, administered the consent process for a hypothetical HIV prevention trial in Kiswahili, and then administered a comprehension assessment, per their randomization. They were then asked how easy or hard it was to understand each of the informed consent components measured in the comprehension assessment. Women in the closed-ended arm had significantly higher overall comprehension scores than in the open-ended arm. Perceived scores were significantly higher when compared to both open-ended and close-ended scores within arms but were similar between arms. Findings highlight the importance of comprehension assessments in complex clinical trials that go beyond asking participants if they understand or have any questions. They also indicate the need for continued exploration of objective measures of comprehension in international clinical research settings, so that points in need of clarification can be efficiently and effectively identified and addressed. Such measures would reduce burdens on both staff and participants that result from well-intentioned but potentially unnecessary time spent explaining in unwarranted detail things already understood.