Expressed emotion (EE), especially criticism, is an important predictor of outcomes for the patient for a wide range of mental health problems. To understand complex links between EE and various relevant variables in early phase psychosis, this study examined criticism, distress of caregivers, other patients', and caregivers' variables, and links between criticism and these variables in those with at-risk mental state (ARMS) for psychosis and first-episode psychosis (FEP). The participants were 56 patients (mean age 18.8 ± 4.2 years) with ARMS and their caregivers (49.4 ± 5.8 years) and 43 patients (21.7 ± 5.2 years) with FEP and their caregivers (49.3 ± 7.4 years). We investigated criticisms made by caregivers using the Japanese version of the Family Attitude Scale and caregiver depressive symptoms via the self-report Beck Depression Inventory. We also assessed psychiatric symptoms and functioning of the patients. Approximately one-third of caregivers of patients with ARMS or FEP had depressive symptoms, predominately with mild-to-moderate symptom levels, whereas only a small portion exhibited high criticism. The level of criticism and depression were comparable between ARMS and FEP caregivers. The link between criticism, caregivers' depression, and patients' symptoms were observed in FEP but not in ARMS caregivers. These findings imply that the interaction between criticism and caregivers' and patients' mental states may develop during or after the onset of established psychosis and interventions for the caregivers should be tailored to the patient's specific stage of illness. Interventions for FEP caregivers should target their emotional distress and include education about patient's general symptoms.