Patient‐derived organoids (PDOs) represent promising preclinical models in various tumor types. In the context of prostate cancer (PCa), however, their establishment has been hampered by poor success rates, which impedes their broad use for translational research applications. Along with the necessity to improve culture conditions, there is a need to identify factors influencing outcomes and to determine how to assess success versus failure in organoid generation. In the present study, we report our unbiased efforts to generate PDOs from a cohort of 81 PCa specimens with diverse pathological and clinical features. We comprehensively analyzed histological features of each enrolled sample (Gleason score, tumor content, proliferation index) and correlated them with organoid growth patterns. We identified improved culture conditions favoring the generation of PCa organoids, yet no specific intrinsic tumor feature was broadly associated with sustained organoid growth. In addition, we performed phenotypic and molecular characterization of tumor–organoid pairs using immunohistochemistry, immunofluorescence, fluorescence in situ hybridization, and targeted sequencing. Morphological and immunohistochemical profiles of whole organoids altogether provided a fast readout to identify the most promising ones. Notably, primary samples were associated with an initial take‐rate of 83% (n = 60/72) in culture, with maintenance of cancer cells displaying common PCa alterations, such as PTEN loss and ERG overexpression. These cancer organoids were, however, progressively overgrown by organoids with a benign‐like phenotype. Finally, out of nine metastasis samples, we generated a novel organoid model derived from a hormone‐naïve lung metastasis, which displays alterations in the PI3K/Akt and Wnt/β‐catenin pathways and responds to androgen deprivation. Taken together, our comprehensive study explores determinants of outcome and highlights the opportunities and challenges associated with the establishment of stable tumor organoid lines derived from PCa patients. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.