Nothing says that a song has to stem from any one part. Sometimes it’s more holistic and you barely know what the first words or images were that rushed to mind.
Nor are all good songs containing of especially memorable parts. It is possible to have a serviceable song that has no peaks or troughs and does and goes as you would expect.
You would be no more advised to pursue perfection in parts than to pay heed to those other features that are a distraction, not an aid. Write the song and let it take its own path and display its own parts.
For all you know, the part that matters to people is the grouped chorus or other addition that wasn’t part of your original thought; or even what you’ve taken to the others. The extent to which you can let this go – if it works – is one of the challenges.