If people start using these passphrases too much, then password crackers will just prioritize dictionary brute forcing with dictionaries based on popular passphrase generator word pools. Then, each word is as useless as a single character in a password.

Create a complex password system only you know and you can memorize, so you can have a different password for every account, which can be rotated on a regular bases, and never have to put all your security in one basket with a password manager or use a security defeating short-cut like the passphrases. If done correctly, you should be able to make paper notes, that no one else can decipher, in case your memory falters.