Quick answer
Self-control is infrastructure. Without it a person is not weak so much as undefended, open to whatever arrives.
Read both lines
The image is precise and easy to underestimate. A city without walls in that world was not a city with a problem; it was a city that could be entered by anyone at any time, and everything inside it belonged to whoever walked in. Note that the walls do not attack. They are entirely defensive, which is what the saying claims self-control actually is: not force applied outward but a boundary that decides what gets in.
"Broken down" matters too, since the Hebrew describes a breached city rather than one built without walls. Something was there and gave way, which is nearer most people's experience than a lifetime of having none. The book has already said the strongest version of this at 16:32, where ruling your spirit beats taking a city, and placing the two together says the wall and the conquest are the same skill.
Related: Proverbs 16:32 · Proverbs 29:11 · Proverbs 14:29 · Proverbs 22:24–25
New King James Version®, Copyright © 1982, Thomas Nelson. All rights reserved. Full copyright information. Scripture provided by API.Bible.
Go Deeper
If this verse has been used to put weight on you that it was never carrying, ask about it directly.