Why is the paint coming off? You have to address the root cause of the problem, before addressing the symptoms, otherwise you will be forever painting.
If it is water ingress, then it needs to be addressed first.
For cracks, you have to gouge out the crack even more, even if quite deep, you cannot just polyfilla over it and hope for the best, chances are that it will keep coming back. Once you have widened and deepened the crack, make sure that it is dust and debris free, use a blower to get every last bit of loose dirt out of it. If the mortar is still friable / powdery, you might want to use a bonding agent , paint the widened crack out with it and wait until it dries.
View attachment 1580540
For a crack where there is water ingress, use a latex-based bonding liquid.
Then irrigate the mortar, plaster and brick with water, allow the mortar and brick to soak it up. Use some chicken mesh as reinforcing, shove it into the crack, as deep as possible.
View attachment 1580530
Use multiple layers if the crack is wide enough. For very big cracks, you may consider a bigger and stronger mesh:
View attachment 1580532
Then use a chemical filler, like Smith and Co Patching Plaster, not polyfilla or any Gypsum-based crack filler, to fill the crack.
View attachment 1580546
Make sure that it goes into the deepest parts, pack it tight. Only mix small quantities, and what you can use, unlike pollyfilla you cannot save it, it will set rock hard even when put in an airtight container. Smooth it over, and use plaster skim for an even finish afterwards if needed.