As a developer I can say that yes there is a fragmentation issue with Android esp when you trying to do specific functions or trying to do something that has changed in the recent libs but at the end of the day its my problem not a users one. What I can advice for people is to look at an Android like looking at a PC.
If you have software like say a PDF Reader and a game like Assassins Creed:
You not gonna expect to run AC on your old Pentium 3 running Windows XP, yet you can run the PDF Reader on it. If you have a recent computer you can certainly buy the game knowing it will run, doesn't matter if you using AMD or Intel, ATI or Nvidia.
Now the same goes for Android phones. If you get a well known brand, you can certainly know that it will be covered. If you just want to surf the Internet, edit documents use a currency converter, edit a document listen to music, play reversi or solitaire and shoot off some email, the old and even slow Android phones going to do the job just fine but if you going to play 3D intense games make sure you get something recent + decent.
I know some people would say that is what make iPhone so good cause you can do everything*. The problem is what about the people that only want email + solitaire, and dont have enough money to buy the biggest thing on the market. Shall we deny them phones just because they dont want latest games or Photo Editor? If you buy an Android phone at the same price as the latest iPhone you can be sure it will run whatever apps you place on it. If you go cheap and get a R99 special at MTN dont expect it to run a top of the line FPS or resource hungry apps.
* I have come across plenty of software in the iPhone market that also don't work on the older devices so this argument is not totally true.