I think people are not aware of the power of podcasts that's all . I use them extensively, especially while driving.
I think a big limitation was the bandwidth required to download these podcasts. An hour long audio podcast can be in the 60MB range and most podcast subscriptions are WEEKLY (i.e. Engadget/PCMag )
There's also VIDEO podcasts that are becoming very popular, all the ipods/zunes can play these things and they are actually perfect for small screen devices since they're episodic content which you can consume "on the move" .