Ad relevancy and usefulness - YPN was regularly serving me ads that had no relevance to the topic of this site at all.
While this impacted the earnings as less people clicked--my main concern with it was that it impacts the usefulness of the ads and look and feel of the blog. ie it doesn't look very professional for ads about pet grooming (I saw one of these yesterday) when your blog is about making money from blogs.
Darren lists two other reasons why he decided to leave Yahoo Publishing, with the second having something to do with text ad size (really?) and the last with the "Advertise on this site" link.
I've never been a huge fan of Yahoo outside of Flickr, but some bloggers have reported doing well on them.