The concern is not that Apple has this information (this is the same/less info your cellphone company has on you and Apple technically doesn't have it, it's stored only locally on the phone and the computer backup), but that the information is unencrypted. In that way if you loose your phone anyone who knows what they are looking for and can jailbreak the phone, can easily get access to it and tell where you have been (or they break into your computer and gain access to your iPhone backup). That's the privacy concern.

http://www.cnn.com/2011/TECH/mobile/04/21/iphone.tracker.explainer/index.html?hpt=Sbin

Though the Guardian is usually better oddly the CNN article in this instance is more thorough.

It's true that the location data in consolidated.db is not being uploaded anywhere. That would have been discovered when the whole Pinch Media beaming your personal information into space without end user knowledge or consent was made public. Anyone worried about this general location tracking has no idea how unsecured iOS actually is. (Although, to be fair, any computer is vulnerable to secure access exploits. *cough* Knoppix *cough*)

Even without their iron-fisted contractual schemes (like the 2-year AT&T mobile foot-bind with the original iphone) or gadget DRM, Apple have never interested me. The social clique of apple users has always struck me as obnoxious, a lifestyle choice for urban bourgeois and student yuppie bohos with too much discretionary income. I spy them lined up in one uniform column at the local coffeehouse, Notebooks at the bow, furiously blogging away without bobbing up for breath.

Or to, you know, speak. Some of their products are fine. I do own an ishuffle I was given as a Christmas gift 3 years back and it does have its uses, though I would have been fine with an off-brand MP3 player. And I don't buy any of their overpriced gadgets new either. Actually, the last apple product I bought was one of their first imacsmacs back when I was in high school; you remember, all the easter egg colored ones? Guess their "think different" marketing campaign bilked a Junior out of his summer job money. Since then, I've been reliably apple-free, however.