This phrase has always annoyed me, because I have deduced that the speaker is almost always NOT "just saying", but really means it and is trying to get the hearer to not say anything critical in return.
This commenter sums up my feelings admirably:
cybermaniac | Dec-11-06 12:35PM
This is a phrase which has a two part effect: 1) to deflect what would be a strong disgust/disagreement with that person's argument and 2)inserting the mere placeholder of a possibility that the argument is valid (though hollow).
I really hate that term. It's really an attempt to make an end run an extremely weak argument to make the declarant's point good in the face of overwhelming impeaching evidence or argument. The person using this term doesn't have to or want to invest any amount critical thinking skills and logic to support their position.
I'd be dismissive of that type of person as being an academically dishonest cretin.