Excuse Me, Ma'am

Are you a Miss or a Ma'am?

When I worked in a grocery store, part of being a cashier was being polite to customers. Mostly this was the easiest part of the job, but the question of which term, "miss" or "ma'am", would come up almost daily. Some women would laugh and say that being called "ma'am" makes them look around for their mother or even worse, feel that they are too old for shopping without a walker.
Of course, I didn't mean to call them old, but in my head the rule was simple--"Miss" means an unmarried woman and "ma'am" is a married one. It corresponds to the titles, Miss and Mrs. That they would take offense to something so simple and mundane was confusing to me. They wouldn't feel old if I called them Mrs. Soandso, right? Wouldn't a married woman correct me if I called her, Miss Soandso? That is assuming that I enunciated clearly enough to distinguish the difference.
Now that I'm in my thirties, I am getting a better idea of the point, but from the other side of the fence. Ma'am is starting to feel patronizing, especially from a younger person, but Miss seems like transparent flattery, and it is murky waters with the marriage rules. Some women in their early thirties are not married and never have been, so "miss."
But honestly, I think that the problem is less with the terms and more about confronting aging. Thirty isn't old, but I never thought that I would get to it so quickly. And that college-aged clerk calling me ma'am who can tell that I am not 21 anymore is more damaging than the term she uses.