Totally agree. I got the best fitting jeans of my life, essentially new, from a thrift shop on my way down. Wonderful way to stay quite well dressed for very little money, and then not feel bad when one's size changes. For years I avoided thrift stores; personally, I would never throw out anything that might still be marginally useful, and therefore thought that others are the same. It was amazing to me that that is not true, and that very nice items can be found in such shops.
As to the original question, I have kept clothes that are made from good fabric, that I might be able to remake into something else. Other than that the only other "fat" clothes I have kept are, as someone else mentioned, old jeans for doing yard work, and the like.