Here's a simple test to see if you're allergic to the milk sugar- lactose, or the milk protein- casein.

Take natural plain yogourt (that is, no additives whatsoever- just the milk and the bacteria as ingredients), and put some on part of your face like a face mask. If it makes you itchy after a while, it's the protein your skin is actually allergic to. If you don't feel itchy then it could be just you were having problems digesting the lactose before, and if you're having less problems now, then it should be okay to eat some.

That being said, it is not natural for humans to eat the breast products of cows. And lots of toxins are concentrated in products such as cheese and butter because toxins are usually fat-soluble: fat that is transferred to the milk then concentrated. Educate yourself about dairy, then do what you want accordingly for however much you want to incorporate into your diet.