I suspect it depends on the type of fibre. Eating lots of vegetables should not be a problem, eating lots of wheat bran could be.

There is an inverse relationship between the amount of fibre eaten, and the amount of testosterone in the body. One reason you see body builders eating eggs and meat, and not so many bran muffins,,,,