No, you don’t need to worry about soy.

There is no difference supplementing with soy over animal protein in terms of gains in muscle mass or strength in response to resistance training.

Randomized interventions with dietary soy show decreasing prostate specific antigens but not testosterone.

When scientists combine all of the soy intervention studies that have been done that look at soy’s impact on testosterone levels, they find that dietary soy has no impact on reproductive hormone levels in men at all.

There are some studies out there that suggest soy may reduce free testosterone, but the bulk randomized trials do not support this assertion when soy is consumed at normal dietary levels.

If you want to worry about phytoestrogens, stop drinking beer. Not only is alcohol a carcinogen, but the hops used in beer production have phytoestrogens that are orders of magnitude more powerful than soy.