The study your doctor's referring to was one of the worst I've ever seen. Share this link with him:
ExcelMale.com Identifies Major Flaws in Recently Published Testosterone Study in Veterans
Also, we have more than 80yrs of data revolving around testosterone. Why so many physicians ignore this I can't answer. Moreover, the amount of women undergoing HRT isn't decreasing, it's increasing. Methods have evolved from what they were but that doesn't mean they're decreasing the number of patients.
Overall, regardless of what your numbers are, if you feel great, you're blood's not getting thick and you're in good cardiovascular health, there shouldn't be much to worry about.