It matters to your general health. Yolks contain alot of saturated fat as do butters and margerines. Too much can cause arteries to start filling with fatty deposits and can also raise your cholesterol. The cholesterol alone won't affect your levels but the sat fat will.

Fat is an essential element to your diet but it should be controlled. Healthy fats are the better option and, if you are bulking, should constitute approx 20% of your caloric intake.