The jury is out, at the present time, on the degree of harm that fats do to our bodies. There is, as you probably know, a diet that relies on fats for nutrition, with other food types being cut from the diet instead. Saturated fats are always cited as harmful whereas oily fish is cited as healthful.
Medical advice has changed on the subject of fats and therefore I am unhappy answering this question until more research has clarified the position. However, it does seem to be a feature of long-lived persons that they keep oily fish in their diet and may have little dairy foods.