It is shown in the last harry potter book(and technically the chamber of secrets as well), that horcruxes warp your mind while in your possession. They corrupt you into an bitter, grumpy, little prick over time. The locket turned the main trio against each other over a few weeks, the diary almost made Ginny resurrect the dark lord over the course of the year.
Harry technically turned into an horcrux himself after Voldemort tried to kill him after killing his parents. He was then given to the dursleys who 'cared' for him for the next 11 years in the close confines of their home. Ignoring the fact that Petunia was afraid of magic in the first place, this could bring up the possibility that Harry is responsible for their cruelness. Slowing turning them bitter as a result of the horcrux imbedded inside of him.