I broke up with my boyfrind cuz i found out that he did a older woman 2 yrs b4 we started datin is that dumb?

well my boyfriend did a 48yr old woman when he was 18 and he still goes 2 her house so i dumped him but i really miss him cuz he is my best friend should i take him back