Part of it is the jet stream — strong high-altitude winds generally blow west to east, so eastbound flights fight a headwind in reverse and westbound flights often get a tailwind, making eastbound trips genuinely slightly longer in the air for the same route. The bigger factor is your body clock: flying east "loses" hours and forces you to sleep earlier than your body is ready for, which is a harder adjustment for most people than flying west and gaining hours, where you simply stay up later than usual.