So from what I can remember, Quidditch wasn't important to the plot in book 5. It was more important for character development (Ron's, specifically). But it would be tougher to get rid of Quidditch in book 1 (pretty foundational stuff for the young Harry, finally finding something he is good at, something which helps connect him to his father, a place where he feels like he belongs, not to mention the misdirect with Snape/Quirrel messing with Harry's flight). Or in book 3, when dementors attacked during a game and Harry's broom was destroyed. Or in book 4, where the World Cup was a huge thing (introducing portkeys, deatheaters, Crouch, Bagman, Krum and the whole international scene) and where Harry's flying skills were important for the first task.
Which is why I think they had to include the sport in the films. They could downplay it, sure. No need to have 10 minutes of game time where 5 would do. Harry's fall from Dobby's interference in book 2 wasn't super important that I can recall, so maybe that could have been scrubbed out. But they need to keep some Quidditch stuff in, at least on the periphery, so that they could hit those plot points when needed.