There are plenty, but I am going to list only games that I spent a considerable amount of time on and THEN still couldn't beat.
There is plenty of salt here. If I offended a game you like, sorry. ¯\(°_o)/¯
SpaceChem - this is a really fun puzzle game with a unique premise (that's very hard to describe). It's sort of like programming conveyor belts to combine parts and get a desired result. Anyway, the game's difficulty ramps up considerably and very quickly. I made it to the fourth set of stages and could not finish them; I lack the skill to do it. I love the game, though. It's one of my favorites. I'm just bad at it.
Rayman Origins - I enjoyed a lot of things about this game, but when it comes to needing exact precision for long sequences of time, I give up after a certain point. The final levels of this game are maddening if you aren't that great at platformers. Probably no worse than, say, Super Meat Boy's, but I barely played that game so I'm not listing it.
Resonance of Fate - I made it to the final boss of this weird tactical RPG, and basically just hit a hump. My characters literally were incapable of doing enough damage to beat him before he could beat me. I am actually quite good at JRPGs and the like, but I do not grind in them at all, seeing it as a waste of time. I googled the ending of this game. Not proud of it, but I was not going to run through the unskippable final area again to take more cracks at the guy after upgrading my weapons.
Super Mario Bros 2 (Japan) aka The Lost Levels - This game is bull, total bull.
Dragon Quest II - Pretty much the same story as Resonance of Fate. The game isn't that hard for 3/4 of the way if you are familiar with the other DQ games, but it takes a sudden turn for the crazy at the end. The last boss was impenetrable for my party even using savestates, and I just decided it wasn't worth it.
New Little King's Story - I had never been more disappointed in a game, as the original is one of my absolute favorites (if you think there was nothing good on the Wii, give it a try). NLKS removed almost all of the charm the original had, and I endured it for a long time until it finally got to me and I had to stop.
Mother/Earthbound Zero - I plan to finish this for real some day, but it has a maddening encounter rate that got on my nerves big time after a while. Recent patches allow one to alleviate this some.
The Legend of Zelda: Skyward Sword - Trigger Warning, Zelda fanboys. I hated Twilight Princess, thought Phantom Hourglass was mediocre, didn't even bother with Spirit Tracks. I got this game hoping maybe the new controls would breathe some life in to this series for me, but it turned out they were awful, the overworld was awful, the dungeons were bland, and Fi ruined every surprise the game had. I made it halfway through the Cistern dungeon and called it quits; just wasn't feeling this game and felt ripped off. Fortunately, Link Between Worlds saved the series for me and now they're remaking my favorite game in the series, so I have not entirely given up on Zelda after this. (I did watch footage of the story at the end, and I think it was a bad idea, for the record. I will say that I regret finishing Metroid: Other M, and my refusal to let that disappointment happen to me again contributed to my hatred for Skyward Sword)
Ys 1 - The last few bosses in this game are the type that require precision I don't have, nothing to see here. It's a boring game anyway.
Saga 2 (Final Fantasy Legend 2) - I played to the very end of this game as a kid, but I didn't actually understand the mechanics and lacked the party composition to finish the game.
The Legend of Zelda 1 and 2 - I have played both of these most of the way through multiple times over the years, but at some point I tend to just get bored with them. Link to the Past fixed most of the issues I had with the first game, and to be frank I am just really bad at Zelda 2. Love its music, though.
Viewtiful Joe - Too hard for me, even on kids mode
Plants vs zombies 2 - Once it became clear EA had ruined one of my favorite games ever with its greed, I uninstalled this and warned all my friends to never go near it.
Etrian Odyssey 4 - I respect this series a lot more than I like it. Brutally hard classic-style JRPGs with enormous, confusing dungeons and ridiculous enemy encounters. I have played 3 and 4, given up on both, but I made it 3/5 of the way through 4. After a while, it just got taxing on me. I saw another huge dungeon ahead and couldn't go on. For the record, I have beaten Shin Megami Tensei: Strange Journey and other first-person dungeon crawlers, so it's not like I was inexperienced here. EO simply broke my patience, plain and simple.
Lufia 2 - I am convinced this is the most overrated game on the SNES. I am an RPG nut and manage to get bored with this game every single time I give it a go. To each his own, I guess. I even enjoyed Lufia 1 for the most part.