In my opinion, Age I military is very important.
Firstly you say the events 'don't hurt much' but I disagree. Suppose in an average 4 player game 8 age I events get resolved, and say perhaps 6 of those will be during Age I. I dunno if that's really an accurate figure, but I think it seems reasonable. Even excluding territories for now, 7 events out of 15 care about strength - and naturally, every territory indirectly cares about it as well. So really that's around 13/21 events, or about 2/3rds which care about strength. In a typical game that means if you neglect strength, you're willingly either taking a penalty or giving away a reward around four times.
Secondly is the fact that neglecting military completely leads to a big problem with aggressions. You say those Age I aggresions don't hurt much, well, they still hurt somewhat AND give a reward to the aggressor. Enslave costs you 2-3 food and a CA typically (to get that Pop back), much more if you need to disband. That's around a turn's tempo of pop production lost, as well as a valuable earlygame CA. Raid varies, but at the very least losing a Philosophy is going to set you back 1 CA and 3 resources if you want to rebuild it (and most likely you will), which is around a turn's tempo of resource production, and again also the valuable earlygame CA. That also only costs 1 MA to play, making it doubly dangerous, AND if you've upgraded to Philosophy or other Age I urban buildings, it's even more damaging. Plunder is a little less scary - 3 resources/food is, again, around a turn's tempo lost but you don't lose a CA rebuilding, and it's 2 MA.
You say you often have bonus cards, but this won't always be the case. There are 6 bonus cards in the Age I military deck - the same number of aggressions. In a four player game, if you're weakest, you can't expect to consistently defend yourself this way.
Secondly you don't need to invest a huge amount in Age I to have a 'decent' military. One military tech, one tactics and one-two units in military keeps you up to a decent strength. And of course, these investments stay as a bonus to your strength in Age II and beyond, as well as giving you the techs. You don't need Cavalrymen if you have Knights, or Riflemen if you have Swordsmen, since you can make Age II tactics with those. This also lets you get onto the Age II tactics quicker - you might only need to snatch up one age II military tech to make your tactics, instead of two, and you might be able to make an antiquated unit much more easily (especially the case with e.g. Conquistadors and Classic Army).
I could say more in defence of Age I military but I really need to go out shopping now...