I have 3 boys - 9, 7 and 1. I guess I would say that I have some guilt because we haul our 1-year-old all over the place just about every weekend for his older brothers wrestling. It is not fair to him, having to be tied down and unable to move around much for an entire weekend! He is just getting up walking pretty good so not being able to move around much gets to him at times.
I also look at it as family time and that helps me not feel so guilty. I guess in a few more weeks my 1 year old will get to be outside at baseball practices (then football practices) and he will like that a ton! He just needs to survive 2 more weeks . . . and then he will have a break until November!!