They aren't toiling they are working and getting paid damn well for working part of the year and not the whole year
Modern athletes do basically work year round. The games themselves don't take place year round, but there are a lot of mandatory and voluntarily-mandatory (i.e. You can't be cut specifically for not attending the ones in the "voluntary" category, but it influences your chances of holding on to a job if you don't show) off-season training and conditioning activities, mini-camps, training camps, pre-season games, regular season game, possible post-season games, and of course practices through the week on game weeks and plenty of traveling. I am thinking of the NFL in particular, but it probably applies to most big league sports. And when there isn't a specific thing you've got to be doing or be at, you're expected to maintain peak physical condition on your own through your own exercise regime. I've seen guys get cut or take severe pay cuts for simply showing up to training camp out of shape (In the old days, you used training camp to get into shape, not any more).
Also, athletes take extra risks with their physical health. Not only do you have the usual risks of athletic activity like broken bones, torn ligaments, hernias, hamstring issues, ACL injuries, rotator cuff injuries, etc., you also have some sport specific injuries like in the NFL, where there are frequent concussions and the like. There are players who get out of the league and wind up with long-term brain trauma that results in dementia and several depression (that leads to some suicides and autopsies reveal brain injuries), are physically unable to walk around very well or pick up their children, can't feel or get full movement in some limbs, etc.. There are rare instances of players being paraylzed or the like, though you don't see it often, statistically. These players also deal with an incredible amount of pain and some wind up addicted to prescription pain killers and the like.
Anyhow, I know the NFL has done a ton of work lately to make the game safer through rule changes and putting doctors on the field to examine players who may have concussions and keep them out of the game and future practices until they are medically cleared, etc.. Things are starting to become safer. A real effort is being made there. But the fact is, these players do take risks that a lot of people don't. And if they are on the lower end of the pay spectrum for athletes in the top leagues, which granted is still a great living, and only last a few years in the league, they find themselves needing a regular job, but being unable to do it due to the injuries they've sustained.
Anyhow, I don't begrudge the players a dime, especially when I know it'd otherwise just go into the owners' pockets. As folks have mentioned, Nike pays workers pennies an hour to produce shoes in Asia, and they still charge $120 for them.

Production cost doesn't always have a ton to do with the end cost of the product. What has raised the price of sports tickets and television rights to expanding demand for them and a greater willingness to pay. Football taking off on television, for example, generated a lot of this. It's become a big deal for networks who can use it as a platform for promoting their shows, and can get very high ad rates. If my favorite team plays on CBS on Sunday afternoon, I always seem to know what's on "How I Met Your Mother" and "Rules of Engagement" the next night.
I also think that there was a time where sports fans were mainly blue collar types, whereas today I think you see big sports fans from across the economic spectrum, some of whom of course have more disposable income. I have a relative with tickets to a sports franchise, and the guys who sit in front of him are all lawyers, and one of the guys seated to his right is a major real estate developer- and those are in the "cheap seats". I don't think that would have been all that common years ago. I also think there are more "geeks" or "nerds" involved in sports fandom than there used to be. It's not all ex-jocks living vicariously through their favorite teams now that their high school careers are over. Fantasy football, statistical analysis, and the like has been a big driver of interest from all sorts of different personality types. I saw an article a few years that said 40% of the NFL's attendance comes from women, who used to be a much smaller minority of sports fans. Basically, there are a lot more people who are potentially going to tune into their televisions or buy tickets and merchandise, and so on and so forth.
Sporting events used to be mainly blue collar fathers and sons watching games. Now, it's a whole family thing for people in all socio-economic brackets, in many cases. Of course, that increase interest drives increased revenues.