I've never had any damage to an Apple Watch, but, I've never been very hard on any watch. Wear mine all the time when working outside, hiking, pickleball.
I prefer the Sport Loop band for comfort. I do have a couple colorful cheap rubber knockoffs I use for kayaking or beach activities.