After having LTE for a month now, I would have to agree with you. I went from around 8MB download to 30MB download with LTE and it really didn't make much of a difference unless I am downloading large apps.
I also have been eyeing the Nexus 4 and might just make the purchase for a Christmas...