You will need to feed the audio to the receiver via an optical cable and the video to the TV via the HDMI cable. Depending on the receiver you may be able to feed the HDMI to the receiver then out from it to the TV on another HDMI cable.
The only way you are going to get 5.1 out of the PS3 is...