A Malaysian religious court sentenced five men on Thursday to jail, caning and fines for attempting gay sex, media and a rights group said.