Malaysia’s ruling coalition lost a state constituency in a by-election on Saturday in a further sign of declining public support for Prime Minister Mahathir Mohamad’s alliance.