Malaysia’s ruling coalition lost a state constituency in a by-election on Saturday, in a sign of waning popularity of Prime Minister Mahathir Mohamad’s alliance.