Violence in northwest Nigeria has forced around 20,000 refugees to flee to neighboring Niger since April, the United Nations refugee agency said on Tuesday.