It's a bit more complicated than that. Magic was banned out of fear. I grew up on a prison island because people were terrified I'd be like my mother. She used her magic to try to murder someone, long before I was born. Most of the people on that island were magic users in some way who abused their abilities to harm others.

So it wasn't us that put the ban on ourselves. Others did, because they'd seen how much harm magic could cause, never mind the fact that some of their very own were very good people with magic.

Fear does funny things to people. Their reactions, while... very dramatic and probably in hindsight very over the top...were entirely understandable.