How long have you been riding with the magnetic "fidlock"? I had the same feelings about mine at first, felt like such a goon every time I put my helmet on. It's been a couple years now, I don't think twice about it anymore. It's become pretty natural, and I like it now.
The one trick I found is to make sure the left side fidlock buckle (opposite the red tag end side) is down. Took me a bit to realize it was often flipped up, tucked behind the cheek pad, and I'd be ridiculously swinging that red tag magnet all over trying to get it to latch. Now I always grab that left buckle with the strap when pulling the helmet on, ensuring it's always down and ready. Sounds tedious, but it's actually not, I do it too without thinking now.
Anyhow, food for thought. Maybe replacing it with a conventional buckle is possible, I haven't looked into it.