Yes, it's a bad idea to do it this way. The most likely time a RAID array will fail is during a rebuild as that is a whole bunch of drive activity over a sustained timeframe.
Better to perform a backup or copy, power down, remove all the old drives, install the new ones, power back up, configure a new array (most people recommend to use RAID 6 at a minimum, no hot spare, so you have two drive redundancy) then restore or copy back the data.
This way you can also keep the old drives as a cold backup of sorts, potentially reimporting the configuration if needed.