It took the matter from the Nebula condensed it rapidly into a planet.
Or then it just finally hit its intended target, the nearest planet (namely, the D class rock Regula), and did its magic with that one. The nebula appeared to be gone in an eyeblink (when the
Reliant explodes, it's no longer there, it apparently being too much of a VFX hassle to include it in the shot).
But having expanding spheres of whatnot in space is perfectly realistic. Calling them "shockwaves" might be taking liberties with the term, as there may well be shock involved, but the sphere isn't doing any sort of wave motion in- or outside a medium. But just
having them is fine and well, especially if they move at any speed below c. In Star Trek, speeds above c are also allowed, of course.
As for blowing up planets, it may well be a case of the advanced and powerful aliens not bothering to downscale their doomsday weapon for this special application. An M60 machine gun might be excessive for killing a wolf, but if that's the thing you are carrying when the shepherd comes asking for a way to deal with the beast, there you have it.
It's not as if the Xindi designed or even truly built this weapon. They just acted as the physical hands of the real makers, who supposedly already were well beyond blowing up mere planets.
Timo Saloniemi