Giving your speakers or subs less power will not hurt anything?

Actually you can damage a sub by trying to run a cheap amp that is too small. What happens is, as the amplifier tries to put out more wattage there is not enough voltage or input signal available to create the desired wattage. This drives the amp into what is called "saturation", this is where clipping occurrs. Basically, you are now sending out more distortion than actual sound signal which is very damaging to any speaker.