it's good that you have replaced armrest pannel of your car.and you are very right we should check by pressing with finger wehther it's working or not.
you should be careful while playing the music it should not be so much loud that it will scratch your speakers.now change your speakers and play carefully in future.