A capacitor is two conductors separated by a dielectric (an insulator in English). Kinda explains why the symbol on a schematic is two straight lines separated by a space. The conductors can be all sorts of things, but in most of 'em you see it is a metal foil. The dielectric can be lots of things too (air, paper, oil, polyethylene, polypropylene...). The conductors in a capacitor are also called "plates" (probably referring to cool old capacitors that were just a pair of metal plates separated by air... if you have ever looked inside an old tuner the odd set of metal fins that rotate between each other for tuning is an air-gap capacitor )
When you connect different potentials (voltages in English) to either side, electrons try to go from the higher potential side to the lower.. but there is an insulator in the way -- the dielectric. Because the electrons can't go through the insulator and because there is still a voltage present trying to push them through, the capacitor stores up those electrons -- also called storing up a charge. FWIW, capacitors have also been called condensers.
If you swap the voltages (high where there used to be low, low where there used to be high) the stored electrons (charge) will rush out from the plate where they were stored and will also start to collect up on the other plate in response to the new situation.
Current (electrons) cannot pass through a capacitor because of the insulator. However, this charge/discharge behavior in response to an applied voltage makes them act as if AC can pass through. It is exactly this behavior that makes them great for coupling tube stages -- they block the high DC voltage from the plate of the previous stage, but transmit the AC information through the charge/discharge cycle.
It is also this "block DC, pass AC" behavior that makes 'em great for "filter" applications.
The "DC" in your amp is actually rectified AC. That means it is almost DC but has some ripple in it from the rectification process (subject of another diatribe next time I have insomnia). The rectified signal looks like a small AC voltage applied over (offset by) a large DC voltage. The tubes (or, gaak, transistors) in yoru amp can't tell the difference between this ripple AC signal (again, purely an artifact of making "DC") and the guitar's AC signal, so they would amplify both. The amplified ripple sounds like hum.
If you look at a power supply schematic you will see that the capacitors are connected from the DC mains to ground. The action of blocking DC while transmitting AC by these capacitors is used to shunt the ripple to ground while preserving the DC component for use by the rest of the amp. This is filtering.
Recall also that capacitors will store a charge (this is why they can kill you if you don't take care to drain 'em before playing with them). Under short term demand -- when the amp wants more power now! and the power transformer cannot respond fast enough, the capacitors can make up some of that demand by discharging. The "stored" characteristic tends to hold the DC voltage stable.
There are lots of other nifty bits about 'em, but I think this mostly answers your question... and the insomnia is wearing off.