With every second, your e-mail list is aging. If you do not send mail on a regular basis, the validity level of your e-mail addresses is decreasing.
An old list causes lots of problems. Increased bounces is only one of them- it can also lead to increased complaints.
E-mail addresses are much more transient than physical addresses. They can be changed easily and with the proliferation of free accounts they are often changed frequently. Changed addresses not only means that you will have fewer deliverable messages due to bounces, but it could also cause some e-mail providers to filter your e-mail from everyone else!
Also, if the original address was forwarded to another person due to a change of jobs, the new recipient is more likely to complain. Lastly, if you wait too long, valid recipients may not remember opting into your list and they may report your message as spam.
Remember, you should be sending mail to your list on a regular basis. This not only keeps your list clean, it also help build customer loyalty and recognition.