To expand upon FreewheelinFrank's note, malware theoretically can exploit a security flaw in anything that touches it, not just a browser. Before your browser sees a web page, it passes through several layers of hardware, firmware, and software. Malware could exploit a security flaw in any of these layers to infect your machine.
Imagine that, for example, your operating system's ethernet or wifi driver has a bug that writes some data onto a portion of the stack it doesn't own when it receives a certain kind of IP packet. Imagine also that an attacker has discovered how to exploit this flaw, and has put her exploit (malware) onto a website. When you browse to that site, your browser asks the operating system to request data from it. The website sends, among other things, the malware back to your computer. Your computer's ethernet driver reads the malware, malfunctions, and eventually begins to execute the malware, which can then do anything it wishes.
