If you have a modem, you will need to plug one computer into it then share that connection via that computer.
If you have a router, all your computers will connect via that router, either wired or wireless, and the router will either plug into the modem, or it will have a built-in modem.
From what you describe I think you have an ADSL modem router.
If you have cables, you need to connect each computer to the router.
If any of the computers has a wireless card, you will be able to connect wirelessly if the router is a wireless router.