Internet problems -- virus related?

Recently I've been having trouble with my internet connection. Most of the time it seems to run fine, but recently I've noticed that it slows down a little before loading the website. Sometimes it won't even connect to the websites until I refresh and try again. :/

I've had bad experiences with viruses before and this one is certainly filling the bill. I have Norton Internet Security and so far it is not logging anything suspicious and all full-scans (including in safe-mode) shows that the PC is clean. But I'm pretty sure internet doesn't bog by itself.

If I use the internet right after I start up the PC, I usually get these "Cannot connect to server" pages when it does happen. But after a while, they seem to go away until I restart my PC again. And it doesn't matter what part of the day I'm using the internet.

Some guys at the Symantec forums say it might be an ISP issue. But I'm kinda hesitant in going with that story. Is it really just an ISP issue or did I just grab some malware from a site (I haven't visited any new sites of the late, just fyi. Just all the ones I've visited for about a year and over now) and Norton is not detecting it?

Please help? If it really is a virus, I'd like to take care of this as soon as possible. And preferably without having to do a clean reboot.

OS: Windows 7
Antivirus: Norton Internet Security
Browser: Firefox 3.5
Typical download speed: (30-40 kb, due to being stuck in a rural area. Btw, this hasn't changed much from before and after this weird internet issue. I've ran the speedtest.net for an hour and nothing significant has changed.)

The rest of the PC is running at normal capacity. No extra usage of CPU has been detected and there's no suspicious-looking processes. Just this internet thing.

If you need more info, just lemme know.