[PrestoChango]

The Squid proxy (2.5.STABLE6) that I am using on a server running CentOS 4 reports an interesting/incredibly frustrating error twice a day. It reads:
-----------------------------------
ERROR
Cache Access Denied
While trying to retrieve the URL: http://www.google.com/search?
The following error was encountered:

• Cache Access Denied

Sorry, you are not currently allowed to request:
http://www.google.com/search?
from this cache until you have authenticated yourself
-----------------------------------
This error comes up when using IE6 at ~7-7:30AM and ~6-6:30PM. It occurs for everyone on the network. If anyone is using Firefox or other browser, a popup will open asking for the user to type in user name and password. No matter what is done (reboots of client or server, etc.), this occurs for about 30 minutes. No one can access the internet at this time. Also, there are times when this message will come up during the weekends for hours on end BUT there are times, such as this Thanksgiving weekend, when the message does not come up in the last 5 days. IE6 automatically authenticates using the user's Windows logon information.

Does anyone know why this happens or how to fix it?
Thank you

[RobLinux]

What TZ are you in? I'm wondering if that's midnight GMT.

May be squid and Windows are disagreeing about the day in some time window?

[PrestoChango]

I'm in EDT (-5 GMT) so that could be an issue at night. The strange thing is that this happens twice a day and not consistently.

Something I forgot to mention earlier, but is probably important is about the BIOS time. The problem was showing up around 4-5 every night for a few weeks (or 13 out of 15 weekdays). I rebooted the server and set the BIOS time to be 3 hours earlier than what it was set to, and now this problem happens at about 6-7PM. It seems to be related, but not directly.

Could there be some kind of issue with the automatic authentication (since the issue seems to be the authentication) in the authentication server as opposed to the proxy server? Or, could it be the communication between the two? I wonder what would cause something like this.

[RobLinux]

Can you run xntp to synchronise with a Network Time Protocol server? There's also Windows NTP clients, and I think some kind of time service that network windows uses.

I remember there were problems with Kerberos if you didn't have accurate time keeping on your machines. Perhaps something screwy is going on due to that with this protocol to.

Credentials have some kind of TTL and shifts from local to gmtime and back, plus clock errors are possibly exposing some underlying bug that's hidden on most systems. Probably disagreement about which day, by more than a few hours.

Computer clocks are supposed to run a little slow (so you never have to move time backwards) then may be the effect is scewed a little due to that, later in morning, and earlier at night when there's some disagreement. It is odd though.

Perhaps the machine is just trying to tell ppl not to work so hard, but arriver later and leave earlier?

Just a theory, when it's time related it's nearly always down to broken machines, not running on UTC but being localtime and getting conversions wrong. If you can find something authoriative on squid site, go with that rather than this idea.

[PrestoChango]

Thank you. NTP was installed, but not running. I started ntpd after checking the configuration file ntp.conf. I'm not sure of what xntp is, but am looking into it right now.

Ntp.conf reports that NTP will sync with 0.pool.ntp.org,1.pool.ntp.org, and 2.pool.ntp.org. I know that this is a M$ question, but is there a way to sync NTP with a domain controller running Windows Server 2003 (probably running w32time)?

In my research, I came across a page (http://support.microsoft.com/kb/816042) that confirms that you know what you are talking about (not that I would doubt a competent technician such as yourself, but rather would like to give you recognition for your help thus far).

Also, I did not set this proxy up myself (only a small fraction of it) because we were busy enough to outsource some network setup, but the company helping us was a bit suspicious. I would not put it past them to set up some kind of set hours (outside of them originally limiting network/logon access). Humorous, but a very good point.

I am continuing research on the squid wiki also. Thank you for your help. I will see how this works and try to find a way to input correct values into the ntp.conf file.

[RobLinux]

Well I've seen some funny bugs in my time. Once it was a printer which would take an hour to print in Holland. The cause was hidden during DST, but in winter when they were GMT+1, a UTC v localtime bug surfaced!

Thanks for the appreciation, but I was really firing "hunches" at you, as I know what it's like when these weird stuff goes on and it is crazy but you cannot find answers.

xntp is the NTP implementation that is installed with most Linux distro's, I also use chrony in cases where the machine is irregularly running Linux or does but not connected to the Internet that often.

ntpdate is a command to set, your clock to Network Time. You should find some NTP servers, that are local to you, perhaps your ISP has something like ntp.some.where.net defined, usually they are ntp0, ntp1 or ntp2. The ntp.org pool is OK, but there's regional versions.

If the time is the problem, then actually it'll be the Squid cache server, and the Win Credentials logon server that need to be in agreement.

Rather than configure server to use a Win time service, you set up the Win time server's clock to be set to NTP on boot.

Reading the M$ explanation, I see they use Kerberos, well I did not know that till now, the problem was seen with BIND/Hesiod. Part of well run network IMO is to have automatic time synchronisation, to reduce maintenance, and ease problem tracking through log files, eliminating clock skew.

http://www.ntp.org/ has public timeserver lists, you may need to open up a UDP port in your network firewall. http://www.pool.ntp.org/zone/north-america

http://en.wikipedia.org/wiki/NTP_pool is a nice explanation

Try

ntpdate -q {2,3}.north-america.pool.ntp.org

Which should give you "offsets" from your clock to the servers, but not update your RTC clock. These pool entries are selected at random by DNS round robbin, so it is better in ntp.conf to use a stable ISP NTP server or two, may be a Uni one somewhere that's Stratum 2 or lower, if you're going to run NTP service in your network.

You should know whether your RTC is set to localtime or UTC as is usual on servers.

Synchronising via command, daily or every few hours via cron should be more than accurate though for most needs, I do like to run NTP server in tandem with DNS, for convenience reasons in network. xntpd supports broadcast, so it's possible to have Zeroconf on workstations.

[PrestoChango]

On Tuesday, I set the ntp.conf file to sync to the domain controller (M$ Server 2003) or to the North America pool time servers intentionally. I left a computer on overnight (because nothing happened during the day) with Firefox running because #1) I like it and #2) if the authentication error occurs the popup asking for username and password stays on the screen. This produced interesting results:

1) The last three days, the message did not come up during working hours.
2) The message came up overnight all three nights.
3) Because of an increase in work, I have brought work home and napped in between long processes. This has helped me notice that the problem is now occurring between 10PM and 2AM in my time zone. This would not be a problem, but it may be in the future and will start working on putting Server 2003 and the Linux server on the same time server.

I don't know for sure about whether this will continue to only occur between 10PM-2AM because the problem is not always consistent, but will continue to watch it.

Thanks for your help and expertise. It is very much appreciated.

[RobLinux]

You'll only need them to be within reasonable agreement, not < 10ms.
M$ have a few time standard options, last time I post-OEM installed a (Vista) PC that included NTP. Think you mean you run NTP server on Domain controller and sync the cache to it.

You have checked that the TZ's are set right on the servers to?

Sorry I didn't user authentication on the cache, but IP address based access, and frankly I threw away host info in log files, as without Management Policy, I didn't want to become the Net Cop, snooping on ppl's surfing.

Once you have the time in sync, perhaps you should try posting to the Squid Users mailing-list, after searching the archive of course. Bugzilla search, http://www.squid-cache.org/bugs/bugl...ed+twice+daily
doesn't seem to turn up anything, obviously relevant.

[briealeida]

Forgive me if I missed it but I didn't see any references to the Squid.conf file. Are you certain that the problem is not there? Are you using the default squid.conf file or one supplied to you by someone else? Are you attempting to block access to the cache at some point in time?

[PrestoChango]

Rob, that is correct. NTP is on Domain Controller and the cache server is syncing to it. The servers are all running the same time zone.

Problem was solved...the authentication fails at 11:30-midnight roughly (no problem because 99% of users are kicked out at that time). What I just found out today is that the internet fails at about 7-7:30...so problem is only solved by 50%.

I decided to run NTP on the domain controller to sync to a regional time server and am also running ntpd on the squid server to sync to the same regional time server and restarted the process on both servers). I will see if this changes anything. Ironically, the time is now synced to the second where as before it might be off by a few (not that this is a big deal).

Bri, in the Squid.conf file, pretty much everything is at the default setting. It has a lot of commenting, but other than that, it is almost completely default. Is there another file I could check to see if access to the cache is blocked?

Will report on how the new NTP settings work tomorrow.
Thanks

[PrestoChango]

Since 5PM last night, the authentication has not failed...a good sign as long as things do not turn sour during the day.

[PrestoChango]

Immediately after midnight today, the authentication failed for about 30 minutes. It failed again this morning at a little before 7 until a little after 7 (which is noon GMT). The first person who tried to get on the internet this morning sent me an email at 7:10 saying that the internet was working (because I asked her to check the internet when she came in).

Maybe this has something to do with a windows service? I can't imagine what could be wrong, but there is a service in Server 2003 called WinHttpAutoProxySvc that stops itself and restarts itself for no known reason. This does not affect connectivity though.

[RobLinux]

Weird. I think you should ask on squid mail list, presuming you did check for odd policies, in squid.conf and also for any cron jobs in the root crontab file, /etc/crontab blah blah blah.
Doesn't make sense 5am GMT and 12 noon.

Somone might have seen something screwy before and know what it is, I can't comment on the win services.

Because the problems moves with the time of the machines, you have eliminated a badly set up box on some timer. if the authentication fails quickly then its' a reject rather than a timeout.