[AdvancedSetup]

Mystery infestation strikes Linux/Apache Web sites

Mystery infestation strikes Linux/Apache Web sites
By Joe Barr on January 24, 2008 (7:18:05 PM)


According to a press release issued earlier this month by Finjan, a security research firm, compromised Web servers are infecting thousands of visitors daily with malware that turns their Windows machines into unwitting bots to do the bidding of an as yet unidentified criminal organization. Security firms ScanSafe and SecureWorks have since added their own takes on the situation, though with varying estimates on the number of sites affected. All reports thus far say the compromised servers are running Linux and Apache.

According to an article on ServerTune.com, the exploit involves a rootkit installed on the compromised server that replaces several system binaries with infected versions. When the system is booted, the infected binaries are executed, and as a result, dynamically created JavaScript payloads are randomly and intermittently served to site visitors. The malware JavaScript attempts to exploit vulnerabilites in Windows, QuickTime, and Yahoo! Messenger on the visitor's machine in order to infect them.

We asked the Apache Software Foundation if it had any advice on how to detect the rootkit or cleanse a server when it's found. According to Mark Cox of the Apache security team, "Whilst details are thin as to how the attackers gained root access to the compromised servers, we currently have no evidence that this is due to an unfixed vulnerability in the Apache HTTP Server."

We sent a similar query to Red Hat, the largest vendor of Linux, but all its security team could tell us was that "At this point in time we have not had access to any affected machines and therefore cannot give guidance on which tools would reliably detect the rootkit."

cPanel, a popular administration tool used by hosting companies that allows clients to manage their hosted sites, has posted a security note describing what the rootkit does after it's installed, and suggests two ways to check a server for the rootkit.

According to cPanel, if you are unable to create a directory name beginning with a numeral -- as in mkdir 1 -- you're infected. Another test is to monitor the packets from the server with the following tcpdump command:

tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]\{5\}\.js'"

One great unknown thus far is how the servers come to be infected. Absent any forensic evidence of break-ins, the current thinking is that the malware authors gained access to the servers using stolen root passwords. The earliest known victims, according to quotes by researchers in this ComputerWorld story, were sites run by large hosting companies, which could give attackers root access to hundreds or even thousands of Web sites when compromised.

Other than using and safeguarding secure root passwords, not much can be done at this time to be proactive in preventing servers from being compromised, so searching techniques similar to the tcpdump command above, which check to see if a server has already been compromised, is probably the best course of action available to administrators. We haven't found a good answer yet for disinfecting compromised servers, but a complete reinstall of Linux, Apache, and a new root password would certainly do the trick.

[bearqst]

Why even allow remote root access would be my first question.

[lotuseclat79]

Originally Posted by bearqst

Why even allow remote root access would be my first question.

Hi bearqst,

The very definition of a web server is to allow remote access, but not remote root access, so, you question would be moot. The issue is not that remote root access was allowed - it wasn't, but it was gained by nefarious method.

The malware penetrated the systems by attacking and specifically targeting Linux/Apache web servers with as yet an undetectable method, unless the root passwords were stolen as the article seems to infer from large hosting companies.

When root kits hide their presence, it is then very difficult to detect them.

-- Tom

[bearqst]

Agreed, it was the comment

"Whilst details are thin as to how the attackers gained root access to the compromised servers,

that made me ask.

[RobLinux]

Originally Posted by lotuseclat79

The very definition of a web server is to allow remote access, but not remote root access, so, you question would be moot. The issue is not that remote root access was allowed - it wasn't, but it was gained by nefarious method.

No, the definition of a web server, is that the host servers http requests. "root" access is a policy issue.

The malware penetrated the systems by attacking and specifically targeting Linux/Apache web servers with as yet an undetectable method, unless the root passwords were stolen as the article seems to infer from large hosting companies.

In past CGI scripts were often to blame and exploitable, once you're running a shell program on a machine a knowledgeable attacker can probably find an exploit to root access.

When root kits hide their presence, it is then very difficult to detect them.

The suggesion was that stolen root passwords were used to log in. Holes in ftp servers may also be responsible, hosted sites giving access to 1000's of sites tend to have many features any of which could be exploitable.

Configuring, an Intrusion Detection System (I used tripwire long time ago) would actually detect most root kits, though to be 100% you'ld need to boot off read-only media, with a trusted kernel & C library, rather than rely on the "possibly" comprimised machine.

Probably these hosts are run fairly laxly, there's a lot of web servers and it would be suprising if none of them, have been configured imperfactly, or been neglected and allowing a root exploit.

[lotuseclat79]

Hi Rob,

Duh! The httpd server services "both remote and local web requests" implied to be protocols covered under the http protocol suite (i.e. not just http, but https, ftp, etc.) - which is normally not via root access, but as you say, root access is a policy issue.

-- Tom