Live Chat & Podcast at 1:00PM Eastern on Sunday!
There's no such thing as a stupid question, but they're the easiest to answer.
JoinTour
Login
Search
Web Design & Development
Tag Cloud
access acer asus bios bsod crash desktop driver drivers error ethernet excel freeze gaming hard drive hardware hdmi internet laptop mac malware memory monitor motherboard network operating system printer problem ram registry router security slow software sound svchost.exe trojan ubuntu 11.10 uninstall usb video virus vista wifi windows windows 7 windows 7 32 bit windows 7 64 bit windows xp wireless
Search
Search for:
Tech Support Guy Forums > Internet & Networking > Web Design & Development >
How To Save Web Page From Command Line

Reply  
Thread Tools
Somnamna's Avatar
Computer Specs
Member with 128 posts.
 
Join Date: Jun 2004
Location: Phuket
Experience: Intermediate
19-Nov-2009, 02:04 PM #1
How To Save Web Page From Command Line
I need to download a web page as "Web page, HTML only" and save it to my hard drive. The problem is, it needs to be done from a command line - not by me sitting at the computer reading and clicking.

Here's the situation: I get emails at all times, day & night, telling me that a file is available on a web site. The emails contain a link to the "download page".

The "download page" has a link to the file. It is that link that I need to extract from the "download page", which I can do if I can save the "download page" to my hard drive.

Any ideas gratefully received.
MMJ's Avatar
MMJ MMJ is offline
Senior Member with 3,637 posts.
 
Join Date: Oct 2006
19-Nov-2009, 03:46 PM #2
Think you need a 3rd party program.

http://www.google.com/#hl=en&q=wget+windows
Somnamna's Avatar
Computer Specs
Member with 128 posts.
 
Join Date: Jun 2004
Location: Phuket
Experience: Intermediate
19-Nov-2009, 04:42 PM #3
Quote:
Originally Posted by MMJ View Post
Think you need a 3rd party program.

http://www.google.com/#hl=en&q=wget+windows
Yes, I've tried wget, but I can't get it to work with the web site.

The page I want to download the source of is: http://www.website.com/details.php?id=123456

(I've changed the name and details).

It may be that the "?" in the URL is causing a problem. I've tried changing it to "%3F", but no luck.

If anyone can give the correct syntax to get this single page source, I'd be very grateful.

Edit:

I've just tried WinWget and got the same error, so at least I know I was creating the same command line as WinWget does.

I think the problem may be because I need to log in. When I put the URL into my browser, it works, but I am logged in. If I log out and put the same URL in, I get a "not logged in" error page.

So I must check out how to added username and password to the Wget line.

Last edited by Somnamna; 19-Nov-2009 at 04:59 PM.. Reason: Added WinWget info
MMJ's Avatar
MMJ MMJ is offline
Senior Member with 3,637 posts.
 
Join Date: Oct 2006
19-Nov-2009, 06:57 PM #4
Using http://www.christopherlewis.com/WGet/WGetFiles.htm I downloaded any page very easily, even with a query string.

Code:
wget "http://www.google.com/#hl=en&source=hp&q=asd"
Are you getting an error message?
Somnamna's Avatar
Computer Specs
Member with 128 posts.
 
Join Date: Jun 2004
Location: Phuket
Experience: Intermediate
20-Nov-2009, 01:28 AM #5
Quote:
Originally Posted by MMJ View Post
Using http://www.christopherlewis.com/WGet/WGetFiles.htm I downloaded any page very easily, even with a query string.

Code:
wget "http://www.google.com/#hl=en&source=hp&q=asd"
Are you getting an error message?
Yes - here's the end of the log:
Quote:
Resolving www.xxxxxxx.xxx... 80.xxx.xxx.xxx
Connecting to www.xxxxxxx.xxx[80.xxx.xxx.xxx]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
03:54:25 ERROR 404: Not Found.
The 404 error message confused me. I'm sure I need to log in or send my user name and password. I think wget can do this, and I need to find the correct syntax.
Somnamna's Avatar
Computer Specs
Member with 128 posts.
 
Join Date: Jun 2004
Location: Phuket
Experience: Intermediate
10-Dec-2009, 03:07 PM #6
The best solution I've found is to use the "iMacros" extension for Firefox which allows you to write a macro to save the current page, as well as do lots of other things.

Can be got here: http://www.iopus.com/imacros/firefox/
jlhaslip's Avatar
Junior Member with 24 posts.
 
Join Date: Nov 2009
Experience: Advanced
11-Dec-2009, 12:20 AM #7
<?php
$source = file_get_contents('http://www.google.com');
echo $source;
?>
PLay with the file_get_contents function. You should be able to open a text file and save the page contents inside it instead of an echo.
Reply

THIS THREAD HAS EXPIRED.
Are you having the same problem? We have volunteers ready to answer your question, but first you'll have to join for free. Need help getting started? Check out our Welcome Guide.

Search Tech Support Guy

Find the solution to your
computer problem!




Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
WELCOME TO TECH SUPPORT GUY! Are you looking for the solution to your computer problem? Join our site today to ask your question -- for free! Our site is run completely by volunteers who want to help you solve your computer problems. See our Welcome Guide to get started.
Thread Tools



Facebook Facebook Twitter Twitter TechGuy.tv TechGuy.tv Mobile TSG Mobile
You Are Using:
Server ID
Advertisements do not imply our endorsement of that product or service.
All times are GMT -4. The time now is 03:01 AM.
Copyright © 1996 - 2011 TechGuy, Inc. All rights reserved.

Powered by Cermak Technologies, Inc.