Monday, July 23, 2012

Working with Wget command to download a single file or to download files from a ftp with a username and password

Wget stands for World Wide Web and get.

Usually Wget is used to download a file from a http server (or https), and even ftp servers.
Wget can also be used to download an entire directory recursively.

In this post, I will show you 5 ways for using wget

1- Using Wget to download a single file

To download a single file, simply write wget followed by the url, for this example, I will download a test file from thinkbroadband.com
wget http://download.thinkbroadband.com/100MB.zip



Below, you will see the file being downloaded
Saving to: '100MB.zip'

34% [============>                          ] 36,639,898  5.90M/s  eta 13s


Saving to: 100MB.zip : This is the filename that is being saved on your server/computer
34%:  Percentage of the completion of the file being downloaded
36,639,898 : Number of bytes downloaded so far
5.90M/s : Current download speed
eta 13s : The remaining estimation time left to complete the download

Once the download is completed, you will see something that look like this:
Saving to: '100MB.zip'

100%[======================================>] 104,857,600 4.17M/s   in 23s

2012-07-23 23:29:16 (4.33 MB/s) - '100MB.zip '

In the last line, you will see:
2012-07-23 23:29:16 : the date the file was completely downloaded
(4.33 MB/s) : Average download speed for the entire download
100MB.zip : the filename of the downloaded file


2- Using Wget to resume a single file (-c option)

If you were downloading a file, and the download was interrupted whether you had to restart your computer, or internet disconnected or any other reason, you can use wget command with -c option to resume it:
wget -c http://download.thinkbroadband.com/512MB.zip

Saving to: '512MB.zip'

100%[++++++================================>] 536,870,912 8.99M/s   in 54s

2012-07-23 23:42:16 (8.04 MB/s) - '512MB.zip'

Notice the ++++ in the download bar, this means that this portion of the file was already downloaded, the ======= is the new portion being downloaded.



3- Using Wget to download in the background (-b option)

If you want to download a file while doing other stuff in the command line, you can use the -b option, for example:

wget -b http://download.thinkbroadband.com/512MB.zip
Continuing in background, pid 26530.
Output will be written to 'wget-log'

This will start the download in the background while giving back the shell command to you to allow you to write other commands.

To check the status of the download, simply enter the following command:

tail -f wget-log

and you will see the download status, for example:
509450K .......... .......... .......... .......... .......... 97% 25.2M 3s
509500K .......... .......... .......... .......... .......... 97% 10.8M 3s
509550K .......... .......... .......... .......... .......... 97% 6.25M 2s
509600K .......... .......... .......... .......... .......... 97% 22.4M 2s
509650K .......... .......... .......... .......... .......... 97% 11.5M 2s



4- Using Wget to download from a FTP

For this example, I will download a file from the mozilla ft:p
wget ftp://ftp.mozilla.org/index.html



5- Using Wget to recursively download a directory from a FTP with a username and password

To recursively download all the files from a ftp with a username and password, use the following wget command:
wget -r ftp://test:test@ftp.secureftp-test.com:21/subdir1/*

username = test
password = test
host = ftp.secureftp-test.com
port=21
directory to download = subdir1

 

Installing Wget (on CentOS 6.2)

If you get the following message while using wget:
-bash: wget: command not found

This means that your Wget is not installed.

To install wget, run the below command:
[root@myserver ~]# yum install wget
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.beyondhosting.net
 * extras: centos.corenetworks.net
 * updates: mirrors.serveraxis.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.12-1.4.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch             Version                   Repository      Size
================================================================================
Installing:
 wget           x86_64           1.12-1.4.el6              base           481 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 481 k
Installed size: 1.8 M
Is this ok [y/N]: y
Downloading Packages:
wget-1.12-1.4.el6.x86_64.rpm                             | 481 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : wget-1.12-1.4.el6.x86_64                                     1/1

Installed:
  wget.x86_64 0:1.12-1.4.el6

Complete!

Happy WGETting :)