Learn All About Linux wget and HTTP authentication
We covered how to enable HTTP authentication in Nginx web server in a previous article
Nginx HTTP Authentication on your Linux VPS
As a result, we also need to cover how to fetch web objects with wget when the objects require http basic authentication.
HTTP Basic authentication allows to protect of web locations or subdomains with a basic user/password authentication schema.
Wget is the tool to download http/https pages or objects from your Linux VPS CLI and, fortunately, it can fetch these resources even if they are protected with HTTP basic auth. For resources protected with session/cookie systems, it is more complex.
Wget files from HTTP authentication websites
$ wget --http-user=[HTTP-USER] --http-passwd=[HTTP-PASSWORD] https://server1.domain.com/vault/file.xml
Since the username and password used by wget to perform the HTTP authentication are used in a CLI, they will remain in your shell’s history. Make sure you clear these lines by logging out and logging back in and editing the history file. For bash, it is .bash_history in your root directory.
You can actually create these settings on our platform in a few minutes utilizing our PCS (Private Cloud Solution) which allows you to have VPSie(s) on a private network – NAT – Port forward – traffic control for inbound and outbound – multiple gateway IPs which you could use for the load-balancing and failover.
FAQ
wget is a command-line utility used for downloading files from the internet. It supports downloading files over HTTP, HTTPS, and FTP protocols.
To use wget for HTTP authentication, you need to provide the username and password as part of the URL. For example:
wget --http-user=USERNAME --http-password=PASSWORD http://example.com/file.zip
Alternatively, you can create a .netrc file in your home directory and add the following lines:
machine example.com
login USERNAME
password PASSWORD
Then run wget without any authentication flags:
wget http://example.com/file.zipYou can use the -i flag to download multiple files listed in a text file, each on a separate line. You will need to provide the HTTP authentication details in the URL for each file. For example:
wget --http-user=USERNAME --http-password=PASSWORD -i files.txt
Where files.txt contains a list of URLs, each with their own authentication details:
http://example.com/file1.zip
http://example.com/file2.zip
http://example.com/file3.zipYou can use the -i flag to download multiple files listed in a text file, each on a separate line. You will need to provide the HTTPS authentication details in the URL for each file. For example:
CSS
wget --user=USERNAME --password=PASSWORD -i files.txt
Where files.txt contains a list of URLs, each with its own authentication details:
https://example.com/file1.zip
https://example.com/file2.zip
https://example.com/file3.zipYes, you can create a .wgetrc file in your home directory and add the following lines:
user=USERNAME
password=PASSWORD
This will save your authentication details for all future wget commands. However, it is not recommended for security reasons, especially if you are sharing the computer with other users. It is better to use the .netrc file or provide the authentication details in the command line.
