Fun with wget

Mon, 07 Feb 2022 00:00:00 +0000

wget multiple files

wget -r -l1 -A.mp3 http://aaa.com/directory

In the above example, -r and -l1 options together enable 1-level deep recursive retrieval, and -A option specifies lists of file name suffixes to accept during recursive download (.mp3 in this case).

ways to wget entire webpage

This one works so well, I’ve created an alias for it:

wgetMirror='/usr/bin/wget -o wget.log -mkEpnp –wait=9 –user-agent='\''Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'\'' –no-check-certificate'

Other options:

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains example.com --no-parent <url>

Modify User Agent

$ wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" <url>

wget from webdav with authentication

wget --http-user=user-id --http-password=password <URL>

Reference