Class Downloader

java.lang.Object

org.owasp.dependencycheck.utils.Downloader

public final class Downloader
extends Object

A Utility class to centralize download logic like HTTP(S) proxy configuration and proxy- and server credential handling.

Author:

Jeremy Long, Hans Aikema

Method Summary

Modifier and Type	Method	Description
void	configure(Settings settings)	Initialize the Downloader from the settings.
<T> T	fetchAndHandle(@NonNull URL url, @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler)	Download a resource from the given URL and have its content handled by the given ResponseHandler.
<T> T	fetchAndHandle(@NonNull URL url, @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler, @NonNull List<org.apache.hc.core5.http.Header> hdr)	Download a resource from the given URL and have its content handled by the given ResponseHandler.
<T> T	fetchAndHandle(@NonNull URL url, @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler, @NonNull List<org.apache.hc.core5.http.Header> hdr, boolean useProxy)	Download a resource from the given URL and have its content handled by the given ResponseHandler.
<T> T	fetchAndHandle(@NonNull org.apache.hc.client5.http.impl.classic.CloseableHttpClient client, @NonNull URL url, @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler, @NonNull List<org.apache.hc.core5.http.Header> hdr)	Download a resource from the given URL and have its content handled by the given ResponseHandler.
String	fetchContent(URL url, boolean useProxy, Charset charset)	Retrieves a file from a given URL and returns the contents.
String	fetchContent(URL url, Charset charset)	Retrieves a file from a given URL and returns the contents.
void	fetchFile(URL url, File outputPath)	Retrieves a file from a given URL and saves it to the outputPath.
void	fetchFile(URL url, File outputPath, boolean useProxy)	Retrieves a file from a given URL and saves it to the outputPath.
void	fetchFile(URL url, File outputPath, boolean useProxy, String userKey, String passwordKey, String tokenKey)	Retrieves a file from a given URL using an ad-hoc created CredentialsProvider if needed and saves it to the outputPath.
org.apache.hc.client5.http.impl.classic.CloseableHttpClient	getHttpClient(boolean useProxy)	Gets a pre-configured HttpClient.
static Downloader	getInstance()	The singleton instance for downloading file resources.
org.apache.hc.client5.http.protocol.HttpClientContext	getPreEmptiveAuthContext()	Gets a HttpClientContext that supports pre-emptive authentication.
String	postBasedFetchContent(URI url, String payload, org.apache.hc.core5.http.ContentType payloadType, List<org.apache.hc.core5.http.Header> hdr)	Posts a payload to the URL and returns the response as a string.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

getInstance

public static Downloader getInstance()

The singleton instance for downloading file resources.

Returns:

The singleton instance managing download credentials and proxy configuration

configure

public void configure(Settings settings)
               throws InvalidSettingException

Initialize the Downloader from the settings. Extracts the configured proxy- and credential information from the settings and system properties and caches those for future use by the Downloader.

Parameters:

settings - The settings to configure from

Throws:

InvalidSettingException - When improper configurations are found.

fetchFile

public void fetchFile(URL url,
 File outputPath)
               throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException,
URLConnectionFailureException

Retrieves a file from a given URL and saves it to the outputPath.

Parameters:

url - the URL of the file to download

outputPath - the path to the save the file to

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

URLConnectionFailureException - is thrown when certificate-chain trust errors occur downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

fetchFile

public void fetchFile(URL url,
 File outputPath,
 boolean useProxy)
               throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException,
URLConnectionFailureException

Retrieves a file from a given URL and saves it to the outputPath.

Parameters:

url - the URL of the file to download

outputPath - the path to the save the file to

useProxy - whether to use the configured proxy when downloading files

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

URLConnectionFailureException - is thrown when certificate-chain trust errors occur downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

fetchFile

public void fetchFile(URL url,
 File outputPath,
 boolean useProxy,
 String userKey,
 String passwordKey,
 String tokenKey)
               throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException,
URLConnectionFailureException

Retrieves a file from a given URL using an ad-hoc created CredentialsProvider if needed and saves it to the outputPath.

Parameters:

url - the URL of the file to download

outputPath - the path to the save the file to

useProxy - whether to use the configured proxy when downloading files

userKey - The settings property key for a potentially configured configured Basic-auth username

passwordKey - The settings property key for a potentially configured configured Basic-auth password

tokenKey - The settings property key for a potentially configured Bearer-auth token

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

URLConnectionFailureException - is thrown when certificate-chain trust errors occur downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

Implementation Note:

This method should only be used in cases where the target host cannot be determined beforehand from settings, so that ad-hoc Credentials needs to be constructed for the target URL when the user/password keys point to configured credentials. The method delegates to fetchFile(URL, File, boolean) when credentials are not configured for the given keys or the resource points to a file.

postBasedFetchContent

public String postBasedFetchContent(URI url,
 String payload,
 org.apache.hc.core5.http.ContentType payloadType,
 List<org.apache.hc.core5.http.Header> hdr)
                             throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException,
URLConnectionFailureException

Posts a payload to the URL and returns the response as a string.

Parameters:

url - the URL to POST to

payload - the Payload to post

payloadType - the string describing the payload's mime-type

hdr - Additional headers to add to the HTTP request

Returns:

the content of the response

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

URLConnectionFailureException - is thrown when certificate-chain trust errors occur downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

fetchContent

public String fetchContent(URL url,
 Charset charset)
                    throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException

Retrieves a file from a given URL and returns the contents.

Parameters:

url - the URL of the file to download

charset - The characterset to use to interpret the binary content of the file

Returns:

the content of the file

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

fetchContent

public String fetchContent(URL url,
 boolean useProxy,
 Charset charset)
                    throws DownloadFailedException,
TooManyRequestsException,
ResourceNotFoundException

Retrieves a file from a given URL and returns the contents.

Parameters:

url - the URL of the file to download

useProxy - whether to use the configured proxy when downloading files

charset - The characterset to use to interpret the binary content of the file

Returns:

the content of the file

Throws:

DownloadFailedException - is thrown if there is an error downloading the file

TooManyRequestsException - thrown when a 429 is received

ResourceNotFoundException - thrown when a 404 is received

getPreEmptiveAuthContext

public org.apache.hc.client5.http.protocol.HttpClientContext getPreEmptiveAuthContext()

Gets a HttpClientContext that supports pre-emptive authentication.

Returns:

A HttpClientContext pre-configured with the authentication cache build from the settings.

getHttpClient

public org.apache.hc.client5.http.impl.classic.CloseableHttpClient getHttpClient(boolean useProxy)

Gets a pre-configured HttpClient. Mainly targeted for use in paged resultset scenarios with multiple roundtrips.

Parameters:

useProxy - Whether to use the configuration that includes proxy-settings

Returns:

A HttpClient pre-configured with the settings.

fetchAndHandle

public <T> T fetchAndHandle(@NonNull URL url,
 @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler)
                     throws IOException,
TooManyRequestsException,
ResourceNotFoundException,
URISyntaxException,
ForbiddenException

Download a resource from the given URL and have its content handled by the given ResponseHandler.

Type Parameters:

T - The return-type for the responseHandler

Parameters:

url - The url of the resource

handler - The responsehandler to handle the response

Returns:

The response handler result

Throws:

IOException - on I/O Exceptions

TooManyRequestsException - When HTTP status 429 is encountered

ResourceNotFoundException - When HTTP status 404 is encountered

URISyntaxException

ForbiddenException

fetchAndHandle

public <T> T fetchAndHandle(@NonNull URL url,
 @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler,
 @NonNull List<org.apache.hc.core5.http.Header> hdr)
                     throws IOException,
TooManyRequestsException,
ResourceNotFoundException,
URISyntaxException,
ForbiddenException

Download a resource from the given URL and have its content handled by the given ResponseHandler.

Type Parameters:

T - The return-type for the responseHandler

Parameters:

url - The url of the resource

handler - The responsehandler to handle the response

hdr - Additional headers to add to the HTTP request

Returns:

The response handler result

Throws:

IOException - on I/O Exceptions

TooManyRequestsException - When HTTP status 429 is encountered

ResourceNotFoundException - When HTTP status 404 is encountered

URISyntaxException

ForbiddenException

fetchAndHandle

public <T> T fetchAndHandle(@NonNull URL url,
 @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler,
 @NonNull List<org.apache.hc.core5.http.Header> hdr,
 boolean useProxy)
                     throws IOException,
TooManyRequestsException,
ResourceNotFoundException,
URISyntaxException,
ForbiddenException

Download a resource from the given URL and have its content handled by the given ResponseHandler.

Type Parameters:

T - The return-type for the responseHandler

Parameters:

url - The url of the resource

handler - The responsehandler to handle the response

hdr - Additional headers to add to the HTTP request

useProxy - Whether to use the configured proxy for the connection

Returns:

The response handler result

Throws:

IOException - on I/O Exceptions

TooManyRequestsException - When HTTP status 429 is encountered

ResourceNotFoundException - When HTTP status 404 is encountered

URISyntaxException

ForbiddenException

fetchAndHandle

public <T> T fetchAndHandle(@NonNull org.apache.hc.client5.http.impl.classic.CloseableHttpClient client,
 @NonNull URL url,
 @NonNull org.apache.hc.core5.http.io.HttpClientResponseHandler<T> handler,
 @NonNull List<org.apache.hc.core5.http.Header> hdr)
                     throws IOException,
TooManyRequestsException,
ResourceNotFoundException,
ForbiddenException

Download a resource from the given URL and have its content handled by the given ResponseHandler.

Type Parameters:

T - The return-type for the responseHandler

Parameters:

client - The HTTP Client to reuse for the request

url - The url of the resource

handler - The responsehandler to handle the response

hdr - Additional headers to add to the HTTP request

Returns:

The response handler result

Throws:

IOException - on I/O Exceptions

TooManyRequestsException - When HTTP status 429 is encountered

ResourceNotFoundException - When HTTP status 404 is encountered

ForbiddenException