src.prepare_data.task_download_data

Module Contents

Functions

_is_download_necessary(path, response)

Check whether a download is necessary.

_downloader(file: pathlib.Path, url: str, response: int)

Download url in URLS[position] to disk with possible resumption.

download_file(url: str, path: str)

Execute the correct download operation.

task_download_file(url, produces)

Attributes

PARAMETRIZED_DOWNLOADS

PARAMETRIZED_DOWNLOADS = [None, None, None, None, None, None, None, None, None][source]
_is_download_necessary(path, response)[source]

Check whether a download is necessary.

There three criteria.

  1. If the file is missing, download it.

  2. The following two checks depend on each other.

    1. Some files have an entry in the header which specifies when the file was modified last. If the file has been modified, download it.

    2. If the header has no entry for the last modified date, we compare file sizes. If the file sizes do not match, the file is downloaded.

_downloader(file: pathlib.Path, url: str, response: int)[source]

Download url in URLS[position] to disk with possible resumption.

Parameters

  • file (str) – Path of file on disk

  • url (str) – URL of file

download_file(url: str, path: str)[source]

Execute the correct download operation.

If offline and online filesize differ, download the file again.

task_download_file(url, produces)[source]