Tasks

The Tasks API provides scalable compute capabilities to parallelize your computations. It works by packaging your Python code and executing the code on nodes hosted by Descartes Labs in our cloud infrastructure. These nodes are able to access imagery at extremely high rates of throughput which, paired with horizontal scaling, allow you to execute computations over nearly any spatio-temporal scale.

You can view your task groups through the Task Monitor UI.

All features described here require a recent version of the Descartes Labs Python client. See these instructions for installing the latest client.

Note

For information about API Quotas and limits see our Quotas & Limits page.

Basic Example

This basic example shows how to create a new task group and submit a single task to the group.

Note

All the following examples use Python 3.7. You may need to adapt these to your Python version by changing the image argument to match your Python version. See Choosing Your Environment for the available images.

Note

The source for the task function must be available to tasks. These examples must be run by placing the code in a file and executing that file with Python.

from descarteslabs import tasks

def hello(i):
    import geopandas

    print(geopandas)
    return "hello {}".format(i)

print("creating function")
async_func = tasks.create_function(
    hello,
    name='my-task-hello',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    requirements=[
        'geopandas==0.4.0',
    ],
)

# submit a task to the task group
print("submitting a task")
task = async_func(5)

# print the task result and logs
print("waiting for the task to complete")
print(task.result)
print(task.log)

We define a function in called hello which prints out information about the geopandas package, and returns the string hello <argument>.

Then we generate a new task group using the create_function method which specifies the entrypoint function hello, gives the task group a name, and specifies a Docker image that defines the environment in which the code will be executed.

Finally, we call the async function to submit a single task. This submits the task to the task group created by the create_function call, and the reference to the task is stored in the task variable. This also triggers instances to spin up on the backend to execute the task. Instance management is handled in the background. Instances are created or destroyed as needed to match the compute resources required by the job.

A few important features of the Tasks API are highlighted by this example:

• You can pass any JSON-serializable argument to a task, e.g. arguments with type str, dict, list, None, or any numeric data type.
• Your can import non-standard Python packages to be used in your entrypoint if the packages are specified as requirements or already present in the image you’ve selected.
• You can access any logging or debugging information, including print statements executed inside your function, through the logs stored in task.log. Logs and details for individual tasks are also available through the Task Monitor UI.

Advanced Tasks Usage

Advanced features of Tasks allow you to

• organize your code using standard Python package and module conventions instead of writing all of your code inside a single function
• add Python dependencies and specify particular version requirements
• include data files that your group requires to run

We recommend that you use these features to improve the readability of your code and better control the environment your code executes on.

Python Package Example

This example shows all the features you can use when using Python packages to organize your code. This and the following examples require some example code. Download the example code.

See scripts/complete_example.py.

from descarteslabs import tasks

print("creating function")
async_func = tasks.create_function(
    "task_examples.complete.simplify",
    name='my-task-hello',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    include_modules=[
        'task_examples'
    ],
    requirements=[
        "descarteslabs[complete]>=1.3.0",
        'geopandas==0.4.0',
    ],
    include_data=[
        'task_examples/data/*.json'
    ]
)

# submit a task to the task group
print("submitting a task")
task = async_func(5)

# print the task result and logs
print("waiting for the task to complete")
print(task.result)
print(task.log)

Instead of defining our entrypoint function in the deployment script, we’ve organized our code using common Python conventions. We’ve created a task_examples.complete module which contains the simplify function. Additionally, we tell the group to include this package, some additional data, and specific Python requirements for it to run successfully.

Including local packages (include_modules). Your function can make use of any local modules and packages. Specify them by the name you would use to import them. This includes cython module source files (with some restrictions, see the section on Cython Code). In this example the assumption is there is a local directory task_examples with a complete.py file that defines a simplify function. All submodules of the task_examples package will be included. The total size of these modules and any included data files cannot exceed 10MB.

Making Python dependencies available to your code (requirements). Your function can make use of any external Python dependencies that you specify as requirements. In this example, we specify a descarteslabs client version and a geopandas version. As long as you pick an image with your desired Python version (Python 3.7 in this case), you can upgrade or downgrade any of your other package dependencies as needed.

Including data files (include_data). You can include local data files that your function and included code can read. Wildcard patterns such as the * (asterisk) - meaning any string - are supported. Your code must use the pkg_resources API to read data files (see below). The total size of this data and any included local packages cannot exceed 10MB.

Code Organization

We suggest that you use customary ways of organizing the code for a Python project. A common way to organize your source repository looks like this:

myproject/
├── my_package/
|   ├── data/
|   |   └── my_data.txt
|   ├── __init__.py
|   ├── models.py
|   └── utils.py
|   └── cython_module.pyx
├── scripts/
|   └── deploy_task_group.py
└── requirements.txt

• The project’s Python code is all contained within a package called my_package.
• Data is co-located with code within my_package so it can be referenced relative to the source code.
• A requirements file at the top level lists all the dependencies for the the source code. The same requirements file can be given when creating a task group.
• A deploy_task_group.py script creates a new task group and kicks off tasks. It contains an entrypoint function (see below) which imports code from my_package to use.

This example follows some general guidelines. But you are not restricted to a single package and you can organize your code in any way you want, as long as you can put it together as a list of module names importable in your current local Python environment.

Entrypoint Function

You can specify an entrypoint function two ways. As a referenced function:

from descarteslabs import tasks

def f(x):
    from my_package import my_entrypoint

    return my_entrypoint(x)

async_func = tasks.create_function(
    f,
    name='hello-world',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    include_modules=[
        'my_package',
    ],
)

Alternatively, you can use a fully-qualified function name:

from descarteslabs import tasks

async_func = tasks.create_function(
    'my_package.my_entrypoint',
    name='hello-world',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    include_modules=[
        'my_package',
    ],
)

Some restrictions apply to one or both methods of passing an entrypoint function:

• *function references only* The function needs to be completely self-contained. Globals (variables defined in the top-level module namespace) cannot be referenced. Define any variables and constants within the function’s local scope. All modules it uses need to be imported within the function. The function can’t be decorated.
• *fully-qualified function name* Any modules referenced in your packages and submodules need to be locally importable.
• You can only return JSON-serializable values from the function. If a function returns a value that cannot be JSON-serialized, your tasks will fail.
• You can only pass JSON-serializable arguments to the function, e.g. arguments with type str, dict, list, None, or any numeric data type.

Python Dependencies

You can specify your Python dependencies in two ways. You can give a list of dependencies:

from descarteslabs import tasks

async_func = tasks.create_function(
    requirements=[
        "descarteslabs[complete]==1.3.0",
        "scikit-image==0.13.1".
        "scipy>=1.0.0",
    ],
    ...
)

If you already have your dependencies in a standard requirements file you can give a path (absolute or relative to the current working directory) to that:

from descarteslabs import tasks

async_func = tasks.create_function(
    requirements="path/to/requirements.txt",
    ...
)

The dependency specification and requirements file use the same format you are used to from standard Python packaging tools such as pip. For exhaustive details on this see PEP 508 for dependency specification and the pip documentation on requirements files.

If you specify a different version for a requirement that already exists on the image, your specified version will take precedence over the existing version, allowing you to upgrade or downgrade dependencies as required.

Cython Code

Cython extension modules can be included in your code in much the same way as regular Python modules. See tasks_examples/scripts/cython_example.py. The source files (.pyx) will be compiled into extension modules (.so) during the build phase. However, there are a few restrictions:

• Source cython files in the working directory (where the deploy script is being run from) cannot be included. Instead, simply create a subdirectory e.g. my_package and import the cython module as my_package.cython_module as in the examples.
• Tasks cannot directly execute a function from within a cython module as the Entrypoint Function. Instead of executing cython_example.fib, create a wrapper function in the deployment script that imports and executes cython_example.fib. Use the wrapper function as the entrypoint.
• numpy.get_include() will be added to the cythonize’s include_dirs argument to allow the compiler to find numpy header and library files. If you request a specific version of numpy in tasks requirements while using numpy in a cython module, the task may fail.

• Cython modules will be compiled using the default settings (except for adding numpy include dirs, discussed above). Cython compiler options are not currently supported.

Life Cycle

When you submit a task group, it goes through several states.

• awaiting_bundle – This means that the request was received but is waiting for the corresponding code to be uploaded
• building – This means that the code was received and a task group image is being created
• build_failed – This means that the task group image could not be created, see the next section for an explanation of Build Failures
• pending – The task group image has been successfully built but waiting for resources
• running – The task group is ready to receive requests for tasks
• terminated – The task group has been shut down

Build Failures

If you give Python dependencies for your task, they are essentially installed with pip from PyPI in your image before a task is run. There is a chance that this dependency build fails. Here are a few reasons why it might fail:

• You have a typo in your list of requirements and the package doesn’t exist
• A package version you request is not compatible with the environment (e.g. incompatible Python version)
• A package needs system libraries or tools to build that are not present in the environment
• The package fails to download from PyPI because of a transient network problem
• Data or code files you included are too large (they shouldn’t exceed a total of 10MB)

If a problem occurs during the build, the task group will be in a “build failed” state and not accept tasks anymore. In the Task Monitor UI you can find a URL to the build log that will help you diagnose the problem before attempting to create a group again.

Data Files

You can specify data files to be included as a list of patterns:

from descarteslabs import tasks

 async_func = tasks.create_function(
     include_data=[
         'my_package/data/*.txt',
         'my_package/data/image??.png',
         'my_package/data/document.rst',
     ],
     ...
 )

This supports Unix-style pattern expansion as per the glob module in the Python standard library.

In your code you must read data files using the standard pkg_resources API - not by looking for and opening files directly:

import pkg_resources
import my_package

# Read a file as a string
text = pkg_resources.resource_string(my_package.__name__, "data/data.txt")

# Open a file as a file-like object
file_like = pkg_resources.resource_stream(my_package.__name__, "data/data.txt")

We reference data files relative to the package they are contained in. For example, the original inclusion path for the file referenced here would have been my_package/data/data.txt - in the package my_package. Colocate your data with your code in a package as much as possible.

The pkg_resources API is part of setuptools, read more details about it in its documentation.

Tasks Best Practices

Make the task function idempotent and deterministic

The Tasks service guarantees that every submitted task will run at least once. Because tasks run on scalable cloud infrastructure it is possible for a task to be preempted occasionally - this means a task can foceably abort at any point in time. If this happens, it will be restarted from the beginning.

From this follows that the task function should be idempotent and (usually) deterministic: if it’s aborted at any point and restarted it should still work and it should produce the same result for the same input. If a task is long-running and produces an intermediate result (which is for example persisted to the storage service) it’s a good practice to check for the presence of the intermediate result before expensively producing it again. This saves time in case the previous run for the same input was preempted.

Make a task a small unit of work

A single task has low overhead and startup cost from the point of view of the Tasks service. There is little penalty for having lots of small tasks, so it’s recommended to break up work into the smallest parallelizable unit per task. Tasks with a shorter runtime are also less likely to be preempted. A good typical task runtime is a few minutes, though it is possible to have a task run for hours.

The code in the task itself may have a high startup cost. A typical example is a task that needs to download a Tensorflow model over the network and load it into memory. In this case there may be a balance to strike between many tasks, each of which has the same model loading overhead, and fewer tasks that run several independent inputs against the Tensorflow model, ammortizing some of the model loading cost. The right balance depends on your constraints on total runtime and cost.

Use task results

Each task produces a result when it completes. The result includes the return value of the task function, any output written by the code to stdout/stderr, and - in case of a failure - details about raised exceptions. Results for a task group are persisted and can be browsed in the Task Monitor UI as well as queried through the Tasks API.

Typically the outcome of a task is some new piece of data such as metrics, a classification or a geometry. If that data needs to be persisted and is easily JSON-serializable the simplest solution is to return it from the task function as the result. iter_task_results() can then iterate over all results, and get_task_result() retrieves individual results by id. See Retries, reruns and task results for example code.

Use retries and reruns to handle failures

If a task is doing anything that may occasionally fail by raising an exception, for example network requests through the Raster API, it’s often a good idea not to do explicit error handling. Instead, a task group can handle occasional failures by giving a retry_count during task group creation (i.e., create_function()); if any uncaught exceptions are raised during the execution of a task it is retried this many times before it is finally considered a failure. This works particularly well if tasks are small, idempotent units of work as recommended above.

As an alternative or in addition to retries, a set of tasks can also be rerun through the client. rerun_failed_tasks() reruns all tasks in a group that have failed. For more control, rerun_matching_tasks() reruns tasks with specific failure types or those submitted after a given timestamp. See Retries, reruns and task results for example code.

More Examples

Multiple Tasks Example

This example illustrates the more typical use case of submitting multiple tasks to a new group.

See scripts/multiple_tasks.py

from descarteslabs import tasks
from descarteslabs.client.services.tasks import as_completed

# create the task group
print("creating function")
async_func = tasks.create_function(
    "task_examples.basic.generate_random_image",
    name='my-task-random-image',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    include_modules=[
        'task_examples'
    ]
)

# submit 20 tasks to the task group
print("submitting tasks")
tasks = async_func.map(range(20))

# print the shape of the image array returned by each task
print("starting to wait for task completions")
for task in as_completed(tasks):
    if task.is_success:
        print(task.result.shape)
    else:
        print(task.exception)
        print(task.log)

Here, we reference the "task_examples.basic.generate_random_image" function which generates a random image using numpy with the same number of bands as the value passed to the num_bands parameter.

This example highlights a few additional features of the Tasks API:

• To submit tasks to the task group, we are using the map method to submit a task for each of the elements in the list. This is typically the most efficient way to submit tasks to a task group, particularly if the number of tasks is large. You are also able to submit tasks one at a time, e.g. within in a for-loop.
• We use the as_completed method to retrieve the task results for each task as it is completed. Within this loop, we also catch exceptions and print the logs of any failed task.

It’s important to note that the return value from the entrypoint function is converted to a list because return values must be JSON-serializable.

GPU-enabled Task Example

In this example, we use GPU-enabled tasks to do TensorFlow matrix multiplication on a GPU. We do this by defining a TensorFlow task that explicitly places operations on the GPU, and we request a GPU when creating the task group. We use a GPU-enabled container image for this task.

See scripts/gpu_example.py

from descarteslabs import tasks

# We create our task function, which creates a new task group in the process.
# Note that we use a public GPU-enabled image that Descartes provides: py3.7-gpu.
print("creating function")
async_function = tasks.create_function(
    "task_examples.gpu.gpu_tf_ex",
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7-gpu:v2023.02.04-3-g883c0377",
    name='gpu_tf_ex',
    # You can request GPUs exactly as you would request CPUs.
    gpus=1,
    include_modules=[
        'task_examples'
    ]
)

# We launch a task, wait for it to complete, and print the result.  This task will only
# succeed if TensorFlow was able to execute the matrix multiplication on the GPU.
print("submitting a task")
task = async_function(3)

print("waiting for the task to complete")
async_function.wait_for_completion(show_progress=True)
print(task.result)

When creating a GPU-enabled task group, add the gpus=1 keyword argument to your client.create_function, client.create_or_get_function, or client.new_group function call. The only other thing you’ll need to run GPU-enabled tasks is a container image that contains the NVIDIA CUDA library and GPU-supporting libraries for whatever computation you wish to do (e.g. the tensorflow-gpu library for TensorFlow with GPU support.) We provide several public GPU-oriented container images; see the GPU-enabled images in the Choosing Your Environment section.

This task will only succeed (and return a result) if the TensorFlow matrix multiplication is able to be placed on a GPU. We only do this for demonstration purposes, however. If you’re using a machine learning framework with a backend that does automatic device placement for operators (e.g. Keras with a TensorFlow backend), you won’t have to explicitly place operations on the GPU; the backend will automatically do that for you. If you’ve already defined a task function that does Keras model training or inference, you shouldn’t have to changed that code to take advantage of GPU training/inference within our tasks service.

Retries, reruns and task results

This examples demonstrates how to use retries and reruns to make tasks more robust and how to make use of task results.

This is a function that takes a single Descartes Labs tile as input and returns a histogram of the pixel values of the NIR band of a Sentinel-2 mosaic around July 2018:

def nir_histogram(tile):
    import descarteslabs.scenes
    import numpy as np

    scenes, ctx = descarteslabs.scenes.search(
        tile,
        products=["sentinel-2:L1C"],
        start_datetime="2018-07-01",
        end_datetime="2019-01-01",
        cloud_fraction=0.2,
        sort_field="acquired",
        sort_order="asc",
        limit=10,
    )
    tile_mosaic = scenes.mosaic("nir", ctx)
    histogram, _ = np.histogram(
        tile_mosaic,
        bins=100,
        range=(0, 10000),
        density=False,
    )
    return histogram.tolist()

Each histogram is an array of 100 elements corresponding to pixel counts in 100 bins, evenly spaced from pixel values 0 to 10000. For example, the first bin is the total number of pixels in a tile that have values 0 to 100.

We can create a task group from this function and run it with tiles covering the state of New Mexico:

from descarteslabs.client.services.tasks import Tasks
from descarteslabs.client.services.raster import Raster

tasks = Tasks()
raster = Raster()

async_func = tasks.create_function(
    nir_histogram,
    name="nir-histogram",
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    retry_count=3,
)

nm_geom = {
     "type": "Polygon",
     "coordinates": [[
         [-109.039306640625, 37.00255267215955], [-109.039306640625, 31.3348710339506],
         [-108.21533203125, 31.344254455668054], [-108.19335937499999, 31.784216884487385],
         [-106.490478515625, 31.784216884487385], [-106.490478515625, 31.99875937194732],
         [-103.062744140625, 31.99875937194732], [-102.996826171875, 37.00255267215955],
         [-109.039306640625, 37.00255267215955]
     ]]
}
resolution = 10
tile_size = 2000
tiles = raster.iter_dltiles_from_shape(resolution, tile_size, 0, nm_geom)

async_func.map(tiles)
async_func.wait_for_completion()

Segmenting a large geographic area into tiles and processing one tile per task like this is a common pattern to parallelize work. This will kick off and wait for the completion of 867 tasks, each computing a histogram for one 2000x2000 pixel tile at full resolution of the Sentinel-2 NIR band (10m per pixel).

When creating the task group, we passed an argument retry_count=3. The nir_histogram function uses a raster call - there’s a small chance that this will raise an exception, e.g., because of network instability or exceeded rate/quota limits. Rather than doing explicit error handling in the function, we can rely on the retry feature of tasks. If a task raises an exception here it is retried 3 times before it is discarded as a failure. Using retries instead of explicit error handling is recommended if task runtimes are reasonably short.

In the very unlikely case that some tasks failed even with retries, there is a quick way to rerun all failed tasks:

tasks.rerun_failed_tasks(async_func.group_id, retry_count=3)
async_func.wait_for_completion()

rerun_failed_tasks() takes the task group id as the first argument. In this case we take it from the previously created task group function. In other cases we might look it up in the Task Monitor UI.

We broke up our geographic area into tiles so we can run a computation on high-resolution imagery without running into memory limits and to speed it up through parallelization. In the end we are after the aggregate computation across the whole area of the state of New Mexico. We returned the histograms for each tile from the task function, so they are now stored as task results. We can retrieve and aggregate them:

import json
import numpy as np
import requests

success_results = tasks.iter_task_results(
    async_func.group_id,
    status="SUCCESS",
    include=["result_url"],
)
aggregated_histogram = np.zeros((100,))
for result in success_results:
    histogram_list = json.loads(requests.get(result.result_url).text)
    aggregated_histogram += np.asarray(histogram_list)

iter_task_results() iterates over all completed tasks in a group, optionally restricted by some search criteria. Here we only query for successful tasks. We ask for the result_url to be included because we want to download the result.

aggregated_histogram is now a numpy histogram across the whole state of New Mexico. This shows how it is often natural to rely on task results if indeed the return value of the function is the crucial outcome of a task - as opposed to other cases where the return value is insignificant because the side effect of the task matters, such as typically the upload of a new raster scene into the catalog.

In-Memory file storage

You can use the directory located at /cache for file storage while your task is running. This directory is a memory-backed filesystem and is a good place to write temporary files. Keep in mind that any data written to /cache counts against the memory limit of your task. Use /cache if you can tolerate the memory usage and you have a lot of IO or you need high-performance IO.

See scripts/cache_example.py

from descarteslabs import tasks

def hello(i):
    from task_examples.cache import hello

    # specify a file location in the cache to write files to
    return hello(i, "/cache/geometry.wkt")

print("creating function")
async_func = tasks.create_function(
    hello,
    name='my-task-hello',
    image="us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377",
    include_modules=[
        "tasks_examples"
    ]
)

# submit a task to the task group
print("submitting a task")
task = async_func(5)

# print the task result and logs
print("waiting for the task to complete")
print(task.result)
print(task.log)

Troubleshooting Tasks

Understanding Tasks Concurrency

Task concurrency can be an indicator of the health of your task group, but can it can be tricky to interpret what the concurrency of your group means. Here are some guidelines to help you understand your task concurrency.

If your task group has been running for a while, but you have 0 concurrency, check that you have submitted tasks and there are still tasks pending to be processed.

If you know you have have tasks pending, but the concurrency is 0 or lower than your minimum_concurrency it’s likely that your task group is waiting for resources to become available. If the Tasks service is under heavy load, or you’ve asked for a large resource allocation for your tasks, it can take some time for resources to become available for scheduling your workers.

If you’ve set a minimum_concurrency, a best effort is made to schedule the minimum number of workers requested as soon as your task group starts up. If resources to schedule your minimum_concurrency are not available, we will continually attempt to meet at least the minumum. Once scheduled, your group should not drop below the minimum_concurrency until all the remaining tasks are completed. Certain events may cause your group to reschedule workers, which causes the process to start over. These cases are rare, but may account for missing workers.

The default for the maximum_concurrency is 5. You can set it to anything greater than or equal to the minimum_concurrency, up to a maximum of 500. As you are developing your task code, it is recommended that you stick with this default until you are sure that things are working correctly.

You can change the concurrency of your task group (and many other task group properties) while it is running using the update_group() method.

Choosing Your Environment

The execution environment for your function in the cloud is defined by the docker image you pick when creating the function. The below images are available covering typical use cases.

Match your local Python version to the image you choose. Your function will be rejected or might not run successfully if there is a mismatch between your local Python version and the Python version in the image. A differing bug release version (the “x” in Python version “3.7.x”) is fine.

If you need GPU support, choose one of the GPU-enabled images. All GPU-supporting images end with -gpu.

Current Images

Python 3.9.16, Ubuntu 18.04

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.9:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.11.1
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==1.6.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.12.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2023.1.23
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.22.4
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.3
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.4.1
python-apt==1.6.6
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.4.1
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1+cpu
torchvision==0.14.1+cpu
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1

Python 3.9.16, Ubuntu 18.04, NVIDIA/CUDA GPU-enabled

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.9-gpu:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: cuda 11.2, cuDNN 8.1.1, TensorRT 7.2.3, GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.11.1
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==1.6.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.12.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2023.1.23
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.22.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.3
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.4.1
python-apt==1.6.5+ubuntu0.7
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.4.1
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1
torchvision==0.14.1
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1

Python 3.8.16, Ubuntu 18.04

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.8:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.11.1
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==1.6.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.12.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2023.1.23
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.22.4
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.3
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.4.1
python-apt==1.6.6
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.4.1
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1+cpu
torchvision==0.14.1+cpu
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1

Python 3.8.16, Ubuntu 18.04, NVIDIA/CUDA GPU-enabled

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.8-gpu:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: cuda 11.2, cuDNN 8.1.1, TensorRT 7.2.3, GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.11.1
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==1.6.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.12.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2023.1.23
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.22.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.3
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.4.1
python-apt==1.6.5+ubuntu0.7
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.4.1
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1
torchvision==0.14.1
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1

Python 3.7.16, Ubuntu 18.04

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.10.6
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==0.4.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.10.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2021.11.20
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.21.6
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.3.5
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.2.1
python-apt==1.6.6
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.3.0
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1+cpu
torchvision==0.14.1+cpu
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1

Python 3.7.16, Ubuntu 18.04, NVIDIA/CUDA GPU-enabled

Image: us.gcr.io/dl-ci-cd/images/tasks/public/py3.7-gpu:v2023.02.04-3-g883c0377

Date: 02/01/2023

Python highlights: GDAL, numpy, pandas, scikit-image, scikit-learn, scipy, Tensorflow, PyTorch

Other libraries and tools: cuda 11.2, cuDNN 8.1.1, TensorRT 7.2.3, GEOS 3.8.0, proj 7.0.0

absl-py==1.4.0
affine==2.4.0
astunparse==1.6.3
attrs==22.2.0
blosc==1.10.6
cachetools==4.2.4
certifi==2022.12.7
charset-normalizer==3.0.1
click==8.1.3
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==0.4.0
Cython==0.29.33
decorator==5.1.1
descarteslabs==1.12.1
dynaconf==3.1.11
Fiona==1.8.22
flatbuffers==23.1.21
gast==0.4.0
GDAL==3.0.4
geojson==3.0.0
geopandas==0.10.2
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.51.1
h5py==3.7.0
ibis-framework==1.4.0
idna==3.4
imagecodecs==2021.11.20
imageio==2.25.0
importlib-metadata==6.0.0
joblib==1.2.0
keras==2.11.0
lazy-object-proxy==1.9.0
libclang==15.0.6.1
Markdown==3.4.1
MarkupSafe==2.1.2
mercantile==1.2.1
multipledispatch==0.6.0
munch==2.5.0
networkx==2.4
numpy==1.21.6
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauthlib==3.2.2
opencv-python-headless==4.7.0.68
opt-einsum==3.3.0
packaging==23.0
pandas==1.3.5
pex==2.1.61
Pillow==9.4.0
protobuf==3.19.6
psutil==5.9.4
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyGObject==3.26.1
pyparsing==3.0.9
pyproj==3.2.1
python-apt==1.6.5+ubuntu0.7
python-dateutil==2.8.2
pytz==2022.7.1
PyWavelets==1.3.0
PyYAML==6.0
rasterio==1.2.10
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.7.3
Shapely==1.8.5.post1
six==1.16.0
snuggs==1.4.7
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
threadpoolctl==3.1.0
tifffile==2021.4.8
toolz==0.12.0
torch==1.13.1
torchvision==0.14.1
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==1.26.14
Werkzeug==2.2.2
wrapt==1.14.1
xarray==0.20.2
zipp==3.12.1