Execution

Exceptions

JobComputeError(job)

Generic error raised when a job computation fails.

JobTimeoutError

Raised when a computation took longer to complete than a specified timeout.

Classes

Workflow(id[, title, description, public, …])

A collection of versions of a proxy object, with associated metadata.

VersionedGraft(version, proxy_object[, …])

A specific version of a Workflow.

Job(proxy_object, parameters[, format, …])

A Job represents the computation of a proxy object’s graft within a specific environment of parameters.

XYZ(proxy_object, proto_message[, client])

Stores proxy objects to be rendered by an XYZ tile server.

XYZErrorListener(xyz_id[, client])

Calls callback functions in a background thread when XYZ errors occur.

Functions

compute(obj[, geoctx, format, destination, …])

Compute a proxy object and wait for its result.

publish(id, version[, obj, title, …])

Publish a proxy object as a Workflow with the given version.

use(workflow_id, version[, client])

Use like import: load the proxy object of a published Workflow version.

exception JobComputeError(job)[source]

Generic error raised when a job computation fails.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

args
property code
property id
property message
exception JobTimeoutError[source]

Raised when a computation took longer to complete than a specified timeout.

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

args
class Workflow(id, title='', description='', public=False, labels=None, tags=None, client=None)[source]

A collection of versions of a proxy object, with associated metadata.

Examples

Attributes

created_timestamp

The UTC date this Workflow was created, or None if it hasn’t been saved yet.

description

A long-form description of this Workflow.

id

The globally unique identifier for the Workflow, of the format

labels

The labels associated with this Workflow, expressed as key-value pairs.

name

The name of this Workflow (everything following the first colon in id).

public

Whether this Workflow is publicly accessible.

tags

The tags associated with this Workflow, expressed as a list of stirngs.

title

The user-friendly title of this Workflow.

updated_timestamp

The UTC date this Workflow was most recently modified, or None if it hasn’t been saved yet.

version_names

versions

Methods

delete()

Delete this Workflow and every VersionedGraft it contains.

delete_id(id[, client])

Delete the Workflow that has the provided ID, and every VersionedGraft it contains.

duplicate(new_id[, include_versions, client])

Duplicate this Workflow object with a new ID.

get(id[, client])

Get an existing Workflow by ID.

get_version(version)

Get the VersionedGraft object for an existing version.

save([client])

Save the current state of this Workflow and its VersionedGrafts.

search([email, name, tags, client])

Iterator over Workflows you have access to.

set_version(version[, obj, docstring, labels])

Register a version to an object.

update_version(version[, docstring, labels])

Update the docstring and/or labels for a Workflow version.

>>> from descarteslabs.workflows import Int, Workflow
>>> workflow = Workflow(
...     id="bob@gmail.com:cool_addition_model",
...     title="Bob's Super Cool Addition Model",
...     description="The result of 1 plus 1",
...     public=True,
...     labels={"project": "arithmetic"},
...     tags=["test", "cool", "deep learning", "AI", "digital twin"])  
>>> num = Int(1) + 1  
>>> workflow.set_version(num, version="0.0.1")  
>>> workflow  
Workflow: "Bob's Super Cool Addition Model"
    - id: bob@gmail.com:cool_addition_model
    - public: True
    - labels: {'project': 'arithmetic'}
    - tags: ['test', 'cool', 'deep learning', 'AI', 'digital twin']
    - versions: '0.0.1'
    The result of 1 plus 1
>>> workflow.get_version("0.0.1").object  
<descarteslabs.workflows.types.primitives.number.Int object at 0x...>
>>> workflow.get_version("0.0.1").object.inspect()  
2
>>> workflow.set_version(num + 2, version="0.0.2", docstring="1 + 2")  
>>> workflow.description = "The result of 1 + 2"  
>>> workflow.save()  
>>> workflow.get_version("0.0.2").object.inspect()  
3
>>> workflow  
Workflow: "Bob's Super Cool Addition Model"
    - id: bob@gmail.com:cool_addition_model
    - public: True
    - labels: {'project': 'arithmetic'}
    - tags: ['test', 'cool', 'deep learning', 'AI', 'digital twin']
    - versions: '0.0.1', '0.0.2'
    The result of 1 plus 2

Construct a Workflow object referencing a new or existing Workflow.

Parameters
  • id (str) – ID for the new Workflow. This should be of the form "email:workflow_name" and should be globally unique. If this ID is not of the proper format, you will not be able to save the Workflow. Cannot be changed once set.

  • title (str, default "") – User-friendly title for the Workflow.

  • description (str, default "") – Long-form description of this Workflow. Markdown is supported.

  • public (bool, default False) – Whether this Workflow will be publicly accessible, or only visible to you.

  • labels (dict, optional) – Key-value pair labels to add to the Workflow.

  • tags (list, optional) – A list of strings of tags to add to the Workflow.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

Returns

Return type

Workflow

Example

>>> from descarteslabs.workflows import Workflow, Int
>>> workflow = Workflow(
...     id="bob@gmail.com:cool_addition_model",
...     title="Bob's Super Cool Addition Model",
...     description="The result of 1 plus 1",
...     public=True,
...     labels={"project": "arithmetic"},
...     tags=["test", "cool", "deep learning", "AI", "digital twin"]) 
>>> workflow 
Workflow: "Bob's Super Cool Addition Model"
    - id: bob@gmail.com:cool_addition_model
    - public: True
    - labels: {'project': 'arithmetic'}
    - tags: ['test', 'cool', 'deep learning', 'AI', 'digital twin']
    - versions: '0.0.1', '0.0.2'
    The result of 1 plus 1
delete()[source]

Delete this Workflow and every VersionedGraft it contains.

After deletion, all fields on self are reset.

Warning: this cannot be undone!

static delete_id(id, client=None)[source]

Delete the Workflow that has the provided ID, and every VersionedGraft it contains.

Warning: this cannot be undone!

Parameters
  • id (str) – The ID of the Workflow that we wish to delete.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

duplicate(new_id, include_versions=True, client=None)[source]

Duplicate this Workflow object with a new ID.

This does not save the new object to the backend; to do so, call save.

Parameters
  • new_id (str) – ID for the copied Workflow

  • include_versions (bool, optional, default True) – Whether to also copy over all the versions currently set on this Workflow

Returns

new_workflow – Copy of self with a new ID

Return type

Workflow

classmethod get(id, client=None)[source]

Get an existing Workflow by ID.

Parameters
  • id (string) – The ID of the Workflow (like name@example.com:workflow_id)

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

Returns

Return type

Workflow

Example

>>> from descarteslabs.workflows import Workflow
>>> workflow = Workflow.get('bob@gmail.com:cool_addition_model') 
>>> workflow 
Workflow: "Bob's Super Cool Addition Model"
    - id: bob@gmail.com:cool_addition_model
    - public: True
    - labels: {'project': 'arithmetic'}
    - tags: ['test', 'cool', 'deep learning', 'AI', 'digital twin']
    - versions: '0.0.1', '0.0.2'
    The result of 1 plus 1
get_version(version)[source]

Get the VersionedGraft object for an existing version.

Raises a KeyError if this Workflow object doesn’t contain that version. Note that that doesn’t necessarily mean the version doesn’t exist—if you constructed this Workflow object directly and haven’t called save yet, the object won’t contain existing versions that are defined on the backend, but not in this session.

However, if you constructed this Workflow from Workflow.get, then an error here means the version actually doesn’t exist (unless it was just added, after the get call).

Note

You can also use indexing syntax as shorthand for get_version: workflow["1.0.0"] is equivalent to workflow.get_version("1.0.0").

Parameters

version (str) – A version of this Workflow.

Returns

version – The version requested, represented as a VersionedGraft.

Return type

VersionedGraft

Raises

KeyError – If this object doesn’t contain the given version.

save(client=None)[source]

Save the current state of this Workflow and its VersionedGrafts.

If the Workflow doesn’t exist yet, it will be created.

If it does exist, it’ll attempt to update the VersionedGrafts with the ones set on this object. Any new versions will be added. Since VersionedGrafts are immutable, if a version already exists on the backend that is also set on this object, save will raise an error if the grafts of those version are different, otherwise updaing that version will be a no-op if the grafts are the same.

If the Workflow does exist, all Workflow-level metadata (title, description, labels, etc) will be overwritten with the values set on this object.

Parameters

client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

Returns

self – Same Workflow object, after its state has been updated to match the backend’s state

Return type

Workflow

classmethod search(email='', name='', tags=None, client=None)[source]

Iterator over Workflows you have access to.

All search options are logically ANDed together.

Parameters
  • email (str, optional) – Restrict to Workflows belonging to this email address (exact match).

  • name (str, optional) – Restrict to Workflow names that start with this string (prefix search). (The name follows the colon in a Workflow ID: email@example.com:this_is_the_name.)

  • tags (List[str], optional) – Restrict to Workflows that have _any_ of these tags.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

Yields

workflow (Workflow) – Workflow matching the search criteria

Example

>>> import descarteslabs.workflows as wf
>>> for workflow in wf.Workflow.search():  
...     print(workflow.id)  
you@example.com:my_workflow
someone_else@example.com:my_workflow
someone_else@example.com:my_other_workflow
set_version(version, obj=None, docstring='', labels=None)[source]

Register a version to an object. Can also be used as a decorator.

Note: this doesn’t save the new version to the backend. Call save afterwords to do so.

Parameters
  • version (str) – The version to be set, tied to the given obj. This should adhere to the semantic versioning schema (https://semver.org).

  • obj (Proxytype, optional) – The object to store as this version. If not provided, it’s assumed that set_version is being used as a decorator on a function.

  • docstring (str, default "") – The docstring for this version. If not provided and set_version is not being used as a decorator, then the version’s docstring will be set to the empty string. If not provided and set_version is being used as a decorator, then the version’s docstring will be set to the docstring of the decorated Python function.

  • labels (dict, optional) – Key-value pair labels to add to the version.

Returns

version – The version set. If used as a decorator, returns the Function instead.

Return type

VersionedGraft or Function

update_version(version, docstring=None, labels=None)[source]

Update the docstring and/or labels for a Workflow version.

Note: this doesn’t save the updated version to the backend. Call save afterwords to do so.

Parameters
  • version (str) – The version to be updated.

  • docstring (str, optional) – The new docstring for this version. One of docstring and labels must be provided.

  • labels (dict, optional) – Key-value pair labels to add to the version. These labels will overwrite the existing labels for this version. One of docstring and labels must be provided.

Returns

version – The updated version.

Return type

VersionedGraft

property created_timestamp

The UTC date this Workflow was created, or None if it hasn’t been saved yet. Cannot be modified.

Type

datetime.datetime or None

property description

A long-form description of this Workflow. Markdown is supported.

Type

str

property id

The globally unique identifier for the Workflow, of the format "user@domain.com:workflow-name"

Type

str

property labels

The labels associated with this Workflow, expressed as key-value pairs.

Type

dict

property name

The name of this Workflow (everything following the first colon in id).

Type

str

property public

Whether this Workflow is publicly accessible.

Type

bool

property tags

The tags associated with this Workflow, expressed as a list of stirngs.

Type

list

property title

The user-friendly title of this Workflow.

Type

str

property updated_timestamp

The UTC date this Workflow was most recently modified, or None if it hasn’t been saved yet. Updated automatically.

Type

datetime.datetime or None

property version_names
property versions
class VersionedGraft(version, proxy_object, docstring='', labels=None, client=None)[source]

A specific version of a Workflow.

Except in advanced cases, you shouldn’t need to interact with this object much—you’ll primarily use the Workflow object and wf.use.

Construct a VersionedGraft object from a proxy object.

You shouldn’t construct a VersionedGraft directly; use Workflow.set_version or wf.publish instead.

Parameters
  • version (str) – Version of the graft. This should adhere to the semantic versioning schema (https://semver.org).

  • proxy_object (Proxytype) – The proxy object source of the graft.

  • docstring (str, default "") – Docstring for the VersionedGraft.

  • labels (dict, optional) – Key-value pair labels to add to the VersionedGraft.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

Returns

Return type

VersionedGraft

Attributes

channel

The channel under which this VersionedGraft was created.

docstring

The docstring for this VersionedGraft.

labels

The labels attached to this VersionedGraft.

object

The proxy object of this VersionedGraft.

type

The type of the proxy object.

version

The version of this VersionedGraft.

Methods

get(workflow_id, version[, client])

Get a specific VersionedGraft of a Workflow.

classmethod get(workflow_id, version, client=None)[source]

Get a specific VersionedGraft of a Workflow.

Parameters
  • workflow_id (str) – The ID of the Workflow.

  • version (str) – The version of the Workflow that you wish to fetch.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters.

Returns

Return type

VersionedGraft

property channel

The channel under which this VersionedGraft was created.

Type

str

property docstring

The docstring for this VersionedGraft.

Type

str

property labels

The labels attached to this VersionedGraft.

Type

dict

property object

The proxy object of this VersionedGraft.

Raises ValueError if the VersionedGraft is not compatible with the current client version.

Type

Proxytype

property type

The type of the proxy object.

Raises ValueError if the VersionedGraft is not compatible with the current client version.

Type

type

property version

The version of this VersionedGraft.

Type

str

class Job(proxy_object, parameters, format='pyarrow', destination='download', client=None, cache=True)[source]

A Job represents the computation of a proxy object’s graft within a specific environment of parameters.

Example

Attributes

cache_enabled

Whether caching is enabled for this job.

cancelled

Whether the job has been cancelled.

channel

The channel name where this Job will execute.

created_datetime

The time the Job was created.

destination

The destination for the Job results, as a dictionary.

done

Whether the Job has completed or not.

error

The error of the Job, or None if it finished successfully.

format

The serialization format of the Job, as a dictionary.

id

The globally unique identifier for the Job, or None if it hasn’t been executed yet.

object

The proxy object this Job computes.

parameters

The parameters of the Job, as a graft.

result_type

Name of the type of object that will be used to hold the result

runtime

The total time it took the Job to run.

stage

The current stage of the Job (queued, preparing, running, saving, succeeded, failed).

type

The type of the proxy object.

updated_datetime

The time of the most recent Job update.

url

The download URL for this Job’s results.

Methods

cancel()

Cancel a running job.

get(id[, client])

Get a currently-running Job by its ID.

refresh()

Refresh the attributes and state of the job.

resubmit()

Resubmit this job, returning a new Job object.

result([timeout, progress_bar])

Get the result of the job.

result_to_file(file[, timeout, progress_bar])

Save the result of the job to a file.

wait([timeout, progress_bar, …])

Block until the Job is complete, optionally displaying a progress bar.

watch([timeout])

Generator that yields self each time an update to the Job occurs.

>>> from descarteslabs.workflows import Int, Job
>>> num = Int(1) + 1
>>> job = num.compute(block=False)  
>>> job 
<descarteslabs.workflows.models.job.Job object at 0x...>
>>> job.id 
'3754676080bbb2b857fbc04a3e48f6312732e1bc42e0bd7b'
>>> job.result() 
2
>>> same_job = Job.get('3754676080bbb2b857fbc04a3e48f6312732e1bc42e0bd7b') 
>>> same_job.stage 
'STAGE_DONE'
>>> same_job.result 
2

Creates a new Job to compute the provided proxy object with the given parameters.

Parameters
  • proxy_object (Proxytype) – Proxy object to compute

  • parameters (dict[str, Proxytype]) – Python dictionary of parameter names and values

  • format (str or dict, default "pyarrow") – The serialization format for the result.

  • destination (str or dict, default "download") – The destination for the result.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

  • cache (bool, default True) – Whether to use the cache for this job.

Returns

The job that’s executing.

Return type

Job

Example

>>> from descarteslabs.workflows import Job, Int, parameter
>>> my_int = Int(1) + parameter("other_int", Int)
>>> job = Job(my_int, {"other_int": 10}) 
>>> job.stage 
QUEUED
cancel()[source]

Cancel a running job.

Example

>>> from descarteslabs.workflows import Job, Int, parameter
>>> my_int = Int(1) + parameter("other_int", Int)
>>> job = Job(my_int, {"other_int": 10}) 
>>> job.cancel() 
classmethod get(id, client=None)[source]

Get a currently-running Job by its ID.

Parameters
  • id (string) – The ID of a running job.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

Example

>>> from descarteslabs.workflows import Job
>>> job = Job.get('3754676080bbb2b857fbc04a3e48f6312732e1bc42e0bd7b') 
refresh()[source]

Refresh the attributes and state of the job.

Example

>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(1), {}) 
>>> job.stage 
QUEUED
>>> job.refresh() 
>>> job.stage 
SUCCEEDED
resubmit()[source]

Resubmit this job, returning a new Job object.

Example

>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(1), {}) 
>>> job.id 
abc123
>>> job.result() 
1
>>> new_job = job.resubmit() 
>>> new_job.id 
xyz456
>>> new_job.result() 
1
result(timeout=None, progress_bar=None)[source]

Get the result of the job. This blocks until the job is complete.

Only the “download” destination can be retrieved. Raises NotImplementedError for other destinations.

Parameters
  • timeout (int, optional) – The number of seconds to wait for the result.

  • progress_bar (bool, optional) – Flag to draw the progress bar. Default is to True if in Jupyter Notebook.

Returns

result – When the Job’s format is “pyarrow”, returns a Python object representing the result, either as a plain Python type, or object from descarteslabs.workflows.result_types. For other formats, returns raw bytes. Consider using result_to_file in that case to save the results to a file.

Return type

Python object or bytes

Example

>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(1), {}) 
>>> job.result() 
1
result_to_file(file, timeout=None, progress_bar=None)[source]

Save the result of the job to a file. This blocks until the job is complete.

Only the “download” destination can be written to a file. For destinations like “catalog”, where the data is handed off to another service, you’ll need to use that service to retrieve it. (In the “catalog” case, that’s Raster and Metadata.)

Parameters
  • file (path or file-like object) – Path or file where results will be written

  • timeout (int, optional) – The number of seconds to wait for the result.

  • progress_bar (bool, optional) – Flag to draw the progress bar. Default is to True if in Jupyter Notebook.

Example

>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(1), {}, format="json") 
>>> job.result_to_file("one.json") 
>>> import io
>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(2), {}, format="json") 
>>> bytestream = io.BytesIO() 
>>> job.result_to_file(bytestream) 
>>> print(bytestream.read()) 
b'2'
wait(timeout=None, progress_bar=False, cancel_on_timeout=True, cancel_on_interrupt=True)[source]

Block until the Job is complete, optionally displaying a progress bar.

Raises any error that occurs with the Job, or JobTimeoutError if the timeout passes before the Job is complete.

Parameters
  • timeout (int, optional) – The number of seconds to wait for the result.

  • progress_bar (bool, optional) – Flag to draw the progress bar. Default is to True if in Jupyter Notebook.

  • cancel_on_timeout (bool, optional) – Whether to cancel the job on client timeout. Default is True.

  • cancel_on_interrupt (bool, optional) – Whether to cancel the job on interrupt (e.g. ctrl + c). Default is True.

watch(timeout=None)[source]

Generator that yields self each time an update to the Job occurs.

Parameters

timeout (int, optional) – The number of seconds to watch for Job updates. Defaults to self._client.STREAM_TIMEOUT, which is also the maximum allowed.

Example

>>> from descarteslabs.workflows import Job, Int
>>> job = Job(Int(1), {}) 
>>> for job in job.watch(): 
...     print(job.stage)
QUEUED
PREPARING
RUNNING
RUNNING
SAVING
SUCCEEDED
property cache_enabled

Whether caching is enabled for this job.

property cancelled

Whether the job has been cancelled.

property channel

The channel name where this Job will execute.

Type

str

property created_datetime

The time the Job was created.

Type

datetime

property destination

The destination for the Job results, as a dictionary.

property done

Whether the Job has completed or not.

Type

bool

property error

The error of the Job, or None if it finished successfully.

property format

The serialization format of the Job, as a dictionary.

property id

The globally unique identifier for the Job, or None if it hasn’t been executed yet.

Type

str or None

property object

The proxy object this Job computes.

Type

Proxytype

property parameters

The parameters of the Job, as a graft.

property result_type

Name of the type of object that will be used to hold the result

Type

str

property runtime

The total time it took the Job to run.

Type

datetime

property stage

The current stage of the Job (queued, preparing, running, saving, succeeded, failed).

property type

The type of the proxy object.

Type

type

property updated_datetime

The time of the most recent Job update.

Type

datetime

property url

The download URL for this Job’s results.

If format is not “download” or “email”, url will be None.

class XYZ(proxy_object, proto_message, client=None)[source]

Stores proxy objects to be rendered by an XYZ tile server.

Similar to a Workflow, but meant for storing proxy objects so the XYZ tile service can display them, rather than for persisting and sharing workflows between users.

Use url to generate an XYZ URL template, and iter_tile_errors or error_listener to retrieve error messages that happen while computing them.

Examples

Methods

build(proxy_object[, name, description, client])

Construct a new XYZ from a proxy object.

error_listener()

An XYZErrorListener to trigger callbacks when errors occur computing tiles.

get(xyz_id[, client])

Get an existing XYZ by id.

iter_tile_errors(session_id[, start_datetime])

Iterator over errors generated while computing tiles

save()

Persist this XYZ layer.

url([session_id, colormap, scales, checkerboard])

XYZ tile URL format-string, like https://workflows.descarteslabs.com/v0-5/xyz/1234567/{z}/{x}/{y}.png

Attributes

channel

The channel name this XYZ is compatible with.

created_timestamp

The UTC date this XYZ was created, or None if it hasn’t been saved yet.

description

A long-form description of this xyz.

id

The globally unique identifier for the XYZ, or None if it hasn’t been saved yet.

name

The name of this XYZ.

object

The proxy object of this XYZ.

type

The type of the proxy object.

updated_timestamp

The UTC date this XYZ was most recently modified, or None if it hasn’t been saved yet.

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> rgb = img.pick_bands("red green blue")
>>> xyz = XYZ.build(rgb, name="My RGB") 
>>> xyz.save() 
>>> xyz 
<descarteslabs.workflows.models.xyz.XYZ object at 0x...>
>>> xyz.id 
'24d0e79c5c1e1f10a0b1177ef3974d7edefd5988291cf2c6'
>>> same_xyz = XYZ.get('24d0e79c5c1e1f10a0b1177ef3974d7edefd5988291cf2c6') 
>>> same_xyz.url() 
'https://workflows.descarteslabs.com/master/xyz/24d0e79c5c1e1f10a0b1177ef3974d7edefd5988291cf2c6/{z}/{x}/{y}.png'
>>> same_xyz.object 
<descarteslabs.workflows.types.geospatial.image.Image object at 0x...>

Construct a XYZ object from a proxy object and Protobuf message.

Do not use this method directly; use the XYZ.build and XYZ.get classmethods instead.

Parameters
  • proxy_object (Proxytype) – The proxy object to store in this XYZ

  • proto_message (xyz_pb2.XYZ message) – Protobuf message for the XYZ

  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters

classmethod build(proxy_object, name='', description='', client=None)[source]

Construct a new XYZ from a proxy object.

Note that this does not persist the XYZ, call save() on the returned XYZ to do that.

Parameters
  • proxy_object (Proxytype) – The proxy object to store in this XYZ

  • name (str, default "") – Name for the new XYZ

  • description (str, default "") – Long-form description of this XYZ. Markdown is supported.

  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters

Returns

Return type

XYZ

Example

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> rgb = img.pick_bands("red green blue")
>>> xyz = XYZ.build(rgb, name="My RGB") 
>>> xyz 
<descarteslabs.workflows.models.xyz.XYZ object at 0x...>
error_listener()[source]

An XYZErrorListener to trigger callbacks when errors occur computing tiles.

Example

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> rgb = img.pick_bands("red green blue")
>>> xyz = XYZ.build(rgb, name="My RGB") 
>>> url = xyz.url(session_id="some_session") 
>>> listener = xyz.error_listener() 
>>> errors_log = []
>>> listener.add_callback(lambda error: errors_log.append(error.message)) 
>>> listener.listen("some_session") 
>>> # any errors that occur loading tiles from the generated URL will be appended
>>> # to `errors_log` in the background
classmethod get(xyz_id, client=None)[source]

Get an existing XYZ by id.

Parameters
  • id (string) – The unique id of a XZY object

  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters

Returns

Return type

XYZ

Example

>>> from descarteslabs.workflows import XYZ
>>> xyz = XYZ.get('24d0e79c5c1e1f10a0b1177ef3974d7edefd5988291cf2c6') 
>>> xyz 
<descarteslabs.workflows.models.xyz.XYZ object at 0x...>
iter_tile_errors(session_id, start_datetime=None)[source]

Iterator over errors generated while computing tiles

Parameters
  • session_id (str) – Unique, client-generated that error logs are stored under.

  • start_datetime (datetime.datetime) – Only return errors occuring after this datetime

Yields

error (descarteslabs.common.proto.xyz_pb2.XYZError) – Errors in protobuf message objects, with fields code, message, timestamp, session_id.

Example

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> rgb = img.pick_bands("red green blue")
>>> xyz = XYZ.build(rgb, name="My RGB") 
>>> url = xyz.url(session_id="some_session") 
>>> for error in xyz.iter_tile_errors("some_session", start_datetime=datetime.datetime.now()): 
...     print(error.code, error.message)
>>>     # any errors that occur loading tiles from the generated URL will be printed here
save()[source]

Persist this XYZ layer.

After saving, self.id will contain the new ID of the XYZ layer.

Example

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> rgb = img.pick_bands("red green blue")
>>> xyz = XYZ.build(rgb, name="My RGB") 
>>> xyz.save() 
>>> xyz.id 
'24d0e79c5c1e1f10a0b1177ef3974d7edefd5988291cf2c6'
url(session_id=None, colormap=None, scales=None, checkerboard=False, **parameters)[source]

XYZ tile URL format-string, like https://workflows.descarteslabs.com/v0-5/xyz/1234567/{z}/{x}/{y}.png

Parameters
  • session_id (str, optional, default None) – Unique, client-generated ID that error logs will be stored under. Since multiple users may access tiles from the same XYZ object, each user should set their own session_id to get individual error logs.

  • colormap (str, optional, default None) – Name of the colormap to use. If set, the displayed Image must have 1 band.

  • scales (list of lists, optional, default None) –

    The scaling to apply to each band in the Image this XYZ object will display.

    If the Image contains 3 bands, scales must be a list like [(0, 1), (0, 1), (-1, 1)].

    If the Image contains 1 band, scales must be a list like [(0, 1)], or just (0, 1) for convenience

    If None, each 256x256 tile will be scaled independently.

  • checkerboard (bool, default False) – Whether to display a checkerboarded background for missing or masked data.

  • parameters (dict[str, Union[Proxytype, json_serializable_value]]) –

    Parameters to use while computing.

    Each argument must be the name of a parameter created with parameter. Each value must be a JSON-serializable type (bool, int, float, str, list, dict, etc.), a Proxytype (like Image or Timedelta), or a value that proxify can handle (like a datetime.datetime).

Returns

url – Tile URL containing {z}, {x}, and {y} as Python format string parameters, and query arguments URL-quoted.

Return type

str

Raises
  • ValueError – If the XYZ object has no id and save has not been called yet.

  • TypeError – If the scales or parameters are of invalid type.

Example

>>> import descarteslabs.workflows as wf
>>> img = wf.Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> red = img.pick_bands("red")
>>> viz = red ** wf.parameter("exponent", wf.Float)
>>> xyz = wf.XYZ.build(viz, name="Red band raised to exponent")
>>> xyz.save() 
>>> xyz.url("some_session", colormap="magma", scales=[(0.2, 0.8)], exponent=2.5) 
'https://workflows.descarteslabs.com/master/xyz/0d21037edb4bdd16b735f24bb3bff6d4202a71c20404b101/
 {z}/{x}/{y}.png?session_id=some_session&colormap=magma&scales=[[0.2, 0.8]]&exponent=2.5'
property channel

The channel name this XYZ is compatible with.

Type

str

property created_timestamp

The UTC date this XYZ was created, or None if it hasn’t been saved yet. Cannot be modified.

Type

datetime.datetime or None

property description

A long-form description of this xyz. Markdown is supported.

Type

str

property id

The globally unique identifier for the XYZ, or None if it hasn’t been saved yet.

Type

str or None

property name

The name of this XYZ.

Type

str

property object

The proxy object of this XYZ.

Raises ValueError if the XYZ is not compatible with the current channel.

Type

Proxytype

property type

The type of the proxy object.

Type

type

property updated_timestamp

The UTC date this XYZ was most recently modified, or None if it hasn’t been saved yet. Updated automatically.

Type

datetime.datetime or None

class XYZErrorListener(xyz_id, client=None)[source]

Calls callback functions in a background thread when XYZ errors occur.

Note: the thread is automatically cleaned up on garbage collection.

Example

Methods

add_callback(callback)

Function will be called with descarteslabs.common.proto.xyz_pb2.XYZError on each error.

listen(session_id[, start_datetime])

Start listening for errors.

running()

bool: whether this is an active listener

stop([timeout])

Cancel and clean up the listener.

>>> from descarteslabs.workflows import Image, XYZ
>>> img = Image.from_id("landsat:LC08:PRE:TOAR:meta_LC80270312016188_v1")
>>> xyz = XYZ.build(img)
>>> xyz.save()  
>>> listener = xyz.error_listener()
>>> def callback(msg):
...     print(msg.code, msg.message)
>>> listener.add_callback(callback)
>>> listener.listen("session_id", start_datetime=datetime.datetime.now())  
>>> # later
>>> listener.stop()  
add_callback(callback)[source]

Function will be called with descarteslabs.common.proto.xyz_pb2.XYZError on each error.

Parameters

callback (callable) –

Function that takes one argument, a descarteslabs.common.proto.xyz_pb2.XYZError protobuf message object. This message contains the fields code, message, timestamp, session_id.

The function will be called within a separate thread, therefore it must behave thread-safely. Any errors raised by the function will terminate the listener.

Example

>>> from descarteslabs.workflows import XYZErrorListener
>>> listener = XYZErrorListener("xyz_id") 
>>> def callback(msg):
...     print(msg.code, msg.message)
>>> listener.add_callback(callback) 
listen(session_id, start_datetime=None)[source]

Start listening for errors.

Parameters
  • session_id (str) – Unique, client-generated ID that error logs are stored under. See XYZ.url for more information.

  • start_datetime (datetime.datetime) – Only listen for errors occuring after this datetime. Must be tz-aware.

Example

>>> from descarteslabs.workflows import XYZErrorListener
>>> listener = XYZErrorListener("xyz_id") 
>>> listener.listen("session-id", start_datetime=datetime.datetime.now(datetime.timezone.utc)) 
running()[source]

bool: whether this is an active listener

Example

>>> from descarteslabs.workflows import XYZErrorListener
>>> listener = XYZErrorListener("xyz_id") 
>>> listener.listen("session-id", start_datetime=datetime.datetime.now()) 
>>> listener.running() 
True
stop(timeout=None)[source]

Cancel and clean up the listener. Blocks up to timeout seconds, or forever if None.

Returns True if the background thread stopped successfully.

Example

>>> from descarteslabs.workflows import XYZErrorListener
>>> listener = XYZErrorListener("xyz_id") 
>>> listener.listen("session-id", start_datetime=datetime.datetime.now()) 
>>> listener.stop() 
>>> listener.running() 
False
compute(obj, geoctx=None, format='pyarrow', destination='download', file=None, timeout=None, block=True, progress_bar=None, client=None, cache=True, **params)[source]

Compute a proxy object and wait for its result.

Parameters
  • obj (Proxytype) – A proxy object to compute.

  • geoctx (GeoContext, or None) – The GeoContext parameter under which to run the computation. Almost all computations will require a GeoContext, but for operations that only involve non-geospatial types, this parameter is optional.

  • format (str or dict, default "pyarrow") – The serialization format for the result. See the formats documentation for more information. If “pyarrow” (the default), returns an appropriate Python object, otherwise returns raw bytes.

  • destination (str or dict, default "download") – The destination for the result. See the destinations documentation for more information.

  • file (path or file-like object, optional) – If specified, writes results to the path or file instead of returning them.

  • timeout (int, optional) – The number of seconds to wait for the result, if block is True. Raises JobTimeoutError if the timeout passes.

  • block (bool, default True) – If True (default), block until the job is completed, or timeout has passed. If False, immediately returns a Job (which has already had execute called).

  • progress_bar (bool, default None) – Whether to draw the progress bar. If None (default), will display a progress bar in Jupyter Notebooks, but not elsewhere. Ignored if block==False.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

  • cache (bool, default True) – Whether to use the cache for this job.

  • **params (Proxytype) – Parameters under which to run the computation, such as geoctx.

Returns

result – When format="pyarrow" (the default), returns an appropriate Python object representing the result, either as a plain Python type, or object from descarteslabs.workflows.result_types. For other formats, returns raw bytes. Consider using file in that case to save the results to a file. If the destination doesn’t support retrieving results (like “email”), returns None

Return type

Python object, bytes, or None

Example

>>> import descarteslabs.workflows as wf
>>> num = wf.Int(1) + 1
>>> wf.compute(num) 
2
>>> # same computation but do not block
>>> job = wf.compute(num, block=False) 
>>> job 
<descarteslabs.workflows.models.job.Job object at 0x...>
>>> job.result() 
2
>>> # pass multiple proxy objects to `wf.compute` to compute all at once
>>> wf.compute((num, num, num)) 
(2, 2, 2)
>>> # specifying a format
>>> img = wf.Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1").pick_bands("red")
>>> wf.compute(img, geoctx=ctx, format="pyarrow") # default 
ImageResult:
...
>>> # same computation but with json format
>>> wf.compute(img, geoctx=ctx, format="json") 
b'{"ndarray":[[[0.39380000000000004,0.3982,0.3864,...
>>> # same computation but with geotiff format (and some format options)
>>> bytes_ = wf.compute(img, geoctx=ctx, format={"type": "geotiff", "tiled": False}) 
>>> # you probably want to save the geotiff to a file:
>>> wf.compute(img, geoctx=ctx, file="my_geotiff.tif", format={"type": "geotiff", "tiled": False}) 
>>> # specifying a destination
>>> num = wf.Int(1) + 1
>>> wf.compute(num, destination="download") # default 
2
>>> # same computation but with email destination
>>> wf.compute(num, destination="email") 
>>> # now with some destination options
>>> wf.compute(
...     num,
...     destination={
...         "type": "email",
...         "subject": "My Computation is Done"
...     },
...     format="json",
... ) 
publish(id, version, obj=None, title='', description='', public=False, labels=None, tags=None, docstring='', version_labels=None, client=None)[source]

Publish a proxy object as a Workflow with the given version. Can also be used as a decorator.

Parameters
  • id (str) – ID for the new Workflow. This should be of the form "email:workflow_name" and should be globally unique. If this ID is not of the proper format, you will not be able to save the Workflow.

  • version (str) – The version to be set, tied to the given proxy_object. This should adhere to the semantic versioning schema.

  • obj (Proxytype) – The object to store as this version. If not provided, it’s assumed that publish is being used as a decorator on a function.

  • title (str, default "") – User-friendly title for the Workflow.

  • description (str, default "") – Long-form description of this Workflow. Markdown is supported.

  • public (bool, default False) – Whether this Workflow will be publicly accessible.

  • labels (dict, optional) – Key-value pair labels to add to the Workflow.

  • tags (list, optional) – A list of strings to add as tags to the Workflow.

  • docstring (str, default "") – The docstring for this version.

  • version_labels (dict, optional) – Key-value pair labels to add to the version.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

Returns

workflow – The saved Workflow object. workflow.id contains the ID of the new Workflow. If used as a decorator, returns the Function instead.

Return type

Workflow or Function

Example

>>> import descarteslabs.workflows as wf
>>> @wf.publish("bob@gmail.com:ndvi", "0.0.1") 
... def ndvi(img: wf.Image) -> wf.Image:
...     "Compute the NDVI of an Image"
...     nir, red = img.unpack_bands("nir red")
...     return (nir - red) / (nir + red)
>>> # `ndvi` becomes a Function proxy object
>>> ndvi  
<descarteslabs.workflows.types.Function[Image, {}, Image] object at 0x...>
>>> two = wf.Int(1) + 1
>>> workflow = wf.publish("bob@gmail.com:two", "1.0.0", two) 
>>> workflow 
<descarteslabs.workflows.models.workflow.Workflow object at 0x...>
>>> workflow.version_names 
["1.0.0"]
use(workflow_id, version, client=None)[source]

Use like import: load the proxy object of a published Workflow version.

Parameters
  • workflow_id (str) – ID of the Workflow to retrieve

  • version (str) – Version of the workflow to retrive

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters

Returns

obj – Proxy object of the Workflow version.

Return type

Proxytype

Example

>>> import descarteslabs.workflows as wf
>>> @wf.publish("bob@gmail.com:ndvi", "0.0.1") 
... def ndvi(img: wf.Image) -> wf.Image:
...     nir, red = img.unpack_bands("nir red")
...     return (nir - red) / (nir + red)
>>> same_function = wf.use("bob@gmail.com:ndvi", "0.0.1") 
>>> same_function 
<descarteslabs.workflows.types.function.function.Function[Image, {}, Image] object at 0x...>
>>> img = wf.Image.from_id("sentinel-2:L1C:2019-05-04_13SDV_99_S2B_v1")
>>> same_function(img).compute(geoctx) # geoctx is an arbitrary geocontext for 'img' 
ImageResult:
...