Execution

Exceptions

JobComputeError(job)
TimeoutError

Classes

Workflow(proxy_object, proto_message[, client]) A proxy object, and metadata about it.
Job(message[, client]) A Job represents the computation of a proxy object’s graft within a specific environment of parameters.
XYZ(proxy_object, proto_message[, client]) Stores proxy objects to be rendered by an XYZ tile server.
XYZErrorListener(xyz_id[, client]) Calls callback functions in a background thread when XYZ errors occur.

Functions

compute(obj[, timeout, block, progress_bar, …]) Compute a proxy object and wait for its result.
publish(obj[, name, description, client]) Publish a proxy object as a Workflow.
retrieve(workflow_id[, client]) Load a published Workflow object.
use(workflow_id[, client]) Use like import: load the proxy object of a published Workflow.
exception JobComputeError(job)[source]
args
code
id
message
exception TimeoutError[source]
args
message
class Workflow(proxy_object, proto_message, client=None)[source]

A proxy object, and metadata about it.

To construct, use:

Methods

build(proxy_object[, name, description, client]) Construct a new Workflow from a proxy object.
get(workflow_id[, client]) Get an existing workflow by id.
save() Persist this Workflow.
update([proxy_object, name, description]) Update the proxy object and/or metadata of this Workflow.

Attributes

channel The channel name this Workflow is compatible with.
created_timestamp The UTC date this Workflow was created, or None if it hasn’t been saved yet.
description A long-form description of this workflow.
id The globally unique identifier for the Workflow, or None if it hasn’t been saved yet.
name The name of this Workflow.
object The proxy object of this Workflow.
owners
readers
type The type of the proxy object.
updated_timestamp The UTC date this Workflow was most recently modified, or None if it hasn’t been saved yet.
writers

Construct a Workflow object from a proxy object and Protobuf message.

Do not use this method directly; use the Workflow.build and Workflow.get classmethods instead.

Parameters:
  • proxy_object (Proxytype) – The proxy object to store in this Workflow
  • proto_message (workflow_pb2.Workflow message) – Protobuf message for the Workflow
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
classmethod build(proxy_object, name='', description='', client=None)[source]

Construct a new Workflow from a proxy object.

Note that this does not persist the Workflow, call save() on the returned Workflow to do that.

Parameters:
  • proxy_object (Proxytype) – The proxy object to store in this Workflow
  • name (str, default "") – Name for the new Workflow
  • description (str, default "") – Long-form description of this Workflow. Markdown is supported.
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

Return type:

Workflow

classmethod get(workflow_id, client=None)[source]

Get an existing workflow by id.

Parameters:
  • id (string) – The unique id of a Workflow
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

Return type:

Workflow

save()[source]

Persist this Workflow.

After saving, self.id will contain the new ID of the Workflow.

update(proxy_object=None, name=None, description=None)[source]

Update the proxy object and/or metadata of this Workflow.

Values given as None are left unchanged.

Parameters:
  • proxy_object (Proxytype or None, default None) – New proxy object for this workflow
  • name (str or None, default None) – New name for the Workflow
  • description (str or None, default None) – New long-form description of this Workflow. Markdown is supported.
channel

The channel name this Workflow is compatible with.

Type:str
created_timestamp

The UTC date this Workflow was created, or None if it hasn’t been saved yet. Cannot be modified.

Type:datetime.datetime or None
description

A long-form description of this workflow. Markdown is supported.

Type:str
id

The globally unique identifier for the Workflow, or None if it hasn’t been saved yet.

Type:str or None
name

The name of this Workflow.

Type:str
object

The proxy object of this Workflow.

Raises ValueError if the Workflow is not compatible with the current channel.

Type:Proxytype
owners
readers
type

The type of the proxy object.

Type:type
updated_timestamp

The UTC date this Workflow was most recently modified, or None if it hasn’t been saved yet. Updated automatically.

Type:datetime.datetime or None
writers
class Job(message, client=None)[source]

A Job represents the computation of a proxy object’s graft within a specific environment of parameters.

Example

Attributes

BUCKET_PREFIX str(object=’‘) -> string
WAIT_INTERVAL float(x) -> floating point number
WATCH_TIMEOUT int(x=0) -> int or long
channel The channel name where this Job will execute.
created_datetime
done
error
id The globally unique identifier for the Job, or None if it hasn’t been executed yet.
object The proxy object this Job computes.
parameters
result_type Name of the type of object that will be used to hold the result
runtime
stage
status
type The type of the proxy object.
updated_datetime

Methods

build(proxy_object, parameters[, channel, …]) Build a new Job for computing a proxy object under certain parameters.
cancel() Cancel a running job.
execute() Asynchronously submit a job for execution.
get(id[, client]) Get a currently-running Job by its ID.
refresh() Refresh the attributes and status of the job.
result([timeout, progress_bar]) Get the result of the job.
watch()
>>> import descarteslabs.workflows as wf
>>> img = wf.Image.from_id("landsat:LC08:PRE:TOAR:meta_LC80270312016188_v1")
>>> job = img.compute(geoctx, block=False)  # doctest: +SKIP
>>> result = job.result()  # doctest: +SKIP
classmethod build(proxy_object, parameters, channel=None, client=None)[source]

Build a new Job for computing a proxy object under certain parameters. Does not actually trigger computation; call Job.execute on the result to do so.

Parameters:
  • proxy_object (Proxytype) – Proxy object to compute
  • parameters (dict[str, Proxytype]) – Python dictionary of parameter names and values
  • channel (str or None, optional) –

    Channel name to submit the Job to. If None, uses the default channel for this client (descarteslabs.workflows.__channel__).

    Channels are different versions of the backend, to allow for feature changes without breaking existing code. Not all clients are compatible with all channels. This client is only guaranteed to work with its default channel, whose name can be found under descarteslabs.workflows.__channel__.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

The job waiting to be executed.

Return type:

Job

cancel()[source]

Cancel a running job.

execute()[source]

Asynchronously submit a job for execution.

After submission, self.id will be the ID of the running job.

This method is idempotent: calling it multiple times on the same Job object will only trigger execution once.

classmethod get(id, client=None)[source]

Get a currently-running Job by its ID.

refresh()[source]

Refresh the attributes and status of the job.

result(timeout=None, progress_bar=None)[source]

Get the result of the job. This blocks until the job is complete.

Parameters:
  • timeout (int, optional) – The number of seconds to wait for the result.
  • progress_bar (bool, optional) – Flag to draw the progress bar. Default is to True if in Jupyter Notebook.
Returns:

Appropriate Python object representing the result, either as a plain Python type, or object from descarteslabs.common.workflows.containers.

Return type:

result

Example

>>> import descarteslabs.workflows as wf
>>> job = wf.Int(1).compute()  # doctest: +SKIP
>>> job.result(timeout=10)  # doctest: +SKIP
1
watch()[source]
BUCKET_PREFIX = 'https://storage.googleapis.com/dl-compute-dev-results/{}'
WAIT_INTERVAL = 0.1
WATCH_TIMEOUT = 30
channel

The channel name where this Job will execute.

Type:str
created_datetime
done
error
id

The globally unique identifier for the Job, or None if it hasn’t been executed yet.

Type:str or None
object

The proxy object this Job computes.

Type:Proxytype
parameters
result_type

Name of the type of object that will be used to hold the result

Type:str
runtime
stage
status
type

The type of the proxy object.

Type:type
updated_datetime
class XYZ(proxy_object, proto_message, client=None)[source]

Stores proxy objects to be rendered by an XYZ tile server.

Similar to a Workflow, but meant for storing proxy objects so the XYZ tile service can display them, rather than for persisting and sharing workflows between users.

Use url to generate an XYZ URL template, and iter_tile_errors or error_listener to retrieve error messages that happen while computing them.

Construct a XYZ object from a proxy object and Protobuf message.

Do not use this method directly; use the XYZ.build and XYZ.get classmethods instead.

Parameters:
  • proxy_object (Proxytype) – The proxy object to store in this XYZ
  • proto_message (xyz_pb2.XYZ message) – Protobuf message for the XYZ
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters

Attributes

BASE_URL str(object=’‘) -> string
channel The channel name this XYZ is compatible with.
created_timestamp The UTC date this XYZ was created, or None if it hasn’t been saved yet.
description A long-form description of this xyz.
id The globally unique identifier for the XYZ, or None if it hasn’t been saved yet.
name The name of this XYZ.
object The proxy object of this XYZ.
owners
readers
type The type of the proxy object.
updated_timestamp The UTC date this XYZ was most recently modified, or None if it hasn’t been saved yet.
writers

Methods

build(proxy_object[, name, description, client]) Construct a new XYZ from a proxy object.
error_listener() An XYZErrorListener to trigger callbacks when errors occur computing tiles
get(xyz_id[, client]) Get an existing XYZ by id.
iter_tile_errors(session_id[, start_datetime]) Iterator over errors generated while computing tiles
save() Persist this XYZ layer.
update(*args, **kwargs)
url([session_id]) XYZ tile URL format-string, like https://workflows.descarteslabs.com/v0-5/xyz/1234567/{z}/{x}/{y}.png
classmethod build(proxy_object, name='', description='', client=None)[source]

Construct a new XYZ from a proxy object.

Note that this does not persist the XYZ, call save() on the returned XYZ to do that.

Parameters:
  • proxy_object (Proxytype) – The proxy object to store in this XYZ
  • name (str, default "") – Name for the new XYZ
  • description (str, default "") – Long-form description of this XYZ. Markdown is supported.
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

Return type:

XYZ

error_listener()[source]

An XYZErrorListener to trigger callbacks when errors occur computing tiles

classmethod get(xyz_id, client=None)[source]

Get an existing XYZ by id.

Parameters:
  • id (string) – The unique id of a XZY object
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

Return type:

XYZ

iter_tile_errors(session_id, start_datetime=None)[source]

Iterator over errors generated while computing tiles

Parameters:
  • session_id (str) – Unique, client-generated that error logs are stored under.
  • start_datetime (datetime.datetime) – Only return errors occuring after this datetime
Yields:

error (descarteslabs.common.proto.xyz_pb2.XYZError) – Errors in protobuf message objects, with fields code, message, timestamp, session_id.

save()[source]

Persist this XYZ layer.

After saving, self.id will contain the new ID of the XYZ layer.

update(*args, **kwargs)[source]
url(session_id=None, **query_args)[source]

XYZ tile URL format-string, like https://workflows.descarteslabs.com/v0-5/xyz/1234567/{z}/{x}/{y}.png

Parameters:
  • session_id (str, optional, default None) – Unique, client-generated ID that error logs will be stored under. Since multiple users may access tiles from the same XYZ object, each user should set their own session_id to get individual error logs.
  • query_args (dict[str, str], optional, default None) – Additional query arguments to add to the URL. Keys and values must be strings.
Returns:

url

Return type:

str

Raises:

ValueError – If the XYZ object has no id and save has not been called yet.

BASE_URL = 'https://workflows.descarteslabs.com'
channel

The channel name this XYZ is compatible with.

Type:str
created_timestamp

The UTC date this XYZ was created, or None if it hasn’t been saved yet. Cannot be modified.

Type:datetime.datetime or None
description

A long-form description of this xyz. Markdown is supported.

Type:str
id

The globally unique identifier for the XYZ, or None if it hasn’t been saved yet.

Type:str or None
name

The name of this XYZ.

Type:str
object

The proxy object of this XYZ.

Raises ValueError if the XYZ is not compatible with the current channel.

Type:Proxytype
owners
readers
type

The type of the proxy object.

Type:type
updated_timestamp

The UTC date this XYZ was most recently modified, or None if it hasn’t been saved yet. Updated automatically.

Type:datetime.datetime or None
writers
class XYZErrorListener(xyz_id, client=None)[source]

Calls callback functions in a background thread when XYZ errors occur.

Note: the thread is automatically cleaned up on garbage collection.

Example

Methods

add_callback(callback) Function will be called with descarteslabs.common.proto.xyz_pb2.XYZError on each error.
listen(session_id[, start_datetime]) Start listening for errors.
running() bool: whether this is an active listener
stop([timeout]) Cancel and clean up the listener.
>>> import descarteslabs.workflows as wf
>>> xyz = wf.XYZ.build(wf.Image.from_id("landsat:LC08:PRE:TOAR:meta_LC80270312016188_v1"))
>>> xyz.save()  # doctest: +SKIP
>>> listener = xyz.error_listener()
>>> def callback(msg):
...     print(msg.code, msg.message)
>>> listener.add_callback(callback)
>>> listener.listen("my_session_id", start_datetime=datetime.datetime.now())  # doctest: +SKIP
>>> # later
>>> listener.stop()  # doctest: +SKIP
add_callback(callback)[source]

Function will be called with descarteslabs.common.proto.xyz_pb2.XYZError on each error.

Parameters:callback (callable) –

Function that takes one argument, a descarteslabs.common.proto.xyz_pb2.XYZError protobuf message object. This message contains the fields code, message, timestamp, session_id.

The function will be called within a separate thread, therefore it must behave thread-safely. Any errors raised by the function will terminate the listener.

listen(session_id, start_datetime=None)[source]

Start listening for errors.

Parameters:
  • session_id (str) – Unique, client-generated ID that error logs are stored under. See XYZ.url for more information.
  • start_datetime (datetime.datetime) – Only listen for errors occuring after this datetime.
running()[source]

bool: whether this is an active listener

stop(timeout=None)[source]

Cancel and clean up the listener. Blocks up to timeout seconds, or forever if None.

Returns True if the background thread stopped successfully.

compute(obj, timeout=None, block=True, progress_bar=None, channel=None, client=None, **params)[source]

Compute a proxy object and wait for its result.

Parameters:
  • obj (Proxytype) – A proxy object to compute
  • timeout (int, optional) – The number of seconds to wait for the result, if block is True. Raises TimeoutError if the timeout passes.
  • block (bool, default True) – If True (default), block until the job is completed, or timeout has passed. If False, immediately returns a Job (which has already had execute called).
  • progress_bar (bool, default None) – Whether to draw the progress bar. If None (default), will display a progress bar in Jupyter Notebooks, but not elsewhere. Ignored if block==False.
  • channel (str or None, optional) –

    Channel name to submit the Job to. If None, uses the default channel for this client (descarteslabs.workflows.__channel__).

    Channels are different versions of the backend, to allow for feature changes without breaking existing code. Not all clients are compatible with all channels. This client is only guaranteed to work with its default channel, whose name can be found under descarteslabs.workflows.__channel__.

  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters
  • **params (Proxytype) – Parameters under which to run the computation, such as geoctx.
Returns:

Appropriate Python object representing the result, either as a plain Python type, or object from descarteslabs.common.workflows.containers.

Return type:

result

publish(obj, name='', description='', client=None)[source]

Publish a proxy object as a Workflow.

Parameters:
  • obj (Proxytype) – A proxy object to compute
  • name (str, default "") – Name for the new Workflow
  • description (str, default "") – Long-form description of this Workflow. Markdown is supported.
  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

workflow – The saved Workflow object. workflow.id contains the ID of the new Workflow.

Return type:

Workflow

retrieve(workflow_id, client=None)[source]

Load a published Workflow object.

Parameters:
  • workflow_id (str) – ID of the Workflow to retrieve
  • client (Compute, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

workflow – Object representing the workflow, including both its metadata (like workflow.name, workflow.description) and proxy object (workflow.object).

Return type:

Workflow

use(workflow_id, client=None)[source]

Use like import: load the proxy object of a published Workflow.

Shorthand for retrieve(workflow_id).object.

Parameters:
  • workflow_id (str) – ID of the Workflow to retrieve
  • client (workflows.client.Client, optional) – Allows you to use a specific client instance with non-default auth and parameters
Returns:

obj – Proxy object of the Workflow.

Return type:

Proxytype