Compute

The Batch Compute API can be found as Python package descarteslabs.compute. To install please refer to Installation.

Classes:

`ComputeClient`([url, auth, catalog_client, ...])
`ComputeResult`([value, description, expires, ...])	Used to store the result of a compute function with additional attributes.
`Function`([function, requirements, ...])	The serverless cloud function that you can call directly or submit many jobs to.
`FunctionStatus`(value)	The status of the Function.
`Job`(function_id[, args, kwargs, client, ...])	A single invocation of a Function.
`JobSearch`(document, client[, url])
`JobStatus`(value)	The status of the Job.
`Search`(document, client[, url])	A search request that iterates over its search results.
`Serializable`()	Interface for serializing objects to bytes as a result of a Function invocation.

class ComputeClient(url=None, auth=None, catalog_client=None, retries=None)[source]

Bases: ApiService, DefaultClientMixin

Methods:

`check_credentials`()	Determine if valid credentials are already set for the user.
`clear_all_default_clients`()	Clear all default clients of this class and all its subclasses.
`get_default_client`()	Retrieve the default client.
`get_default_session_class`()	Get the default session class for `ApiService`.
`get_namespace`(function_id)
`iter_log_lines`(url[, timestamps])
`iter_pages`(url[, params])
`set_credentials`()
`set_default_client`(client)	Change the default client to the given client.
`set_default_session_class`(session_class)	Set the default session class for `ApiService`.
`set_namespace`(function_id, namespace)

Attributes:

`CONNECT_TIMEOUT`
`READ_TIMEOUT`
`RETRY_CONFIG`
`TIMEOUT`
`session`	The session instance used by this service.
`token`	The bearer token used in the requests.

check_credentials()[source]: Determine if valid credentials are already set for the user.

classmethod clear_all_default_clients(): Clear all default clients of this class and all its subclasses.

classmethod get_default_client()

Retrieve the default client.

This client is used whenever you don’t explicitly set the client.

classmethod get_default_session_class()

Get the default session class for ApiService.

Returns:: The default session class, which is ApiService itself or a derived class from ApiService.
Return type:: ApiService

get_namespace(function_id: str) → str | None[source]

iter_log_lines(url: str, timestamps: bool = True) → Iterator[str][source]

iter_pages(url: str, params: dict | None = None)

set_credentials()[source]

classmethod set_default_client(client)

Change the default client to the given client.

This is the client that will be used whenever you don’t explicitly set the client

classmethod set_default_session_class(session_class)

Set the default session class for ApiService.

The default session is used for any ApiService that is instantiated without specifying the session class.

Parameters:: session_class (class) – The session class to use when instantiating the session. This must be the class ApiSession itself or a derived class from ApiSession.

set_namespace(function_id: str, namespace: str)[source]

CONNECT_TIMEOUT = 9.5

READ_TIMEOUT = 30

RETRY_CONFIG = Retry(total=3, connect=2, read=2, redirect=None, status=2)

TIMEOUT = (9.5, 30)

property session: Session

The session instance used by this service.

Type:: Session

property token

The bearer token used in the requests.

Type:: str

Bases: object

Used to store the result of a compute function with additional attributes.

When returned from a compute function, the result will be serialized and stored in the Storage service with the given attributes.

Notes

Results that are None and have no attributes will not be stored. If you want to store a None result with attributes, you can do so by passing in a None value as well as any attributes you wish to set.

Examples

Result with raw binary data: >>> from descarteslabs.services.compute import ComputeResult >>> result = ComputeResult(value=b”result”, description=”result description”)

Null result with attributes: >>> from descarteslabs.services.compute import ComputeResult >>> result = ComputeResult(None, geometry=geometry, tags=[“tag1”, “tag2”])

Parameters:

value (bytes, Serializable, or Any) – The resulting value of a compute function. This can be any bytes, any JSON serializable type or any type implementing the Serializable interface.
description (str or None) – A description with further details on this result blob. The description can be up to 80,000 characters and is used by Search.find_text().
expires (datetime, str, or None) – The date the result should expire and be deleted from storage. If a string is given, it must be in ISO 8601 format.
extra_properties (dict or None) –
A dictionary of up to 50 key/value pairs.

The keys of this dictionary must be strings, and the values of this dictionary can be strings or numbers. This allows for more structured custom metadata to be associated with objects.
geometry (shapely.geometry.base.BaseGeometry, dict, str, or None) – The geometry associated with the result if any.
tags (set, list, or None) – The tags to set on the catalog object for the result.

Bases: Document

The serverless cloud function that you can call directly or submit many jobs to.

Parameters:

function (Callable) – The function to be called in a Compute Job.
requirements (List[str], optional) – A list of Python dependencies required by this function.
include_data (List[str], optional) – Non-Python data files to include in the compute function.
name (str, optional) – Name of the function, will take name of function if not provided.
image (str) – The location of a docker image to be used for the environment where the function will be executed.
cpus (Cpus) – The number of CPUs requested for a single Job. CPUs can be specified as an integer, float, or string. Supported CPU options include: 0.25, 0.5, 1, 2, 4, 8, 16, '0.25vCPU', '0.5vCPU', '1vCPU', '2vCPU', '4vCPU', '8vCPU', '16vCPU'.
memory (Memory) – The maximum memory requirement for a single Job. Memory can be specified as an integer or string. If an integer is provided, it is assumed that the units are megabytes. For instance, 1024 is equivalent to one 1GB or 1024MB of memory. Alternatively, memory can be specified as a case-insensitive memory string, such as '1GB', '1Gi', '1024MB', or '1024Mi', all of which are equivalent. The allowable memory for a Function is determined by the number of CPUs requested. Supported memory options per CPUs requested include: 0.25vCPU: 0.5GB, 1GB, or 2GB, 0.5vCPU: 1 - 4GB in 1GB increments, 1vCPU: 2 - 8GB in 1GB increments, 2vCPU: 4 - 16GB in 1GB increments, 4vCPU: 8 - 30GB in 1GB increments, 8vCPU: 16 - 60GB in 4GB increments, 16vCPU: 32 - 120GB in 8GB increments.
maximum_concurrency (int) – The maximum number of jobs to run in parallel.
timeout (int, optional) – Maximum runtime for a single job in seconds. Job will be killed if it exceeds this limit.
retry_count (int, optional) – Number of times to retry a job if it fails.
client (ComputeClient, optional) – If set, operations on the function will be performed using the configured client. Otherwise, the default client will be used.

Examples

Retrieving an existing function and executing it.

>>> fn = Function.get(<function-id>)
>>> fn()
Job <job id>: "pending"

Creating a new function.

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn()
Job <job id>: "pending"

Methods:

`as_completed`(jobs[, timeout, interval])	Yields jobs as they complete.
`build_log`()	Retrieves the build log for the Function.
`cancel_jobs`([query, job_ids])	Cancels all jobs for the Function matching the given query.
`delete`([delete_results])	Deletes the Function and all associated Jobs.
`delete_jobs`([query, job_ids, delete_results])	Deletes all non-running jobs for the Function matching the given query.
`disable`()	Disables the Function so that new jobs cannot be submitted.
`enable`()	Enables the Function so that new jobs may be submitted.
`get`(id[, client])	Get Function by id.
`iter_results`([cast_type])	Iterates over all successful job results.
`list`([page_size, client])	Lists all Functions for a user.
`map`(args[, kwargs, tags, batch_size, ...])	Submits multiple jobs efficiently with positional args to each function call.
`refresh`([includes])	Updates the Function instance with data from the server.
`rerun`([query, job_ids])	Submits all the unsuccessful jobs matching the query to be rerun.
`results`([cast_type])	Retrieves all the job results for the Function as a list.
`save`()	Creates the Function if it does not already exist.
`search`([client])	Creates a search for Functions.
`start`()	Starts Function so that pending jobs can be executed.
`stop`()	Stops Function so that pending jobs cannot be executed.
`to_dict`([only_modified, exclude_readonly, ...])	Converts the document to a dictionary.
`update`([ignore_missing])	Updates the document setting multiple attributes at a time.
`update_credentials`([client])	Updates the credentials for the Functions and Jobs run by this user.
`wait_for_completion`([timeout, interval])	Waits until all submitted jobs for a given Function are completed.

Attributes:

`auto_start`	Whether the Function will be placed into READY status once building is complete.
`cpus`	The number of cpus to request when executing the Function.
`creation_date`	The date the Function was created.
`enabled`	Whether the Function accepts job submissions and reruns.
`environment`	The environment variables to set for Jobs run by the Function.
`id`	The ID of the Function.
`image`	The base image used to create the Function.
`is_modified`	Determines if the document has been modified.
`job_statistics`	Statistics about the Job statuses for this Function.
`jobs`	Returns a JobSearch for all the Jobs for the Function.
`maximum_concurrency`	The maximum number of Jobs that execute at the same time for this Function.
`memory`	The amount of memory, in megabytes, to request when executing the Function.
`modified_date`	The date the Function was last modified or processed a job submission.
`name`	The name of the Function.
`namespace`	The storage namespace for the Function.
`owner`	The owner of the Function.
`retry_count`	The total number of retries requested for a Job before it failed.
`state`	Returns the state of the current document instance.
`status`	The status of the Function.
`timeout`	The number of seconds Jobs can run before timing out for this Function.

as_completed(jobs: Iterable[Job], timeout=None, interval=10)[source]

Yields jobs as they complete.

Completion includes success, failure, timeout, and canceled.

Can be used in any iterable context.

Parameters:

jobs (Iterable[Job]) – The jobs to wait for completion. Jobs must be associated with this Function.
timeout (int, default=None) – Maximum time to wait before timing out. If not set, this will continue polling until all jobs have completed.
interval (int, default=10) – Interval in seconds for how often to check if jobs have been completed.

build_log()[source]: Retrieves the build log for the Function.

cancel_jobs(query: JobSearch | None = None, job_ids: List[str] | None = None)[source]

Cancels all jobs for the Function matching the given query.

If both query and job_ids are None, all jobs for the Function will be canceled. If both are provided, they will be combined with an AND operator. Any jobs matched by query or job_ids which are not associated with this function will be ignored.

Parameters:

query (JobSearch, optional) – Query to filter jobs to cancel.
job_ids (List[str], optional) – List of job ids to cancel.

delete(delete_results: bool = False)[source]

Deletes the Function and all associated Jobs.

If any jobs are in a running state, the deletion will fail.

Please see the :meth:~descarteslabs.compute.Function.delete_jobs method for more information on deleting large numbers of jobs.

Parameters:: delete_results (bool, default=False) – If True, deletes the job result blobs as well.

delete_jobs(query: JobSearch | None = None, job_ids: List[str] | None = None, delete_results: bool = False) → List[str][source]

Deletes all non-running jobs for the Function matching the given query.

If both query and job_ids are None, all jobs for the Function will be deleted. If both are provided, they will be combined with an AND operator. Any jobs matched by query or job_ids which are not associated with this function will be ignored.

Also deletes any job log blobs for the jobs. Use delete_results=True to delete the job result blobs as well.

There is a limit to how many jobs can be deleted in a single request before the request times out. If you need to delete a large number of jobs and experience timeouts, consider using a loop to delete batches, using the query parameter with a limit (e.g. async_func.delete_jobs(async_func.jobs.limit(10000)), or use the job_ids parameter to limit the number of jobs to delete.

Parameters:

query (JobSearch, optional) – Query to filter jobs to delete.
job_ids (List[str], optional) – List of job ids to delete.
delete_results (bool, default=False) – If True, deletes the job result blobs as well.

Returns:

List of job ids that were deleted.

Return type:

List[str]

disable()[source]: Disables the Function so that new jobs cannot be submitted.

enable()[source]: Enables the Function so that new jobs may be submitted.

classmethod get(id: str, client: ComputeClient | None = None, **params)[source]

Get Function by id.

Parameters:

id (str) – Id of function to get.
client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.
include (List[str], optional) –
List of additional attributes to include in the response. Allowed values are:
- ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
<Function name="test_name" image=test_image cpus=1 memory=16 maximum_concurrency=5 timeout=3 retries=1

iter_results(cast_type: Type[Serializable] | None = None)[source]

Iterates over all successful job results.

Parameters:: cast_type (Type[Serializable], optional) – If set, the results will be cast to the given type.

classmethod list(page_size: int = 100, client: ComputeClient | None = None, **params) → Search[Function][source]

Lists all Functions for a user.

If you would like to filter Functions, use Function.search().

Parameters:

page_size (int, default=100) – Maximum number of results per page.
client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.
include (List[str], optional) –
List of additional attributes to include in the response. Allowed values are:
- ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.list()

map(args: Iterable[Iterable[Any]], kwargs: Iterable[Mapping[str, Any]] = None, tags: List[str] = None, batch_size: int = 1000, environments: Iterable[Mapping[str, str]] = None) → JobBulkCreateResult[source]

Submits multiple jobs efficiently with positional args to each function call.

Preferred over repeatedly calling the function, such as in a loop, when submitting multiple jobs.

If supplied, the length of kwargs must match the length of args. All parameter values must be JSON serializable.

As an example, if the function takes two positional arguments and has a keyword argument x, you can submit multiple jobs like this:

>>> async_func.map([['a', 'b'], ['c', 'd']], [{'x': 1}, {'x': 2}])  

is equivalent to:

>>> async_func('a', 'b', x=1)  
>>> async_func('c', 'd', x=2)  

Notes

Map is idempotent for the initial call such that request errors that occur once started, will not cause duplicate jobs to be submitted. However, if the method is called again with the same arguments, it will submit duplicate jobs.

You should always check the return value to ensure all jobs were submitted successfully and handle any errors that may have occurred.

Parameters:

args (Iterable[Iterable[Any]]) – An iterable of iterables of arguments. For each outer element, a job will be submitted with each of its elements as the positional arguments to the function. The length of each element of the outer iterable must match the number of positional arguments to the function.
kwargs (Iterable[Mapping[str, Any]], optional) – An iterable of Mappings with keyword arguments. For each outer element, the Mapping will be expanded into keyword arguments for the function.
environments (Iterable[Mapping[str, str]], optional) – AN iterable of Mappings of Environment variables to be set in the environment of the running Jobs. The values for each job will be merged with environment variables set on the Function, with the Job environment variables taking precedence.
tags (List[str], optional) – A list of tags to apply to all jobs submitted.
batch_size (int, default=1000) – The number of jobs to submit in each batch. The maximum batch size is 1000.

Returns:

An object containing the jobs that were submitted and any errors that occurred. This object is compatible with a list of Job objects for backwards compatibility.

If the value of JobBulkCreateResult.is_success is False, you should check JobBulkCreateResult.errors and handle any errors that occurred.

Return type:

JobBulkCreateResult

Raises:

ClientError, ServerError – If the request to create the first batch of jobs, fails after all retries have been exhausted. Otherwise, any errors will be available in the returned JobBulkCreateResult.

refresh(includes: str | None = None)[source]

Updates the Function instance with data from the server.

Parameters:

includes (Optional[str], optional) –

List of additional attributes to include in the response. Allowed values are:

”job.statistics”: Include statistics about the Job statuses for this Function.

rerun(query: JobSearch | None = None, job_ids: List[str] | None = None)[source]

Submits all the unsuccessful jobs matching the query to be rerun.

If both query and job_ids are None, all rerunnable jobs for the Function will be rerun. If both are provided, they will be combined with an AND operator. Any jobs matched by query or job_ids which are not associated with this function will be ignored.

Parameters:

query (JobSearch, optional) – Query to filter jobs to rerun.
job_ids (List[str], optional) – List of job ids to rerun.

results(cast_type: Type[Serializable] | None = None)[source]

Retrieves all the job results for the Function as a list.

Notes

This immediately downloads all results into a list and could run out of memory. If the result set is large, strongly consider using Function.iter_results() instead.

Parameters:: cast_type (Type[Serializable], optional) – If set, the results will be cast to the given type.

save()[source]

Creates the Function if it does not already exist.

If the Function already exists, it will be updated on the server if the Function instance was modified.

Examples

Create a Function without creating jobs:

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn.save()

Updating a Function:

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
>>> fn.memory = 4096  # 4 Gi
>>> fn.save()

classmethod search(client: ComputeClient | None = None) → Search[Function][source]

Creates a search for Functions.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Parameters:: client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.

Example

>>> from descarteslabs.compute import Function, FunctionStatus
>>> fns: List[Function] = (
...     Function.search()
...     .filter(Function.status.in_([
...         FunctionStatus.BUILDING, FunctionStatus.AWAITING_BUNDLE
...     ])
...     .collect()
... )
Collection([Function <fn-id1>: building, Function <fn-id2>: awaiting_bundle])

start()[source]: Starts Function so that pending jobs can be executed.

stop()[source]: Stops Function so that pending jobs cannot be executed.

to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:

only_modified (bool, False) – If set, only modified attributes and their values will be included.
exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
exclude_none (bool, False) – If set, attributes with a value of None will be excluded.

Returns:

The attributes matching the call parameters. The result of this function can be json encoded without modification.

Return type:

Dict[str, Any]

update(ignore_missing=False, **kwargs)

Updates the document setting multiple attributes at a time.

Parameters:: ignore_missing (bool, False) – If set, unknown attributes will be ignored.

classmethod update_credentials(client: ComputeClient | None = None)[source]

Updates the credentials for the Functions and Jobs run by this user.

These credentials are used by other Descarteslabs services.

If the user invalidates existing credentials and needs to update them, you should call this method.

Notes

Credentials are automatically updated when a new Function is created.

Parameters:: client (ComputeClient, optional) – If set, the operation will be performed using the configured client. Otherwise, the default client will be used.

wait_for_completion(timeout=None, interval=10)[source]

Waits until all submitted jobs for a given Function are completed.

Completion includes success, failure, timeout, and canceled.

Parameters:

timeout (int, default=None) – Maximum time to wait before timing out. If not set, this method will block indefinitely.
interval (int, default=10) – Interval in seconds for how often to check if jobs have been completed.

auto_start: bool

Whether the Function will be placed into READY status once building is complete.

Type:: bool or None

cpus: float

The number of cpus to request when executing the Function.

Type:: Cpus or None

creation_date: datetime

The date the Function was created.

The attribute is readonly and cannot be modified.

Type:: datetime or None

enabled: bool

Whether the Function accepts job submissions and reruns.

Type:: bool or None

environment: Dict[str, str]

The environment variables to set for Jobs run by the Function.

Type:: dict or None

id: str

The ID of the Function.

The attribute is readonly and cannot be modified.

Type:: str or None

image: str

The base image used to create the Function.

The attribute is immutable and cannot be modified once set.

Type:: str or None

property is_modified: bool: Determines if the document has been modified.

job_statistics: Dict

Statistics about the Job statuses for this Function. This attribute will only be available if includes=’job.statistics’ is specified in the request.

The attribute is readonly and cannot be modified.

Type:: dict or None

property jobs: JobSearch: Returns a JobSearch for all the Jobs for the Function.

maximum_concurrency: int

The maximum number of Jobs that execute at the same time for this Function.

Type:: int or None

memory: int

The amount of memory, in megabytes, to request when executing the Function.

Type:: Memory or None

modified_date: datetime

The date the Function was last modified or processed a job submission.

The attribute is readonly and cannot be modified.

Type:: datetime or None

name: str

The name of the Function.

Type:: str or None

namespace: str

The storage namespace for the Function.

The attribute is readonly and cannot be modified.

Type:: str or None

owner: str

The owner of the Function.

The attribute is readonly and cannot be modified.

Type:: str or None

retry_count: int

The total number of retries requested for a Job before it failed.

Type:: int or None

property state: DocumentState

Returns the state of the current document instance.

Return type:: DocumentState

status: FunctionStatus

The status of the Function.

The attribute is readonly and cannot be modified.

Type:: FunctionStatus or None

timeout: int

The number of seconds Jobs can run before timing out for this Function.

Type:: int or None

class FunctionStatus(value)[source]

Bases: StrEnum

The status of the Function.

Attributes:

`AWAITING_BUNDLE`
`BUILDING`
`BUILD_FAILED`
`READY`
`STOPPED`

AWAITING_BUNDLE = 'awaiting_bundle'

BUILDING = 'building'

BUILD_FAILED = 'build_failed'

READY = 'ready'

STOPPED = 'stopped'

class Job(function_id: str, args: List | None = None, kwargs: Dict | None = None, client: ComputeClient | None = None, environment: Dict[str, str] | None = None, **extra)[source]

Bases: Document

A single invocation of a Function.

Parameters:

function_id (str) – The id of the Function. A function must first be created to create a job.
args (List, optional) – A list of positional arguments to pass to the function.
kwargs (Dict, optional) – A dictionary of named arguments to pass to the function.
environment (Dict[str, str], optional) – Environment variables to be set in the environment of the running Job. Will be merged with environment variables set on the Function, with the Job environment variables taking precedence.
client (ComputeClient, optional) – The compute client to use for requests. If not set, the default client will be used.

Methods:

`cancel`()	Cancels the Job.
`delete`([delete_result])	Deletes the Job.
`get`(id[, client])	Retrieves the Job by id.
`iter_log`([timestamps])	Retrieves the log for the job, returning an iterator over the lines.
`list`([page_size, client])	Retrieves an iterable of all jobs matching the given parameters.
`log`([timestamps])	Retrieves the log for the job, returning a string.
`refresh`([client])	Update the Job instance with the latest information from the server.
`result`([cast_type, catalog_client])	Retrieves the result of the Job.
`result_blob`([catalog_client])	Retrieves the Catalog Blob holding the result of the Job.
`save`()	Creates the Job if it does not already exist.
`search`([client])	Creates a search for Jobs.
`to_dict`([only_modified, exclude_readonly, ...])	Converts the document to a dictionary.
`update`([ignore_missing])	Updates the document setting multiple attributes at a time.
`wait_for_completion`([timeout, interval])	Waits until the Job is completed.

Attributes:

`args`	The arguments provided to the Job.
`creation_date`	The date the Job was created.
`environment`	The environment variables provided to the Job.
`error_reason`	The reason the Job failed.
`execution_count`	The number of attempts made to execute this job.
`exit_code`	The exit code of the Job.
`function`	Returns the Function the Job belongs to.
`function_id`	The ID of the Function the Job belongs to.
`id`	The ID of the Job.
`is_modified`	Determines if the document has been modified.
`kwargs`	The parameters provided to the Job.
`last_completion_date`	The date the Job was last completed or canceled.
`last_execution_date`	The date the Job was last executed.
`provisioning_time`	("The time it took to provision the Job.
`pull_time`	("The time it took to load the user code in the Job.
`runtime`	The time it took the Job to complete.
`state`	Returns the state of the current document instance.
`statistics`	The runtime utilization statistics for the Job.
`status`	The current status of the Job.
`tags`	A list of tags associated with the Job.

cancel()[source]

Cancels the Job.

If the Job is already canceled or completed, this will do nothing.

If the Job is still pending, it will be canceled immediately.

If the job is running, it will be canceled as soon as possible. However, it may complete before the cancel request is processed.

delete(delete_result: bool = False)[source]

Deletes the Job.

Also deletes any job log blob for the job. Use delete_result=True to delete the job result blob as well.

Parameters:: delete_result (bool, False) – If set, the result of the job will also be deleted.

classmethod get(id, client: ComputeClient | None = None, **params) → Job[source]

Retrieves the Job by id.

Parameters:

id (str) – The id of the Job to fetch.
client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.
include (List[str], optional) –
List of additional attributes to include in the response. Allowed values are:
- ”timings”: Include additional debugging timing information about the Job.

Example

>>> from descarteslabs.compute import Job
>>> job = Job.get(<job-id>)
Job <job-id>: pending

iter_log(timestamps: bool = True)[source]

Retrieves the log for the job, returning an iterator over the lines.

Parameters:

timestamps (bool, True) –

If set, log timestamps will be included and converted to the users system timezone from UTC.

You may consider disabling this if you use a structured logger.

classmethod list(page_size: int = 100, client: ComputeClient | None = None, **params) → JobSearch[source]

Retrieves an iterable of all jobs matching the given parameters.

If you would like to filter Jobs, use Job.search().

Parameters:

page_size (int, default=100) – Maximum number of results per page.
client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.

Example

>>> from descarteslabs.compute import Job
>>> fn = Job.list(<function_id>)
[Job <job-id1>: pending, Job <job-id2>: pending, Job <job-id3>: pending]

log(timestamps: bool = True)[source]

Retrieves the log for the job, returning a string.

As logs can potentially be unbounded, consider using Job.iter_log().

Parameters:

timestamps (bool, True) –

If set, log timestamps will be included and converted to the users system timezone from UTC.

You may consider disabling this if you use a structured logger.

refresh(client: ComputeClient | None = None) → None[source]

Update the Job instance with the latest information from the server.

Parameters:: client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.

result(cast_type: Type[Serializable] | None = None, catalog_client: CatalogClient | None = None)[source]

Retrieves the result of the Job.

Parameters:

cast_type (Type[Serializable], None) – If set, the result will be deserialized to the given type.
catalog_client (CatalogClient, None) – If set, the result will be retrieved using the configured catalog client. Otherwise, the default catalog client will be used.

Raises:

ValueError – When job has not completed successfully or when cast_type does not implement Serializable.

result_blob(catalog_client: CatalogClient | None = None)[source]

Retrieves the Catalog Blob holding the result of the Job.

If there is no result Blob, None will be returned.

Parameters:: catalog_client (CatalogClient, None) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.
Raises:: ValueError – When job has not completed successfully or when cast_type does not implement Serializable.

save()[source]

Creates the Job if it does not already exist.

If the job already exists, it will be updated on the server if modifications were made to the Job instance.

classmethod search(client: ComputeClient | None = None) → JobSearch[source]

Creates a search for Jobs.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Parameters:: client (ComputeClient, optional) – If set, the result will be retrieved using the configured client. Otherwise, the default client will be used.

Example

>>> from descarteslabs.compute import Job, JobStatus
>>> jobs: List[Job] = Job.search().filter(Job.status == JobStatus.SUCCESS).collect()
Collection([Job <job-id1>: success, <job-id2>: success])

to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:

only_modified (bool, False) – If set, only modified attributes and their values will be included.
exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
exclude_none (bool, False) – If set, attributes with a value of None will be excluded.

Returns:

The attributes matching the call parameters. The result of this function can be json encoded without modification.

Return type:

Dict[str, Any]

update(ignore_missing=False, **kwargs)

Updates the document setting multiple attributes at a time.

Parameters:: ignore_missing (bool, False) – If set, unknown attributes will be ignored.

wait_for_completion(timeout=None, interval=10)[source]

Waits until the Job is completed.

Parameters:

timeout (int, default=None) – Maximum time to wait before timing out. If not set, the call will block until job completion.
interval (int, default=10) – Interval in seconds for how often to check if jobs have been completed.

args: List | None

The arguments provided to the Job.

Type:: list or None

creation_date: datetime

The date the Job was created.

The attribute is readonly and cannot be modified.

Type:: datetime or None

environment: Dict[str, str] | None

The environment variables provided to the Job.

Type:: dict or None

error_reason: str | None

The reason the Job failed.

The attribute is readonly and cannot be modified.

Type:: str or None

execution_count: int | None

The number of attempts made to execute this job.

The attribute is readonly and cannot be modified.

Type:: int or None

exit_code: int | None

The exit code of the Job.

The attribute is readonly and cannot be modified.

Type:: int or None

property function: Function: Returns the Function the Job belongs to.

function_id: str

The ID of the Function the Job belongs to.

The attribute is immutable and cannot be modified once set.

Type:: str or None

id: str

The ID of the Job.

The attribute is readonly and cannot be modified.

Type:: str or None

property is_modified: bool: Determines if the document has been modified.

kwargs: Dict[str, Any] | None

The parameters provided to the Job.

Type:: dict or None

last_completion_date: datetime | None

The date the Job was last completed or canceled.

The attribute is readonly and cannot be modified.

Type:: datetime or None

last_execution_date: datetime | None

The date the Job was last executed.

The attribute is readonly and cannot be modified.

Type:: datetime or None

provisioning_time: int | None

(“The time it took to provision the Job. This attribute will only be available if include=’timings’ is specified in the request by setting params.”,)

The attribute is readonly and cannot be modified.

Type:: int or None

pull_time: int | None

(“The time it took to load the user code in the Job. This attribute will only be available if include=’timings’ is specified in the request by setting params.”,)

The attribute is readonly and cannot be modified.

Type:: int or None

runtime: int | None

The time it took the Job to complete.

The attribute is readonly and cannot be modified.

Type:: int or None

property state: DocumentState

Returns the state of the current document instance.

Return type:: DocumentState

statistics: JobStatistics | None

The runtime utilization statistics for the Job.

The statistics include the cpu, memory, and network usage of the Job.

The attribute is readonly and cannot be modified.

Type:: JobStatistics or None

status: JobStatus

The current status of the Job.

The status may occasionally need to be refreshed by calling Job.refresh()

The attribute is readonly and cannot be modified.

Type:: JobStatus or None

tags: List[str]

A list of tags associated with the Job.

Type:: str or []

class JobSearch(document: T, client: ApiService, url: str = None, **params)[source]

Bases: Search[Job]

Methods:

`cancel`()
`collect`(**kwargs)	Execute the search query and return the appropriate collection.
`count`()	Fetch the number of documents that match the search.
`delete`([delete_results])
`filter`(expression)	Filter results by the values of various fields.
`limit`(limit)	Limit the number of search results returned by the search execution.
`param`(**params)	Add additional parameters to the search request.
`rerun`()
`sort`(*sorts)	Sort the returned results by the given fields.

cancel()[source]

collect(**kwargs) → Collection[T]

Execute the search query and return the appropriate collection.

Returns:

Collection of objects that match the type of document beng searched.

Return type:

Collection

Raises:

BadRequestError – If any of the query parameters or filters are invalid
ClientError or ServerError – Spurious exception that can occur during a network request.

count() → int

Fetch the number of documents that match the search.

Returns:

Number of matching records

Return type:

int

Raises:

BadRequestError – If any of the query parameters or filters are invalid
ClientError or ServerError – Spurious exception that can occur during a network request.

Example

>>> from descarteslabs.compute import Function
>>> search = Function.search().filter(Function.status == "building")
>>> count = search.count() 

delete(delete_results: bool = False)[source]

filter(expression: Expression | LogicalExpression) → AnySearch

Filter results by the values of various fields.

Successive calls to filter will add the new filter(s) using the and Boolean operator (&).

Parameters:: expression (Expression) – Expression used to filter objects in the search by their attributes, built from class attributes ex. Job.id == ‘some-id’. You can construct filter expressions using the ==, !=, <, >, <= and >= operators as well as the in_() or any_of() method. You cannot use the boolean keywords and and or because of Python language limitations; instead combine filter expressions using & (boolean “and”) and | (boolean “or”).
Returns:: A new Search instance with the new filter(s) applied (using and if there were existing filters)
Return type:: Search
Raises:: ValueError – If the filter expression provided is not supported.

Example

>>> from descarteslabs.compute import Job
>>> search = Job.search().filter(
...     (Job.runtime > 60) | (Job.status == "failure")
... )
>>> list(search) 

limit(limit: int) → AnySearch

Limit the number of search results returned by the search execution.

Successive calls to limit will overwrite the previous limit parameter.

Parameters:: limit (int) – The maximum number of records to return.
Return type:: Search

param(**params) → AnySearch

Add additional parameters to the search request.

Parameters:: params (dict) – The parameters to add to the search request.
Return type:: Search

rerun()[source]

sort(*sorts: List[Attribute | Sort]) → AnySearch

Sort the returned results by the given fields.

Parameters:: sorts (List[Union[Attribute, Sort]]) – The attributes and direction to sort by.
Return type:: Search

Example

>>> from descarteslabs.compute import Function
>>> Function.search().sort(Function.id, -Function.creation_date) 
>>> list(search) 

class JobStatus(value)[source]

Bases: StrEnum

The status of the Job.

Methods:

terminal()

Attributes:

`CANCEL`
`CANCELED`
`CANCELING`
`FAILURE`
`PENDING`
`RUNNING`
`SUCCESS`
`TIMEOUT`

classmethod terminal()[source]

CANCEL = 'cancel'

CANCELED = 'canceled'

CANCELING = 'canceling'

FAILURE = 'failure'

PENDING = 'pending'

RUNNING = 'running'

SUCCESS = 'success'

TIMEOUT = 'timeout'

class Search(document: T, client: ApiService, url: str = None, **params)[source]

Bases: Generic[T]

A search request that iterates over its search results.

The search can be narrowed by using the methods on the search object.

Example

>>> search = Search(Model).filter(Model.name == "test")
>>> list(search) 
>>> search.collect() 

Methods:

`collect`(**kwargs)	Execute the search query and return the appropriate collection.
`count`()	Fetch the number of documents that match the search.
`filter`(expression)	Filter results by the values of various fields.
`limit`(limit)	Limit the number of search results returned by the search execution.
`param`(**params)	Add additional parameters to the search request.
`sort`(*sorts)	Sort the returned results by the given fields.

collect(**kwargs) → Collection[T][source]

Execute the search query and return the appropriate collection.

Returns:

Collection of objects that match the type of document beng searched.

Return type:

Collection

Raises:

BadRequestError – If any of the query parameters or filters are invalid
ClientError or ServerError – Spurious exception that can occur during a network request.

count() → int[source]

Fetch the number of documents that match the search.

Returns:

Number of matching records

Return type:

int

Raises:

BadRequestError – If any of the query parameters or filters are invalid
ClientError or ServerError – Spurious exception that can occur during a network request.

Example

>>> from descarteslabs.compute import Function
>>> search = Function.search().filter(Function.status == "building")
>>> count = search.count() 

filter(expression: Expression | LogicalExpression) → AnySearch[source]

Filter results by the values of various fields.

Successive calls to filter will add the new filter(s) using the and Boolean operator (&).

Parameters:: expression (Expression) – Expression used to filter objects in the search by their attributes, built from class attributes ex. Job.id == ‘some-id’. You can construct filter expressions using the ==, !=, <, >, <= and >= operators as well as the in_() or any_of() method. You cannot use the boolean keywords and and or because of Python language limitations; instead combine filter expressions using & (boolean “and”) and | (boolean “or”).
Returns:: A new Search instance with the new filter(s) applied (using and if there were existing filters)
Return type:: Search
Raises:: ValueError – If the filter expression provided is not supported.

Example

>>> from descarteslabs.compute import Job
>>> search = Job.search().filter(
...     (Job.runtime > 60) | (Job.status == "failure")
... )
>>> list(search) 

limit(limit: int) → AnySearch[source]

Limit the number of search results returned by the search execution.

Successive calls to limit will overwrite the previous limit parameter.

Parameters:: limit (int) – The maximum number of records to return.
Return type:: Search

param(**params) → AnySearch[source]

Add additional parameters to the search request.

Parameters:: params (dict) – The parameters to add to the search request.
Return type:: Search

sort(*sorts: List[Attribute | Sort]) → AnySearch[source]

Sort the returned results by the given fields.

Parameters:: sorts (List[Union[Attribute, Sort]]) – The attributes and direction to sort by.
Return type:: Search

Example

>>> from descarteslabs.compute import Function
>>> Function.search().sort(Function.id, -Function.creation_date) 
>>> list(search) 

class Serializable[source]

Bases: object

Interface for serializing objects to bytes as a result of a Function invocation.

Methods:

`deserialize`(data)
`serialize`()

classmethod deserialize(data: bytes)[source]

serialize() → bytes[source]