Compute

The Batch Compute API can be found as Python package descarteslabs.compute. To install please refer to Installation.

Classes:

Function([function, requirements, …]) The serverless cloud function that you can call directly or submit many jobs to.
Job(function_id[, args, kwargs]) A single invocation of a Function.
JobStatus The status of the Job.
FunctionStatus The status of the Function.
class Function(function: Callable = None, requirements: List[str] = None, include_data: List[str] = None, include_modules: List[str] = None, name: str = None, image: str = None, cpus: descarteslabs.compute.compute.Cpus = None, memory: descarteslabs.compute.compute.Memory = None, maximum_concurrency: int = None, timeout: int = None, retry_count: int = None, **extra)[source]

Bases: descarteslabs.common.client.document.Document

The serverless cloud function that you can call directly or submit many jobs to.

Parameters:
  • function (Callable) – The function to be called in a Compute Job.
  • requirements (List[str], optional) – A list of Python dependencies required by this function.
  • include_data (List[str], optional) – Non-Python data files to include in the compute function.
  • name (str, optional) – Name of the function, will take name of function if not provided.
  • image (str) – The location of a docker image to be used for the environment where the function will be executed.
  • cpus (Cpus) – The number of CPUs requested for a single Job.
  • memory (Memory) – The maximum memory requirement for a single Job.
  • maximum_concurrency (str) – The maximum number of jobs to run in parallel.
  • timeout (int, optional) – Maximum runtime for a single job in seconds. Job will be killed if it exceeds this limit.
  • retry_count (int, optional) – Number of times to retry a job if it fails.

Examples

Retrieving an existing function and executing it.

>>> fn = Function.get(<function-id>)
>>> fn()
Job <job id>: "pending"

Creating a new function.

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn()
Job <job id>: "pending"

Methods:

build_log() Retrieves the build log for the Function.
delete() Deletes the Function and all associated Jobs.
get(id, **params) Get Function by id.
iter_results([cast_type]) Iterates over all successful job results.
list([page_size]) Lists all Functions for a user.
map(args[, iterargs]) Submits multiple jobs efficiently with positional args to each function call.
refresh() Updates the Function instance with data from the server.
rerun() Submits all the failed and timed out jobs to be rerun.
results([cast_type]) Retrieves all the job results for the Function as a list.
save() Creates the Function if it does not already exist.
search() Creates a search for Functions.
to_dict([only_modified, exclude_readonly, …]) Converts the document to a dictionary.
update([ignore_missing]) Updates the document setting multiple attributes at a time.
update_credentials() Updates the credentials for the Functions and Jobs run by this user.
wait_for_completion([timeout, interval]) Waits until all submitted jobs for a given Function are completed.

Attributes:

cpus The number of cpus to request when executing the Function.
creation_date The date the Function was created.
id The ID of the Function.
image The base image used to create the Function.
is_modified Determines if the document has been modified.
job_statistics Statistics about the Job statuses for this Function.
jobs Returns all the Jobs for the Function.
maximum_concurrency The maximum number of Jobs that execute at the same time for this Function.
memory The amount of memory, in megabytes, to request when executing the Function.
name The name of the Function.
retry_count The total number of retries requested for a Job before it failed.
state Returns the state of the current document instance.
status The status of the Function.
timeout The number of seconds Jobs can run before timing out for this Function.
build_log()[source]

Retrieves the build log for the Function.

delete()[source]

Deletes the Function and all associated Jobs.

classmethod get(id: str, **params)[source]

Get Function by id.

Parameters:
  • id (str) – Id of function to get.
  • include (List[str], optional) –

    List of additional attributes to include in the response. Allowed values are:

    • ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
<Function name="test_name" image=test_image cpus=1 memory=16 maximum_concurrency=5 timeout=3 retries=1
iter_results(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]

Iterates over all successful job results.

classmethod list(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]

Lists all Functions for a user.

If you would like to filter Functions, use Function.search().

Parameters:
  • page_size (int, default=100) – Maximum number of results per page.
  • include (List[str], optional) –

    List of additional attributes to include in the response. Allowed values are:

    • ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.list()
map(args, iterargs=None) → List[descarteslabs.compute.compute.Job][source]

Submits multiple jobs efficiently with positional args to each function call.

Preferred over repeatedly calling the function, such as in a loop, when submitting multiple jobs.

Parameters:
  • args (iterable) – An iterable of arguments. A job will be submitted with each element as the first positional argument to the function.
  • kwargs (List[iterable], optional) – If additional iterable arguments are passed, the function must take that many arguments and is applied to the items from all iterables in parallel.
refresh()[source]

Updates the Function instance with data from the server.

rerun()[source]

Submits all the failed and timed out jobs to be rerun.

results(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]

Retrieves all the job results for the Function as a list.

Notes

This immediately downloads all results into a list and could run out of memory. If the result set is large, strongly consider using Function.refresh() instead.

save()[source]

Creates the Function if it does not already exist.

If the Function already exists, it will be updated on the server if the Function instance was modified.

Examples

Create a Function without creating jobs:

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn.save()

Updating a Function:

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
>>> fn.memory = 4096  # 4 Gi
>>> fn.save()
classmethod search() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]

Creates a search for Functions.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Example

>>> from descarteslabs.compute import Function, FunctionStatus
>>> fns: List[Function] = (
...     Function.search()
...     .filter(Function.status.in_([
...         FunctionStatus.BUILDING, FunctionStatus.AWAITING_BUNDLE
...     ])
...     .collect()
... )
Collection([Function <fn-id1>: building, Function <fn-id2>: awaiting_bundle])
to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:
  • only_modified (bool, False) – If set, only modified attributes and their values will be included.
  • exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
  • exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns:

The attributes matching the call parameters. The result of this function can be json encoded without modification.

Return type:

Dict[str, Any]

update(ignore_missing=False, **kwargs)

Updates the document setting multiple attributes at a time.

Parameters:ignore_missing (bool, False) – If set, unknown attributes will be ignored.
classmethod update_credentials()[source]

Updates the credentials for the Functions and Jobs run by this user.

These credentials are used by other Descarteslabs services.

If the user invalidates existing credentials and needs to update them, you should call this method.

Notes

Credentials are automatically updated when a new Function is created.

wait_for_completion(timeout=None, interval=10)[source]

Waits until all submitted jobs for a given Function are completed.

Parameters:
  • timeout (int, default=None) – Maximum time to wait before timing out. If not set, this will hang until completion.
  • interval (int, default=10) – Interval for how often to check if jobs have been completed.
cpus

The number of cpus to request when executing the Function.

Type:Cpus or None
creation_date

The date the Function was created.

The attribute is readonly and cannot be modified.

Type:datetime or None
id

The ID of the Function.

The attribute is readonly and cannot be modified.

Type:str or None
image

The base image used to create the Function.

The attribute is immutable and cannot be modified once set.

Type:str or None
property is_modified

Determines if the document has been modified.

job_statistics

Statistics about the Job statuses for this Function. This attribute will only be available if includes=’job.statistics’ is specified in the request.

The attribute is readonly and cannot be modified.

Type:dict or None
property jobs

Returns all the Jobs for the Function.

maximum_concurrency

The maximum number of Jobs that execute at the same time for this Function.

Type:int or None
memory

The amount of memory, in megabytes, to request when executing the Function.

Type:Memory or None
name

The name of the Function.

Type:str or None
retry_count

The total number of retries requested for a Job before it failed.

Type:int or None
property state

Returns the state of the current document instance.

Returns:
Return type:DocumentState
status

The status of the Function.

The attribute is readonly and cannot be modified.

Type:FunctionStatus or None
timeout

The number of seconds Jobs can run before timing out for this Function.

Type:int or None
class Job(function_id: str, args: Optional[List] = None, kwargs: Optional[Dict] = None, **extra)[source]

Bases: descarteslabs.common.client.document.Document

A single invocation of a Function.

Parameters:
  • function_id (str) – The id of the Function. A function must first be created to create a job.
  • args (List, optional) – A list of positional arguments to pass to the function.
  • kwargs (Dict, optional) – A dictionary of named arguments to pass to the function.

Attributes:

args The arguments provided to the Job.
creation_date The date the Job was created.
function Returns the Function the Job belongs to.
function_id The ID of the Function the Job belongs to.
id The ID of the Job.
is_modified Determines if the document has been modified.
kwargs The parameters provided to the Job.
runtime The time it took the Job to complete.
state Returns the state of the current document instance.
status The current status of the Job.

Methods:

delete() Deletes the Job.
get(id) Retrieves the Job by id.
list([page_size]) Retrieves an iterable of all jobs matching the given parameters.
log([timestamps]) Retrieves the log for the job.
refresh() Update the Job instance with the latest information from the server.
result([cast_type]) Retrieves the result of the Job.
save() Creates the Job if it does not already exist.
search() Creates a search for Jobs.
to_dict([only_modified, exclude_readonly, …]) Converts the document to a dictionary.
update([ignore_missing]) Updates the document setting multiple attributes at a time.
wait_for_completion([timeout, interval]) Waits until the Job is completed.
delete()[source]

Deletes the Job.

classmethod get(id) → descarteslabs.compute.compute.Job[source]

Retrieves the Job by id.

Parameters:id (str) – The id of the Job to fetch.

Example

>>> from descarteslabs.compute import Job
>>> job = Job.get(<job-id>)
Job <job-id>: pending
classmethod list(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]

Retrieves an iterable of all jobs matching the given parameters.

If you would like to filter Jobs, use Job.search().

Parameters:page_size (int, default=100) – Maximum number of results per page.

Example

>>> from descarteslabs.compute import Job
>>> fn = Job.list(<function_id>)
[Job <job-id1>: pending, Job <job-id2>: pending, Job <job-id3>: pending]
log(timestamps: bool = True)[source]

Retrieves the log for the job.

Parameters:timestamps (bool, True) –

If set, log timestamps will be included and converted to the users system timezone from UTC.

You may consider disabling this if you use a structured logger.

refresh() → None[source]

Update the Job instance with the latest information from the server.

result(cast_type: Optional[Type[descarteslabs.compute.compute.Serializable]] = None)[source]

Retrieves the result of the Job.

Parameters:cast_type (Type[Serializable], None) – If set, the result will be deserialized to the given type.
Raises:ValueError – When cast_type does not implement Serializable.
save()[source]

Creates the Job if it does not already exist.

If the job already exists, it will be updated on the server if modifications were made to the Job instance.

classmethod search() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]

Creates a search for Jobs.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Example

>>> from descarteslabs.compute import Job, JobStatus
>>> jobs: List[Job] = Job.search().filter(Job.status == JobStatus.SUCCESS).collect()
Collection([Job <job-id1>: success, <job-id2>: success])
to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:
  • only_modified (bool, False) – If set, only modified attributes and their values will be included.
  • exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
  • exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns:

The attributes matching the call parameters. The result of this function can be json encoded without modification.

Return type:

Dict[str, Any]

update(ignore_missing=False, **kwargs)

Updates the document setting multiple attributes at a time.

Parameters:ignore_missing (bool, False) – If set, unknown attributes will be ignored.
wait_for_completion(timeout=None, interval=10)[source]

Waits until the Job is completed.

Parameters:
  • timeout (int, default=None) – Maximum time to wait before timing out. If not set, the call will block until job completion.
  • interval (int, default=10) – Interval for how often to check if jobs have been completed.
args

The arguments provided to the Job.

Type:list or None
creation_date

The date the Job was created.

The attribute is readonly and cannot be modified.

Type:datetime or None
property function

Returns the Function the Job belongs to.

function_id

The ID of the Function the Job belongs to.

The attribute is immutable and cannot be modified once set.

Type:str or None
id

The ID of the Job.

The attribute is readonly and cannot be modified.

Type:str or None
property is_modified

Determines if the document has been modified.

kwargs

The parameters provided to the Job.

Type:dict or None
runtime

The time it took the Job to complete.

The attribute is readonly and cannot be modified.

Type:int or None
property state

Returns the state of the current document instance.

Returns:
Return type:DocumentState
status

The current status of the Job.

The status may occasionally need to be refreshed by calling Job.refresh()

The attribute is readonly and cannot be modified.

Type:JobStatus or None
class JobStatus[source]

Bases: strenum.StrEnum

The status of the Job.

Attributes:

FAILURE
PENDING
RUNNING
SUCCESS
TIMEOUT
FAILURE = 'failure'
PENDING = 'pending'
RUNNING = 'running'
SUCCESS = 'success'
TIMEOUT = 'timeout'
class FunctionStatus[source]

Bases: strenum.StrEnum

The status of the Function.

Attributes:

AWAITING_BUNDLE
BUILDING
BUILD_FAILED
FAILURE
RUNNING
SUCCESS
AWAITING_BUNDLE = 'awaiting_bundle'
BUILDING = 'building'
BUILD_FAILED = 'build_failed'
FAILURE = 'failure'
RUNNING = 'running'
SUCCESS = 'success'