Compute¶

The Batch Compute API can be found as Python package descarteslabs.compute. To install please refer to Installation.

Classes:

`Function`([function, requirements, …])	The serverless cloud function that you can call directly or submit many jobs to.
`Job`(function_id[, args, kwargs])	A single invocation of a Function.
`JobStatus`	The status of the Job.
`FunctionStatus`	The status of the Function.

class Function(function: Callable = None, requirements: List[str] = None, include_data: List[str] = None, include_modules: List[str] = None, name: str = None, image: str = None, cpus: descarteslabs.compute.compute.Cpus = None, memory: descarteslabs.compute.compute.Memory = None, maximum_concurrency: int = None, timeout: int = None, retry_count: int = None, **extra)[source]¶

Bases: descarteslabs.common.client.document.Document

The serverless cloud function that you can call directly or submit many jobs to.

Parameters:

function (Callable) – The function to be called in a Compute Job.
requirements (List[str], optional) – A list of Python dependencies required by this function.
include_data (List[str], optional) – Non-Python data files to include in the compute function.
name (str, optional) – Name of the function, will take name of function if not provided.
image (str) – The location of a docker image to be used for the environment where the function will be executed.
cpus (Cpus) – The number of CPUs requested for a single Job.
memory (Memory) – The maximum memory requirement for a single Job.
maximum_concurrency (str) – The maximum number of jobs to run in parallel.
timeout (int, optional) – Maximum runtime for a single job in seconds. Job will be killed if it exceeds this limit.
retry_count (int, optional) – Number of times to retry a job if it fails.

Examples

Retrieving an existing function and executing it.

>>> fn = Function.get(<function-id>)
>>> fn()
Job <job id>: "pending"

Creating a new function.

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn()
Job <job id>: "pending"

Methods:

`build_log`()	Retrieves the build log for the Function.
`delete`()	Deletes the Function and all associated Jobs.
`get`(id, **params)	Get Function by id.
`iter_results`([cast_type])	Iterates over all successful job results.
`list`([page_size])	Lists all Functions for a user.
`map`(args[, iterargs])	Submits multiple jobs efficiently with positional args to each function call.
`refresh`()	Updates the Function instance with data from the server.
`rerun`()	Submits all the failed and timed out jobs to be rerun.
`results`([cast_type])	Retrieves all the job results for the Function as a list.
`save`()	Creates the Function if it does not already exist.
`search`()	Creates a search for Functions.
`to_dict`([only_modified, exclude_readonly, …])	Converts the document to a dictionary.
`update`([ignore_missing])	Updates the document setting multiple attributes at a time.
`update_credentials`()	Updates the credentials for the Functions and Jobs run by this user.
`wait_for_completion`([timeout, interval])	Waits until all submitted jobs for a given Function are completed.

Attributes:

`cpus`	The number of cpus to request when executing the Function.
`creation_date`	The date the Function was created.
`id`	The ID of the Function.
`image`	The base image used to create the Function.
`is_modified`	Determines if the document has been modified.
`job_statistics`	Statistics about the Job statuses for this Function.
`jobs`	Returns all the Jobs for the Function.
`maximum_concurrency`	The maximum number of Jobs that execute at the same time for this Function.
`memory`	The amount of memory, in megabytes, to request when executing the Function.
`name`	The name of the Function.
`retry_count`	The total number of retries requested for a Job before it failed.
`state`	Returns the state of the current document instance.
`status`	The status of the Function.
`timeout`	The number of seconds Jobs can run before timing out for this Function.

build_log()[source]¶: Retrieves the build log for the Function.

delete()[source]¶: Deletes the Function and all associated Jobs.

classmethod get(id: str, **params)[source]¶

Get Function by id.

Parameters:	id (str) – Id of function to get. include (List[str], optional) – List of additional attributes to include in the response. Allowed values are: ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
<Function name="test_name" image=test_image cpus=1 memory=16 maximum_concurrency=5 timeout=3 retries=1

iter_results(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]¶: Iterates over all successful job results.

classmethod list(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]¶

Lists all Functions for a user.

If you would like to filter Functions, use Function.search().

Parameters:	page_size (int, default=100) – Maximum number of results per page. include (List[str], optional) – List of additional attributes to include in the response. Allowed values are: ”job.statistics”: Include statistics about the Job statuses for this Function.

Example

>>> from descarteslabs.compute import Function
>>> fn = Function.list()

map(args, iterargs=None) → List[descarteslabs.compute.compute.Job][source]¶

Submits multiple jobs efficiently with positional args to each function call.

Preferred over repeatedly calling the function, such as in a loop, when submitting multiple jobs.

Parameters:	args (iterable) – An iterable of arguments. A job will be submitted with each element as the first positional argument to the function. kwargs (List[iterable], optional) – If additional iterable arguments are passed, the function must take that many arguments and is applied to the items from all iterables in parallel.

refresh()[source]¶: Updates the Function instance with data from the server.

rerun()[source]¶: Submits all the failed and timed out jobs to be rerun.

results(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]¶

Retrieves all the job results for the Function as a list.

Notes

This immediately downloads all results into a list and could run out of memory. If the result set is large, strongly consider using Function.refresh() instead.

save()[source]¶

Creates the Function if it does not already exist.

If the Function already exists, it will be updated on the server if the Function instance was modified.

Examples

Create a Function without creating jobs:

>>> from descarteslabs.compute import Function
>>> def test_func():
...     print("Hello :)")
>>> fn = Function(
...     test_func,
...     requirements=[],
...     name="my_func",
...     image="test_image",
...     cpus=1,
...     memory=16,
...     maximum_concurrency=5,
...     timeout=3600,
...     retry_count=1,
... )
>>> fn.save()

Updating a Function:

>>> from descarteslabs.compute import Function
>>> fn = Function.get(<func_id>)
>>> fn.memory = 4096  # 4 Gi
>>> fn.save()

classmethod search() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]¶

Creates a search for Functions.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Example

>>> from descarteslabs.compute import Function, FunctionStatus
>>> fns: List[Function] = (
...     Function.search()
...     .filter(Function.status.in_([
...         FunctionStatus.BUILDING, FunctionStatus.AWAITING_BUNDLE
...     ])
...     .collect()
... )
Collection([Function <fn-id1>: building, Function <fn-id2>: awaiting_bundle])

to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]¶

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:	only_modified (bool, False) – If set, only modified attributes and their values will be included. exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded. exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns:	The attributes matching the call parameters. The result of this function can be json encoded without modification.
Return type:	Dict[str, Any]

update(ignore_missing=False, **kwargs)¶

Updates the document setting multiple attributes at a time.

Parameters:	ignore_missing (bool, False) – If set, unknown attributes will be ignored.

classmethod update_credentials()[source]¶

Updates the credentials for the Functions and Jobs run by this user.

These credentials are used by other Descarteslabs services.

If the user invalidates existing credentials and needs to update them, you should call this method.

Notes

Credentials are automatically updated when a new Function is created.

wait_for_completion(timeout=None, interval=10)[source]¶

Waits until all submitted jobs for a given Function are completed.

Parameters:	timeout (int, default=None) – Maximum time to wait before timing out. If not set, this will hang until completion. interval (int, default=10) – Interval for how often to check if jobs have been completed.

cpus¶

The number of cpus to request when executing the Function.

Type:	Cpus or None

creation_date¶

The date the Function was created.

The attribute is readonly and cannot be modified.

Type:	datetime or None

id¶

The ID of the Function.

The attribute is readonly and cannot be modified.

Type:	str or None

image¶

The base image used to create the Function.

The attribute is immutable and cannot be modified once set.

Type:	str or None

property is_modified¶: Determines if the document has been modified.

job_statistics¶

Statistics about the Job statuses for this Function. This attribute will only be available if includes=’job.statistics’ is specified in the request.

The attribute is readonly and cannot be modified.

Type:	dict or None

property jobs¶: Returns all the Jobs for the Function.

maximum_concurrency¶

The maximum number of Jobs that execute at the same time for this Function.

Type:	int or None

memory¶

The amount of memory, in megabytes, to request when executing the Function.

Type:	Memory or None

name¶

The name of the Function.

Type:	str or None

retry_count¶

The total number of retries requested for a Job before it failed.

Type:	int or None

property state¶

Returns the state of the current document instance.

Returns:
Return type:	`DocumentState`

status¶

The status of the Function.

The attribute is readonly and cannot be modified.

Type:	FunctionStatus or None

timeout¶

The number of seconds Jobs can run before timing out for this Function.

Type:	int or None

class Job(function_id: str, args: Optional[List] = None, kwargs: Optional[Dict] = None, **extra)[source]¶

Bases: descarteslabs.common.client.document.Document

A single invocation of a Function.

Parameters:	function_id (str) – The id of the Function. A function must first be created to create a job. args (List, optional) – A list of positional arguments to pass to the function. kwargs (Dict, optional) – A dictionary of named arguments to pass to the function.

Attributes:

`args`	The arguments provided to the Job.
`creation_date`	The date the Job was created.
`function`	Returns the Function the Job belongs to.
`function_id`	The ID of the Function the Job belongs to.
`id`	The ID of the Job.
`is_modified`	Determines if the document has been modified.
`kwargs`	The parameters provided to the Job.
`runtime`	The time it took the Job to complete.
`state`	Returns the state of the current document instance.
`status`	The current status of the Job.

Methods:

`delete`()	Deletes the Job.
`get`(id)	Retrieves the Job by id.
`list`([page_size])	Retrieves an iterable of all jobs matching the given parameters.
`log`([timestamps])	Retrieves the log for the job.
`refresh`()	Update the Job instance with the latest information from the server.
`result`([cast_type])	Retrieves the result of the Job.
`save`()	Creates the Job if it does not already exist.
`search`()	Creates a search for Jobs.
`to_dict`([only_modified, exclude_readonly, …])	Converts the document to a dictionary.
`update`([ignore_missing])	Updates the document setting multiple attributes at a time.
`wait_for_completion`([timeout, interval])	Waits until the Job is completed.

delete()[source]¶: Deletes the Job.

classmethod get(id) → descarteslabs.compute.compute.Job[source]¶

Retrieves the Job by id.

Parameters:	id (str) – The id of the Job to fetch.

Example

>>> from descarteslabs.compute import Job
>>> job = Job.get(<job-id>)
Job <job-id>: pending

classmethod list(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]¶

Retrieves an iterable of all jobs matching the given parameters.

If you would like to filter Jobs, use Job.search().

Parameters:	page_size (int, default=100) – Maximum number of results per page.

Example

>>> from descarteslabs.compute import Job
>>> fn = Job.list(<function_id>)
[Job <job-id1>: pending, Job <job-id2>: pending, Job <job-id3>: pending]

log(timestamps: bool = True)[source]¶

Retrieves the log for the job.

Parameters:

timestamps (bool, True) –

If set, log timestamps will be included and converted to the users system timezone from UTC.

You may consider disabling this if you use a structured logger.

refresh() → None[source]¶: Update the Job instance with the latest information from the server.

result(cast_type: Optional[Type[descarteslabs.compute.compute.Serializable]] = None)[source]¶

Retrieves the result of the Job.

Parameters:	cast_type (Type[Serializable], None) – If set, the result will be deserialized to the given type.
Raises:	ValueError – When cast_type does not implement Serializable.

save()[source]¶

Creates the Job if it does not already exist.

If the job already exists, it will be updated on the server if modifications were made to the Job instance.

classmethod search() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]¶

Creates a search for Jobs.

The search is lazy and will be executed when the search is iterated over or Search.collect() is called.

Example

>>> from descarteslabs.compute import Job, JobStatus
>>> jobs: List[Job] = Job.search().filter(Job.status == JobStatus.SUCCESS).collect()
Collection([Job <job-id1>: success, <job-id2>: success])

to_dict(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]¶

Converts the document to a dictionary.

Attributes will be serialized to JSON encodable types.

Parameters:	only_modified (bool, False) – If set, only modified attributes and their values will be included. exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded. exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns:	The attributes matching the call parameters. The result of this function can be json encoded without modification.
Return type:	Dict[str, Any]

update(ignore_missing=False, **kwargs)¶

Updates the document setting multiple attributes at a time.

Parameters:	ignore_missing (bool, False) – If set, unknown attributes will be ignored.

wait_for_completion(timeout=None, interval=10)[source]¶

Waits until the Job is completed.

Parameters:	timeout (int, default=None) – Maximum time to wait before timing out. If not set, the call will block until job completion. interval (int, default=10) – Interval for how often to check if jobs have been completed.

args¶

The arguments provided to the Job.

Type:	list or None

creation_date¶

The date the Job was created.

The attribute is readonly and cannot be modified.

Type:	datetime or None

property function¶: Returns the Function the Job belongs to.

function_id¶

The ID of the Function the Job belongs to.

The attribute is immutable and cannot be modified once set.

Type:	str or None

id¶

The ID of the Job.

The attribute is readonly and cannot be modified.

Type:	str or None

property is_modified¶: Determines if the document has been modified.

kwargs¶

The parameters provided to the Job.

Type:	dict or None

runtime¶

The time it took the Job to complete.

The attribute is readonly and cannot be modified.

Type:	int or None

property state¶

Returns the state of the current document instance.

Returns:
Return type:	`DocumentState`

status¶

The current status of the Job.

The status may occasionally need to be refreshed by calling Job.refresh()

The attribute is readonly and cannot be modified.

Type:	JobStatus or None

class JobStatus[source]¶

Bases: strenum.StrEnum

The status of the Job.

Attributes:

`FAILURE`
`PENDING`
`RUNNING`
`SUCCESS`
`TIMEOUT`

FAILURE = 'failure'¶

PENDING = 'pending'¶

RUNNING = 'running'¶

SUCCESS = 'success'¶

TIMEOUT = 'timeout'¶

class FunctionStatus[source]¶

Bases: strenum.StrEnum

The status of the Function.

Attributes:

`AWAITING_BUNDLE`
`BUILDING`
`BUILD_FAILED`
`FAILURE`
`RUNNING`
`SUCCESS`

AWAITING_BUNDLE = 'awaiting_bundle'¶

BUILDING = 'building'¶

BUILD_FAILED = 'build_failed'¶

FAILURE = 'failure'¶

RUNNING = 'running'¶

SUCCESS = 'success'¶