Compute¶
The Batch Compute API can be found as Python package descarteslabs.compute
.
To install please refer to Installation.
Classes:
Function ([function, requirements, …]) |
The serverless cloud function that you can call directly or submit many jobs to. |
Job (function_id[, args, kwargs]) |
A single invocation of a Function. |
JobStatus |
The status of the Job. |
FunctionStatus |
The status of the Function. |
-
class
Function
(function: Callable = None, requirements: List[str] = None, include_data: List[str] = None, include_modules: List[str] = None, name: str = None, image: str = None, cpus: descarteslabs.compute.compute.Cpus = None, memory: descarteslabs.compute.compute.Memory = None, maximum_concurrency: int = None, timeout: int = None, retry_count: int = None, **extra)[source]¶ Bases:
descarteslabs.common.client.document.Document
The serverless cloud function that you can call directly or submit many jobs to.
Parameters: - function (Callable) – The function to be called in a Compute Job.
- requirements (List[str], optional) – A list of Python dependencies required by this function.
- include_data (List[str], optional) – Non-Python data files to include in the compute function.
- name (str, optional) – Name of the function, will take name of function if not provided.
- image (str) – The location of a docker image to be used for the environment where the function will be executed.
- cpus (Cpus) – The number of CPUs requested for a single Job.
- memory (Memory) – The maximum memory requirement for a single Job.
- maximum_concurrency (str) – The maximum number of jobs to run in parallel.
- timeout (int, optional) – Maximum runtime for a single job in seconds. Job will be killed if it exceeds this limit.
- retry_count (int, optional) – Number of times to retry a job if it fails.
Examples
Retrieving an existing function and executing it.
>>> fn = Function.get(<function-id>) >>> fn() Job <job id>: "pending"
Creating a new function.
>>> from descarteslabs.compute import Function >>> def test_func(): ... print("Hello :)") >>> fn = Function( ... test_func, ... requirements=[], ... name="my_func", ... image="test_image", ... cpus=1, ... memory=16, ... maximum_concurrency=5, ... timeout=3600, ... retry_count=1, ... ) >>> fn() Job <job id>: "pending"
Methods:
build_log
()Retrieves the build log for the Function. delete
()Deletes the Function and all associated Jobs. get
(id, **params)Get Function by id. iter_results
([cast_type])Iterates over all successful job results. list
([page_size])Lists all Functions for a user. map
(args[, iterargs])Submits multiple jobs efficiently with positional args to each function call. refresh
()Updates the Function instance with data from the server. rerun
()Submits all the failed and timed out jobs to be rerun. results
([cast_type])Retrieves all the job results for the Function as a list. save
()Creates the Function if it does not already exist. search
()Creates a search for Functions. to_dict
([only_modified, exclude_readonly, …])Converts the document to a dictionary. update
([ignore_missing])Updates the document setting multiple attributes at a time. update_credentials
()Updates the credentials for the Functions and Jobs run by this user. wait_for_completion
([timeout, interval])Waits until all submitted jobs for a given Function are completed. Attributes:
cpus
The number of cpus to request when executing the Function. creation_date
The date the Function was created. id
The ID of the Function. image
The base image used to create the Function. is_modified
Determines if the document has been modified. job_statistics
Statistics about the Job statuses for this Function. jobs
Returns all the Jobs for the Function. maximum_concurrency
The maximum number of Jobs that execute at the same time for this Function. memory
The amount of memory, in megabytes, to request when executing the Function. name
The name of the Function. retry_count
The total number of retries requested for a Job before it failed. state
Returns the state of the current document instance. status
The status of the Function. timeout
The number of seconds Jobs can run before timing out for this Function. -
classmethod
get
(id: str, **params)[source]¶ Get Function by id.
Parameters: - id (str) – Id of function to get.
- include (List[str], optional) –
List of additional attributes to include in the response. Allowed values are:
- ”job.statistics”: Include statistics about the Job statuses for this Function.
Example
>>> from descarteslabs.compute import Function >>> fn = Function.get(<func_id>) <Function name="test_name" image=test_image cpus=1 memory=16 maximum_concurrency=5 timeout=3 retries=1
-
iter_results
(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]¶ Iterates over all successful job results.
-
classmethod
list
(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]¶ Lists all Functions for a user.
If you would like to filter Functions, use
Function.search()
.Parameters: - page_size (int, default=100) – Maximum number of results per page.
- include (List[str], optional) –
List of additional attributes to include in the response. Allowed values are:
- ”job.statistics”: Include statistics about the Job statuses for this Function.
Example
>>> from descarteslabs.compute import Function >>> fn = Function.list()
-
map
(args, iterargs=None) → List[descarteslabs.compute.compute.Job][source]¶ Submits multiple jobs efficiently with positional args to each function call.
Preferred over repeatedly calling the function, such as in a loop, when submitting multiple jobs.
Parameters: - args (iterable) – An iterable of arguments. A job will be submitted with each element as the first positional argument to the function.
- kwargs (List[iterable], optional) – If additional iterable arguments are passed, the function must take that many arguments and is applied to the items from all iterables in parallel.
-
results
(cast_type: Type[descarteslabs.compute.compute.Serializable] = None)[source]¶ Retrieves all the job results for the Function as a list.
Notes
This immediately downloads all results into a list and could run out of memory. If the result set is large, strongly consider using
Function.refresh()
instead.
-
save
()[source]¶ Creates the Function if it does not already exist.
If the Function already exists, it will be updated on the server if the Function instance was modified.
Examples
Create a Function without creating jobs:
>>> from descarteslabs.compute import Function >>> def test_func(): ... print("Hello :)") >>> fn = Function( ... test_func, ... requirements=[], ... name="my_func", ... image="test_image", ... cpus=1, ... memory=16, ... maximum_concurrency=5, ... timeout=3600, ... retry_count=1, ... ) >>> fn.save()
Updating a Function:
>>> from descarteslabs.compute import Function >>> fn = Function.get(<func_id>) >>> fn.memory = 4096 # 4 Gi >>> fn.save()
-
classmethod
search
() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Function][descarteslabs.compute.compute.Function][source]¶ Creates a search for Functions.
The search is lazy and will be executed when the search is iterated over or
Search.collect()
is called.Example
>>> from descarteslabs.compute import Function, FunctionStatus >>> fns: List[Function] = ( ... Function.search() ... .filter(Function.status.in_([ ... FunctionStatus.BUILDING, FunctionStatus.AWAITING_BUNDLE ... ]) ... .collect() ... ) Collection([Function <fn-id1>: building, Function <fn-id2>: awaiting_bundle])
-
to_dict
(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]¶ Converts the document to a dictionary.
Attributes will be serialized to JSON encodable types.
Parameters: - only_modified (bool, False) – If set, only modified attributes and their values will be included.
- exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
- exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns: The attributes matching the call parameters. The result of this function can be json encoded without modification.
Return type: Dict[str, Any]
-
update
(ignore_missing=False, **kwargs)¶ Updates the document setting multiple attributes at a time.
Parameters: ignore_missing (bool, False) – If set, unknown attributes will be ignored.
-
classmethod
update_credentials
()[source]¶ Updates the credentials for the Functions and Jobs run by this user.
These credentials are used by other Descarteslabs services.
If the user invalidates existing credentials and needs to update them, you should call this method.
Notes
Credentials are automatically updated when a new Function is created.
-
wait_for_completion
(timeout=None, interval=10)[source]¶ Waits until all submitted jobs for a given Function are completed.
Parameters: - timeout (int, default=None) – Maximum time to wait before timing out. If not set, this will hang until completion.
- interval (int, default=10) – Interval for how often to check if jobs have been completed.
-
cpus
¶ The number of cpus to request when executing the Function.
Type: Cpus or None
-
creation_date
¶ The date the Function was created.
The attribute is readonly and cannot be modified.
Type: datetime or None
-
id
¶ The ID of the Function.
The attribute is readonly and cannot be modified.
Type: str or None
-
image
¶ The base image used to create the Function.
The attribute is immutable and cannot be modified once set.
Type: str or None
-
property
is_modified
¶ Determines if the document has been modified.
-
job_statistics
¶ Statistics about the Job statuses for this Function. This attribute will only be available if includes=’job.statistics’ is specified in the request.
The attribute is readonly and cannot be modified.
Type: dict or None
-
property
jobs
¶ Returns all the Jobs for the Function.
-
maximum_concurrency
¶ The maximum number of Jobs that execute at the same time for this Function.
Type: int or None
-
memory
¶ The amount of memory, in megabytes, to request when executing the Function.
Type: Memory or None
-
name
¶ The name of the Function.
Type: str or None
-
retry_count
¶ The total number of retries requested for a Job before it failed.
Type: int or None
-
property
state
¶ Returns the state of the current document instance.
Returns: Return type: DocumentState
-
status
¶ The status of the Function.
The attribute is readonly and cannot be modified.
Type: FunctionStatus or None
-
timeout
¶ The number of seconds Jobs can run before timing out for this Function.
Type: int or None
-
class
Job
(function_id: str, args: Optional[List] = None, kwargs: Optional[Dict] = None, **extra)[source]¶ Bases:
descarteslabs.common.client.document.Document
A single invocation of a Function.
Parameters: - function_id (str) – The id of the Function. A function must first be created to create a job.
- args (List, optional) – A list of positional arguments to pass to the function.
- kwargs (Dict, optional) – A dictionary of named arguments to pass to the function.
Attributes:
args
The arguments provided to the Job. creation_date
The date the Job was created. function
Returns the Function the Job belongs to. function_id
The ID of the Function the Job belongs to. id
The ID of the Job. is_modified
Determines if the document has been modified. kwargs
The parameters provided to the Job. runtime
The time it took the Job to complete. state
Returns the state of the current document instance. status
The current status of the Job. Methods:
delete
()Deletes the Job. get
(id)Retrieves the Job by id. list
([page_size])Retrieves an iterable of all jobs matching the given parameters. log
([timestamps])Retrieves the log for the job. refresh
()Update the Job instance with the latest information from the server. result
([cast_type])Retrieves the result of the Job. save
()Creates the Job if it does not already exist. search
()Creates a search for Jobs. to_dict
([only_modified, exclude_readonly, …])Converts the document to a dictionary. update
([ignore_missing])Updates the document setting multiple attributes at a time. wait_for_completion
([timeout, interval])Waits until the Job is completed. -
classmethod
get
(id) → descarteslabs.compute.compute.Job[source]¶ Retrieves the Job by id.
Parameters: id (str) – The id of the Job to fetch. Example
>>> from descarteslabs.compute import Job >>> job = Job.get(<job-id>) Job <job-id>: pending
-
classmethod
list
(page_size: int = 100, **params) → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]¶ Retrieves an iterable of all jobs matching the given parameters.
If you would like to filter Jobs, use
Job.search()
.Parameters: page_size (int, default=100) – Maximum number of results per page. Example
>>> from descarteslabs.compute import Job >>> fn = Job.list(<function_id>) [Job <job-id1>: pending, Job <job-id2>: pending, Job <job-id3>: pending]
-
log
(timestamps: bool = True)[source]¶ Retrieves the log for the job.
Parameters: timestamps (bool, True) – If set, log timestamps will be included and converted to the users system timezone from UTC.
You may consider disabling this if you use a structured logger.
-
result
(cast_type: Optional[Type[descarteslabs.compute.compute.Serializable]] = None)[source]¶ Retrieves the result of the Job.
Parameters: cast_type (Type[Serializable], None) – If set, the result will be deserialized to the given type. Raises: ValueError – When cast_type does not implement Serializable.
-
save
()[source]¶ Creates the Job if it does not already exist.
If the job already exists, it will be updated on the server if modifications were made to the Job instance.
-
classmethod
search
() → descarteslabs.common.client.search.Search[descarteslabs.compute.compute.Job][descarteslabs.compute.compute.Job][source]¶ Creates a search for Jobs.
The search is lazy and will be executed when the search is iterated over or
Search.collect()
is called.Example
>>> from descarteslabs.compute import Job, JobStatus >>> jobs: List[Job] = Job.search().filter(Job.status == JobStatus.SUCCESS).collect() Collection([Job <job-id1>: success, <job-id2>: success])
-
to_dict
(only_modified: bool = False, exclude_readonly: bool = False, exclude_none: bool = False) → Dict[str, Any]¶ Converts the document to a dictionary.
Attributes will be serialized to JSON encodable types.
Parameters: - only_modified (bool, False) – If set, only modified attributes and their values will be included.
- exclude_readonly (bool, False) – If set, readonly attributes and their values are excluded.
- exclude_none (bool, False) – If set, attributes with a value of None will be excluded.
Returns: The attributes matching the call parameters. The result of this function can be json encoded without modification.
Return type: Dict[str, Any]
-
update
(ignore_missing=False, **kwargs)¶ Updates the document setting multiple attributes at a time.
Parameters: ignore_missing (bool, False) – If set, unknown attributes will be ignored.
-
wait_for_completion
(timeout=None, interval=10)[source]¶ Waits until the Job is completed.
Parameters: - timeout (int, default=None) – Maximum time to wait before timing out. If not set, the call will block until job completion.
- interval (int, default=10) – Interval for how often to check if jobs have been completed.
-
args
¶ The arguments provided to the Job.
Type: list or None
-
creation_date
¶ The date the Job was created.
The attribute is readonly and cannot be modified.
Type: datetime or None
-
property
function
¶ Returns the Function the Job belongs to.
-
function_id
¶ The ID of the Function the Job belongs to.
The attribute is immutable and cannot be modified once set.
Type: str or None
-
id
¶ The ID of the Job.
The attribute is readonly and cannot be modified.
Type: str or None
-
property
is_modified
¶ Determines if the document has been modified.
-
kwargs
¶ The parameters provided to the Job.
Type: dict or None
-
runtime
¶ The time it took the Job to complete.
The attribute is readonly and cannot be modified.
Type: int or None
-
property
state
¶ Returns the state of the current document instance.
Returns: Return type: DocumentState
-
status
¶ The current status of the Job.
The status may occasionally need to be refreshed by callingJob.refresh()
The attribute is readonly and cannot be modified.
Type: JobStatus or None
-
class
JobStatus
[source]¶ Bases:
strenum.StrEnum
The status of the Job.
Attributes:
FAILURE
PENDING
RUNNING
SUCCESS
TIMEOUT
-
FAILURE
= 'failure'¶
-
PENDING
= 'pending'¶
-
RUNNING
= 'running'¶
-
SUCCESS
= 'success'¶
-
TIMEOUT
= 'timeout'¶
-
-
class
FunctionStatus
[source]¶ Bases:
strenum.StrEnum
The status of the Function.
Attributes:
AWAITING_BUNDLE
BUILDING
BUILD_FAILED
FAILURE
RUNNING
SUCCESS
-
AWAITING_BUNDLE
= 'awaiting_bundle'¶
-
BUILDING
= 'building'¶
-
BUILD_FAILED
= 'build_failed'¶
-
FAILURE
= 'failure'¶
-
RUNNING
= 'running'¶
-
SUCCESS
= 'success'¶
-