Workers

There is a tight coupling between Temporal Task Queues and Worker Processes.

Worker

In day-to-day conversations, the term Worker is used to denote either a Worker ProgramWhat is a Worker Program?

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

Learn more, a Worker ProcessWhat is a Worker Process?

A Worker Process is responsible for polling a Task Queue, dequeueing a Task, executing your code in response to a Task, and responding to the Temporal Server with the results.

Learn more, or a Worker EntityWhat is a Worker Entity?

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Learn more. Temporal documentation aims to be explicit and differentiate between them.

Task

A Task is the context that a Worker needs to progress with a specific Workflow Execution or Activity ExecutionWhat is an Activity Execution?

An Activity Execution is the full chain of Activity Task Executions.

Learn more.

There are two types of Tasks:

Activity TaskWhat is an Activity Task?

An Activity Task contains the context needed to make an Activity Task Execution.

Learn more
Workflow TaskWhat is a Workflow Task?

A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution.

Learn more

Worker Program

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

Worker Entity

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

A Worker Entity listens and polls on a single Task Queue. A Worker Entity contains both a Workflow Worker and an Activity Worker so that it may make progress of either a Workflow Execution or an Activity Execution.

Can a Worker handle more Workflow Executions than its cache size or number of supported threads?

Yes it can. However, the trade off is added latency.

Workers are stateless, so any Workflow Execution in a blocked state can be safely removed from a Worker. Later on, it can be resurrected on the same or different Worker when the need arises (in the form of an external event). Therefore, a single Worker can handle millions of open Workflow Executions, assuming it can handle the update rate and that a slightly higher latency is not a concern.

Operation guides:

How to tune Workers

Worker Process

Component diagram of a Worker Process and the Temporal Server

A Worker Process is responsible for polling a Task QueueWhat is a Task Queue?

A Task Queue is a first-in, first-out queue that a Worker Process polls for Tasks.

Learn more, dequeueing a TaskWhat is a Task?

A Task is the context needed to make progress with a specific Workflow Execution or Activity Execution.

Learn more, executing your code in response to a Task, and responding to the Temporal ClusterWhat is a Temporal Cluster?

A Temporal Cluster is the Temporal Server paired with persistence.

Learn more with the results.

More formally, a Worker Process is any process that implements the Task Queue Protocol and the Task Execution Protocol.

A Worker Process is a Workflow Worker Process if the process implements the Workflow Task Queue Protocol and executes the Workflow Task Execution Protocol to make progress on a Workflow Execution. A Workflow Worker Process can listen on an arbitrary number of Workflow Task Queues and can execute an arbitrary number of Workflow Tasks.
A Worker Process is an Activity Worker Process if the process implements the Activity Task Queue Protocol and executes the Activity Task Processing Protocol to make progress on an Activity Execution. An Activity Worker Process can listen on an arbitrary number of Activity Task Queues and can execute an arbitrary number of Activity Tasks.

Worker Processes are external to a Temporal Cluster. Temporal Application developers are responsible for developing Worker ProgramsWhat is a Worker Program?

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

Learn more and operating Worker Processes. Said another way, the Temporal ClusterWhat is a Temporal Cluster?

A Temporal Cluster is the Temporal Server paired with persistence.

Learn more (including the Temporal Cloud) doesn't execute any of your code (Workflow and Activity Definitions) on Temporal Cluster machines. The Cluster is solely responsible for orchestrating State TransitionsWhat is a State Transition?

A State Transition is a unit of progress by a Workflow Execution.

Learn more and providing Tasks to the next available Worker EntityWhat is a Worker Entity?

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Learn more.

While data transferred in Event Histories is secured by mTLS, by default, it is still readable at rest in the Temporal Cluster.

To solve this, Temporal SDKs offer a Data Converter APIWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more that you can use to customize the serialization of data going out of and coming back in to a Worker Entity, with the net effect of guaranteeing that the Temporal Cluster cannot read sensitive business data.

In many of our tutorials, we show you how to run both a Temporal Cluster and one Worker on the same machine for local development. However, a production-grade Temporal Application typically has a fleet of Worker Processes, all running on hosts external to the Temporal Cluster. A Temporal Application can have as many Worker Processes as needed.

A Worker Process can be both a Workflow Worker Process and an Activity Worker Process. Many SDKs support the ability to have multiple Worker Entities in a single Worker Process. (Worker Entity creation and management differ between SDKs.) A single Worker Entity can listen to only a single Task Queue. But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues.

Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks

Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions that are defined in Activity Definitions, such as the following:

Network access for external API calls.
Credentials for infrastructure provisioning.
Specialized GPUs for machine learning utilities.

The Temporal Cluster itself has internal workers for system Workflow Executions. However, these internal workers are not visible to the developer.

Workflow Task

A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution.

Every time a new external event that might affect a Workflow state is recorded, a Workflow Task that contains the event is added to a Task Queue and then picked up by a Workflow Worker.
After the new event is handled, the Workflow Task is completed with a list of CommandsWhat is a Command?

A Command is a requested action issued by a Worker to the Temporal Cluster after a Workflow Task Execution completes.

Learn more.
Handling of a Workflow Task is usually very fast and is not related to the duration of operations that the Workflow invokes.

Workflow Task Execution

A Workflow Task Execution occurs when a WorkerWhat is a Worker Entity?

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Learn more picks up a Workflow TaskWhat is a Workflow Task?

A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution.

Learn more and uses it to make progress on the execution of a Workflow DefinitionWhat is a Workflow Definition?

A Workflow Definition is the code that defines the constraints of a Workflow Execution.

Learn more (also known as a Workflow function).

Activity Task

An Activity Task contains the context needed to proceed with an Activity Task ExecutionWhat is an Activity Task Execution?

An Activity Task Execution occurs when a Worker uses the context provided from the Activity Task and executes the Activity Definition.

Learn more. Activity Tasks largely represent the Activity Task Scheduled Event, which contains the data needed to execute an Activity Function.

If Heartbeat data is being passed, an Activity Task will also contain the latest Heartbeat details.

Activity Task Execution

An Activity Task Execution occurs when a WorkerWhat is a Worker Entity?

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Learn more uses the context provided from the Activity TaskWhat is an Activity Task?

An Activity Task contains the context needed to make an Activity Task Execution.

Learn more and executes the Activity DefinitionWhat is an Activity Definition?

An Activity Definition is the code that defines the constraints of an Activity Task Execution.

Learn more (also known as the Activity Function).

The ActivityTaskScheduled EventEvents reference

Events are created by the Temporal Cluster in response to external occurrences and Commands generated by a Workflow Execution.

Learn more corresponds to when the Temporal Cluster puts the Activity Task into the Task Queue.

The ActivityTaskStarted EventEvents reference

Events are created by the Temporal Cluster in response to external occurrences and Commands generated by a Workflow Execution.

Learn more corresponds to when the Worker picks up the Activity Task from the Task Queue.

Either ActivityTaskCompletedEvents reference

Events are created by the Temporal Cluster in response to external occurrences and Commands generated by a Workflow Execution.

Learn more or one of the other Closed Activity Task Events corresponds to when the Worker has yielded back to the Temporal Cluster.

The API to schedule an Activity Execution provides an "effectively once" experience, even though there may be several Activity Task Executions that take place to successfully complete an Activity.

Once an Activity Task finishes execution, the Worker responds to the Cluster with a specific Event:

ActivityTaskCanceled
ActivityTaskCompleted
ActivityTaskFailed
ActivityTaskTerminated
ActivityTaskTimedOut

Task Queue

A Task Queue is a lightweight, dynamically allocated queue that one or more Worker EntitiesWhat is a Worker Entity?

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Learn more poll for TasksWhat is a Task?

A Task is the context needed to make progress with a specific Workflow Execution or Activity Execution.

Learn more.

Task Queues do not have any ordering guarantees. It is possible to have a Task that stays in a Task Queue for a period of time, if there is a backlog that wasn't drained for that time.

There are two types of Task Queues, Activity Task Queues and Workflow Task Queues.

Task Queue component

Task Queues are very lightweight components. Task Queues do not require explicit registration but instead are created on demand when a Workflow Execution or Activity spawns or when a Worker Process subscribes to it. When a Task Queue is created, both a Workflow Task Queue and an Activity Task Queue are created under the same name. There is no limit to the number of Task Queues a Temporal Application can use or a Temporal Cluster can maintain.

Workers poll for Tasks in Task Queues via synchronous RPC. This implementation offers several benefits:

A Worker Process polls for a message only when it has spare capacity, avoiding overloading itself.
In effect, Task Queues enable load balancing across many Worker Processes.
Task Queues enable what we call Task RoutingWhat is Task Routing?

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

Learn more, which is the routing of specific Tasks to specific Worker Processes or even a specific process.
Task Queues support server-side throttling, which enables you to limit the Task dispatching rate to the pool of Worker Processes while still supporting Task dispatching at higher rates when spikes happen.
When all Worker Processes are down, messages simply persist in a Task Queue, waiting for the Worker Processes to recover.
Worker Processes do not need to advertise themselves through DNS or any other network discovery mechanism.
Worker Processes do not need to have any open ports, which is more secure.

All Workers listening to a given Task Queue must have identical registrations of Activities and/or Workflows. The one exception is during a Server upgrade, where it is okay to have registration temporarily misaligned while the binary rolls out.

Where to set Task Queues

There are four places where the name of the Task Queue can be set by the developer.

A Task Queue must be set when spawning a Workflow Execution:

A Task Queue name must be set when creating a Worker Entity and when running a Worker Process:

Note that all Worker Entities listening to the same Task Queue name must be registered to handle the exact same Workflows Types and Activity Types.

If a Worker Entity polls a Task for a Workflow Type or Activity Type it does not know about, it will fail that Task. However, the failure of the Task will not cause the associated Workflow Execution to fail.

A Task Queue name can be provided when spawning an Activity Execution:

This is optional. An Activity Execution inherits the Task Queue name from its Workflow Execution if one is not provided.

A Task Queue name can be provided when spawning a Child Workflow Execution:

This is optional. A Child Workflow Execution inherits the Task Queue name from its Parent Workflow Execution if one is not provided.

Sticky Execution

A Sticky Execution is when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task Queue to listen on.

A Sticky Execution occurs after a Worker Entity completes the first Workflow Task in the chain of Workflow Tasks for the Workflow Execution.

The Worker Entity caches the Workflow Execution Event History and begins polling the dedicated Task Queue for Workflow Tasks that contain updates, rather than the entire Event History.

If the Worker Entity does not pick up a Workflow Task from the dedicated Task Queue in an appropriate amount of time, the Cluster will resume Scheduling Workflow Tasks on the original Task Queue. Another Worker Entity can then resume the Workflow Execution, and can set up its own Sticky Execution for future Workflow Tasks.

How to set a StickyScheduleToStartTimeout on a Worker Entity in Go

Sticky Executions are the default behavior of the Temporal Platform.

Task Routing

Task Routing is simply when a Task Queue is paired with one or more Workers, primarily for Activity Task Executions.

This could also mean employing multiple Task Queues, each one paired with a Worker Process.

Task Routing has many applicable use cases.

Some SDKs provide a Session APIWhat is a Worker Session

A Worker Session is a feature provided by some SDKs that provides a straightforward way to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names.

Learn more that provides a straightforward way to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. It also includes features like concurrent session limitations and worker failure detection.

Flow control

A Worker that consumes from a Task Queue asks for an Activity Task only when it has available capacity, so it is never overloaded by request spikes. If Activity Tasks get created faster than Workers can process them, they are backlogged in the Task Queue.

Throttling

The rate at which each Activity Worker polls for and processes Activity Tasks is configurable per Worker. Workers do not exceed this rate even if it has spare capacity. There is also support for global Task Queue rate limiting. This limit works across all Workers for the given Task Queue. It is frequently used to limit load on a downstream service that an Activity calls into.

Specific environments

In some cases, you might need to execute Activities in a dedicated environment. To send Activity Tasks to this environment, use a dedicated Task Queue.

Route Activity Tasks to a specific host

In some use cases, such as file processing or machine learning model training, an Activity Task must be routed to a specific Worker Process or Worker Entity.

For example, suppose that you have a Workflow with the following three separate Activities:

Download a file.
Process the file in some way.
Upload a file to another location.

The first Activity, to download the file, could occur on any Worker on any host. However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file.

In a real-life scenario, you might have many Worker Processes scaled over many hosts. You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed.

Code samples:

Route Activity Tasks to a specific process

Some Activities load large datasets and cache them in the process. The Activities that rely on those datasets should be routed to the same process.

In this case, a unique Task Queue would exist for each Worker Process involved.

Workers with different capabilities

Some Workers might exist on GPU boxes versus non-GPU boxes. In this case, each type of box would have its own Task Queue and a Workflow can pick one to send Activity Tasks.

Multiple priorities

If your use case involves more than one priority, you can create one Task Queue per priority, with a Worker pool per priority.

Versioning

Task Routing is the simplest way to version your code.

If you have a new backward-incompatible Activity Definition, start by using a different Task Queue.

Worker Session

A Worker Session is a feature provided by some SDKs that provides a straightforward API for Task RoutingWhat is Task Routing?

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

Learn more to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. It also includes features like concurrent session limitations and Worker failure detection.

How to use Worker SessionsHow to use Worker Session APIs

To use Worker Sessions for Activity Executions the Worker must be enabled to use Sessions for the Workflows and Activities it is registered with.

Learn more

Worker​

Task​

Worker Program​

Worker Entity​

Worker Process​

Workflow Task​

Workflow Task Execution​

Activity Task​

Activity Task Execution​

Task Queue​

Where to set Task Queues​

Sticky Execution​

Task Routing​

Flow control​

Throttling​

Specific environments​

Route Activity Tasks to a specific host​

Route Activity Tasks to a specific process​

Workers with different capabilities​

Multiple priorities​

Versioning​

Worker Session​