Use Polly inside an activity
This recipe protects a price service with a circuit breaker and a concurrency limit from Polly, inside a workflow activity. The workflow's retry policy still decides when to try again and when to give up. Polly decides whether a call is sent at all.
Write a workflow explains activities and retry policies. The complete
programs are in samples/resilience/fsharp and samples/resilience/csharp. They reference the
projects under src/, Polly.Core and Polly.RateLimiting 8.6.5, and need FCQRS.Workflows 7.8.0 or
later, where every activity names a time limit.
Build one pipeline for the process
A circuit breaker counts failed calls. After enough of them it refuses calls for a while instead of sending them to a service that is already failing. A concurrency limiter refuses a call when too many are in flight. Both keep their counts in memory and share them across calls, so the pipeline is built once and every workflow instance uses it.
Shared setup
module Price =
open System
open System.Threading
open System.Threading.Tasks
open Polly
open Polly.CircuitBreaker
open Polly.RateLimiting
open FCQRS.Common
open FCQRS.Workflows
type IPriceService =
abstract Fetch: key: string * symbol: string * cancel: CancellationToken -> Task<decimal>
Fcqrs_550-how-to_015-use-polly-in-an-activity.md_page.PriceSystemThreadingTasksPollyCircuitBreakerRateLimitingFCQRSFCQRS.CommonContains common types like Events and Commands Functionality for Write Side.
WorkflowsFcqrs_550-how-to_015-use-polly-in-an-activity.md_page.Price.IPriceServiceFetch: IPriceService -> string * string * CancellationToken -> Task<decimal>keystringAn abbreviation for the CLI type . Basic Types
symbolcancelSystem.Threading.CancellationTokenPropagates notification that operations should be canceled.
System.Threading.Tasks.Task`1Represents an asynchronous operation that can return a value. The type of the result produced by this .
decimalAn abbreviation for the CLI type . Basic Types
// One pipeline for the process, shared by every workflow instance. Its state
// is in memory: a restart closes the breaker and empties the limiter.
let pricePipeline: ResiliencePipeline =
ResiliencePipelineBuilder()
// At most four calls at once; a fifth is refused at once.
.AddConcurrencyLimiter(4, 0)
// After half of at least two calls in ten seconds fail, refuse
// calls for one second instead of sending them.
.AddCircuitBreaker(
CircuitBreakerStrategyOptions(
FailureRatio = 0.5,
MinimumThroughput = 2,
SamplingDuration = TimeSpan.FromSeconds 10.0,
BreakDuration = TimeSpan.FromSeconds 1.0))
.Build()
pricePipeline: ResiliencePipelinePolly.ResiliencePipelineResilience pipeline is used to execute the user-provided callbacks. Resilience pipeline supports various types of callbacks and provides a unified way to execute them. This includes overloads for synchronous and asynchronous callbacks, generic and non-generic callbacks.
``.ctor``: unit -> unitAddConcurrencyLimiter: int * int -> 'TBuilderAdds the concurrency limiter. The builder type. The builder instance. Maximum number of permits that can be leased concurrently. Maximum number of permits that can be queued concurrently. The builder instance with the concurrency limiter added. Thrown when is . Thrown when the options constructed from the arguments are invalid. Thrown when or is invalid.
FailureRatio: floatGets or sets the failure-to-success ratio at which the circuit will break. A number between zero and one (inclusive) e.g. 0.5 represents breaking if 50% or more of actions result in a handled failure. A ratio number higher than 0, up to 1. The default value is 0.1 (i.e. 10%).
MinimumThroughput: intGets or sets the minimum throughput: this many actions or more must pass through the circuit in the time-slice, for statistics to be considered significant and the circuit-breaker to come into action. The default value is 100. The value must be 2 or greater.
SamplingDuration: TimeSpanGets or sets the duration of the sampling over which failure ratios are assessed. The default value is 30 seconds. Value must be greater than 0.5 seconds.
System.TimeSpanRepresents a time interval.
FromSeconds: float -> TimeSpanReturns a that represents a specified number of seconds, where the specification is accurate to the nearest millisecond. A number of seconds, accurate to the nearest millisecond. is less than TimeSpan.MinValue or greater than TimeSpan.MaxValue. -or- is . -or- is . is equal to . An object that represents .
BreakDuration: TimeSpanGets or sets the duration of break the circuit will stay open before resetting. The default value is 5 seconds. Value must be greater than 0.5 seconds.
// One pipeline for the process, shared by every workflow instance. Its
// state is in memory: a restart closes the breaker and empties the limiter.
private static readonly ResiliencePipeline PricePipeline =
new ResiliencePipelineBuilder()
// At most four calls at once; a fifth is refused at once.
.AddConcurrencyLimiter(4, 0)
// After half of at least two calls in ten seconds fail, refuse
// calls for one second instead of sending them.
.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
MinimumThroughput = 2,
SamplingDuration = TimeSpan.FromSeconds(10),
BreakDuration = TimeSpan.FromSeconds(1),
})
.Build();
The pipeline has no retry strategy. Retries stay in the step's policy, because its deadline is measured from the stored start of the step and survives a restart. Polly's retries run in memory, and they multiply with the step's: three Polly retries inside six attempts can send 24 calls.
Let the step retry what Polly refuses
An open breaker throws BrokenCircuitException, and a full limiter throws
RateLimiterRejectedException. The step's ShouldRetry accepts both, so the step waits for its
schedule and tries again, by which time the breaker may have closed. A filter that accepts only
the service's own errors would fail the step at the first refusal.
// The step retries, not Polly: six attempts, half a second apart, each
// allowed five seconds for the whole pipeline. An open breaker and a full
// limiter are worth another attempt; so is a service error.
let priceRetry = {
Retry.times 6 (TimeSpan.FromSeconds 5.0) with
Schedule = FixedInterval(TimeSpan.FromMilliseconds 500.0)
ShouldRetry = fun error ->
error :? BrokenCircuitException
|| error :? RateLimiterRejectedException
|| error :? TimeoutException
|| error :? InvalidOperationException
}
priceRetry: RetryPolicyFCQRS.Workflows.RetryCommon retry policies.
times: int -> TimeSpan -> RetryPolicyAt most `attempts` attempts, with backoff from one second to one minute. Each attempt fails when it has not finished within `attemptTimeout`.
System.TimeSpanRepresents a time interval.
FromSeconds: float -> TimeSpanReturns a that represents a specified number of seconds, where the specification is accurate to the nearest millisecond. A number of seconds, accurate to the nearest millisecond. is less than TimeSpan.MinValue or greater than TimeSpan.MaxValue. -or- is . -or- is . is equal to . An object that represents .
Schedule: RetryScheduleThe wait between attempts.
FixedIntervalRe-send at a fixed interval.
FromMilliseconds: float -> TimeSpanReturns a that represents a specified number of milliseconds. A number of milliseconds. is less than TimeSpan.MinValue or greater than TimeSpan.MaxValue. -or- is . -or- is . is equal to . An object that represents .
ShouldRetry: exn -> boolWhether an exception is worth another attempt. `false` fails the step at once.
error: exnPolly.CircuitBreaker.BrokenCircuitExceptionException thrown when a circuit is broken.
(||): bool -> bool -> boolBinary 'or'. When used as a binary operator the right hand value is evaluated only on demand The first value. The second value. The result of the operation.
Polly.RateLimiting.RateLimiterRejectedExceptionException thrown when a rate limiter rejects an execution.
System.TimeoutExceptionThe exception that is thrown when the time allotted for a process or operation has expired.
System.InvalidOperationExceptionThe exception that is thrown when a method call is invalid for the object's current state.
// The step retries, not Polly: six attempts, half a second apart, each
// allowed five seconds for the whole pipeline. An open breaker and a full
// limiter are worth another attempt; so is a service error.
private static readonly RetryPolicy PriceRetry =
Retry.times(6, TimeSpan.FromSeconds(5))
.WithSchedule(RetrySchedules.Fixed(TimeSpan.FromMilliseconds(500)))
.RetryWhen(error =>
error is BrokenCircuitException
or RateLimiterRejectedException
or TimeoutException
or InvalidOperationException);
TimeoutException is what a step sees when an attempt runs past its AttemptTimeout.
Run the pipeline in the activity
The activity passes the attempt's cancellation token to the pipeline, and the pipeline passes its own token to the service.
let definition (prices: IPriceService) =
Workflow.define "price" (fun (symbol: string) -> workflow {
return!
activity "fetch" priceRetry (fun attempt ->
// The attempt's token stops the pipeline when the attempt
// times out, the deadline passes, or the workflow stops.
let call (cancel: CancellationToken) =
ValueTask<decimal>(
prices.Fetch(attempt.IdempotencyKey, symbol, cancel))
pricePipeline
.ExecuteAsync(call, attempt.Cancellation)
.AsTask())
})
definition: IPriceService -> WorkflowDefinition<string,decimal>prices: IPriceServiceFcqrs_550-how-to_015-use-polly-in-an-activity.md_page.Price.IPriceServiceFCQRS.Workflows.WorkflowDefines workflows.
define: string -> ('Input -> Workflow<'Result>) -> WorkflowDefinition<'Input,'Result>A workflow named `name` whose instances run `body` on their input.
symbol: stringstringAn abbreviation for the CLI type . Basic Types
workflow: WorkflowBuilderBuilds a workflow: `workflow { let! x = activity ... }`.
activity: string -> RetryPolicy -> (AttemptContext -> Task<'T>) -> Workflow<'T>Calls an external system. The call runs at least once: a crash after the call and before its result is stored repeats it, so pass `IdempotencyKey` to the external system. `name` is stored and must not change while instances are running. The policy needs an `AttemptTimeout` or a `Deadline`: an attempt with neither would wait as long as its call, also for a call that ignores its cancellation token.
priceRetry: RetryPolicyattempt: AttemptContextcall: CancellationToken -> ValueTask<decimal>cancel: CancellationTokenSystem.Threading.CancellationTokenPropagates notification that operations should be canceled.
``.ctor``: Task<decimal> -> unitInitializes a new instance of the class using the supplied task that represents the operation. The task. The argument is .
decimalAn abbreviation for the CLI type . Basic Types
Fetch: string * string * CancellationToken -> Task<decimal>IdempotencyKey: stringThe same for every attempt of this step, across restarts: workflow name, workflow ID, and step path. Pass it to the external system so a repeated call is recognised.
pricePipeline: ResiliencePipelineCancellation: CancellationTokenCancelled when the attempt times out, the step's deadline passes, or the workflow stops.
public override async WorkflowTask<decimal> Run(
WorkflowContext ctx, string symbol) =>
await ctx.Activity("fetch", PriceRetry, attempt =>
// The attempt's token stops the pipeline when the attempt times
// out, the deadline passes, or the workflow stops.
PricePipeline.ExecuteAsync(
cancel => new ValueTask<decimal>(
prices.Fetch(attempt.IdempotencyKey, symbol, cancel)),
attempt.Cancellation).AsTask());
The stand-in service in the programs fails its first two calls. The second failure opens the breaker, so the next attempt is refused without a call, and a later attempt succeeds:
call 1: the service failed
call 2: the service failed
call 3: ABC costs 12.5
price 12.5 after 3 calls to the service
What each part guarantees
AttemptTimeoutbounds the whole pipeline call. When it runs out, the attempt fails withTimeoutExceptioneven if the pipeline or the service ignores the token. The call keeps running in the background, and its result is ignored.- The step stores only the result of a successful attempt. The breaker's state, the limiter's count, and the attempts are not stored: a restart starts them again.
- The pipeline is shared within one process. FCQRS runs on one node, so it is shared by every workflow instance that can call the service.
- A call can reach the service more than once: a retry repeats it, and so does a restart before the result is stored. Pass the idempotency key to the service, as for any activity.
When the step's policy runs out, for example because the breaker stayed open past the last attempt,
the body receives WorkflowStepFailedException. The body can catch it and choose another path,
such as a cached price or a person's decision.
Run scheduled jobs can start many runs at once after downtime. A limiter like this one keeps them from calling a service at the same moment.
Next: Run a free-monad program as a workflow carries out a program's instructions as workflow steps.