Observe your system
FCQRS reports command, event, saga, dispatch, and projection activity through ILogger and
ActivitySource. Configure both before production so one correlation id can be followed across the
complete workflow.
Message-flow logs
At Information level, the FCQRS.MessageFlow category records aggregate decisions, persisted event
versions, saga transitions, and commands issued by sagas. Every line contains the correlation id.
These lines are the first transfer of the saga sample
transfer-with-a-saga,
one correlation id from Alice's command to Bob's stored event:
info: FCQRS.MessageFlow[0]
Command SendTransfer ("t1", "bob", 30M) to aggregate alice (v2) yielded PersistEvent (TransferSent ("t1", "bob", 30M)) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
Aggregate alice persisted event TransferSent ("t1", "bob", 30M) (v3) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf changed state to Delivering [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf sent command ReceiveTransfer ("t1", "alice", 30M) to bob [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
Aggregate bob persisted event TransferReceived ("t1", "alice", 30M) (v2) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf changed state to Completed [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
Disable the process-wide narrative with
FCQRS.Common.Telemetry.MessageFlowLogging <- false or
builder.WithMessageFlowLogging(false). Standard logger filtering also applies:
{
"Logging": {
"LogLevel": {
"FCQRS.MessageFlow": "None"
}
}
}
When the category is disabled, FCQRS skips formatting the message payload.
Distributed traces
Aggregates, sagas, and projections use three activity sources. A W3C traceparent is copied into
command metadata and carried through later events and saga commands. Register all three sources:
// "FCQRS", "FCQRS.Saga", and "FCQRS.Query"
tracing.AddSource(FCQRS.Common.Telemetry.AllActivitySources);
ActivitySource avoids creating activities when no listener is attached. Restart-detection aborts and
fatal errors set span status to Error.
Start an activity at the application boundary before constructing the first command. The resulting trace should contain the initial command, stored event, saga states, follow-up commands, and projection handler. The CID remains a domain correlation value; trace context travels beside it in metadata.
Span names are low-cardinality
Span names contain the case name, such as Command:SendTransfer, Event:TransferSent,
Saga:Delivering, or Abort:TransferSent. Payload values do not appear in the span name,
so trace backends can group operations without creating one name per entity. On .NET 11, tracing rules
can select a source and operation:
using Microsoft.Extensions.Diagnostics.Tracing;
builder.Services.AddTracing(tracing =>
{
tracing.EnableTracing(sourceName: "FCQRS.Saga");
tracing.DisableTracing(
sourceName: "FCQRS", operationName: "Command:HealthPing");
});
Payload detail may still appear in tags and logs as described below.
Workflows
FCQRS.Workflows adds a fourth activity source, FCQRS.Workflow, which is already in
Telemetry.AllActivitySources, and a meter with the same name:
metrics.AddMeter(FCQRS.Workflows.WorkflowTelemetry.MeterName);
A workflow stores the trace context of the caller that started it. Spans emitted after a restart use that context as their parent, so a resumed workflow continues the same trace.
| Span | Emitted when |
|---|---|
Workflow.Start:<workflow> |
the start is stored |
Step:<step> |
a step ends; for an activity, a command, or a child, it covers the step |
Attempt:<step> |
one attempt of an activity, a command, or a child request runs; a child of its step |
Workflow.End:<workflow> |
the workflow ends; its status is Error when the workflow faulted |
- A step that is replayed from the journal emits no span.
- A timer or a signal wait emits one short span when it ends, with the time it waited in
fcqrs.workflow.waited_ms. No span stays open across a wait. - The span of a signal wait has a link to the span of the caller that sent the signal. The wait itself stays in the trace of the caller that started the workflow.
- A child started with
childjoins its parent's trace, under the attempt that started it. A child started withstartChildbegins a trace of its own: itsWorkflow.Startspan has no parent and links to the parent'sStepspan, and its other spans are children of that start span. This holds also when the parent awaits the handle later. - An attempt span is
Activity.Currentinside the activity, soHttpClientand database instrumentation attach to it. A command sent by a workflow carries it intraceparent. - Spans carry
fcqrs.workflow.name,fcqrs.workflow.id,fcqrs.workflow.step.path,fcqrs.workflow.step.name, andfcqrs.workflow.attempt. Inputs, results, and signal payloads are never written to a span.
| Instrument | Use |
|---|---|
fcqrs.workflow.started, .finished, .faulted, .cancelled |
throughput and failure rate, by workflow name |
fcqrs.workflow.running |
instances in memory that have not ended |
fcqrs.workflow.parked |
instances whose journal does not match their code; alert when above zero |
fcqrs.workflow.step.duration |
milliseconds from a step's stored start to its end, by step kind |
fcqrs.workflow.activity.retries |
attempts that failed and were retried |
fcqrs.workflow.recovery.duration, .recovery.records |
workflows whose journal has grown long |
A faulted workflow is final, and a parked one runs nothing until matching code is deployed. Alert on both.
fcqrs.workflow.running counts instances. To see which steps are running now, and for how long,
call Workflows.running api (C#: WorkflowHost.Running(actor)) from a diagnostics endpoint: it
lists each activity call, command waiting for its reply, and child start, oldest first, with its
workflow, instance, step, attempt, and start time. A step that has run far longer than its timeout,
with CancellationRequested true, is a call that ignores its cancellation token.
A shutdown waits up to akka.fcqrs.workflow-drain-timeout for the calls it cancelled, then logs a
Warning in the FCQRS.Workflows.Shutdown category for each one still running, naming its
workflow, instance, step, and attempt. Such a call can still reach the external system, and the step
runs again after the restart. See what is running, and what a shutdown waits
for explains both.
Workflow logs
A workflow writes its progress to the FCQRS.MessageFlow category at Information. These are the
lines of the quote workflow from Write a workflow, instance q-1001:
| Line | Written when |
|---|---|
Workflow quote q-1001 started |
the start is stored |
Workflow quote q-1001 completed step 0 price (Activity) |
a step's result is stored; 0 is the step's path |
Workflow quote q-1001 received signal approval |
a signal is stored |
Workflow quote q-1001 timed out at step 1 approval (Signal) |
a signal wait's timeout passes before a signal arrives |
Workflow quote q-1001 was asked to cancel |
a cancel request is stored |
Workflow quote q-1001 finished or ... was cancelled |
the workflow ends |
- A line is written after its record is stored. A process that stops in between loses the line, not the record.
- A step that is replayed from the journal writes no line.
- The lines never contain an input, a step result, a signal payload, or the workflow's result.
Telemetry.MessageFlowLoggingand a filter on the category turn these lines off, as they do for aggregates and sagas.
Failures go to the FCQRS.Workflows category. The message-flow switch does not affect them:
| Level | Line | Meaning |
|---|---|---|
| Warning | Workflow quote q-1001: attempt 1 of step price failed and will be retried. |
one attempt failed and the retry policy allows another |
| Warning | Workflow quote q-1001: step 0 price failed as Exhausted: <message> |
the step failed for good; the body may still catch the failure |
| Error | Workflow quote q-1001 faulted: BodyFailed: <message> |
the body ended with an exception it did not handle |
| Error | Workflow quote q-1001 is parked: its journal does not match its code. |
the instance runs nothing until matching code is deployed |
The first three lines carry the exception that an activity or the body threw. The journal stores only an exception's type and message, so the warning for a failed step is the one place that keeps its stack trace. A step that fails because the workflow was cancelled writes no warning. An exception's message appears in these lines; keep sensitive values out of the exceptions an activity throws.
Each of these lines is written in a log scope that holds fcqrs.workflow.name and
fcqrs.workflow.id, and for a step fcqrs.workflow.step.path and fcqrs.workflow.step.name. An
activity runs inside a scope that also holds fcqrs.workflow.attempt, so a line the activity writes
through the same logger factory names the workflow and the step that wrote it. Log providers leave
scopes out by default: set IncludeScopes on the console logger or on OpenTelemetry's logger options
to export them.
When a trace listener is attached, a line about a step, the start, or the end is written while that span is current, so it carries the span's trace and span IDs. The lines for a received signal and a cancel request belong to no span of the workflow and carry none.
Keep payloads out of diagnostics
Rendered payloads appear in span tags and message-flow logs by default. Disable them before processing sensitive values when detailed payload diagnostics are not acceptable:
FCQRS.Common.Telemetry.IncludePayloads <- false
FCQRSIncludePayloads: boolProcess-wide switch for including rendered message *payloads* in diagnostics. Default: on. Span *names* are always low-cardinality case names (e.g. "Command:Register") regardless of this switch — payload values never appear there, so per-operation tracing rules and trace-viewer grouping always work, and nothing sensitive leaks into the indexed span name. This switch governs the *detail*: the payload rendered into span tags (command.type / event.type) and into the message-flow log lines. Turn it off (Telemetry.IncludePayloads <- false, or FcqrsBuilder.WithPayloadDiagnostics(false)) for sensitive domains — tags and log lines then carry the case name only, matching the span name.
FCQRS.CommonContains common types like Events and Commands Functionality for Write Side.
FCQRS.Common.TelemetryFCQRS's ActivitySource names — register them with your tracing pipeline (e.g. OpenTelemetry: tracing.AddSource(Telemetry.AllActivitySources)).
builder.WithPayloadDiagnostics(false);
Tags and log lines then contain the case name only. This switch affects diagnostics, not persisted events. A secret stored in an event remains in the journal regardless of the diagnostics setting.
Flush telemetry on a fatal exit
FCQRS terminates the process when a fold, aggregate handler, saga handler, effect runner, or projection handler fails in a way that could leave state processing inconsistent. It also terminates when a payload cannot be serialized or the journal rejects an event. When FCQRS stops the process lists every case and explains why. Fail-fast skips normal finalizer and process-exit flushing. Register a bounded flush hook for buffered telemetry:
Shared setup
open OpenTelemetry.Trace
open OpenTelemetry.Logs
let configureFlush (tracerProvider: TracerProvider) (loggerProvider: LoggerProvider) =
OpenTelemetryTraceLogsconfigureFlush: TracerProvider -> LoggerProvider -> unittracerProvider: TracerProviderOpenTelemetry.Trace.TracerProviderTracerProvider is the entry point of the OpenTelemetry API. It provides access to .
loggerProvider: LoggerProviderOpenTelemetry.Logs.LoggerProviderLoggerProvider is the entry point of the OpenTelemetry API. It provides access to .
FCQRS.Common.Telemetry.FatalFlush <- System.Action(fun () ->
tracerProvider.ForceFlush(3000) |> ignore
// With Serilog, call Log.CloseAndFlush() instead.
loggerProvider.ForceFlush(3000) |> ignore)
FCQRSFatalFlush: System.Action | nullOptional hook invoked right before FCQRS kills the process on a fatal error. FailFast skips finalizers and ProcessExit handlers, so without this everything still sitting in a batch exporter or buffered log sink is silently dropped — including the span and log entry of the fatal flow itself (every fatal site logs before invoking this hook). Flush your whole pipeline here, e.g. Telemetry.FatalFlush <- Action(fun () -> tracerProvider.ForceFlush(3000) |> ignore loggerProvider.ForceFlush(3000) |> ignore) // or Serilog's Log.CloseAndFlush() It runs on a background thread with a 5-second cap so a hung exporter cannot block the kill.
FCQRS.CommonContains common types like Events and Commands Functionality for Write Side.
FCQRS.Common.TelemetryFCQRS's ActivitySource names — register them with your tracing pipeline (e.g. OpenTelemetry: tracing.AddSource(Telemetry.AllActivitySources)).
SystemSystem.ActionEncapsulates a method that has no parameters and does not return a value.
tracerProvider: TracerProviderForceFlush: int -> boolFlushes all the processors registered under TracerProviderSdk, blocks the current thread until flush completed, shutdown signaled or timed out. TracerProviderSdk instance on which ForceFlush will be called. The number (non-negative) of milliseconds to wait, or Timeout.Infinite to wait indefinitely. Returns true when force flush succeeded; otherwise, false. Thrown when the timeoutMilliseconds is smaller than -1. This function guarantees thread-safety.
(|>): 'T1 -> ('T1 -> 'U) -> 'UApply a function to a value, the value being on the left, the function on the right The argument. The function. The function result. let doubleIt x = x * 2 3 |> doubleIt // Evaluates to 6
ignore: 'T -> unitIgnore the passed value. This is often used to throw away results of a computation. The value to ignore. ignore 55555 // Evaluates to ()
loggerProvider: LoggerProviderForceFlush: int -> boolFlushes all the processors registered under , blocks the current thread until flush completed, shutdown signaled or timed out. instance on which ForceFlush will be called. The number (non-negative) of milliseconds to wait, or Timeout.Infinite to wait indefinitely. Returns true when force flush succeeded; otherwise, false. Thrown when the timeoutMilliseconds is smaller than -1. This function guarantees thread-safety.
FCQRS.Common.Telemetry.FatalFlush = new Action(() =>
{
tracerProvider.ForceFlush(3000);
// With Serilog, call Serilog.Log.CloseAndFlush() instead.
loggerProvider.ForceFlush(3000);
});
The hook runs on a background thread with a five-second cap.
Alerts to add
At minimum, alert on:
- process fail-fast and repeated restarts;
- projection handler failure and growing projection lag;
- a saga remaining in one state beyond its domain timeout;
- unreachable cluster members and shard movement that does not settle;
- journal or snapshot storage latency and errors;
- exhausted retries or workflows sent to manual intervention.
FCQRS supplies the message and trace context. Domain timeouts, projection-lag metrics, and manual-intervention counters belong to the application because only it knows the expected duration and business impact.
Akka's own logging
Akka.NET internal logging defaults to OFF; FCQRS logs still use the application's ILoggerFactory.
Enable Akka.NET internals with builder.WithAkkaLogging(AkkaLogLevel.Info)
from the hosting builder, or set config:akka:loglevel in configuration. See
Configuration for the config-key details.