Observe your system

FCQRS reports command, event, saga, dispatch, and projection activity through ILogger and ActivitySource. Configure both before production so one correlation id can be followed across the complete workflow.

Message-flow logs

At Information level, the FCQRS.MessageFlow category records aggregate decisions, persisted event versions, saga transitions, and commands issued by sagas. Every line contains the correlation id. These lines are the first transfer of the saga sample transfer-with-a-saga, one correlation id from Alice's command to Bob's stored event:

info: FCQRS.MessageFlow[0]
      Command SendTransfer ("t1", "bob", 30M) to aggregate alice (v2) yielded PersistEvent (TransferSent ("t1", "bob", 30M)) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
      Aggregate alice persisted event TransferSent ("t1", "bob", 30M) (v3) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
      Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf changed state to Delivering [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
      Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf sent command ReceiveTransfer ("t1", "alice", 30M) to bob [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
      Aggregate bob persisted event TransferReceived ("t1", "alice", 30M) (v2) [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]
info: FCQRS.MessageFlow[0]
      Saga alice~Saga~01a0cf1b-8485-72e5-9e41-6011435b56bf changed state to Completed [cid: 01a0cf1b-8485-72e5-9e41-6011435b56bf]

Disable the process-wide narrative with FCQRS.Common.Telemetry.MessageFlowLogging <- false or builder.WithMessageFlowLogging(false). Standard logger filtering also applies:

{
  "Logging": {
    "LogLevel": {
      "FCQRS.MessageFlow": "None"
    }
  }
}

When the category is disabled, FCQRS skips formatting the message payload.

Distributed traces

Aggregates, sagas, and projections use three activity sources. A W3C traceparent is copied into command metadata and carried through later events and saga commands. Register all three sources:

// "FCQRS", "FCQRS.Saga", and "FCQRS.Query"
tracing.AddSource(FCQRS.Common.Telemetry.AllActivitySources);

ActivitySource avoids creating activities when no listener is attached. Restart-detection aborts and fatal errors set span status to Error.

Start an activity at the application boundary before constructing the first command. The resulting trace should contain the initial command, stored event, saga states, follow-up commands, and projection handler. The CID remains a domain correlation value; trace context travels beside it in metadata.

Span names are low-cardinality

Span names contain the case name, such as Command:SendTransfer, Event:TransferSent, Saga:Delivering, or Abort:TransferSent. Payload values do not appear in the span name, so trace backends can group operations without creating one name per entity. On .NET 11, tracing rules can select a source and operation:

using Microsoft.Extensions.Diagnostics.Tracing;

builder.Services.AddTracing(tracing =>
{
    tracing.EnableTracing(sourceName: "FCQRS.Saga");
    tracing.DisableTracing(
        sourceName: "FCQRS", operationName: "Command:HealthPing");
});

Payload detail may still appear in tags and logs as described below.

Workflows

FCQRS.Workflows adds a fourth activity source, FCQRS.Workflow, which is already in Telemetry.AllActivitySources, and a meter with the same name:

metrics.AddMeter(FCQRS.Workflows.WorkflowTelemetry.MeterName);

A workflow stores the trace context of the caller that started it. Spans emitted after a restart use that context as their parent, so a resumed workflow continues the same trace.

Span Emitted when
Workflow.Start:<workflow> the start is stored
Step:<step> a step ends; for an activity, a command, or a child, it covers the step
Attempt:<step> one attempt of an activity, a command, or a child request runs; a child of its step
Workflow.End:<workflow> the workflow ends; its status is Error when the workflow faulted
  • A step that is replayed from the journal emits no span.
  • A timer or a signal wait emits one short span when it ends, with the time it waited in fcqrs.workflow.waited_ms. No span stays open across a wait.
  • The span of a signal wait has a link to the span of the caller that sent the signal. The wait itself stays in the trace of the caller that started the workflow.
  • A child started with child joins its parent's trace, under the attempt that started it. A child started with startChild begins a trace of its own: its Workflow.Start span has no parent and links to the parent's Step span, and its other spans are children of that start span. This holds also when the parent awaits the handle later.
  • An attempt span is Activity.Current inside the activity, so HttpClient and database instrumentation attach to it. A command sent by a workflow carries it in traceparent.
  • Spans carry fcqrs.workflow.name, fcqrs.workflow.id, fcqrs.workflow.step.path, fcqrs.workflow.step.name, and fcqrs.workflow.attempt. Inputs, results, and signal payloads are never written to a span.
Instrument Use
fcqrs.workflow.started, .finished, .faulted, .cancelled throughput and failure rate, by workflow name
fcqrs.workflow.running instances in memory that have not ended
fcqrs.workflow.parked instances whose journal does not match their code; alert when above zero
fcqrs.workflow.step.duration milliseconds from a step's stored start to its end, by step kind
fcqrs.workflow.activity.retries attempts that failed and were retried
fcqrs.workflow.recovery.duration, .recovery.records workflows whose journal has grown long

A faulted workflow is final, and a parked one runs nothing until matching code is deployed. Alert on both.

fcqrs.workflow.running counts instances. To see which steps are running now, and for how long, call Workflows.running api (C#: WorkflowHost.Running(actor)) from a diagnostics endpoint: it lists each activity call, command waiting for its reply, and child start, oldest first, with its workflow, instance, step, attempt, and start time. A step that has run far longer than its timeout, with CancellationRequested true, is a call that ignores its cancellation token.

A shutdown waits up to akka.fcqrs.workflow-drain-timeout for the calls it cancelled, then logs a Warning in the FCQRS.Workflows.Shutdown category for each one still running, naming its workflow, instance, step, and attempt. Such a call can still reach the external system, and the step runs again after the restart. See what is running, and what a shutdown waits for explains both.

Workflow logs

A workflow writes its progress to the FCQRS.MessageFlow category at Information. These are the lines of the quote workflow from Write a workflow, instance q-1001:

Line Written when
Workflow quote q-1001 started the start is stored
Workflow quote q-1001 completed step 0 price (Activity) a step's result is stored; 0 is the step's path
Workflow quote q-1001 received signal approval a signal is stored
Workflow quote q-1001 timed out at step 1 approval (Signal) a signal wait's timeout passes before a signal arrives
Workflow quote q-1001 was asked to cancel a cancel request is stored
Workflow quote q-1001 finished or ... was cancelled the workflow ends
  • A line is written after its record is stored. A process that stops in between loses the line, not the record.
  • A step that is replayed from the journal writes no line.
  • The lines never contain an input, a step result, a signal payload, or the workflow's result.
  • Telemetry.MessageFlowLogging and a filter on the category turn these lines off, as they do for aggregates and sagas.

Failures go to the FCQRS.Workflows category. The message-flow switch does not affect them:

Level Line Meaning
Warning Workflow quote q-1001: attempt 1 of step price failed and will be retried. one attempt failed and the retry policy allows another
Warning Workflow quote q-1001: step 0 price failed as Exhausted: <message> the step failed for good; the body may still catch the failure
Error Workflow quote q-1001 faulted: BodyFailed: <message> the body ended with an exception it did not handle
Error Workflow quote q-1001 is parked: its journal does not match its code. the instance runs nothing until matching code is deployed

The first three lines carry the exception that an activity or the body threw. The journal stores only an exception's type and message, so the warning for a failed step is the one place that keeps its stack trace. A step that fails because the workflow was cancelled writes no warning. An exception's message appears in these lines; keep sensitive values out of the exceptions an activity throws.

Each of these lines is written in a log scope that holds fcqrs.workflow.name and fcqrs.workflow.id, and for a step fcqrs.workflow.step.path and fcqrs.workflow.step.name. An activity runs inside a scope that also holds fcqrs.workflow.attempt, so a line the activity writes through the same logger factory names the workflow and the step that wrote it. Log providers leave scopes out by default: set IncludeScopes on the console logger or on OpenTelemetry's logger options to export them.

When a trace listener is attached, a line about a step, the start, or the end is written while that span is current, so it carries the span's trace and span IDs. The lines for a received signal and a cancel request belong to no span of the workflow and carry none.

Keep payloads out of diagnostics

Rendered payloads appear in span tags and message-flow logs by default. Disable them before processing sensitive values when detailed payload diagnostics are not acceptable:

FCQRS.Common.Telemetry.IncludePayloads <- false
builder.WithPayloadDiagnostics(false);

Tags and log lines then contain the case name only. This switch affects diagnostics, not persisted events. A secret stored in an event remains in the journal regardless of the diagnostics setting.

Flush telemetry on a fatal exit

FCQRS terminates the process when a fold, aggregate handler, saga handler, effect runner, or projection handler fails in a way that could leave state processing inconsistent. It also terminates when a payload cannot be serialized or the journal rejects an event. When FCQRS stops the process lists every case and explains why. Fail-fast skips normal finalizer and process-exit flushing. Register a bounded flush hook for buffered telemetry:

Shared setup
open OpenTelemetry.Trace
open OpenTelemetry.Logs
let configureFlush (tracerProvider: TracerProvider) (loggerProvider: LoggerProvider) =
FCQRS.Common.Telemetry.FatalFlush <- System.Action(fun () ->
    tracerProvider.ForceFlush(3000) |> ignore
    // With Serilog, call Log.CloseAndFlush() instead.
    loggerProvider.ForceFlush(3000) |> ignore)
FCQRS.Common.Telemetry.FatalFlush = new Action(() =>
{
    tracerProvider.ForceFlush(3000);
    // With Serilog, call Serilog.Log.CloseAndFlush() instead.
    loggerProvider.ForceFlush(3000);
});

The hook runs on a background thread with a five-second cap.

Alerts to add

At minimum, alert on:

  • process fail-fast and repeated restarts;
  • projection handler failure and growing projection lag;
  • a saga remaining in one state beyond its domain timeout;
  • unreachable cluster members and shard movement that does not settle;
  • journal or snapshot storage latency and errors;
  • exhausted retries or workflows sent to manual intervention.

FCQRS supplies the message and trace context. Domain timeouts, projection-lag metrics, and manual-intervention counters belong to the application because only it knows the expected duration and business impact.

Akka's own logging

Akka.NET internal logging defaults to OFF; FCQRS logs still use the application's ILoggerFactory. Enable Akka.NET internals with builder.WithAkkaLogging(AkkaLogLevel.Info) from the hosting builder, or set config:akka:loglevel in configuration. See Configuration for the config-key details.