Manage Synchronization Domain Entities

Manual vs Automatic Initialization

If you’re running a sync domain in its default configuration as shown previously, it will have a sequencer and mediator embedded and these components will be automatically bootstrapped for you.

However, there are situations where a node should not be automatically initialized, but where you prefer to control each step of the initialization. For example, this might be the case when a node in the setup does not control its own identity, when you do not want to store the identity key on the node for security reasons, or when you want to set your own keys (e.g. when keys are externally stored in a Key Management Service - KMS).

If you want to disable the automatic initialization of domain nodes you have to set:

<node>.init.auto-init = false

This is only applicable for domain/domain-manager nodes.

Keys Initialization

When manual initialization is enabled, cryptographic keys in the nodes are not automatically generated and registered in Canton. The following command manually generates a Canton signing key.

<node>.keys.secret.generate_signing_key(<key_name>)

If you are using a Key Management Service (KMS) to handle Canton’s keys and you want to use a set of pre-generated keys you must use instead the following command.

node.keys.secret.register_kms_signing_key(<kms_key_id>, <key_name>)

Please refer to External Key Storage with a Key Management Service (KMS) for more details.

Note

The following sections assume that you are using Canton keys and that you do not have a KMS set-up. If you are running your environment with KMS and want to provide your own keys, replace all instances of generate_signing_key() by the correct register_kms_signing_key() command.

Embedded Synchronization Domain Initialization

The following steps describe how to manually initialize an embedded sync domain node, where the domain manager, the sequencer and mediator are aggregated into a single entity and deployed together.

// first, let's create a signing key that is going to control our identity
val identityKey =
  if (useKms) {
    mydomain.keys.secret.register_kms_signing_key(
      kmsKeyId = identityKmsKeyId,
      name = mydomain.name + "-namespace",
    )
  } else {
    mydomain.keys.secret.generate_signing_key(name = mydomain.name + "-namespace")
  }

// create a signing key for this entity
val signingKey =
  if (useKms) {
    mydomain.keys.secret.register_kms_signing_key(
      kmsKeyId = signingKmsKeyId,
      name = mydomain.name + "-signing",
    )
  } else {
    mydomain.keys.secret.generate_signing_key(name = mydomain.name + "-signing")
  }

val namespace = identityKey.fingerprint

// initialise the identity of this domain
val uid = mydomain.topology.init_id(identifier = mydomain.name, fingerprint = namespace)

// create the root certificate for this namespace
mydomain.topology.namespace_delegations.authorize(
  ops = TopologyChangeOp.Add,
  namespace = namespace,
  authorizedKey = namespace,
  isRootDelegation = true,
)

val protocolVersion = mydomain.config.init.domainParameters.initialProtocolVersion

// set the initial dynamic domain parameters for the domain
mydomain.topology.domain_parameters_changes
  .authorize(
    domainId = DomainId(uid),
    newParameters = com.digitalasset.canton.admin.api.client.data.DynamicDomainParameters
      .defaultValues(protocolVersion),
    protocolVersion = protocolVersion,
  )

val mediatorId = MediatorId(uid)
Seq[Member](DomainTopologyManagerId(uid), SequencerId(uid), mediatorId).foreach { keyOwner =>
  // in this case, we are using an embedded domain. therefore, we initialise all domain
  // entities at once. in a distributed setup, the process needs to be invoked on
  // the separate entities, and therefore requires a bit more coordination.
  // however, the steps remain the same.

  // then, create a topology transaction linking the entity to the signing key
  mydomain.topology.owner_to_key_mappings.authorize(
    ops = TopologyChangeOp.Add,
    keyOwner = keyOwner,
    key = signingKey.fingerprint,
    purpose = KeyPurpose.Signing,
  )
}

// Register the mediator
mydomain.topology.mediator_domain_states.authorize(
  ops = TopologyChangeOp.Add,
  domain = mydomain.id,
  mediator = mediatorId,
  side = RequestSide.Both,
)

Set up a Distributed Synchronization Domain With a Single Console

If your sync domain operates with external sequencers and mediators, you will need to configure a sync domain manager node (which only runs topology management) and bootstrap your sync domain with at least one external sequencer node and one external mediator node.

First make sure the nodes are fresh and have not yet been initialized:

@ mediator1.health.initialized()
res1: Boolean = false

@ sequencer1.health.initialized()
res2: Boolean = false

@ domainManager1.health.initialized()
res3: Boolean = false

If the sync domain manager has auto-init = true (the default value) you need to make sure that the identity is generated and ready before you can bootstrap the sync domain:

@ domainManager1.health.wait_for_identity()

Note

This is technically only required when accessing the sync domain manager through a remote console, but is a good practice regardless.

If the sync domain manager has auto-init = false, then you need to manually initialize it by generating its identity and setting-up its keys:

// create namespace key for the domain manager
val namespaceKey = if (useKms) {
  mydomain_manager.keys.secret.register_kms_signing_key(
    namespaceKmsKeyIdDomainManager,
    name = mydomain_manager.name + "-namespace",
  )
} else {
  mydomain_manager.keys.secret
    .generate_signing_key(name = mydomain_manager.name + "-namespace")
}

// use the fingerprint of namespace key for our identity
val namespace = namespaceKey.fingerprint

// initialise the identity of this domain
val uid =
  mydomain_manager.topology.init_id(identifier = mydomain_manager.name, fingerprint = namespace)

// create the root certificate for this namespace
mydomain_manager.topology.namespace_delegations.authorize(
  ops = TopologyChangeOp.Add,
  namespace = namespace,
  authorizedKey = namespace,
  isRootDelegation = true,
)

val protocolVersion = mydomain_manager.config.init.domainParameters.initialProtocolVersion

// set the initial dynamic domain parameters for the domain
mydomain_manager.topology.domain_parameters_changes
  .authorize(
    domainId = DomainId(uid),
    newParameters = com.digitalasset.canton.admin.api.client.data.DynamicDomainParameters
      .defaultValues(protocolVersion),
    protocolVersion = protocolVersion,
  )

// create signing key for the domain manager
val signingKeyDomainManager =
  if (useKms)
    mydomain_manager.keys.secret.register_kms_signing_key(
      signingKmsKeyIdDomainManager,
      name = mydomain_manager.name + "-signing",
    )
  else
    mydomain_manager.keys.secret
      .generate_signing_key(name = mydomain_manager.name + "-signing")

// create a topology transaction linking the domain manager to its signing key
mydomain_manager.topology.owner_to_key_mappings.authorize(
  ops = TopologyChangeOp.Add,
  keyOwner = DomainTopologyManagerId(uid),
  key = signingKeyDomainManager.fingerprint,
  purpose = KeyPurpose.Signing,
)

If you want, before continuing, you can pre-generate signing keys for mediators and sequencers by running:

mediator1.keys.secret.generate_signing_key(name = mediator1.name + "-signing")
sequencer1.keys.secret.generate_signing_key(name = sequencer1.name + "-signing")

Now you can initialize the distributed sync domain as follows:

@ domainManager1.setup.bootstrap_domain(Seq(sequencer1), Seq(mediator1))

At this point a participant should be able to connect to a sequencer and operate on that sync domain:

@ participant1.domains.connect_local(sequencer1)

@ participant1.health.ping(participant1)
res7: Duration = 1843 milliseconds

Sync domain managers are configured as domain-managers under the canton configuration. Sync domain managers are configured similarly to sync domain nodes, except that there are no sequencer, mediator, public API, or service agreement configs.

Please note that if your sequencer is database-based and you’re horizontally scaling it as described in sequencer high availability, you do not need to pass all sequencer nodes into the command above. Since they all share the same relational database, you only need to run this initialization step on one of them.

For a non-database-based sequencer such as Ethereum or Fabric sequencers you need to have each node initialized individually. You can either initialize such sequencers as part of the initial sync domain bootstrap shown above or dynamically add a new sequencer at a later point as described in operational processes.

Set Up a Distributed Synchronization Domain With Separate Consoles

The process outlined in the previous section only works if all nodes are accessible from the same console environment. If each node has its own isolated console environment, the bootstrapping process must be coordinated in steps with the exchange of data via files using any secure channel of communication between the environments.

Note

Please ensure that all of the nodes in the distributed sync domain are started before proceeding. If you are running the domain manager with auto-init = false, then you need to manually generate its identity using the commands listed in the previous section.

Initially the sync domain manager must transmit its sync domain parameters from its console by saving the parameters to a file. The sync domain ID, serialized as a string, must also be transmitted.

@ domainManager1.service.get_static_domain_parameters.writeToFile("tmp/domain-bootstrapping-files/params.proto")

@ val domainIdString = domainManager1.id.toProtoPrimitive
domainIdString : String = "domainManager1::12201a92491a2e5d17e3b4d0ea7241ea760beec8b64bb1034c593975da363841c60f"

Then the sequencer must receive this file, deserialize it and initialize itself. As part of the initialization, the sequencer creates a signing key pair whose public key it must then transmit via file. Optionally, repeat this for any extra sequencer nodes.

@ val domainParameters = com.digitalasset.canton.admin.api.client.data.StaticDomainParameters.tryReadFromFile("tmp/domain-bootstrapping-files/params.proto")
domainParameters : StaticDomainParameters = StaticDomainParametersV1(
  uniqueContractKeys = true,
  requiredSigningKeySchemes = Set(Ed25519, ECDSA-P256, ECDSA-P384),
  requiredEncryptionKeySchemes = Set(ECIES-P256_HMAC256_AES128-GCM),
  requiredSymmetricKeySchemes = Set(AES128-GCM),
  requiredHashAlgorithms = Set(Sha256),
  requiredCryptoKeyFormats = Set(Tink, Raw, DER),
  protocolVersion = 5
)

@ val domainId = DomainId.tryFromString(domainIdString)
domainId : DomainId = domainManager1::12201a92491a...

@ sequencer1.keys.secret.generate_signing_key(sequencer1.name + "-signing")
res5: SigningPublicKey = SigningPublicKey(id = 1220b326cb64..., format = Tink, scheme = Ed25519)

@ val initResponse = sequencer1.initialization.initialize_from_beginning(domainId, domainParameters)
initResponse : com.digitalasset.canton.domain.sequencing.admin.grpc.InitializeSequencerResponse = InitializeSequencerResponse(
  keyId = "sequencer-id",
  publicKey = SigningPublicKey(id = 1220b326cb64..., format = Tink, scheme = Ed25519),
  replicated = false
)

@ initResponse.publicKey.writeToFile("tmp/domain-bootstrapping-files/seq1-key.proto")

The sync domain manager must then authorize the sequencer’s key. Optionally, repeat this for any extra sequencer keys.

@ val sequencerPublicKey = SigningPublicKey.tryReadFromFile("tmp/domain-bootstrapping-files/seq1-key.proto")
sequencerPublicKey : SigningPublicKey = SigningPublicKey(id = 1220b326cb64..., format = Tink, scheme = Ed25519)

@ domainManager1.setup.helper.authorizeKey(sequencerPublicKey, "sequencer", SequencerId(domainManager1.id))

Now the mediator also needs to create a signing key pair and transmit it. Optionally, repeat this for any extra mediator nodes.

@ mediator1.keys.secret.generate_signing_key(mediator1.name + "-signing").writeToFile("tmp/domain-bootstrapping-files/med1-key.proto")

The sync domain manager must now authorize the mediator’s key and also authorize the mediator to act as part of this sync domain. Optionally, repeat this for any extra mediator nodes.

@ val mediatorKey = SigningPublicKey.tryReadFromFile("tmp/domain-bootstrapping-files/med1-key.proto")
mediatorKey : SigningPublicKey = SigningPublicKey(id = 12201fd701e1..., format = Tink, scheme = Ed25519)

@ val domainId = DomainId.tryFromString(domainIdString)
domainId : DomainId = domainManager1::12201a92491a...

@ domainManager1.setup.helper.authorizeKey(mediatorKey, "mediator1", MediatorId(domainId))

@ domainManager1.topology.mediator_domain_states.authorize(TopologyChangeOp.Add, domainId, MediatorId(domainId), RequestSide.Both)
res14: com.google.protobuf.ByteString = <ByteString@7008854c size=560 contents="\n\255\004\n\333\001\n\326\001\n\323\001\022 s82P0DklMIxRZ7PM06KeSgxUOAoCifjyR...">

After that, still on the sync domain manager’s console, the sync domain manager must collect the list of topology transactions, which includes all the key authorizations and a few other things it needs to broadcast to all sync domain members. This is now saved to a file.

@ domainManager1.topology.all.list().collectOfType[TopologyChangeOp.Positive].writeToFile("tmp/domain-bootstrapping-files/topology-transactions.proto")

The sequencer then reads this set of initial topology transactions and sequences it as the first message to be sequenced in this sync domain. This will allow the sync domain members whose keys were authorized in previous steps to connect to this sequencer and operate with it. The sequencer will then transmit its connection info.

@ val initialTopology = com.digitalasset.canton.topology.store.StoredTopologyTransactions.tryReadFromFile("tmp/domain-bootstrapping-files/topology-transactions.proto").collectOfType[TopologyChangeOp.Positive]
initialTopology : store.StoredTopologyTransactions[TopologyChangeOp.Positive] = Seq(
  StoredTopologyTransaction(
    sequenced = 2024-06-26T12:41:15.934868Z,
    validFrom = 2024-06-26T12:41:15.934868Z,
    op = Add,
    mapping = NamespaceDelegation(
..

@ sequencer1.initialization.bootstrap_topology(initialTopology)

@ SequencerConnections.single(sequencer1.sequencerConnection).writeToFile("tmp/domain-bootstrapping-files/sequencer-connection.proto")

To initialize the mediator, it will need a connection to the sequencer and the sync domain parameters. Optionally, repeat this for any extra mediator nodes.

@ val sequencerConnections = SequencerConnections.tryReadFromFile("tmp/domain-bootstrapping-files/sequencer-connection.proto")
sequencerConnections : SequencerConnections = Sequencer 'DefaultSequencer' -> GrpcSequencerConnection(
  endpoints = http://127.0.0.1:30075,
  transportSecurity = false
)

@ val domainParameters = com.digitalasset.canton.admin.api.client.data.StaticDomainParameters.tryReadFromFile("tmp/domain-bootstrapping-files/params.proto")
domainParameters : StaticDomainParameters = StaticDomainParametersV1(
  uniqueContractKeys = true,
  requiredSigningKeySchemes = Set(Ed25519, ECDSA-P256, ECDSA-P384),
  requiredEncryptionKeySchemes = Set(ECIES-P256_HMAC256_AES128-GCM),
  requiredSymmetricKeySchemes = Set(AES128-GCM),
  requiredHashAlgorithms = Set(Sha256),
  requiredCryptoKeyFormats = Set(Tink, Raw, DER),
  protocolVersion = 5
)

@ mediator1.mediator.initialize(domainId, MediatorId(domainId), domainParameters, sequencerConnections, None)
res21: PublicKey = SigningPublicKey(id = 12201fd701e1..., format = Tink, scheme = Ed25519)

@ mediator1.health.wait_for_initialized()

The sync domain manager will also need a connection to the sequencer in order to complete its initialization.

@ val sequencerConnection = SequencerConnections.tryReadFromFile("tmp/domain-bootstrapping-files/sequencer-connection.proto")
sequencerConnection : SequencerConnections = Sequencer 'DefaultSequencer' -> GrpcSequencerConnection(
  endpoints = http://127.0.0.1:30075,
  transportSecurity = false
)

@ domainManager1.setup.init(sequencerConnection)

@ domainManager1.health.wait_for_initialized()

At this point the distributed sync domain should be completely initialized and a participant should be able to operate on this sync domain by connection to the sequencer.

@ participant1.domains.connect_local(sequencer1)

@ participant1.health.ping(participant1)
res27: Duration = 288 milliseconds

Additionally, please note that if more than one sequencer has been initialized, any mediator node and sychronizer manager can choose to connect to just a subset of them.

Add New Sequencers to a Distributed Synchronization Domain

For non-database-based sequencers such as Ethereum or Fabric sequencers, you can either initialize them as part of the regular distributed sync domain bootstrapping process or dynamically add a new sequencer at a later point as follows:

domainManager1.setup.onboard_new_sequencer(
  initialSequencer = sequencer1,
  newSequencer = sequencer2,
)

Similarly to initializing a distributed sync domain with separate consoles, dynamically onboarding new sequencers (supported by Fabric and Ethereum sequencers) can be achieved in separate consoles as follows:

// Second sequencer's console: write signing key to file
{
  secondSequencer.keys.secret
    .generate_signing_key(s"${secondSequencer.name}-signing")
    .writeToFile(file1)
}

// Domain manager's console: write domain params and current topology
{
  domainManager1.service.get_static_domain_parameters.writeToFile(paramsFile)

  val sequencerSigningKey = SigningPublicKey.tryReadFromFile(file1)

  domainManager1.setup.helper.authorizeKey(
    sequencerSigningKey,
    s"${secondSequencer.name}-signing",
    sequencerId,
  )

  domainManager1.setup.helper.waitForKeyAuthorizationToBeSequenced(
    sequencerId,
    sequencerSigningKey,
  )

  domainManager1.topology.all
    .list(domainId.filterString)
    .collectOfType[TopologyChangeOp.Positive]
    .writeToFile(file1)
}

// Initial sequencer's console: read topology and write snapshot to file
{
  val topologySnapshotPositive =
    StoredTopologyTransactions
      .tryReadFromFile(file1)
      .collectOfType[TopologyChangeOp.Positive]

  val sequencingTimestamp = topologySnapshotPositive.lastChangeTimestamp.getOrElse(
    sys.error("topology snapshot is empty")
  )

  sequencer.sequencer.snapshot(sequencingTimestamp).writeToFile(file2)
}

// Second sequencer's console: read topology, snapshot and domain params
{
  val topologySnapshotPositive =
    StoredTopologyTransactions
      .tryReadFromFile(file1)
      .collectOfType[TopologyChangeOp.Positive]

  val state = SequencerSnapshot.tryReadFromFileUnsafe(file2)

  val domainParameters = StaticDomainParameters.tryReadFromFile(paramsFile)

  secondSequencer.initialization
    .initialize_from_snapshot(
      domainId,
      topologySnapshotPositive,
      state,
      domainParameters,
    )
    .publicKey

  secondSequencer.health.initialized() shouldBe true

}

A newly onboarded sequencer only serves events more recent than the “onboarding snapshot”. In addition some events may belong to transactions initiated before the sequencer was onboarded, but the sequencer is not in a position to sign such events and replaces them with “tombstones”. If a participant (or mediator) connects to a newly onboarded sequencer too soon and the subscription encounters a tombstone, the sequencer subscription aborts with a “FAILED_PRECONDITION” error specifying “InvalidCounter” or “SEQUENCER_TOMBSTONE_ENCOUNTERED”. If this occurs, the participant or mediator should connect to another sequencer with a longer history of sequenced events before switching to the newly onboarded sequencer. To avoid such errors the best practice is to wait at least for the “maximum decision duration” (the sum of the participant_response_timeout and mediator_reaction_timeout dynamic sync domain parameters with a default of 30 seconds each) before connecting nodes to a newly onboarded sequencer.