Manage Synchronization Domains

Permissioned Synchronization Domains

Important

Daml Enterprise license required

Canton as a network is an open virtual shared ledger. Whoever runs a Canton participant node is part of the same virtual shared ledger. However, the network itself is made up of sync domains that are used by participants to run the Canton protocol and communicate to their peers. Such sync domains can be open, allowing any participant with access to a sequencer node to enter and participate in the network. But sync domains can also be permissioned, where the operator of the sync domain topology managers needs to explicitly add the participant to the allow-list before the participant can register with a sync domain.

While the Canton architecture is designed to be resilient against malicious participants, there can never be a guarantee that the implementation of said architecture is absolutely secure. Therefore, it makes sense for most networks to impose control on which participant can be part of the network.

The first layer of control is given by securing access to the public API of the sequencers in the network. This can be done using standard network tools such as firewalls and virtual private networks.

The second layer of control is given by setting the appropriate configuration flag of the sync domain manager (or sync domain):

canton.domain-managers.domainManager1 {
  topology.open = false
  domain-parameters.protocol-version = 5
}

Assuming we have set up a sync domain with this flag turned off, the config for that particular sync domain would read:

@ val config = DomainConnectionConfig("mydomain", sequencer1.sequencerConnection)
config : DomainConnectionConfig = DomainConnectionConfig(
  domain = Domain 'mydomain',
  sequencerConnections = Sequencer 'DefaultSequencer' -> GrpcSequencerConnection(
    endpoints = http://127.0.0.1:30007,
    transportSecurity = false
..

When a participant attempts to join the sync domain, it will be rejected:

@ participant1.domains.register(config)
ERROR com.digitalasset.canton.integration.EnterpriseEnvironmentDefinition$$anon$3 - Request failed for participant1.
  GrpcRequestRefusedByServer: FAILED_PRECONDITION/PARTICIPANT_IS_NOT_ACTIVE(9,5799e3ae): The participant is not yet active
  Request: RegisterDomain(DomainConnectionConfig(
  domain = Domain 'mydomain',
  sequencerConnections = Sequencer 'DefaultSequencer' -> GrpcSequencerConnection(endpoints = http://127.0.0.1:30007, transportSecurity = false),
  manualConnect = false
))
  CorrelationId: 5799e3aeeb9b25587bb76d12c094080f
  Context: HashMap(participant -> participant1, test -> ManagePermissionedDomainsDocumentationManual, serverResponse -> Domain Domain 'mydomain' has rejected our on-boarding attempt, domain -> mydomain, tid -> 5799e3aeeb9b25587bb76d12c094080f)
  Command ParticipantAdministration$domains$.register invoked from cmd10000006.sc:1

In order to allow the participant to join the sync domain, we must first actively enable it on the topology manager. We assume now that the operator of the participant extracts its id into a string:

@ val participantAsString = participant1.id.toProtoPrimitive
participantAsString : String = "PAR::participant1::1220fc0ca9e40a300168eeae518d5d68be93716e046ff59403625074daa7f0b7ea93"

and communicates this string to the operator of the sync domain topology manager:

@ val participantIdFromString = ParticipantId.tryFromProtoPrimitive(participantAsString)
participantIdFromString : ParticipantId = PAR::participant1::1220fc0ca9e4...

This topology manager can now add the participant by enabling it:

@ domainManager1.participants.set_state(participantIdFromString, ParticipantPermission.Submission, TrustLevel.Ordinary)

Note that the participant is not active yet:

@ domainManager1.participants.active(participantIdFromString)
res5: Boolean = false

So far, what we’ve done with setting the state is to issue a “sync domain trust certificate”, where the sync domain topology manager declares that it trusts the participant enough to become a participant of the sync domain. We can inspect this certificate using:

@ domainManager1.topology.participant_domain_states.list(filterStore="Authorized").map(_.item)
res6: Seq[ParticipantState] = Vector(
  ParticipantState(
    From,
    domainManager1::1220ba342473...,
    PAR::participant1::1220fc0ca9e4...,
    Submission,
    Ordinary
  )
)

In order to have the participant become active on the sync domain, we need to register the signing keys and the “sync domain trust certificate” of the participant. The certificate is generated by the participant automatically and sent to the sync domain during the initial handshake.

We can trigger that handshake again by attempting to reconnect to the sync domain:

@ participant1.domains.reconnect_all()

Now, we can check that the participant is active:

@ domainManager1.participants.active(participantIdFromString)
res8: Boolean = true

We can also observe that we now have both sides of the sync domain trust certificate, the From and the To:

@ domainManager1.topology.participant_domain_states.list(filterStore="Authorized").map(_.item)
res9: Seq[ParticipantState] = Vector(
  ParticipantState(
    From,
    domainManager1::1220ba342473...,
    PAR::participant1::1220fc0ca9e4...,
    Submission,
    Ordinary
  ),
  ParticipantState(
    To,
    domainManager1::1220ba342473...,
    PAR::participant1::1220fc0ca9e4...,
    Submission,
    Ordinary
  )
)

Finally, the participant is healthy and can use the sync domain:

@ participant1.health.ping(participant1)
res10: Duration = 897 milliseconds

Synchronization Domain Rules

Every sync domain has its own rules in terms of what parameters are used by the participants while running the protocol. The participants obtain these parameters before connecting to the sync domain. They can be configured using the specific parameter section. An example would be:

init.domain-parameters {
  // example setting
  unique-contract-keys = yes
}

The full set of available parameters can be found in the scala reference documentation.

Dynamic synchronization domain parameters

In addition to the parameters that are specified in the configuration, some parameters can be changed at runtime (i.e., while the sync domain is running); these are called dynamic sync domain parameters. When the sync domain is bootstrapped, default values are used for the dynamic sync domain parameters. They can be changed subsequently using the console commands described below.

A participant can get the current parameters on a sync domain it is connected to using the following command:

mydomain.service.get_dynamic_domain_parameters

Parameters that were transitioned from static to dynamic with protocol version 4 need to be retrieved individually:

mydomain.service.get_reconciliation_interval
mydomain.service.get_max_rate_per_participant
mydomain.service.get_max_request_size
mydomain.service.get_mediator_deduplication_timeout

Dynamic parameters can be set individually using:

mydomain.service.set_reconciliation_interval(5.seconds)
mydomain.service.set_max_rate_per_participant(100)
mydomain.service.set_max_request_size(100000)
mydomain.service.set_mediator_deduplication_timeout(2.minutes)

Alternatively, several can be set at the same time:

mydomain.service.update_dynamic_domain_parameters(
  _.update(
    participantResponseTimeout = 10.seconds,
    topologyChangeDelay = 1.second,
  )
)

Note

When increasing max request size, the sequencer nodes need to be restarted for the new value to be taken into account. If the sync domain is not distributed, it means that the sync domain node needs to be restarted.

Recover From a Small Max Request Size

MaxRequestSize is a dynamic parameter starting from protocol version 4. This parameter configures both the gRPC channel size on the sequencer node and the maximum size that a sequencer client is allowed to transfer.

If the parameter is set to a very small value (roughly under 30kb), Canton can crash because all messages are rejected by the sequencer client or by the sequencer node. This cannot be corrected by setting a higher value within the console, because this change request needs to be sent via the sequencer and will also be rejected.

To recover from this crash, you need to configure override-max-request-size on both the sequencer node and the sequencer clients.

On a non-distributed deployment, this means modifying both the sync domain and the participant configuration as follows:

domains {
  da {
    # overrides the maxRequestSize in bytes on the sequencer node
    public-api.override-max-request-size = 30000
    sequencer-client.override-max-request-size = 30000
  }
}
participants {
  participant1 {
    sequencer-client.override-max-request-size = 30000
  }
  participant2 {
    sequencer-client.override-max-request-size = 30000
  }
}

On a distributed deployment, for each sync domain entity deployed on its own node, you will need to override the max-request-size as follows:

domain-managers {
  domainManager1 {
    sequencer-client.override-max-request-size = 30000
  }
}
participants {
  participant1 {
    sequencer-client.override-max-request-size = 30000
  }
  participant2 {
    sequencer-client.override-max-request-size = 30000
  }
}
mediators {
  mediator1 {
    sequencer-client.override-max-request-size = 30000
  }
}
sequencers {
  sequencer1 {
    # overrides the maxRequestSize in bytes on the sequencer node
    public-api.override-max-request-size = 30000
    sequencer-client.override-max-request-size = 30000
  }
}

After the configuration is modified, disconnect all the participants from the sync domain and then restart all nodes.

On a non-distributed deployment, you can stop Canton by following these steps:

participants.all.domains.disconnect(da.name)
nodes.local.stop()

On a distributed deployment, you can stop Canton by following these steps:

participants.all.domains.disconnect(sequencer1.name)
nodes.local.stop()

Then perform the restart:

nodes.local.start()
participants.all.domains.reconnect_all()

Once Canton has recovered, use the admin command to set the maxRequestSize value, then delete the added configuration in the previous step, and finally perform the restart again.