Frequently Asked Questions

This section covers other questions that frequently arise when using Canton. If your question is not answered here, consider searching the Daml forum and creating a post if you can’t find the answer.

Log Messages

Database task queue full

If you see the log message:

java.util.concurrent.RejectedExecutionException:
Task slick.basic.BasicBackend$DatabaseDef$@... rejected from slick.util.AsyncExecutorWithMetrics$$...
[Running, pool size = 25, active threads = 25, queued tasks = 1000, completed tasks = 181375]

It is likely that the database task queue is full. You can check this by inspecting the log message: if the logged queued tasks is equal to the limit for the database task queue, then the task queue is full. This error message does not indicate that anything is broken, and the task will be retried after a delay.

If the error occurs frequently, consider increasing the size of the task queue:

canton.participants.participant1.storage.config.queueSize = 10000

A higher queue size can lead to better performance, because it avoids the overhead of retrying tasks; on the flip side, a higher queue size comes with higher memory usage.

Console Commands

I received an error saying that the DomainAlias I used was too long. Where I can see the limits of String types in Canton?

Bootstrap Scripts

Why do you have an additional new line between each line in your example scripts?

  • When we write participant1 start the scala compiler translates this into participant1.start(). This works great in the console when each line is parsed independently. However with a script all of it’s content is parsed at once, and in which case if there is anything on the line following participant1 start it will assume it is an argument for start and fail. An additional newline prevents this. Adding parenthesis would also work.

How can I use nested import statements to split my script into multiple files?

  • Ammonite supports splitting scripts into several files using two mechanisms. The old one is interp.load.module(..). The new one is import $file.<fname>. The former will compile the module as a whole, which means that variables defined in one module can not be used in another one as they are not available during compilation. The import $file. syntax however will make all variables accessible in the importing script. However, it only works with relative paths as e.g. ../path/to/foo/bar.sc needs to be converted into import $file.^.path.to.foo.bar and it only works if the script file is named with suffix .sc.

How do I write data to a file and how do I read it back?

  • Canton uses Protobuf for serialization and as a result, you can leverage Protobuf to write objects to a file. Here is a basic example:
       // Obtain the last event.
       val lastEvent: PossiblyIgnoredProtocolEvent =
         participant1.testing.state_inspection
           .findMessage(da.name, LatestUpto(CantonTimestamp.MaxValue))
           .getOrElse(throw new NoSuchElementException("Unable to find last event."))

       // Dump the last event to a file.
       utils.write_to_file(lastEvent.toProtoV0, dumpFilePath)

       // Read the last event back from the file.
       val dumpedLastEventP: v0.PossiblyIgnoredSequencedEvent =
         utils.read_first_message_from_file[v0.PossiblyIgnoredSequencedEvent](
           dumpFilePath
         )

       val dumpedLastEventOrErr: Either[
         ProtoDeserializationError,
         PossiblyIgnoredProtocolEvent,
       ] =
         PossiblyIgnoredSequencedEvent
           .fromProtoV0(cryptoPureApi(participant1.config))(dumpedLastEventP)


- You can also dump several objects to the same file:

       // Obtain all events.
       val allEvents: Seq[PossiblyIgnoredProtocolEvent] =
         participant1.testing.state_inspection.findMessages(da.name, None, None, None)

       // Dump all events to a file.
       utils.write_to_file(allEvents.map(_.toProtoV0), dumpFilePath)

       // Read the dumped events back from the file.
       val dumpedEventsP: Seq[v0.PossiblyIgnoredSequencedEvent] =
         utils.read_all_messages_from_file[v0.PossiblyIgnoredSequencedEvent](
           dumpFilePath
         )

       val dumpedEventsOrErr: Seq[Either[
         ProtoDeserializationError,
         PossiblyIgnoredProtocolEvent,
       ]] =
         dumpedEventsP.map {
           PossiblyIgnoredSequencedEvent.fromProtoV0(cryptoPureApi(participant1.config))(_)
         }


- Some classes do not have a (public) ``toProto*`` method, but they can be serialized to a
  `ByteString <https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/ByteString>`__
  instead. You can dump the corresponding instances as follows:

// Obtain the last acs commitment.
val lastCommitment: AcsCommitment = participant1.commitments
  .received(
    da.name,
    CantonTimestamp.MinValue.toInstant,
    CantonTimestamp.MaxValue.toInstant,
  )
  .lastOption
  .getOrElse(
    throw new NoSuchElementException("Unable to find an acs commitment.")
  )
  .message

// Dump the commitment to a file.
utils.write_to_file(
  lastCommitment.toByteString(ProtocolVersion.latestForTest),
  dumpFilePath,
)

// Read the dumped commitment back from the file.
val dumpedLastCommitmentBytes: ByteString =
  utils.read_byte_string_from_file(dumpFilePath)

val dumpedLastCommitmentOrErr: Either[
  ProtoDeserializationError,
  AcsCommitment,
] =
  AcsCommitment.fromByteString(dumpedLastCommitmentBytes)

How to Setup Canton to Get Best Performance?

In this section, the findings from our internal performance tests are outlined to help you achieve best performance for your Canton application.

System Design / Architecture

Make sure to use Canton Enterprise because it is heavily optimized when compared with the community edition.

Plan your topology such that your DAML parties can be partitioned into independent blocks. That means, most of your DAML commands involve parties of a single block only. It is ok if some commands involve parties of several (or all) blocks, as long as this happens only very rarely. In particular, avoid having a single master party that is involved in every command, because that party would become a bottleneck of the system.

If your participants are becoming a bottleneck, add more participant nodes to your system. Make sure that each block runs on its own participant. If your domain(s) are becoming a bottleneck, add more domain nodes and distribute the load evenly over all domains.

Prefer sending big commands with multiple actions (creates / exercise) over sending numerous small commands. Avoid sending unnecessary commands through the ledger API. Try to minimize the payload of commands.

Further information can be found in Section Scaling and Performance.

Hardware and Database

Do not run Canton nodes with an in-memory storage or with an H2 storage in production or during performance tests. You may observe very good performance in the beginning, but performance can degrade substantially once the data stores fill up.

Measure memory usage, CPU usage and disk throughput and improve your hardware as needed. For simplicity, it makes sense to start on a single machine. Once the resources of a machine are becoming a bottleneck, distribute your nodes and databases to different machines.

Try to make sure that the latency between a Canton node and its database is very low (ideally in the order of microseconds). Prefer hosting a Canton node and its database on the same machine. This is likely faster than running several Canton nodes on the same machine and the databases on a separate machine; for, the latency between Canton nodes is much less performance critical than the latency between a Canton node and its database.

Optimize the configuration of your database, and make sure the database has sufficient memory and is stored on SSD disks with a very high throughput. For Postgres, this online tool is a good starting point for finding reasonable parameters.

Configuration

In the following, we go through the parameters with known impact on performance.

Timeouts. Under high load, you may observe that commands timeout. This will negatively impact throughput, because the commands consume resources without contributing to the number of accepted commands. To avoid this situation increase timeout parameters from the Canton console:

myDomain.service.update_dynamic_parameters(
  _.copy(
    participantResponseTimeout = TimeoutDuration.ofSeconds(60),
    mediatorReactionTimeout = TimeoutDuration.ofSeconds(60),
  )
)

If timeouts keep occurring, change your setup to submit commands at a lower rate. In addition, take the next paragraph on resource limits into account.

Configure generous resource limits. Resource limits are used to prevent ledger applications from overloading Canton by sending commands at an excessive rate. While they may be required to protect the system from denial of service attacks in a production environment, they can get in the way when doing performance measurements. Resource limits can be configured as follows from the Canton console:

participant1.resources.set_resource_limits(
  ResourceLimits(
    maxDirtyRequests = Some(10000),
    maxRate = Some(10000),
  )
)

Size of connection pools. Make sure that every node uses a connection pool to communicate with the database. This avoids the extra cost of creating a new connection on every database query. Canton chooses a suitable connection pool by default. Configure the maximum number of connections such that the database is fully loaded, but not overloaded. Detailed instructions can be found in the Section Max Connection Settings.

Size of database task queue. If you are seeing frequent RejectedExecutionExceptions when Canton queries the database, increase the size of the task queue, as described in Section Database task queue full.

JVM heap size. In case you observe OutOfMemoryErrors or high overhead of garbage collection, you must increase the heap size of the JVM, as described in Section Java Virtual Machine Arguments. Use tools of your JVM provider (such as VisualVM) to monitor the garbage collector to check whether the heap size is tight.

Size of thread pools. Every Canton process has a thread pool for executing internal tasks. By default, the size of the thread-pool is configured as the number of (virtual) cores of the underlying (physical) machine. If the underlying machine runs other processes (e.g., a database) or if Canton runs inside of a container, the thread-pool may be too big, resulting in excessive context switching. To avoid that, configure the size of the thread pool explicitly like this:

"bin/canton -Dscala.concurrent.context.numThreads=12 --config examples/01-simple-topology/simple-topology.conf"

As a result, Canton will log the following line:

"INFO  c.d.c.e.EnterpriseEnvironment - Deriving 12 as number of threads from '-Dscala.concurrent.context.numThreads'."

Asynchronous commits. If you are using a Postgres database, configure the participant’s ledger api server to commit database transactions asynchronously by including the following line into your Canton configuration:

canton.participants.participant1.ledger-api.synchronous-commit-mode = off

Log level. Make sure that Canton outputs log messages only at level INFO and above.

Disable additional consistency checks. Additional consistency-checks would degrade performance substantially. Make sure they are disabled by including the following line into your Canton configuration:

canton.parameters.enable-additional-consistency-checks = false

Why is Canton complaining about my database version?

Postgres

Canton is tested with Postgres 10, 11, 12, 13, and 14 – so these are the recommended versions. Canton is also likely to work with any higher versions, but will WARN when a higher version is encountered. By default, Canton will not start when the Postgres version is below 10.

Oracle

Canton Enterprise additionally supports using Oracle for storage. Only Oracle 19 has been tested, so by default Canton will not start if the Oracle version is not 19.

Note that Canton’s version checks use the v$$version table so, for the version check to succeed, this table must exist and the database user must have SELECT privileges on the table.

Using non-standard database versions

Canton’s database version checks can be disabled with the following config option:

canton.parameters.non-standard-config = "yes"

Note that this will disable all “standard config” checks, not just those for the database.