Daml Integration Kit¶
The Daml integration kit helps third-party ledger developers to implement a Daml Ledger on top of their distributed ledger or database of choice.
We provide the resources in the kit, which include guides to
- Daml Integration Kit status and roadmap
- Implementing your own Daml Ledger
- Deploying a Daml Ledger
- Testing a Daml Ledger
- Benchmarking a Daml Ledger
Using these guides, you can focus on your own distributed-ledger or database and reuse our Daml Ledger server and Daml interpreter code for implementing the Daml Ledger API. For example uses of the integration kit, see below.
Daml Integration Kit status and roadmap¶
The current status of the integration kit is ALPHA. We are working towards BETA, and General Availability (GA) will come quite a bit later. The roadmap below explains what we mean by these different statuses, and what’s missing to progress.
- ALPHA (current status)
In the ALPHA status, the Daml integration kit is ready to be used by third-parties willing to accept the following caveats:
- The architecture includes everything required to run Daml Applications using the Daml Ledger API. However, it misses support for testing Daml Applications in a uniform way against different Daml Ledgers.
- Ledger API authorization, package upload, party on-boarding, ledger reset, and time manipulation are specific to each Daml Ledger, until the uniform administrative Daml ledger access API is introduced, which is different to the uniform per-party Daml ledger access that the Daml Ledger API provides. We will address this before reaching BETA status.
- The architecture is likely to change due to learnings from integrators like you! Where possible we strive to make these changes backwards compatible. though this might not always be possible.
- The documentation might be spotty in some places, and you might have to infer some of the documentation from the code.
- Some of our code might be fresh off the press and might therefore have a higher rate of bugs.
That said: we highly value your feedback and input on where you find Daml software and this integration kit most useful. You can get into contact with us using the feedback form on this documentation page or by creating issues or pull-requests against the digital-asset/daml GitHub repository.
For us, BETA status means that we have architectural stability and solid documentation in place. At this point, third-parties should have everything they need to integrate Daml with their ledger of choice completely on their own.
Before reaching BETA status, we expect to have:
- hardened our test tooling
- built tooling for benchmarking Daml ledgers
- completed several integrations of Daml for different ledgers
- implemented uniform administrative Daml ledger access to provide a portable way for testing Daml applications against different Daml ledgers
For us GA (General Availability) means that there are several production-ready Daml ledgers built using the Daml integration kit. We expect to reach GA in 2019.
Implementing your own Daml Ledger¶
Each X ledger requires at least the implementation of a specific
daml-on-<X>-server, which implements the Daml Ledger API. It might also
require the implementation of a
<X>-daml-validator, which provides the
ability for nodes to validate Daml transactions.
For more about these parts of the architecture, read the Architectural overview.
Before you can decide on an appropriate architecture and implement your own server and validator, you need a significant amount of context about Daml. To acquire this context, you should:
Deciding on the architecture and writing the code¶
Once you have the necessary context, we recommend the steps to implement your own server and validator:
- Clone our example Daml Ledger (which is backed by an in-memory key-value store) from the digital-asset/daml-on-x-example.
- Read the example code jointly with the Architectural overview, Resources we provide, and the Library infrastructure overview below.
- Combine all the knowledge gained to decide on the architecture for your Daml on X ledger.
- Implement your architecture; and let the world know about it by creating a PR against the digital-asset/daml repository to add your ledger to the list of Daml Ledgers built or in development.
If you need help, then feel free to use the feedback form on this documentation page or GitHub issues on the digital-asset/daml repository to get into contact with us.
This section explains the architecture of a Daml ledger backed by a specific ledger X.
The backing ledger can be a proper distributed ledger or also just a database. The goal of a Daml ledger implementation is to allow multiple Daml applications, which are potentially run by different entities, to execute multi-party workflows using the ledger X.
This is a likely architecture for a setup with a distributed ledger:
It assumes that the X ledger allows entities to participate in the evolution of the ledger via particular nodes. In the remainder of this documentation, we call these nodes participant nodes.
In the diagram:
The boxes labeled daml-on-<X>-server denote the Daml Ledger API servers, which implement the Daml Ledger API on top of the services provided by the X participant nodes.
The boxes labeled <X>-daml-validator denote X-specific Daml transaction validation services. In a distributed ledger they provide the ability for nodes to validate Daml transactions at the appropriate stage in the X ledger’s transaction commit process.
Whether they are needed, by what nodes they are used, and whether they are run in-process or out-of-process depends on the X ledger’s architecture. Above we depict a common case where the participant nodes jointly maintain the ledger’s integrity and therefore need to validate Daml transactions.
Resources we provide¶
Scala libraries for validating Daml transactions and serving the Ledger API given implementations of two specific interfaces. See the Library infrastructure overview for an overview of these libraries.
A complete example of a Daml Ledger backed by an in-memory key-value store, in the digital-asset/daml-on-x-example GitHub repository. It builds on our Scala libraries and demonstrates how they can be assembled to serve the Ledger API and validate Daml transactions.
For ledgers where data is shared between all participant nodes, we recommend using this example as a starting point for implementing your server and validator.
For ledgers with stronger privacy models, this example can serve as an inspiration. You will need to dive deeper into how transactions are represented and how to communicate them to implement Daml’s privacy model at the ledger level instead of just at the Ledger API level.
Library infrastructure overview¶
To help you implement your server and validator, we provide the following four Scala libraries as part of Daml Connect. Changes to them are explained as part of the Release Notes.
Contains interfaces abstracting over the state of a participant node relevant for a Daml Ledger API server.
These are the interfaces whose implementation is specific to a particular X ledger. These interfaces are optimized for ease of implementation.
These utilities provide methods to succinctly implement interfaces from
participant-state.jaron top of a key-value state storage.
See documentation in package.scala
ledger-api-server.jar(source code for API server, source code for indexer)
- Contains code that implements a Daml Ledger API server and the SQL-backed indexer
given implementations of the interfaces in
Contains code for serializing and deserializing Daml transactions and for validating them.
An <X>-daml-validator is typically implemented by wrapping this code in the X-ledger’s SDK for building transaction validators.
daml-engine.jaralso contains code for interpreting commands sent over the Ledger API. It is used by the daml-on-<X>-server to construct the transactions submitted to its participant node.
This diagram shows how the classes and interfaces provided by these libraries are typically combined to instantiate a Daml Ledger API server backed by an X ledger:
TODO: Update this diagram to mention ledger server classes above instead of deprecated daml-on-x-server
In the diagram above:
- Boxes labeled with fully qualified class names denote class instances.
- Solid arrows labeled with fully qualified interface names denote that an instance depends on another instance providing that interface.
- Dashed arrows denote that a class instance provides or depends on particular services.
- Boxes embedded in other boxes denote that the outer class instance creates the contained instances.
Explaining this diagram in detail (for brevity, we drop prefixes of their qualified names where unambiguous):
- Ledger API
- is the collection of gRPC services that you would like your daml-on-<X>-server to provide.
- are the services provided by which underly your ledger, which you aim to leverage to build your daml-on-<X>-server.
- is the class whose main method or constructor
creates the contained instances and wires them up to provide the Ledger API
backed by the
<X> services. You need to implement this for your Daml on X ledger.
- is an interface abstracting over the mechanism to submit Daml transactions to the underlying X ledger via a participant node.
- is an interface abstracting over the ability to subscribe to changes of the X ledger visible to a particular participant node. The changes are exposed as a stream that is resumable from any particular offset, which supports restarts of the consumer. We typically expect there to be a single consumer of the data provided on this interface. That consumer is responsible for assembling the streamed changes into a view onto the participant state suitable for querying.
- is a class implementing the
WriteServiceon top of the
<X> services. You need to implement this for your Daml on X ledger.
- is a standalone service that subscribe to ledger changes using
ReadServiceand inserts the data into a SQL backend (“index”) for the purpose of serving the data over the Ledger API.
- is a class containing all the code to implement the
Ledger API on top of an ledger backend. It serves the data from a SQL database populated by the
Deploying a Daml Ledger¶
- explain recommended approach for Ledger API authorization (GitHub issue)
- explain option of using a persistent SQL-backed participant state index (GitHub issue).
- explain how testing of Daml applications (ledger reset, time manipulation, scripted package upload) can be supported by a uniform admin interface (GitHub issue).
Testing a Daml Ledger¶
You can test your Daml ledger implementation using Ledger API Test Tool, which will assess correctness of implementation of the Ledger API. For example, it will show you if there are consistency or conformance problem with your implementation.
Assuming that your Ledger API endpoint is accessible at
localhost:6865, you can use the tool in the following manner:
Download the Ledger API Test Tool from Maven and save it as
ledger-api-test-tool.jarin your current directory.
Obtain the Daml archives required to run the tests:
java -jar ledger-api-test-tool.jar --extract
.darfiles extracted in the current directory into your Ledger.
Run the tool against your ledger:
java -jar ledger-api-test-tool.jar localhost:6865
See more in Ledger API Test Tool.