DAML Integration Kit - ALPHA

DAML Applications run on DAML Ledgers. A DAML Ledger is a server serving the Ledger API as per the semantics defined in the DA Ledger Model and the DAML-LF specification.

The DAML Integration Kit helps third-party ledger developers to implement a DAML Ledger on top of their distributed ledger or database of choice.

We provide the resources in the kit, which include guides to

Using these guides, you can focus on your own distributed-ledger or database and reuse our DAML Ledger server and DAML interpreter code for implementing the DAML Ledger API. For example uses of the integration kit see DAML Ledgers built or in development.

DAML Ledgers built or in development

The following table lists the ledgers that support DAML, or are implementing support for running DAML.

Ledger Status Developer (Managed Offering) More Information
VMware Blockchain In Development VMware press release :: April 2019
Hyperledger Sawtooth In Development Blockchain Technology Partners (Sextant) press release :: April 2019
Hyperledger Fabric In Development Hacera (Hacera Unbounded Network) press release :: June 2019
R3 Corda In Development Digital Asset press release :: June 2019
Amazon Aurora In Development Blockchain Technology Partners (Sextant) press release :: June 2019
DAML Sandbox Stable Digital Asset supports PostgreSQL, see section on persistence in DAML Sandbox docs
Canton In Development Digital Asset www.canton.io :: native support for DAML’s fine-grained privacy model.

DAML Integration Kit status and roadmap

The current status of the integration kit is ALPHA. We are working towards BETA, and General Availability (GA) will come quite a bit later. The roadmap below explains what we mean by these different statuses, and what’s missing to progress.

ALPHA (current status)

In the ALPHA status, the DAML Integration Kit is ready to be used by third-parties willing to accept the following caveats:

  • The architecture includes everything required to run DAML Applications using the DAML Ledger API. However, it misses support for testing DAML Applications in a uniform way against different DAML Ledgers.
  • Ledger API authorization, package upload, party on-boarding, ledger reset, and time manipulation are specific to each DAML Ledger, until the uniform administrative DAML ledger access API is introduced, which is different to the uniform per-party DAML ledger access that the DAML Ledger API provides. We will address this before reaching BETA status.
  • The architecture is likely to change due to learnings from integrators like you! Where possible we strive to make these changes backwards compatible. though this might not always be possible.
  • The documentation might be spotty in some places, and you might have to infer some of the documentation from the code.
  • Some of our code might be fresh off the press and might therefore have a higher rate of bugs.

That said: we highly value your feedback and input on where you find DAML software and this integration kit most useful. You can get into contact with us using the feedback form on this documentation page or by creating issues or pull-requests against the digital-asset/daml GitHub repository.

BETA

For us, BETA status means that we have architectural stability and solid documentation in place. At this point, third-parties should have everything they need to integrate DAML with their ledger of choice completely on their own.

Before reaching BETA status, we expect to have:

  • hardened our test tooling
  • built tooling for benchmarking DAML ledgers
  • completed several integrations of DAML for different ledgers
  • implemented uniform administrative DAML ledger access to provide a portable way for testing DAML applications against different DAML ledgers

Related links

GA

For us GA (General Availability) means that there are several production-ready DAML ledgers built using the DAML Integration Kit. We expect to reach GA in 2019.

Related links

Implementing your own DAML Ledger

Each X ledger requires at least the implementation of a specific daml-on-<X>-server, which implements the DAML Ledger API. It might also require the implementation of a <X>-daml-validator, which provides the ability for nodes to validate DAML transactions.

For more about these parts of the architecture, read the Architectural overview.

Step-by-step guide

Prerequisite knowledge

Before you can decide on an appropriate architecture and implement your own server and validator, you need a significant amount of context about DAML. To acquire this context, you should:

  1. Complete the Quickstart guide.
  2. Get an in-depth understanding of the DA Ledger Model.
  3. Build a mental model of how the Ledger API is used to build DAML Applications.

Deciding on the architecture and writing the code

Once you have the necessary context, we recommend the steps to implement your own server and validator:

  1. Clone our example DAML Ledger (which is backed by an in-memory key-value store) from the digital-asset/daml-on-x-example.
  1. Read the example code jointly with the Architectural overview, Resources we provide, and the Library infrastructure overview below.
  1. Combine all the knowledge gained to decide on the architecture for your DAML on X ledger.
  1. Implement your architecture; and let the world know about it by creating a PR against the digital-asset/daml repository to add your ledger to the list of DAML Ledgers built or in development.

If you need help, then feel free to use the feedback form on this documentation page or GitHub issues on the digital-asset/daml repository to get into contact with us.

Architectural overview

This section explains the architecture of a DAML ledger backed by a specific ledger X.

The backing ledger can be a proper distributed ledger or also just a database. The goal of a DAML ledger implementation is to allow multiple DAML applications, which are potentially run by different entities, to execute multi-party workflows using the ledger X.

This is a likely architecture for a setup with a distributed ledger:

../_images/architecture-overview.svg

It assumes that the X ledger allows entities to participate in the evolution of the ledger via particular nodes. In the remainder of this documentation, we call these nodes participant nodes.

In the diagram:

  • The boxes labeled daml-on-<X>-server denote the DAML Ledger API servers, which implement the DAML Ledger API on top of the services provided by the X participant nodes.

  • The boxes labeled <X>-daml-validator denote X-specific DAML transaction validation services. In a distributed ledger they provide the ability for nodes to validate DAML transactions at the appropriate stage in the X ledger’s transaction commit process.

    Whether they are needed, by what nodes they are used, and whether they are run in-process or out-of-process depends on the X ledger’s architecture. Above we depict a common case where the participant nodes jointly maintain the ledger’s integrity and therefore need to validate DAML transactions.

Message flow

TODO (BETA):

  • explain to readers the life of a transaction at a high-level, so they have a mental framework in place when looking at the example code. (GitHub issue)

Resources we provide

  • Scala libraries for validating DAML transactions and serving the Ledger API given implementations of two specific interfaces. See the Library infrastructure overview for an overview of these libraries.

  • A complete example of a DAML Ledger backed by an in-memory key-value store, in the digital-asset/daml-on-x-example GitHub repository. It builds on our Scala libraries and demonstrates how they can be assembled to serve the Ledger API and validate DAML transactions.

    For ledgers where data is shared between all participant nodes, we recommend using this example as a starting point for implementing your server and validator.

    For ledgers with stronger privacy models, this example can serve as an inspiration. You will need to dive deeper into how transactions are represented and how to communicate them to implement DAML’s privacy model at the ledger level instead of just at the Ledger API level.

Library infrastructure overview

To help you implement your server and validator, we provide the following four Scala libraries as part of the DAML SDK. Changes to them are explained as part of the Release notes.

As explained in Deciding on the architecture and writing the code, this section is best read jointly with the code in digital-asset/daml-on-x-example.

participant-state.jar (source code)

Contains interfaces abstracting over the state of a participant node relevant for a DAML Ledger API server.

These are the interfaces whose implementation is specific to a particular X ledger. These interfaces are optimized for ease of implementation.

participant-state-index.jar (source code)

Contains code for reading the abstract state of a participant node and indexing it to satisfy the read access patterns required for serving the Ledger API.

The library provides both an interface enumerating all read access methods and an in-memory reference implementation of that interface. We expect to provide a persistent, SQL-backed index in the future (tracking GitHub issue).

participant-state-kvutils.jar (source code)

These utilities provide methods to succintly implement interfaces from participant-state.jar on top of a key-value state storage.

See documentation in package.scala

api-server-damlonx.jar (source code)
Contains code that implements a DAML Ledger API server given implementations of the interfaces in participant-state.jar.
daml-engine.jar (source code)

Contains code for serializing and deserializing DAML transactions and for validating them.

An <X>-daml-validator is typically implemented by wrapping this code in the X-ledger’s SDK for building transaction validators. daml-engine.jar also contains code for interpreting commands sent over the Ledger API. It is used by the daml-on-<X>-server to construct the transactions submitted to its participant node.

This diagram shows how the classes and interfaces provided by these libraries are typically combined to instantiate a DAML Ledger API server backed by an X ledger:

../_images/server-classes-and-interfaces.svg

In the diagram above:

  • Boxes labeled with fully qualified class names denote class instances.
  • Solid arrows labeled with fully qualified interface names denote that an instance depends on another instance providing that interface.
  • Dashed arrows denote that a class instance provides or depends on particular services.
  • Boxes embedded in other boxes denote that the outer class instance creates the contained instances.

Explaining this diagram in detail (for brevity, we drop prefixes of their qualified names where unambiguous):

Ledger API
is the collection of gRPC services that you would like your daml-on-<X>-server to provide.
<X> services
are the services provided by which underly your ledger, which you aim to leverage to build your daml-on-<X>-server.
<x>.LedgerApiServer
is the class whose main method or constructor creates the contained instances and wires them up to provide the Ledger API backed by the <X> services. You need to implement this for your DAML on X ledger.
WriteService (source code)
is an interface abstracting over the mechanism to submit DAML transactions to the underlying X ledger via a participant node.
ReadService (source code)
is an interface abstracting over the ability to subscribe to changes of the X ledger visible to a particular participant node. The changes are exposed as a stream that is resumable from any particular offset, which supports restarts of the consumer. We typically expect there to be a single consumer of the data provided on this interface. That consumer is responsible for assembling the streamed changes into a view onto the participant state suitable for querying.
<x>.Backend
is a class implementing the ReadService and the WriteService on top of the <X> services. You need to implement this for your DAML on X ledger.
IndexService (source code)
is an interface specific to the needs that the damlonx.Server class has for querying the participant state exposed by the ReadService. It contains methods for all the different read access patterns the Server uses to serve the Ledger API. We include it in this diagram, as in the future there will be choice on what implementation of the IndexService to choose.
index.v1.impl.reference.Indexer (source code)
is an in-memory implementation of the IndexService interface. We recommend using that until the SQL-based index service is ready. See this GitHub issue for its status.
damlonx.Server (source code)
is a class containing all the code to implement the Ledger API on top of an IndexService and a WriteService. Its constructor also takes additional arguments for configuring among others logging and the port at which the Ledger API is served.

Deploying a DAML Ledger

TODO (BETA):

  • explain recommended approach for Ledger API authorization (GitHub issue)
  • explain option of using a persistent SQL-backed participant state index (GitHub issue).
  • explain how testing of DAML applications (ledger reset, time manipulation, scripted package upload) can be supported by a uniform admin interface (GitHub issue).

Testing a DAML Ledger

You can test your DAML ledger implementation using Ledger API Test Tool, which will assess correctness of implementation of the Ledger API. For example, it will show you if there are consistency or conformance problem with your implementation.

Assuming that your Ledger API endpoint is accessible at localhost:6865, you can use the tool in the following manner:

  1. Obtain the tool:

    curl -L 'https://bintray.com/api/v1/content/digitalassetsdk/DigitalAssetSDK/com/daml/ledger/testtool/ledger-api-test-tool_2.12/$latest/ledger-api-test-tool_2.12-$latest.jar?bt_package=sdk-components' -o ledger-api-test-tool.jar

  2. Obtain the DAML archives required to run the tests:

    java -jar ledger-api-test-tool.jar --extract

  3. Load all .dar files extracted in the current directory into your Ledger.

  4. Run the tool against your ledger:

    java -jar ledger-api-test-tool.jar -h localhost -p 6865

See more in Ledger API Test Tool.

Benchmarking a DAML Ledger

TODO (BETA):

  • explain how to use the ledger-api-bench tool to evaluate the performance of your implementation of the Ledger API (GitHub issue).