Production Setup¶
The vast majority of the prior documentation focuses on ease of testing and running the service in a dev environment. From a production perspective given the wide variety of use-cases there is far less of an established framework for deploying the HTTP-JSON API server. In this document we would try to list some recommendations for production deployments.
The HTTP-JSON API server is a JVM application that by default uses an in-memory backend. This in-memory backend setup is inefficient for larger datasets as for every query it ends up fetching the entire active contract set for the templates referenced in that query. For this reason for production setups at a minimum we recommend to use a database as a Query Store, this will allow for more efficient caching of the data to improve query performance. Details for enabling a query store are highlighted below.
Query Store¶
Note
The Community Edition of Daml Connect only supports PostgreSQL backends for the HTTP JSON API server, but the Enterprise Edition also supports Oracle backends.
Query Store can be described as a cached search index and is useful for use-cases where we need to query large active contract sets(ACS). The HTTP-JSON API server can be configured with PostgreSQL/Oracle(Enterprise Edition only) as the query-store backend.
The Query Store is built by saving the state of the ACS up to the current ledger offset. This allows the HTTP-JSON API to only request the delta on subsequent queries, making it much faster than having to request the entire ACS every time.
Given the cache like semantics of the Query Store it is safe to drop and re-initialize the store at any point, though there may be a temporary performance impact while the Query Store is being rebuilt.
For example to enable the PostgreSQL backend you can use the --query-store-jdbc-config flag, as shown below.
daml json-api --ledger-host localhost --ledger-port 6865 --http-port 7575 \
--query-store-jdbc-config "driver=org.postgresql.Driver,url=jdbc:postgresql://localhost:5432/test?&ssl=true,user=postgres,password=password,start-mode=start-only"
If you are not familiar with JDBC URLs, we recommend reading the PostgreSQL JDBC documentation for more information.
The start-mode above simplifies re-initializing the database as well as allowing
for usage of separate credentials.
For example you can run the HTTP-JSON API server with start-mode=create-only
so that all the necessary tables are created using one set of credentials, you can
then start the server with start-mode=start-only to use these tables.
Alternatively using start-mode=create-if-needed-and-start will create the
tables if missing on startup, whereas using start-mode=create-and-start
will re-initialize the database on startup.
Note
The full list of Query Store configuration flags supported can be seen by running daml json-api --help.
Security and privacy¶
For an HTTP-JSON API server, all data is maintained by the operator of the deployment. Thus, it is their responsibility to ensure that the data abides by the necessary regulations and confidentiality expectations.
It is recommended to use the tools documented by PostgreSQL to protect data at rest and using a secure communication channel between the HTTP-JSON API server and the PostgreSQL server.
To protect data in transit and over untrusted networks, the HTTP-JSON API server provides
TLS support, to enable TLS you need to specify the private key for your server and the
certificate chain via daml json-api --pem server.pem --crt server.crt. You can also
set a custom root CA certificate used to validate client certificates via --cacrt ca.crt
For more details on secure DAML infrastructure setup please refer to this reference implementation
Architecture¶
Components¶
A production setup of the HTTP-JSON API will involve the following components:
- the HTTP-JSON API server
- the Query Store backend database server
- the ledger
HTTP-JSON API server exposes an API to interact with the Ledger and it uses JDBC to interact with its underlying Query Store for caching and serving data efficiently.
The HTTP-JSON API server releases are regularly tested with OpenJDK 8 on a x86_64 architecture, with Ubuntu 20.04, macOS 11.5.2 and Windows Server 2016.
In production, we recommend running on a x86_64 architecture in a Linux environment. This environment should have a Java SE Runtime Environment such as OpenJDK JRE and must be compatible with OpenJDK version 1.8.0_202 or later. We recommend using PostgreSQL server as query-store, most of our tests have been done with servers running version > 10.
Scaling and Redundancy¶
Note
This section of the document only talks about scaling and redundancy setup for the HTTP-JSON API server. In all of the recommendations suggested below we assume that the JSON API always interacts with a single participant on the ledger.
We advise that the HTTP-JSON API server and Query Store components to have dedicated computation and memory resources available to them. This can be achieved via containerization or setting them up on independent physical servers. Ensure that the two components are physically co-located to reduce network latency for communication. The scaling and availability aspects heavily rely on the interactions between the core components listed above.
With respect to scaling we recommend to follow the general advice in trying to understand the bottlenecks and see if adding additional processing power/memory is beneficial.
The HTTP-JSON API can be scaled independently of its Query Store. You can have any number of HTTP-JSON API instances talking to the same Query Store (if, for example, your monitoring indicates that the HTTP-JSON API processing time is the bottleneck), or have each HTTP JSON API instance talk to its own independent Query Store (if the database response times are the bottleneck).
In the latter case, the Daml privacy model ensures that the HTTP-JSON API requests are made using the user-provided token, thus the data stored in a given Query Store will be specific to the set of parties that have made queries through that specific Query Store instance (for a given template). Therefore, if you do run with separate Query Stores, it may be useful to route queries (using a reverse proxy server) based on requesting party (and possibly queried template), which would minimize the amount of data in each Query Store as well as the overall redundancy of said data.
Users may consider running PostgreSQL backend in a high availability configuration. The benefits of this are use-case dependent as this may be more expensive for smaller active contract datasets, where re-initializing the cache is cheap and fast.
Finally we recommend using app orchestration utilities which monitor the health of the service and perform subsequent operations to ensure availability. These utilities can use the healthcheck endpoints provided by the HTTP-JSON API server. This can also be tied into supporting arbitrary autoscaling implementation to ensure minimum number of HTTP-JSON API servers on failures.
Logging¶
HTTP-JSON API server uses the industry-standard Logback for logging. You can read more about that in the Logback documentation.
The logging infrastructure leverages structured logging as implemented by the Logstash Logback Encoder.
Logged events should carry information about the request being served by the HTTP-JSON API server. This includes the details of the commands being submitted, the endpoints being hit and response received highlighting details of failures if any. When using a traditional logging target (e.g. standard output or rotating files) this information will be part of the log description. Using a logging target compatible with the Logstash Logback Encoder allows to have rich logs with structured information about the event being logged.
The default log encoder used is the plaintext one for traditional logging targets.
Metrics¶
Enable and configure reporting¶
To enable metrics and configure reporting, you can use the two following CLI options:
--metrics-reporter: passing a legal value will enable reporting; the accepted values are as follows:console: prints captured metrics on the standard outputcsv://</path/to/metrics.csv>: saves the captured metrics in CSV format at the specified locationgraphite://<server_host>[:<server_port>]: sends captured metrics to a Graphite server. If the port is omitted, the default value2003will be used.prometheus://<server_host>[:<server_port>]: renders captured metrics on a http endpoint in accordance with the prometheus protocol. If the port is omitted, the default value55001will be used. The metrics will be available under the addresshttp://<server_host>:<server_port>/metrics.
--metrics-reporting-interval: metrics are pre-aggregated on the HTTP-JSON API and sent to the reporter, this option allows the user to set the interval. The formats accepted are based on the ISO 8601 duration formatPnDTnHnMn.nSwith days considered to be exactly 24 hours. The default interval is 10 seconds.
Types of metrics¶
This is a list of type of metrics with all data points recorded for each. Use this as a reference when reading the list of metrics.
Counter¶
Number of occurrences of some event.
Meter¶
A meter tracks the number of times a given event occurred (throughput). The following data points are kept and reported by any meter.
<metric.qualified.name>.count: number of registered data points overall<metric.qualified.name>.m1_rate: number of registered data points per minute<metric.qualified.name>.m5_rate: number of registered data points every 5 minutes<metric.qualified.name>.m15_rate: number of registered data points every 15 minutes<metric.qualified.name>.mean_rate: mean number of registered data points
Timers¶
A timer records all metrics registered by a meter and by a histogram, where the histogram records the time necessary to execute a given operation ( in fractional milliseconds).
List of metrics¶
The following is a list of selected metrics that can be particularly important to track.
daml.http_json_api.command_submission_timing¶
A timer. Measures latency (in milliseconds) for processing of a command submission request.
daml.http_json_api.query_all_timing¶
A timer. Measures latency (in milliseconds) for processing of a query GET request.
daml.http_json_api.query_matching_timing¶
A timer. Measures latency (in milliseconds) for processing of a query POST request.
daml.http_json_api.fetch_timing¶
A timer. Measures latency (in milliseconds) for processing of a fetch request.
daml.http_json_api.get_party_timing¶
A timer. Measures latency (in milliseconds) for processing of a get party/parties request.
daml.http_json_api.allocate_party_timing¶
A timer. Measures latency (in milliseconds) for processing of a party management request.
daml.http_json_api.download_package_timing¶
A timer. Measures latency (in milliseconds) for processing of a package download request.
daml.http_json_api.upload_package_timing¶
A timer. Measures latency (in milliseconds) for processing of a package upload request.
daml.http_json_api.incoming_json_parsing_and_validation_timing¶
A timer. Measures latency (in milliseconds) for parsing and decoding of an incoming json payload
daml.http_json_api.response_creation_timing¶
A timer. Measures latency (in milliseconds) for construction of the response json payload.
daml.http_json_api.db_find_by_contract_key_timing¶
A timer. Measures latency (in milliseconds) of the find by contract key database operation.
daml.http_json_api.db_find_by_contract_id_timing¶
A timer. Measures latency (in milliseconds) of the find by contract id database operation.
daml.http_json_api.command_submission_ledger_timing¶
A timer. Measures latency (in milliseconds) for processing the command submission requests on the ledger.
daml.http_json_api.http_request_throughput¶
A meter. Number of http requests
daml.http_json_api.websocket_request_count¶
A Counter. Count of active websocket connections
daml.http_json_api.command_submission_throughput¶
A meter. Number of command submissions
daml.http_json_api.upload_packages_throughput¶
A meter. Number of package uploads
daml.http_json_api.allocation_party_throughput¶
A meter. Number of party allocations