Ports & Network Access


The following ports will need to be open to the internet in order to run Senate Matching:

PortProtocolServiceIn/outReason
443TCPHTTPSOutboundConnection to the Senate Matching Network, Docker registry connection
24224TCPFluentd (SSL)OutboundConnection to DR log server
8200TCPHTTPSOutboundConnection to DR Vault Server
4430TCPHTTPSOutboundConnection to DR Consul Server


Your machine should also be able to resolve DNS entries in order to find the registry server, KMS server, and consul server.

Proxy Support


If you prefer you can configure your Contributor Node to use a HTTPS Proxy for its outbound connections. This is done by setting the environment variable HTTPS_PROXY in the contributor.sh script (see below).

You may also want to set whether the node should use HTTP 1.1 or HTTP 2. Older proxy software often does not support HTTP 2, which your node uses by default when talking to the Matcher Node Network (Senate Matching uses gRPC which uses HTTP 2). To force your node to use HTTP 1, set the variable HITCH_HTTP1COMPATIBILITYMODE to true.

Example

# Set proxy to use for outbound connections. Set the
# protocol to "https" and your internal hostname or IP address for your
# proxy. Ports can be set with ":portno" after the host.
# Using https:// scheme in the URL will imply that the client is expecting a client-hello certificate response: the proxy is supposed to do SSL proxying. 
# If the proxy does not intend to do SSL Proxying, one can just drop the protocol or simply use http://.
export HTTPS_PROXY=https://proxy.internal.organization:4430

# If your proxy only supports HTTP 1 then you can force the Contributor Node
# to fall back to HTTP 1 instead of the default HTTP 2. 

export HITCH_HTTP1COMPATIBILITYMODE=true
BASH


Contributor Script


When you are ready to proceed, contact support@datarepublic.com and request a Contributor Node configuration file. Data Republic support will provide you with a personalised version of following scripts, which will set the necessary environment variables and docker configuration. These scripts presume you have the correct version of Ubuntu / Redhat as per the System Requirements. These should all be placed in the same directory.

Some environment variables can be configured by you to suit your internal requirements and IT policies.

Environment variables

Set these by editing your contributor.sh file (example below)

  • Change your node password (API key): Set the variable HITCH_BASICAUTHPASSWORD. No default value, this must be set.
  • Configure outbound HTTP proxy: Use HTTPS_PROXY to set URL to internal proxy service. Defaults to no proxy (direct connection).
  • Force clients to always hash: By default, your node will only accept data if the client has hashed it first (the web browser UI will always hash data first). Set HITCH_HASHEDRECORDSONLY to false to allow the API to accept plain-text data.
  • Force use of HTTP 1: Some web proxy services do not support HTTP 2, which is needed for gRPC. Set HITCH_HTTP1COMPATIBILITYMODE to true to force your node to fall back to HTTP 1 if you're having connection difficulties.

contributor.sh

contributor.sh

#!/usr/bin/env bash
#
# This script exports the necessary environment variables for running a Senate 
# Matching Contributor Node.
# 
# Confidential information prepared for [TODO:COMPANY_NAME]. Internal use only.
#

# You may change any of these variables according to your IT policies

# You set this to your access password for this node. Initial value will be
# randomly generated by DR and should be changed.
#
export HITCH_BASICAUTHPASSWORD=XXX

# You may set this to your internal proxy service. Your contributor node will then use 
# this to communicate with the rest of the Senate Matching network 
#
export HTTPS_PROXY=
export HTTP_PROXY= 

# DR recommends that you configure your node to only accept pre-hashed data. 
# However during test it might be convenient to use plain-text synthetic data. 
# Comment this line out if you want to be able to send non-hashed data via 
# the API. Note: The browser UI interface always hashes data before calling
# APIs.
#
export HITCH_HASHEDRECORDSONLY=true 

# Uncomment this line if your HTTP proxy does NOT support HTTP2, and you need 
# to force your node to communicate over HTTP/1.1. HTTP2 is recommended if
# possible.
#
#export HITCH_HTTP1COMPATIBILITYMODE=true 

# You may change these from the defaults, but generally won't need to when
# first installing.

# These variables specify the Docker image tag to be installed. DR will advise
# if they need updating and whether updates include important security fixes.
#
export HITCH_DOCKER_IMAGE_TAG=1.6.0
export HITCH_UI_DOCKER_IMAGE_TAG=1.6.0

# This is the name the Docker container will use to refer to itself. Can
# change to an internal hostname as required.
#
export HITCH_LOCALSERVERCOMMONNAME=hitch-contributor-XXX

# You set this for the node to configure its internal MySQL and Redis DB. 
# Defaults are randomly generated by DR.
#
export HITCH_DB_PASSWORD=XXXX
export HITCH_DB_ROOT_PASSWORD=XXXX
export HITCH_REDISPASSWORD=XXXX

# HITCH_PORT is the port the API will listen on for HTTPS requests. 
# HITCH_UI_PORT is the port the browser should connect to if you want to use 
# the browser UI.
#
export HITCH_PORT=9054
export HITCH_UI_PORT=9055 

# FluentD is a logging service. If enabled, some error and reporting
# logs are forwarded to DR for monitoring and support. This is optional
# and requires TCP port 24224 to be opened outbound to the host below.
#
export FLUENT_SHARED_KEY=XXXXXX


# These are supplied by DR. They are specific for your organisation.
# Do not change.

# KMS token and node ID
export HITCH_KMSTOKEN=XXXX
export HITCH_POLICYSERVICETOKEN=XXXX
export HITCH_LOCALNODEID=XXXX
export HITCH_DOCKER_REGISTRY=registry.fpims.datarepublic.com.au

# Senate Matching region configuration
export HITCH_KMSADDRESS=https://vault.hitch.prod-au.datarepublic.io:8200
export HITCH_LOGFORMAT=json
export HITCH_POLICYSERVICEADDRESS=https://consul.hitch.prod-au.datarepublic.io:4430
export JAEGER_AGENT_HOST_PORT=
export JAEGER_ENABLED=false
export FLUENT_PROXY_ADDRESS=logs-dr.ops-au.datarepublic.io

echo "XXXX" | docker login -u XXXX --password-stdin registry.fpims.datarepublic.com.au

exec docker-compose -f docker-compose.yaml -p contributor $@
BASH

docker-compose.yaml

docker-compose.yaml

#
# This file is used to run your Contributor Node. Place in the same directory as configure.sh above.
#

version: '2'

services:
    fluentd:
      image: registry.fpims.datarepublic.com.au/dr-log:2
      volumes:
      - ./fluentd.conf:/fluentd/etc/fluentd.conf
      environment:
        FLUENTD_CONF: fluentd.conf
    contributor:
        depends_on:
            - db
            - redis
            - fluentd
        image: ${HITCH_DOCKER_REGISTRY}/contributor:${HITCH_DOCKER_IMAGE_TAG}
        ports:
            - ${HITCH_PORT}:15426
        environment:
            - HITCH_CACERTFILE=/ca-public/tls.crt
            - HITCH_BASICAUTHPASSWORD
            - HITCH_DATABASEURL=mysql://hitchuser:${HITCH_DB_PASSWORD}@db:3306/hitch
            - HITCH_HASHEDRECORDSONLY
            - HITCH_KMSADDRESS
            - HITCH_KMSTOKEN
            - HITCH_LOGFORMAT
            - HITCH_POLICYSERVICEADDRESS
            - HITCH_POLICYSERVICETOKEN
            - HITCH_LOCALNODEID
            - HITCH_LOCALSERVERCOMMONNAME
            - HITCH_REDISHOSTPORT=redis:6379
            - HITCH_REDISPASSWORD
            - JAEGER_AGENT_HOST_PORT
            - JAEGER_ENABLED
            - HITCH_FLUENT_ENABLED=true
            - HITCH_FLUENT_HOST=fluentd
            - HITCH_FLUENT_PORT=24224
            - HITCH_FLUENT_TAG_PREFIX=hitch-au #comment: AU = hitch-au; US = hitch-us; SG = hitch-sg
            - HITCH_FLUENT_MARSHAL_AS_JSON=true
            - HITCH_HTTP1COMPATIBILITYMODE
            - HTTP_PROXY
            - HTTPS_PROXY
        volumes:
            - ./tls.crt:/ca-public/tls.crt:ro
            - ./wait-for-it.sh:/opt/wait-for-it.sh
        restart: always
        entrypoint: ["/opt/wait-for-it.sh", "fluentd:24224", "--", "/entrypoint.sh", "/usr/local/bin/contributor", "service"]

    ui:
        image: ${HITCH_DOCKER_REGISTRY}/contributor-ui:${HITCH_UI_DOCKER_IMAGE_TAG}
        ports:
            - ${HITCH_UI_PORT}:443
        environment:
            - API_HOST=contributor:15426
            - DNS_RESOLVER=127.0.0.11
        volumes:
            - ./certs:/etc/nginx/certs/
        restart: always

    db:
        image: mysql:5.6
        volumes:
            - contributor-db-data:/var/lib/mysql
        restart: unless-stopped
        environment:
            MYSQL_DATABASE: hitch
            MYSQL_PASSWORD: ${HITCH_DB_PASSWORD}
            MYSQL_ROOT_PASSWORD: ${HITCH_DB_ROOT_PASSWORD}
            MYSQL_USER: hitchuser
        restart: always

    redis:
        image: redis:4.0.9
        restart: always
        command: redis-server --requirepass ${HITCH_REDISPASSWORD}
        restart: always

volumes:
    contributor-db-data:
        external: false
YML

Certificates


  • tls.crt - will be provided by Data Republic, do not replace this
  • Please replace the 2 keys below with your own SSL certificate / private key
    • cert.pem
    • key.pem
  • You can then access your Contributor Node using https://<node's IP address or internal hostname>/