Introduction

This page describes a docker-compose based orchestration running on a server instance in a cloud and accessible publicly via browser (SSL enabled via letsencrypt).

Requirements

  • Access to a server instance with a public IP address and resolvable domain name
  • SSH private/public key pair
  • Terminal with ssh client installed locally
  • POSIX-compatible command line interface (Linux, macOS or WSL for Windows)

Creating Server Instance Example

To address first item in the requirements, we guide you through creation of a server instance on Hetzner infrastructure provider. To follow this example, you will need to create account on Hetzner and be able to login into Hetzner Cloud Console.

Generate SSH Key Pair (optional)

Check if you have existing key pair available on your machine:

$ ls ${HOME}/.ssh/id_rsa*
id_rsa  id_rsa.pub
CODE

If you do not see id_rsa, id_rsa.pub files inside the directory, then proceed with the key generation.

Open a terminal window and run the ssh-keygen command:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/username/.ssh/id_rsa.
Your public key has been saved in /Users/username/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:XsRUD7X+rumbEFdasdfHF/KSQuZLexample820wdY username@User-MacBook-Pro.local
The key's randomart image is:
+---[RSA 2048]----+
|          +BBB*=*|
|         o. =B+OE|
|       .  o. oXB*|
|         .. o..++|
|        S .o. o. |
|       . .   o.. |
|        .   .   .|
|             . + |
|             .*o.|
+----[SHA256]-----+
CODE

This will generate id_rsa (private key) and id_rsa.pub (public key) in the ~/.ssh/ directory. You will need public key to follow this example. An example of public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcIZt/J+Ml7jdVwSUl1iszrvUvXMfWLPVla0uMxg0oYCJvf1sBeyWTw2NRN+2GChk1lesYIeym5Buu4nPKQqATrTfbh3eBX2kDbWhsSmO0+dlsDciqBjda2PUxcjYooO0DuqvASmCqfYWlWwTzJGenR44f8/DVO/iocYcyo10WA0worzoiIE1rD+SPBisC/WwWxkGlRGTZsftirZXcC2PC/lsToo/hVtUBUGHZ3wKWpcuSTohVfDaEIQomHpy4bseBSTeLD0RkCM2BgY4049SDJNOb38LQENeb1YV/nr0/w+p6rKFpbfs9wMm0szrLVAqdRKnBsJP47NA/f2Jkxd3r username@User-MacBook-Pro.local
CODE


Create a Server Instance

  1. Login into Hetzner Cloud Console.
  2. Select default project or create a new one. The projects are reflected in the billing and can be used to manage access control to the instances.
  3. Inside the project click "ADD SERVER".
  4. Configure your server instance as follows:
    1. Location: any
    2. Image: Debian 10
    3. Type: CX41 (minimal recommended size for the Corporate Memory) - 18.92€/month
    4. Volume: not necessary
    5. Network: not necessary
    6. Additional Features: not necessary
    7. SSH key: click "ADD SSH KEY" and copy/paste your public ssh key there
    8. Name: corporate-memory
    9. How many servers? 1 Server
    10. click "CREATE & BUY NOW"

After couple of seconds the instance will be created. Note the IP address of the instance (second column).

Create a DNS Entry for a Server Instance Example

In this example we are using inwx.de DNS provider (you can use corporate DNS server or ask your administrator in case of doubt). To create a new DNS entry you need an account at inwx.de as well as registered domain.

  1. Login into inwx.de
  2. Select "Nameserver" from the left menu
  3. Double click on the domain you want to modify
  4. Click "Add DNS entry"
    1. Name: corporate-memory
    2. Type: A
    3. Value: 49.12.6.225
    4. Click "Save"

Test that your DNS entry was successfully added. From the terminal run the following command:

$ dig corporate-memory.eccenca.dev

; <<>> DiG 9.10.6 <<>> corporate-memory.eccenca.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13166
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;corporate-memory.eccenca.dev.	IN	A

;; ANSWER SECTION:
corporate-memory.eccenca.dev. 3600 IN	A	49.12.6.225

;; Query time: 113 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Mar 31 12:14:35 CEST 2020
;; MSG SIZE  rcvd: 101
CODE

In the "ANSWER SECTION" you should see your entry.

Provision Server Instance

The next step is to install necessary software on the server. Login into the server:

$ ssh root@49.12.6.225
The authenticity of host '49.12.6.225 (49.12.6.225)' can't be established.
ECDSA key fingerprint is SHA256:rBn7zzDn8lJL1Aw5MJx8qMtbi1W+oDeygfOJwu6etOY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '49.12.6.225' (ECDSA) to the list of known hosts.
Linux corporate-memory 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@corporate-memory:~#
CODE

Now you can provision the server using the following commands:

You can copy the whole block of commands into the terminal. Or copy them one by one.

For username:password in curl command use the credentials to access eccenca Artifactory and docker registry.


apt-get update
echo "install ntp and set timezone"
apt-get install -y ntp
timedatectl set-timezone Europe/Berlin
echo "install docker"
apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg2 \
    software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
echo "install docker-compose"
curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
echo "install system utils"
apt-get install -y htop iotop pwgen zip unzip zsh git make vim pass
echo "login into eccenca docker registry"
docker login docker-registry.eccenca.com
echo "download and initialize Corporate Memory docker orchestration"
mkdir -p /opt/corporate-memory && cd /opt/corporate-memory
curl -u username:password https://releases.eccenca.com/docker-orchestration/cmem-orchestration-v20.03.2.zip > cmem-orchestration-v20.03.2.zip
unzip cmem-orchestration-v20.03.2.zip
rm cmem-orchestration-v20.03.2.zip
git config --global user.email "you@example.com" && git init && git add . && git commit -m "stub"
CODE

Install Corporate Memory Instance

Connect to the server and navigate to the directory with the Corporate Memory docker orchestration:

ssh root@49.12.6.225
cd /opt/corporate-memory
CODE

The Corporate Memory docker orchestration is configured with environment files. You will need to create an environment file at /opt/corporate-memory/environments/prod.env as follows (use vim or nano to create the file):

To create file simply run:

vim environments/prod.env


#!/bin/bash

CMEM_SERVICE_ACCOUNT_CLIENT_SECRET=c8c12828-000c-467b-9b6d-2d6b5e16df4a
STARDOG_PASSWORD=admin
# change DEPLOYHOST to your own value! the one you have configured in your DNS
DEPLOYHOST=corporate-memory.eccenca.dev
PROXY_ADDRESS_FORWARDING=true
DATAINTEGRATION_JAVA_TOOL_OPTIONS=-Xmx4g
DATAPLATFORM_JAVA_TOOL_OPTIONS=-Xms2g -Xmx4g
STARDOG_SERVER_JAVA_ARGS=-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g

# letsencrypt:
SSLCONF=ssl.letsencrypt.conf
# change MAIL to your own value! use a common system administration mailbox here
LETSENCRYPT_MAIL=administration@eccenca.com

DI_VERSION=v20.03
DP_VERSION=v20.03
DM_VERSION=v20.03
APACHE2_VERSION=v2.6.0
KEYCLOAK_VERSION=v6.0.1-2
POSTGRES_VERSION=11.5-alpine
STARDOG_VERSION=v7.2.0-1

#################################
# Do NOT CHANGE these settings. #
# ###############################
# NOTE:
#  - these settings differ from http setup but should not be altered
# 
DEPLOYPROTOCOL=https
PORT=443
APACHE_BASE_FILE=docker-compose.apache2-ssl.yml
APACHE_CONFIG=default.ssl.conf
PROXY_ADDRESS_FORWARDING=true
CODE

To see all available configuration options refer to Docker Orchestration configuration page.

Next, request SSL certificates from letsencrypt service:

$ CONFIGFILE=environments/prod.env make letsencrypt-create
#@echo "Make sure this machine is reachable from the internet"
Unable to find image 'certbot/certbot:latest' locally
latest: Pulling from certbot/certbot
c9b1b535fdd9: Pull complete
2cc5ad85d9ab: Pull complete
756a868c4378: Pull complete
444b2fc9a129: Pull complete
ea15f1150254: Pull complete
2966bb4c2979: Pull complete
bef055e88bc6: Pull complete
12a9fc86916b: Pull complete
41db5b0d58d8: Pull complete
bc6b91fbba74: Pull complete
852b5bc6112d: Pull complete
Digest: sha256:d908a5d08108feac2a3a479b1bc7d3f33ff4648bc2dbfcde9d4510a57b3cc296
Status: Downloaded newer image for certbot/certbot:latest
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Skipped user interaction because Certbot doesn't appear to be running in a terminal. You should probably include --non-interactive or --force-interactive on the command line.
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for corporate-memory.eccenca.dev
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/corporate-memory.eccenca.dev/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/corporate-memory.eccenca.dev/privkey.pem
   Your cert will expire on 2020-06-29. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
CODE

The stardog triplestore is a third party application, which requires a license to run. If you have your own license, copy it to the server. Otherwise, you can request a trial license from command line:

To copy license file from local computer to the server use the following command

scp your-license.bin root@49.12.6.225:/opt/corporate-memory/conf/stardog/stardog-license-key.bin


# check validity of your license
$ make stardog-license-check
docker run -it --rm --name stardog-license-check -v data:/data -v /opt/corporate-memory//conf/stardog/stardog-license-key.bin:/data/stardog-license-key.bin docker-registry.eccenca.com/complexible-stardog:v7.2.0-1 stardog-admin license info /data/stardog-license-key.bin
The license is invalid: java.io.EOFException
make: *** [custom.dist.Makefile:5: stardog-license-check] Error 1

# request stardog trial license
$ make stardog-license-request
docker run -it --rm --name stardog-license-check -v data:/data -v /opt/corporate-memory//conf/stardog/stardog-license-key.bin:/data/stardog-license-key.bin docker-registry.eccenca.com/complexible-stardog:v7.2.0-1 stardog-admin license request --force --output /data/stardog-license-key.bin
Thank you for downloading Stardog.
A valid license was not found in /data.
Would you like to download a trial license from Stardog (y/N)? y
Contacting Stardog..............
Please provide a valid email address to start your 60-day trial (we may occasionally contact you with Stardog news):  ivan.ermilov@eccenca.com
Contacting license server...................
Email validated. You now have a 60-day Stardog trial license. Starting Stardog...
                                                         %▄,
                                                       ░░Γ╬▀▀█▓╣⌐
                                                      ▄▓▌░░░░░░╨▓
                          .⌐⌐.                     .½▓█▌░░░░░░░░░▀▄
                     ⌐Γ░░░░░░░░░░Γ«⌐              ≤░▓███▓▓▌▄░░░░░▓▒█Γ⌐
                .»≥░░░░░░░░░░░░░░░░░░░░░≥▒▒▒▒▒▒▒░░░▓████████░░░░▐█▄╙░░≥░░≥[».
             ┌Γ░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒▒▒▒▒▒░░╫█████████▌▒▒▒▒▒█▒▓▓▓▌▌▌▌▓▓█▓⌐
          .≥░░░░░░░░░░░░░░░░░░░░░░░░░░░░░╢▒▒▓▓▒▒▒░░░░╟██████╙ └█b  ████▀▀▒█████▌
 Γ    .∩░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒▒▒▒▒▒▒░░░██████`   ╙   ╟█▓∩  ███▀██▌
├░, .░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒▒▒▒░░░╫██████        ▐█    ██    ╙
├░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▒▒░░░░▓██████          ,  '  ▄
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░▄▄▄░░░░░░░░░░░░░░░░░░░░▓██████▓░░░░╕    ▐█▄   ██
.░░░░░░░░░░░░░░░░╣▄░░░░░▄╣▓██████▓▄░░░░░░░░░░░░░░░░░╢████████▒██░░╣   ██▄▄╣▒▒▌▄▄
 ░░░░░░░░░░░░░░,  └╙▀▀██████████████▓░░░░░░░░░░░░░░░░██████████▌░░░╦  █▒██████▌
 ╙░░░░░░░░░░░░░░░       ╙▀███████████░░░░░░░░░░░░░░▓░░█████▀▀▀░░░░░╬▒█████████
  '╙░░░░░░░░░▄▄▄`          └▀███████░░░░░░░░░░░░░░║██▓▄░▀▀░░░░░░░░░Γ "╙░░░░▀▀
    ╙░░▄╣▓▓██▀▀               ╙▀███▌░░░░░░░░░░░░░╢█████⌐   ╙░░░░░░░
    └╣███▀▀└                     ╙▀░░░░░░░░░░░░░╣████▀       '""`
                                 .░░░░░░░░░░░░╠▓███▀²
                                «░░░░░░░░░░░░╣███▀
                               ≥░░░░░░░░░░░┴▀▀╙
                            .Γ░░░░░░░░░░∩`
                á▀▀╕▄#▌▀▀░≥░░░░░░░╙∩"
                ░░░░░░░░░░∩`
                 └░░░░╙∩`
Thank you!

# check the license again
$ make stardog-license-check
docker run -it --rm --name stardog-license-check -v data:/data -v /opt/corporate-memory//conf/stardog/stardog-license-key.bin:/data/stardog-license-key.bin docker-registry.eccenca.com/complexible-stardog:v7.2.0-1 stardog-admin license info /data/stardog-license-key.bin
Licensee:	Stardog Trial User (ivan.ermilov@eccenca.com), Stardog Union
Version:	Stardog *
Type:		Trial
Issued:		Mon Mar 30 10:47:17 GMT 2020
Expiration:	59 days
Support:	The license does not include maintenance.
Quantity:	3
CODE


Finally deploy the Corporate Memory instance:

$ CONFIGFILE=environments/prod.env make clean-pull-start-bootstrap
make[1]: Entering directory '/opt/corporate-memory'
/usr/local/bin/docker-compose kill
/usr/local/bin/docker-compose stop
/usr/local/bin/docker-compose down --volumes --remove-orphans
Removing network corporatememoryeccencadev_default
Removing volume corporatememoryeccencadev_stardog
/usr/local/bin/docker-compose rm -v --force
No stopped containers
Pulling apache2         ... done
Pulling datamanager     ... done
Pulling dataintegration ... done
Pulling stardog         ... done
Pulling dataplatform    ... done
Pulling postgres        ... done
Pulling keycloak        ... done
Creating network "corporatememoryeccencadev_default" with the default driver
Creating volume "corporatememoryeccencadev_stardog" with default driver
Creating corporatememoryeccencadev_postgres_1    ... done
Creating corporatememoryeccencadev_stardog_1         ... done
Creating corporatememoryeccencadev_datamanager_1 ... done
Creating corporatememoryeccencadev_apache2_1     ... done
Creating corporatememoryeccencadev_dataintegration_1 ... done
Creating corporatememoryeccencadev_keycloak_1        ... done
Creating corporatememoryeccencadev_dataplatform_1    ... done
/opt/corporate-memory//scripts/waitForSuccessfulStart.sh
Waiting for healthy orchestration.......................... done
CMEM-Orchestration successfully started.
Run make logs to see log output
make[1]: Leaving directory '/opt/corporate-memory'
CODE

You have successfully deployed a Corporate Memory instance.

Access Corporate Memory Instance

Open your browser and navigate to the host you have created in DNS server, e.g. https://corporate-memory.eccenca.dev

Click CONTINUE WITH LOGIN and use one of these default accounts:

accountpassworddescription
adminadminIs member of the global admin group (can see and do anything)
useruserIs member of the local user group (can not change access conditions or see internal graphs)

 

After successful login, you will see Corporate Memory interface. You can now proceed to the ► Getting Started section.