Skip to content

Setup and Configuration¤

This section describes which backend components are needed on the DataIntegration server, in order to use python plugins.

Basic Configuration and Dependencies¤

Info

When using the official eccenca docker images, setup and basic configuration is already done.

DataIntegration Configuration

The following DataIntegration configuration section describes how to setup and enable the Python Plugin system.

#################################################
# Plugin Configuration
#################################################

# this (optional) file can be used to hold python plugin specific configuration
include "python-plugins.conf"

com.eccenca.di.scripting = {
  python = {
    PythonPluginRegistry = {
      # Python plugins will only be loaded if 'enabled' is set to true.
      enabled = true

      # Plugins will only be loaded below the following base package.
      basePackage = "cmem"
    }

    PythonPackageManager = {
      # Python package installer executable.
      pipExecutable = "cmem-pip-wrapper.sh"
    }
  }
}
Python Interpreter

An installation of the CPython distribution (at least version 3.3) is required. Although other distributions, such as Anaconda, should work as well, only CPython is officially supported.

The official image ships with a tested python interpreter (currently - 2024 - Python 3.11).

Java Embedded Python (Jep)

The Jep package needs to be installed.

The libraries contained in the Jep module need to be accessible from the Java Virtual Machine running DataIntegration. This can be achieved by setting an environment variable to the directory path where the Jep module is located:

  • Linux: set LD_LIBRARY_PATH.
  • OS X: set DYLD_LIBRARY_PATH.
  • Windows: set PATH.

For alternative installation methods, visit Jep

The official image ships with a tested Jep module.

Specific Changes from the default¤

Package Index Locations¤

The basic setup allows for installation of packages from the pypi.org python package index, maintained by the Python Software Foundation. In order to change the index server, from where you can install python packages, you can use the following environment variables:

  • PIP_INDEX_URL - Base URL of the default python package index Base URL. This should point to a repository which is compliant with PEP 503 (the simple repository API).
    • Default Value: https://pypi.python.org/simple
    • Changing this value means, that you can install packages only from this repository.
  • PIP_EXTRA_INDEX_URL - Extra URLs of package indexes to use in addition to the default package index.
    • Example Value: https://pypi.eccenca.com/simple https://example.org/simple
    • Multiple index URLs have to be given space-separated.
    • Changing this values means you can install packages from the given repositories in addition to the main index.

For individual needs, you can use additional environment variables known by pip (PIP_TRUSTED_HOST, PIP_CERT, …). Please have a look at the pip documentation.

Package Path¤

The basic setup provides a /data directory inside of the DataIntegration container, where all changed files are managed in subdirectories. The environment variable PYTHONPATH defines the directory, where the user-managed python packages are saved. This directory shall be persisted between restarts of DataIntegration. The default value of this variable is /data/python-packages/.

Comments