Skip to content

Lift data from JSON and XML source¤

Introduction¤

This tutorial shows how you can build a Knowledge Graph based on input data from hierarchical sources like a JavaScript Object Notation file (.json) or an Extensible Markup Language file (.xml).

Info

The complete tutorial is available as a project file (XML) and a project file (JSON). You can import these projects:

  • by using the web interface (Create → Project → Import project file) or
  • by using the command line interface

    cmemc -c my-cmem project import tutorial-xml.project.zip xml-transformation
    
    cmemc -c my-cmem project import tutorial-json.project.zip json-transformation
    

The documentation  consists of the following steps, which are described in detail below:

The following material is used in this tutorial:

  • Sample vocabulary which describes the data in the JSON and XML files: products_vocabulary.nt

    Visualization of the "Products Vocabulary".

  • Sample JSON file: services.json

    [
        {
            "Price": "748,40 EUR",
            "ProductManager": "Lambert.Faust@company.org",
            "Products": "O491-3823912, I965-1821441, Z655-3173353, ...",
            "ServiceID": "Y704-9764759",
            "ServiceName": "Product Analysis"
        },
        {
            "Price": "1082,00 EUR",
            "ProductManager": "Corinna.Ludwig@company.org",
            "Products": "Z249-1364492, L557-1467804, C721-7900144, ...",
            "ServiceID": "I241-8776317",
            "ServiceName": "Component Confabulation"
        },
        ...
    ]
    
  • Sample XML file: orgmap.xml

    <orgmap>
        <dept id="73191" name="Engineering">
            <manager>
                <email>Thomas.Mueller@company.org</email>
                <name>Thomas Mueller</name>
                <address>Karl-Liebknecht-Straße 885, 82003 Tettnang</address>
                <phone>+49-8200-38218301</phone>
            </manager>
            <employees>
                <employee>
                    <email>Corinna.Ludwig@company.org</email>
                    <name>Corinna Ludwig</name>
                    <address>Ringstraße 276</address>
                    <phone>+49-1743-24836762</phone>
                    <productExpert>Memristor, Gauge, Encoder</productExpert>
                </employee>
                <employee>
                    <email>Karen.Brant@company.org</email>
                    <name>Karen Brant</name>
                    <address>Friedrichstraße 664, 30805 Willich</address>
                    <phone>(00530) 5040048</phone>
                    <productExpert>Inductor</productExpert>
                </employee>
                ...
            </employees>
            <products>
                <product id="Z249-1364492" />
                <product id="O184-6903943" />
                <product id="V404-9975399" />
                <product id="F344-7012314" />
                <product id="N463-8050264" />
                <product id="M605-5951566" />
                <product id="N733-1946687" />
            </products>
            <services>
                <service id="I241-8776317" />
                <service id="D215-3449390" />
            </services>
        </dept>
        <dept id="22183" name="Product Management">        
            ...
        </dept>
        ...
    </orgmap>
    

1 Register the vocabulary¤

The vocabulary contains the classes and properties needed to map the source data into entities in the Knowledge Graph.

  1. In Corporate Memory, click Vocabularies in the under EXPLORE in the navigation on the left side of the page.

    Menu entry EXPLORE > Vocabularies

  2. Click Register new vocabulary on the top right of the Vocabulary catalog page in Corporate Memory.

    Vocabularies Catalog

  3. Define a Name, a Graph URI and a Description of the vocabulary. In this example we will use:

    • Name: Product Vocabulary
    • Graph URI: http://ld.company.org/prod-vocab/
    • Description: Example vocabulary modeled to describe relations between products and services.

    Dialog to register a new vocabulary.

  4. Click REGISTER.

2 Upload the data file¤

To add the data files, click Projects under BUILD in the navigation on the left side of the page. Follow the steps below for adding JSON and XML datasets.

  1. Click Create at the top of the page.

  2. In Create new item window, select JSON and click Add.

    Dialog to create new JSON dataset

  3. Define a Label for the dataset and upload the services.json file. You can leave all the other fields at default values.

    Dialog to create new JSON dataset Dialog to create new JSON dataset

  4. Click Create.

  1. Press the Create button and select XML

    Dialog to create new XML dataset

  2. Define a Label for the dataset and upload the orgmap.xml example file. You can leave all the other fields at default values.

    Dialog to create new XML dataset

  3. Click Create.

3 Create a Knowledge Graph¤

  1. Click Create at the top of the page.  

  2. In Create new item window, select Knowledge Graph and click Add. The Create new item of type Knowledge Graph window appears.

    Dialog to create new Knowledge Graph dataset

  3. Fill in the required details such as Label and Description.

    Define a Label for the Knowlege Graph and provide Graph uri. You can leave all the other fields at default values. In this example we use:

    • Name: Service Knowledge Graph
    • Graph: http://ld.company.org/prod-instances/

    Dialog to create new Knowledge Graph dataset

    Define a Label for the Knowledge Graph and provide Graph uri. You can leave all the other fields at default values. In this example we will use:

    • Name: Organization Knowledge Graph
    • Graph: http://ld.company.org/organization-data/

    Dialog to create new Knowledge Graph dataset

4 Create a Transformation¤

The transformation defines how an input dataset (e.g.: JSON or XML) will be transformed into an output dataset (e.g.: Knowledge Graph).

  1. Click Create in your project.  

  2. On the Create New Item window, select Transform and click Add to create a new transformation.

    Create new Transformation

  3. In the Create new item of type Transform window, enter the required fields.

    For this example, enter the following:

    • Name: Create Service Triples
    • (optional) Description: Lifts the Service file into the Knowledge Graph
    • Select the Source Dataset: Services JSON
    • Select the Output Dataset: Service_Knowledge_Graph

    Dialog to create new Transformation

    Click Create.

    For this example, enter the following:

    • Name: Create Organization Triples
    • (optional) Description: Lifts the Orgmap XML file into the Knowledge GraphOrgmap XML
    • Select the Source Dataset: Orgmap XML
    • Type: dept (define the Source Type, which defines the XML element that should be iterated when creating resources)
    • Select the Output Dataset: Organization_Knowledge_Graph

    Dialog to create new Transformation

    Click Create.

  4. Expand the Mapping Button menu by clicking the arrow on the right side of the page to expand the menu.

  5. Click Edit to create a base mapping.

    Mapping header configuration.

  6. Define the Target entity type from the vocabulary, the URI pattern and a Label for the mapping.

    Target Entity Type defines the class that will be instantiated when the mapping rule is applied.

    The URI pattern that defines the URI that shall be generated for each individual

    • http://ld.company.org/prod-inst/ is a common prefix for the instances in this use case,
    • service-instances/ complements the instances prefix by adding a common prefix for all service instances
    • and finally {ServiceID} is a placeholder that will resolve to the json-key ServiceID (e.g. “ServiceID”: “Y704-9764759”)

    In this example we will use:

    • Target Entity Type: Service
    • URI Pattern: http://ld.company.org/prod-inst/service-instances/{ServiceID}
    • An optional Label: Service

    Click SAVE.

    Example RDF triple in our Knowledge Graph based on the mapping definition:

    <http://ld.company.org/services-instances/Y704-9764759> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ld.company.org/prod-vocab/Service>
    

    Target Entity Type defines the class that will be instantiated when the mapping rule is applied: Department

    The URI pattern that defines the URI that shall be generated for each individual: http://ld.company.org/department/{@id}

    • http://ld.company.org/department/ is a common prefix for the department instances in this use case,
    • and finally {@id} is a placeholder that will resolve the XML attribute of the XML tag dept, which was configured as the Source Type of this transformation (see previous steps)

    In this example we will use:

    • Target Entity Type: Department
    • URI Pattern: http://ld.company.org/department/{@id}
    • An optional Label: Department

    Click Save.

    Example RDF triple in our Knowledge Graph based on the mapping definition:

    <http://ld.company.org/department/73191 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://ld.company.org/prod-vocab/Department>
    
  7. Evaluate your mapping by pressing on the Button expand button in the Examples of target data property to see at most three generated base URIs.

    Preview of mapping results

    We have now created the Service entities in the Knowledge Graph. Next we will now add the name of our entity.

  8. Click the circular blue + icon on the lower right and select Add value mapping.

    Add a mapping rule

    Define the Target property, the Data type, the Value path (column name) and a Label for your value mapping. In this example, enter the following:

    • Target Property: has product manager
    • Data type: StringValueType
    • Value path: ProductManager/name
      • which corresponds to the following element in the json-file: [ {“ProductManager”: {  “name”: “Corinna Ludwig”} … } …]
    • An optional Label: has Product Manager

    Configuration of a mapping rule

    Click Save.

    Define the Target property, the Data type, the Value path (column name) and a Label for your value mapping. In this example we will use:

    • Target Property: name
    • Data type: StringValueType
    • Value path: dept/@name
      • which corresponds to the department name attribute in the XML file
    • An optional Label: department name

    Configuration of a mapping rule

    Click Save.

By clicking on the Button expand button in the Examples of target data property, you can get a preview for 3x value mapping to be created.

Mapping result

Mapping result

5 Evaluate a Transformation¤

Click Transform evaluation to evaluate the transformed entities.

Transformation evaluation view

6 Build the Knowledge Graph¤

  1. Click Transform execution
  2. Press the Button play button and validate the results. In this example, 9x Service entities were created in our Knowledge Graph based on the mapping.
  3. You can click Knowledge Graphs under EXPLORE to (re-)view of the created Knowledge Graphs
  4. Enter the following URIs in the Enter search term for JSON and XML respectively.

    • JSON / Service: http://ld.company.org/prod-instances/
    • XML / Department: http://ld.company.org/organization-data/

    Service KG

    Organization KG