Setup

This guide explains how to get started with the knowledge pooling tool, including how to access the system, authenticate as a contributor, and prepare for submitting or curating records. The focus of these steps differ depending on your role:

• Contributors/Submitters: Setup access to submit and view records
• Managers/Administrators: Configure authentication and permissions

Overview of access and authentication

Access to the knowledge pooling tool is managed using tokens and Forgejo teams. Forgejo-based authentication ensures that permissions are linked to existing institutional or project accounts. Tokens are used to identify users with interacting with the API or web UI.

Each user can hold different roles, such as:

• Submitter: Can create and modify records in their personal “inbox”.
• Curator: Can view, edit, and approve submitted records.
• Manager: Can configure access settings and team memberships.

Setup for general contributors and submitters

Collaboration hub access and permissions

To report and access records using the knowledge pooling tool, you must have read and write access to the protected collection of the pooling tool on the TRR379 collaboration platform. You must be a member of both:

• metadata-pool-read-protected, and
• metadata-pool-write-protected.

To confirm your team memberships, first sign into the TRR379 collaboration platform (see also How to get a Hub account and log in), then click the links above to see the team members. If you are unable to see the repository, team member list, or if your username is not listed, see below how to request access. Otherwise, proceed to generate a personal access token.

Request access to protected team(s)

If you are not yet a member of the metadata-pool-read-protected and/or metadata-pool-write-protected team(s), please email trr379@lists.fz-juelich.de with a request to be added. Please send the email from an institutional email with the respective TRR379 project lead in CC.

Generate a personal access token

Reading and writing records into the knowledge pooling tool requires a token linked to your Forgejo account.

Create a new token

From your collaboration platform account, navigate to Profile and settings > Settings > Applications > Generate new token. Name the token, choose All for the repository and organization access, and click Select permissions to list further permissions; the token needs to have user:Read and organization:Read permissions. Click Generate token once all permissions are set.

Copy the generated value and store your token securely; it will be used to authenticate your submission through the web UI and in API calls.

Test your token

You can verify that your token works using a simple API request:

curl 'https://pool.v0.trr379.de/api/protected/records/Thing' -H "x-dumpthings-token:<your-token-here>"

If the token is valid, this command will return a (potentially empty) list of entries. If the token is invalid, it will return an invalid token error, together with helpful information what may be wrong (e.g., wrong scopes).

Once your token is working and you have confirmed your access, you can proceed to the general usage guide and relevant workflows for instructions on how to enter or retrieve research metadata and outputs through the knowledge pooling tool.

Setup for managers and administrators

This section will describe how to configure the knowledge pooling tool from a management perspective, including:

• Defining collections (e.g., protected) and configuring the backend API (see also the Dump Things Service documentation)
• Setting up and linking Forgejo teams for read and write access

General usage guide

This guide explains how to use the knowledge pooling tool’s shacl-vue web UI and dump-things API service to submit, edit, and review metadata records. It applies to all record types, such as publications, personnel, projects, and topics.

For workflow-specific instructions (e.g., publication reporting), see the Workflows section.

Submit or edit a publication record using the web UI

The web UI for submitting, editing, and browsing records is provided at https://pool.v0.trr379.de/.

Authenticate with your access token

Submitting publication records will only be possible using a valid access token The token can be entered via the Key button in the web UI.

Create or edit a record

Records correspond to structured metadata entities (e.g., Person, Topic, Publication). Each record is created and managed through a form defined by the metadata schema.

If no record on a given entity already exists, a new record must be created. Click the + (plus) button next to the desired record type (e.g., Publication or Person).

Alternatively, if a previous record already exists, it can be loaded by clicking on the Edit button (pencil icon). You can search for records by title/name, identifier, or other metadata attributes.

Fill out record fields

Context help describing expected values and controlled vocabularies is provided for each field. Hover your mouse pointer over the field label to display help text.

For some fields (e.g., authors or publication venue when filling out a publication record), one or more choices have to be selected. The selector offers type-ahead search of choices. The selection is contrained to choices matching the entered text in their label. Select the appropriate value from the dropdown list.

If no adequate choice exists, click Add New Item to create one.

Save and submit records

Submitting metadata involves two different save actions: local saves and submitting to the server.

Save form sections (local save)

Whenever a record (or sub-field) is completed, it must be saved by clicking the Save button. You must do this for each section you complete. Saving temporarily stores the record in your browser session.

Submit to server

Once the record is complete and saved locally, it can be submitted to the server by clicking the Submit button (cloud icon with up arrow), which will show a green indicator if changes have not yet been submitted to the server. If you haven’t already entered your access token, the interface will prompt you to do so.

The interface will provide you with an overview of the changes you are about to submit.

Possible errors

Authorization issues If the access token you are providing is incorrect or does not have appropriate permissions, the interface will display an error like this during metadata retrieval or submission:

Incomplete information If a metadata record is missing required information, you will not be able to save it. Required fields - highlighted with a red asterix - and an warning sign will alert you to this.

Internal errors If there is an unexpected error during metadata submission, the submission interface will report technical information. Please contact m.hanke@fz-juelich.de and copy the entire error message.

Workflows

The TRR379 knowledge pooling tool supports several metadata-driven workflows to collect and curate research information. Each workflow describes how contributors can create, edit, and/or review records for a specific type of research output or activity.

Before starting a workflow, see the setup and general usage guides for instructions on accessing the tool and working with the web interface and backend API.

Use the links below to learn more about individual workflows:

• Publication reporting

Dump-things API

The metadata service is a small application built on FastAPI that can be deployed running in a virtual environment, managed by Hatch – running under an unprivileged user account. This scenario is described here. However, any other deployment approaches suitable for Python-based applications may work just as fine.

Required software

The only software that is required outside the virtual environment (and the web server) is pipx, which is used to deploy hatch for a user – no need for administrator privileges otherwise.

sudo apt install pipx --no-install-recommends

User account setup

Here we set up a dedicated user dumpthing to run the service. However, the service could also run under any other (existing) user account.

# new user, prohibit login, disable password
sudo adduser dumpthing --disabled-password --disabled-login

# allow this user to run prcoess while not logged in
sudo loginctl enable-linger dumpthing

# allow this user to execute systemd commands interactively.
# this needs XDG_RUNTIME_DIR define.
# the demo below is for ZSH
sudo -u dumpthing -s
cd
echo 'export XDG_RUNTIME_DIR="/run/user/$UID"' >> ~/.zshrc
# put `hatch` in the PATH for convenience
echo 'export PATH="/home/dumpthing/.local/bin:$PATH"' >> ~/.zshrc

Service environment setup

Everything in this section is done under the target user account. Use something like sudo -u dumpthing -s to enter it.

The following commands perform the initial setup, which provides an installation of the dump-things-service to query and encrich the TRR379 knowledge base.

# install `hatch` to run the service in a virtual environment
pipx install hatch

# obtain the source code for the service
git clone https://hub.trr379.de/q02/dump-things-service.git

# obtain the dataset with the (curated) metadata to be served
# by the service
git clone https://hub.trr379.de/q02/trr379-knowledge.git curated_metadata

# set up a directory for receiving metadata submissions
# each subdirectory in it must match a "token" the needs to be
# presented to the service to make it accept a record posting.
mkdir token_realms

# the service expects a particular data organization.
# we opt to create a dedicated root directory for it,
# and symlink all necessary components into it
mkdir server_root
ln -s ../curated_metadata/metadata server_root/global_store
ln -s ../token_realms server_root/token_stores

# now we can test-launch the service
hatch run fastapi:run --port 17345 /home/dumpthing/server_root

If the service comes up with no error, we can ctrl-c it.

Service management with systemd

We use systemd for managing the service process, the launch, and logging. This makes it largely unnecessary to interact with hatch directly, and allows for treating the user-space service like any other system service on the system.

The following service unit specification is all that is needed.

mkdir -p .config/systemd/user/
cat << EOT > .config/systemd/user/dumpthing.service
[Unit]
Description=DumpThing service (hatch environment)
Wants=network-online.target
After=network-online.target


[Service]
Type=simple
WorkingDirectory=/home/dumpthing/dump-things-service
ExecStart=/home/dumpthing/.local/bin/hatch run fastapi:run --port 17345 /home/dumpthing/server_root

[Install]
WantedBy=default.target
EOT

With this setup in place, we can control the service via systemd.

# launch the service
systemctl --user start dumpthing

# configure systemd to auto-launch the service in case of a
# system reboot
systemctl --user enable dumpthing.service

Web server setup

Here we use caddy as a reverse proxy to expose the services via https at metadata.trr379.de.

# append the following configuration to the caddy config
cat << EOT >> /etc/caddy/Caddyfile
# dumpthings service endpoints
metadata.trr379.de {
    reverse_proxy localhost:17345
}
EOT

A matching DNS setup must be configured separately.

Afterwards we can reload the web server configuration and have it expose the service.

# reload the webserver config to enable the reverse proxy setup
# (only necessary once)
sudo systemctl reload caddy

Updates and curation

Whenever there are updates to the to-be-served curated metadata, the setup described here only required the equivalent of a git pull to fetch these updates from the “knowledge” repository.

When records are submitted, they end up in the directory matching the token that was used for submission. Until such records are integrated with the curated metadata in global_store, they are only available for service requests that use that particular token.

An independent workflow must be used to perform this curation (acceptance, correction, rejection) of submitted records.

Neurobagel

NeuroBagel is a collection of containerized services that can be deployed in a variety of way. This page describes a deployment using podman and podman-compose that is confirmed to be working on machine with a basic Debian 12 installation.

For other installation methods, please refer to the NeuroBagel documentation.

The following instruction set up a “full-stack” NeuroBagel deployment. The contains all relevant components

• query front-end
• federation API
• node API
• graph database

This setup is suitable for a self-contained deployment, such as the central TRR379 node. Other deployments may only need a subset of these services.

On the target machine, NeuroBagel services will run “rootless”. This means they operate under a dedicated user account with minimal privileges.

Required software

Only podman, and its compose feature are needed. They can be installed via the system package manager.

sudo apt install podman podman-compose

User setup

We create a dedicated user neurobagel on the target machine. NeuroBagel will be deployed under this user account, and all software and data will be stored in its HOME directory.

# new user, prohibit login, disable password
sudo adduser neurobagel --disabled-password --disabled-login

# allow this user to run prcoess while not logged in
sudo loginctl enable-linger neurobagel

# allow this user to execute systemd commands interactively.
# this needs XDG_RUNTIME_DIR define.
# the demo below is for ZSH
sudo -u neurobagel -s
cd
echo 'export XDG_RUNTIME_DIR="/run/user/$UID"' >> ~/.zshrc
exit

Configure NeuroBagel

In the HOME directory of the neurobagel user we create the complete runtime environment for the service. All configuration is obtained from a Git repository.

# become the `neurobagel` user
sudo -u neurobagel -s
cd

# fetch the setup
git clone https://hub.trr379.de/q02/neurobagel-recipes recipes

# create the runtime directory
mkdir -p run/data/
mkdir -p run/secrets/

# copy over the demo data for testing (can be removed later)
cp recipes/data/* run/data

# generate passwords (using `pwgen` here, but could be any)
pwgen 20 1 > run/secrets/NB_GRAPH_ADMIN_PASSWORD.txt
pwgen 20 1 > run/secrets/NB_GRAPH_PASSWORD.txt

# configure the the address of the "local" NeuroBagel
# node to query
cat << EOT > recipes/local_nb_nodes.json
[
    {
        "NodeName": "TRR379 central node",
        "ApiURL": "https://nb-cnode.trr379.de" 
    }
]
EOT

Web server setup

NeuroBagel comprises a set of services that run on local ports that are routed to the respective containers. Here we use caddy as a reverse proxy to expose the necessary services via https at their canonical locations.

# append the following configuration to the caddy config
cat << EOT >> /etc/caddy/Caddyfile
# neurobagel query tool
nb-query.trr379.de {
    reverse_proxy localhost:13700
}
# neurobagel apis
# graph db api not exposed at 13701
nb-cnode.trr379.de {
    reverse_proxy localhost:13702
}
nb-federation.trr379.de {
    reverse_proxy localhost:13703
}
EOT

A matching DNS setup must be configured separately.

Manage NeuroBagel with systemd

We use systemd for managing the NeuroBagel service processes, the launch, and logging. This makes it largely unnecessary to interact with podman directly, and allows for treating the containerized NeuroBagel like any other system service.

The following service unit specification is all that is needed. With more recent versions of podman and podman-compose better setups are possible. using podman version. However, this one is working with the stock versions that come with Debian 12 (podman 4.3.1 and podman-composer 1.0.3) and requires no custom installations.

mkdir -p .config/systemd/user/
cat << EOT > .config/systemd/user/neurobagel.service
[Unit]
Description=NeuroBagel rootless pod (podman-compose)
Wants=network-online.target
After=network-online.target


[Service]
Type=simple
WorkingDirectory=/home/neurobagel/recipes
EnvironmentFile=/home/neurobagel/recipes/trr379.env
ExecStart=/usr/bin/podman-compose -f ./docker-compose.yml up
ExecStop=/usr/bin/podman-compose -f ./docker-compose.yml down

[Install]
WantedBy=default.target
EOT

Launch

With this setup in place, we can launch NeuroBagel

# reload the webserver config to enable the reverse proxy setup
# (only necessary once)
sudo systemctl reload caddy

# launch the service (pulls all images, loads data, etc).
# after a minutes the service should be up
systemctl --user start neurobagel

# configure systemd to auto-launch NeuroBagel in case of a
# system reboot
systemctl --user enable neurobagel.service

Archive DICOMs

As the “true” raw data, DICOMs are rarely (re)accessed and hardly ever need to change. However, they need to be stored somewhere. Tracking DICOMs in DataLad datasets allows dependency tracking for conversion to NIfTI. However, it’s good to keep DataLad optional (allow DataLad and non-DataLad access).

Historical precedent: ICF at FZJ

The following solution has been proposed for the Imaging Core Facility at FZJ:

• DICOMs are packed into tar files (tarballs) ¹
• the tarballs are placed on a web server (intranet only), organized by project (HTTP Basic Authentication for access management)
• DataLad datasets record availability via archivist and uncurl special remotes, which translates to:
  • a file is available from a tarball (archivist special remote)
  • a tarball is available from a given URL, pointing to the web server (uncurl special remote)².
• Only the Git repository (no annex) is stored by the consuming institute; the ICF web server is the permanent DICOM storage.

The system has been documented in https://inm-icf-utilities.readthedocs.io/en/latest/ and the tarball & dataset generation tools implementation is in https://github.com/psychoinformatics-de/inm-icf-utilities.

TRR reimplementation

One of the TRR sites indicated intent to use a Forgejo instance for DICOM storage. A particular challenge for the underlying system was inode limitation. For this reason, an adaptation of the ICF system has been proposed:

• a dataset is generated upfront, and DICOM tarball is stored with the dataset in Forgejo (as annex)
• we keep using the archivist remote (file in tarball) to avoid using up thousands of inodes for individual files (Git can pack its repository into several files, so we only add one more for the tarball).

A proof of principle for dataset generation (using re-written ICF code) has been proposed in https://hub.trr379.de/q02/dicom-utilities. See the README for more detailed explanations (and commit messages for even more detail).

BIDS conversion

Converting the heterogeneous, site-specific raw MRI data acquisitions into a standardized dataset is an essential precondition for the collaborative work in TRR379. It readies the data for processing with established pipelines, and applies a pseudonymization as a safeguard for responsible use of this personal data.

TRR379 uses the Brain Imaging Data Structure (BIDS) as the standard for its datasets.

Conversion to BIDS

The conversion of raw MRI data in DICOM format to a BIDS-compliant dataset is a largely automated process. The recommended software to be used for conversion is Heudiconv.

Heudiconv uses dcm2niix as the actual DICOM→NIfTI converter. In our experience, dcm2niix is the most robust and most correct tool available for this task.

Heudiconv does the job of mapping DICOM series to BIDS entities (ie. determine BIDS-compliant file names). A key heudiconv concept is a heuristic: a Python program (function) which looks at the DICOM series properties and matches it with a file naming pattern. A heuristic typically relies on DICOM series naming (set at the scanner console), but it can also use other properties such as number of images or acquisition parameters.

Because TRR379 uses its own conventions, a matching heuristic needs to be provided (possibly one for each TRR379 site). An implementation of such a heuristic has been created, and was tested on phantom MRI acquisitions from all sites (see below). Using this heuristic, MRI data from all sites can be BIDS-standardized. As with any automation, caution and oversight is needed for edge cases (e.g. repeated / discarded acquisitions).

Heudiconv tutorials further illustrate the process and capabilities of the software.

Good practices

• Use heudiconv as a containerized application. Q02 provides a readily usable utility dataset with a configured container. See that repository for an example usage.
• DICOMs as subdatasets helps with provenance, even if those DICOMs are never accessed outside
• Heudiconv takes paths and (optionally) intended subject IDs as input
  • if paths contain identifying information, this would leak into DataLad run records
  • having a helper script / lookup table in the (private) DICOM dataset can hide this information

Caveats

• https://hub.trr379.de/q02/phantom-mri-bids used dcm2niix v1.0.20240202
  • current latest is v1.0.20250506
  • potential impact discussed in https://hub.trr379.de/q02/phantom-mri-bids/issues/8

Demonstrators and resources

TRR phantom DICOMs

Scans of MRI phantoms were carried out using the intended sequences (presumably - see caveats section below). These were shared with Q02 and uploaded to the TRR Hub forgejo instance:

• https://hub.trr379.de/q01/phantom-mri-dicom-aachen
• https://hub.trr379.de/q01/phantom-mri-dicom-frankfurt
• https://hub.trr379.de/q01/phantom-mri-dicom-heidelberg
• https://hub.trr379.de/q01/phantom-mri-dicom-mannheim

Note: Aachen did a re-scan which was shared by e-mail / cloud (June 03, 2025). This has not been uploaded to forgejo (permissions + size).

TRR phantom BIDS

• A BIDS-compliant dataset from these dicoms (3/4 sites): https://hub.trr379.de/q02/phantom-mri-bids
• The heuristic used: https://hub.trr379.de/q02/phantom-mri-bids/src/branch/main/code/heuristic-q01.py
• Issue tracker: https://hub.trr379.de/q02/phantom-mri-bids/issues

Conversion of re-scanned Aachen phantom is in https://hub.trr379.de/q02/tmp-phantom-bids (separate from the above because input data is not available as a DataLad dataset)

Data consistency

• the phantom datasets are not the same: https://hub.trr379.de/q02/phantom-mri-bids/issues/6
• re-scan from Aachen has more sequences than the initial scan, but lacks T2w
• heudiconv fails to parse some Heidelberg dicoms, and dcm2niix raises warnings; unclear whether this is data issue or software issue: https://hub.trr379.de/q02/phantom-mri-bids/issues/5

Conversion: technical issues

These are open questions:

• Technical: BIDS validator errors https://hub.trr379.de/q02/phantom-mri-bids/issues/7
• Technical: re-run with the latest dcm2niix https://hub.trr379.de/q02/phantom-mri-bids/issues/8