Topology ToolKit

Tutorials

Disclaimer: this video tutorial uses an older version of our software stack. Apart from updated software (as of February 2019), a few instructions have been modified. These modifications mostly occur when exporting a ParaView pipeline as a Catalyst Python script (c.f. section 6).

TTK in-situ with Catalyst

These notes describe the installation procedure of TTK for an in-situ usage with Catalyst. In particular, this specific version of the notes describe the procedure for a Ubuntu Linux 18.04.1 operating system.

Users of older versions of TTK or ParaView can also check the previous version of this tutorial:
Note that for an in-situ usage with Catalyst, TTK must be installed twice (same version, 0.9.7):
  • In an interactive client mode, on a system dedicated to the interactive design of the data analysis and visualization pipeline, typically a workstation. For this purpose, see the main installation of TTK and add PARAVIEW_USE_MPI=ON at the step 5.a.
  • In a batch server mode, on a system dedicated to the actual execution of the simulation code, typically a high-performance system. The remainder of these notes describe the installation for this batch server mode.

For illustration purposes, the in-situ features of TTK will be showcased from within an existing computational fluid dynamic simulation code, Code_Saturne, which already implements Catalyst interfacing features. See the Catalyst documentation to interface your own simulation code with Catalyst.

1. Downloads

The in-situ usage of TTK requires the installation of several third-party packages, from source:

2. Installing the dependencies

Several dependencies will need to be installed in order to compile the packages from source. Please enter the following command (omit the $ character) in a terminal to install them (please see the documentation of your package manager if your operating system is not Ubuntu Linux 18.04.1):

$ sudo apt install build-essential cmake-curses-gui llvm \
      mpich libboost-all-dev

3. Preparing the sources

Move the tarballs to a working directory (for instance called ~/ttk/) and decompress them by entering the following commands in a terminal (this assumes that you downloaded the tarballs to the ~/Downloads/ directory): 

$ mkdir ~/ttk
$ mv ~/Downloads/mesa-18.3.3.tar.gz ~/ttk/
$ mv ~/Downloads/ParaView-v5.6.0.tar.gz ~/ttk/
$ mv ~/Downloads/ttk-0.9.7.tar.gz ~/ttk/

$ cd ~/ttk/
$ tar xvzf mesa-18.3.3.tar.gz
$ tar xvzf ParaView-v5.6.0.tar.gz
$ tar xvzf ttk-0.9.7.tar.gz
Now , you can delete the tarballs after the source trees have been decompressed. 

$ rm mesa-18.3.3.tar.gz
$ rm ParaView-v5.6.0.tar.gz
$ rm ttk-0.9.7.tar.gz

4. Patching ParaView source tree

In order to enjoy the complete set of TTK features, we recommend at this stage to patch the ParaView source tree. This step is optional. To proceed, go to the patch directory and apply it as follows: 

$ cd ~/ttk/ttk-0.9.7/paraview/patch/
$ ./patch-paraview-5.6.0.sh ~/ttk/ParaView-v5.6.0/

5. Configuring, building and installing

a) Mesa
To configure Mesa’s source tree, enter the following command (for more details about the compilation flags, see this blog entry) : 

$ cd ~/ttk/mesa-18.3.3/
$ mkdir build
$ cd build/
$ ../configure                                          \
      --enable-opengl --disable-gles1 --disable-gles2   \
      --disable-va --disable-xvmc --disable-vdpau       \
      --enable-shared-glapi                             \
      --with-gallium-drivers=swrast                     \
      --disable-dri --with-dri-drivers=                 \
      --disable-egl --with-egl-platforms= --disable-gbm \
      --disable-glx                                     \
      --disable-osmesa --enable-gallium-osmesa          \
      --enable-texture-float
Now you can start the compilation process by entering the following command, where <N> is the number of available cores on your system: 
$ make -j<N>
Once the build is finished, enter the following command to install your build of ParaView on your system: 
$ sudo make install
b) ParaView
To configure ParaView's build and generate build commands, enter the following commands: 

$ cd ~/ttk/ParaView-v5.6.0/
$ mkdir build
$ cd build/
$ cmake                                                           \
      -DCMAKE_BUILD_TYPE=Release                                  \
      -DPARAVIEW_INSTALL_DEVELOPMENT_FILES=ON                     \
      -DPARAVIEW_ENABLE_PYTHON=ON                                 \
      -DVTK_PYTHON_VERSION=3                                      \
      -DPARAVIEW_BUILD_QT_GUI=OFF                                 \
      -DPARAVIEW_USE_MPI=ON                                       \
      -DPARAVIEW_ENABLE_CATALYST=ON                               \
      -DVTK_USE_X=OFF                                             \
      -DVTK_OPENGL_HAS_OSMESA=ON                                  \
      -DVTK_USE_OFFSCREEN=OFF                                     \
      -DOSMESA_INCLUDE_DIR=/usr/local/include                     \
      -DOSMESA_LIBRARY=/usr/local/lib/libOSMesa.so                \
      -DPYTHON_INCLUDE_DIR=/usr/include/python3.6                 \
      -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so \
      ..
The previous cmake options are either required by TTK or used to enable in-situ computation for ParaView. (for further details, see this documentation). Now you can start the compilation process by entering the following command, where <N> is the number of available cores on your system (this will take a LONG time): 
$ make -j<N>
Once the build is finished, enter the following command to install your build of ParaView on your system: 
$ sudo make install
c) TTK
To enter the configuration menu of TTK's build, enter the following commands: 

$ cd ~/ttk/ttk-0.9.7/
$ mkdir build
$ cd build/
$ cmake                                                     \
      -DCMAKE_PREFIX_PATH=/usr/local/lib/cmake/paraview-5.6 \
  ..
At this stage, under Linux, TTK's build should be automatically and correctly configured, and generate build commands directly. Once the generation is completed, start the compilation process by entering the following command, where <N> is the number of available cores on your system (this will take a LONG time): 
$ make -j<N>
If the build process exhausts your system memory, reduce the number of cores used with a lower <N>.

Once the build is finished, enter the following command to install your build of TTK on your system: 
$ sudo make install

6. A computational fluid dynamics use-case: Code_Saturne

In the remainder of this tutorial, we will switch to a concrete use-case of in-situ usage of TTK with Catalyst. For this, we will focus on a computational fluid dynamics simulation code, Code_Saturne, which already implements interfacing features with Catalyst.

In particular, we will go through the following steps:
  1. install all the required software packages [client and server]
  2. generate a first representative data set with Code_Saturne [client]
  3. create an analysis and visualization pipeline for the representative data set with TTK [client]
  4. export the analysis and visualization pipeline with Catalyst [client]
  5. integrate the Catalyst/TTK script into Code_Saturne [client, then server]
a) Downloads (both systems, client and server)
The Code_Saturne source tree (version 5.0.9, released in September 2018) can be downloaded here.
The Code_Saturne tutorial example that we will use in the remainder can be downloaded there.

b) Installing the dependencies (both systems, client and server)
Several dependencies will need to be installed in order to compile the packages from source. Please enter the following command (omit the $ character) in a terminal to install them (please see the documentation of your package manager if your operating system is not Ubuntu Linux): 
$ sudo apt install pyqt5-dev-tools libxml2-dev
c) Preparing the sources (both systems, client and server)
Move the tarballs to a working directory (for instance called ~/ttk/) and decompress them by entering the following commands in a terminal (this assumes that you downloaded the tarballs to the ~/Downloads/ directory): 

$ mv ~/Downloads/code_saturne-5.0.9.tar.gz ~/ttk/
$ cd ~/ttk/
$ tar xvzf code_saturne-5.0.9.tar.gz
$ rm code_saturne-5.0.9.tar.gz
d) Configuring, building and installing Code_Saturne (both systems, client and server)
To configure Code_Saturne’s source tree, enter the following commands: 

$ cd ~/ttk/code_saturne-5.0.9/
$ mkdir build
$ cd build/
$ PYTHON=/usr/bin/python3        \
  ../configure                   \
      --with-catalyst=/usr/local \
      --disable-catalyst-as-plugin
Now you can start the compilation process by entering the following command, where <N> is the number of available cores on your system (this will take a LONG time): 
$ make -j<N>
Once the build is finished, enter the following command to install your build of Code_Saturne on your system: 
$ sudo make install
e) Post-installation configuration (both systems, client and server)
Once Code_Saturne is installed, use the following command to set environment variables in the ~/.bashrc configuration file: 

$ cat >> ~/.bashrc <<"EOF"
# ParaView Catalyst environnment variables
# (ParaView version 5.6)
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib/paraview-5.6:$LD_LIBRARY_PATH
EOF
And to apply these modifications on the current shell, enter: 
$ source ~/.bashrc
f) Creating the representative Code_Saturne data set (client system)
Now, following the Code_Saturne tutorial example, we will prepare the computation directories.Create the study simple_junction and the calculation directory case1 by entering the commands: 

$ cd ~/ttk/
$ code_saturne create -s simple_junction -c case1
More details on how to change the parameters of the study can be found in the tutorial. In the following, we will use the pre-configured study located in Code_Saturne source files ~/ttk/code_saturne-5.0.9/examples/simple_junction/ and launch the simulation. 

$ cp ~/ttk/code_saturne-5.0.9/examples/1-simple_junction/case1/case1.xml \
     ~/ttk/simple_junction/case1/DATA/
$ cp ~/ttk/code_saturne-5.0.9/examples/1-simple_junction/mesh/downcomer.des \
     ~/ttk/simple_junction/MESH/
$ cd ~/ttk/simple_junction/case1/
$ code_saturne run --param case1.xml
Once this step is finished, a representative Code_Saturne data set is generated. We will use this data set next to design our analysis and visualization pipeline with TTK.
To visualize the representative data set, open in ParaView the file RESULTS_ENSIGHT.case in the directory ~/ttk/simple_junction/case1/RESU/<date-time>/postprocessing/

g) Generating the Catalyst script (client system)
We will describe here the general procedure to generate automatically a ParaView Catalyst pipeline, applied to a simple junction testcase.

The first step is to start the interactive ParaView on the client system (typically a workstation). In order to create an analysis and visualization pipeline, we provide a ParaView state file that reproduces the first default built-in example of TTK (critical points, warped view, persistence curve and persistence diagram). Download it here. In ParaView, click on the Load State… entry in the File menu. Select Choose Files Names in the drop-down menu, then modify the Case File Name to match the location of the file RESULTS_ENSIGHT.case. Click on the OK button to confirm.


ParaView will apply the visualization pipeline on the dataset and you should obtain something similar to the screenshot below:


Once the full pipeline has been created, a Python script must be exported from ParaView. This is done by choosing the Generate Script -deprecated entry in the Catalyst menu. Exporting Catalyst scripts should be done in a different way in newer versions of ParaView, but some options are only available by using this deprecated way.

In the form, you must first select the sources (i.e. pipeline objects without any input connections, in this example: RESULTS_ENSIGHT.case) by Adding them to the right column, before confirming by using Next.
In the next page, the simulation name should be rename from RESULTS_ENSIGHT.case to input. This particular name is expected by Code_Saturne to correctly call any ParaView pipeline. Double-click on the text field to modify it.


The next page in the wizard offers several rendering options, select Output rendering components i.e. views, you will be able to modify a few other options such as the type and the name of the output images or the write frequency, etc.
You can iterate through the rendering views with the buttons Previous View and Next View. 

Image Type: png
Write Frequency: 1
Magnification: 1
For the rendering views, use the following parameters (screenshot below):


For the rendering views of the persistence diagram and curve, check also the parameter Fit the Screen as shown by the images below:


The persistence diagram itself is configured to be written every 10 time-steps in the Properties panel of the ParallelUnstructuredGridWriter1 filter in the pipeline (screenshot below).


Finally, click on the Finish button to create the Python script and name it ttk_pipeline.py.

h) Patching the Catalyst script (client system)
To automatically load the TTK plugins, add these lines of code just below the two Python from paraview[] import [] lines in the pre-configured script ttk_pipeline.py like in the image below: 

# --------------------------------------------------------------
# The following loads TTK's plugins.
# Topology Toolkit 0.9.7
# --------------------------------------------------------------
import glob
import os
from os.path import join as ttk_path_join

ttk_plugins_path = "/usr/local/lib/plugins/"
for x in glob.glob(
    ttk_path_join(ttk_plugins_path, "*.so" if os.name == "posix" else "*.dll")
):
    LoadPlugin(x, ns=globals())

Note that /usr/local/lib/plugins/ is the default location where TTK plugins are installed.

i) Integrating the Catalyst script in the Code_Saturne project (client system)
To configure the project to use Catalyst and execute the script in-situ, enter the commands: 

$ cd ~/ttk/simple_junction/case1/DATA/
$ ./SaturneGUI case1.xml
The study configuration tool of Code_Saturne opens. In the left panel, click on Calculation control then in the drop-down menu on Output control. This will update the central part of the window and show the options related to the generation of the results. Select the Writer tab and click on the + button at the bottom to add a new writer module. Configure the writer module with the following parameters (as shown by the screenshot image below): 

Name: ttk_pipeline
Id: 1
Format: Catalyst
Directory: postprocessing
Then modify the writing frequency to suit your needs, here is an example: 

Frequency:
    · Output every ‘n’ time steps
    · 1
    · Output at end of calculation

Now to associate the Catalyst writer with the mesh, click on the Mesh tab and select the mesh part named Fluid domain. Then click on the + button in the section Associated writers, normally ttk_pipeline will appear as the new one. You should obtain something like the screenshot image below:


If you want more details about the configuration of a Code_Saturne study with Catalyst, you can follow this video tutorial.

Finally, save the parameters by clicking on the Save entry in the File menu.

j) Running TTK in-situ (server system)
In the following, we describe how to configure a Code_Saturne / Catalyst project on the server side. Alternatively, jump to this paragraph to directly download a preconfigured mirror package.

First, create a “mirror” project on the server system with the commands: 

$ cd ~/ttk/
$ code_saturne create -s simple_junction -c case1
Now, copy the Catalyst script ttk_pipeline.py (section 6.f) and the parameters file case1.xml from the client system (see the directory ~/ttk/simple_junction/case1/DATA/) to the same mirror directory on the server system (same absolute path). Copy also the mesh downcomer.des located in ~/ttk/simple_junction/MESH/ the same way. Here, please choose the appropriate transfer method between your client and server systems based on the nature of their connection (for instance, on a TCP/IP network, the command scp may be recommended).

Alternatively, you can also simply download the following tarball, which already contains a mirror of the client’s project. To decompress this tarball, enter the following commands (this assumes that you downloaded the tarballs to the ~/Downloads/ directory): 

$ mv ~/Downloads/simple_junction.tar.gz ~/ttk/
$ cd ~/ttk/
$ tar xvzf simple_junction.tar.gz
$ rm simple_junction.tar.gz
You are now ready to run your Code_Saturne simulation with an in-situ usage of TTK. For this, enter the following commands: 

$ cd ~/ttk/simple_junction/case1/
$ code_saturne run --param case1.xml
k) In-situ outputs (on the server system)
The calculation results are stored in ~/ttk/simple_junction/case1/RESU/<date-time>/

Once the simulation has finished, you should have a sequence of image files (*.png), for each view configured in the Catalyst script, as well as persistence diagrams (*.vtu), computed every 10 iterations. To generate videos out of these image sequences, please see the ffmpeg documentation.