😸
OpenCFDのコンパイル
FreeCADのアドオンCfdOFで並列計算するとエラーになったので、試しにOpenCFDをコンパイルして、確認してみます。(コンパイルに必要なものはインストール済みのつもり。)
手元の環境はWindows 11 WSL Ubutun 20.04になります。
ESI-OpenCFD版のv2112を使います。ソースをダウンロードして解凍します。
$ wget https://dl.openfoam.com/source/v2112/OpenFOAM-v2112.tgz
$ wget https://dl.openfoam.com/source/v2112/ThirdParty-v2112.tgz
$ mkdir ~/src
$ cd ~/src
$ tar zxvf ~/OpenFOAM-v2112.tgz
$ tar zxvf ~/ThirdParty-v2112.tgz
標準の設定でコンパイル
とりあえず、デフォルトのままやってみます。
大きな設定はOpenFOAM-v2112/etc/bashrcに記載してあります。一部抜粋すると、デフォルトの設定はこんなです。
OpenFOAM-v2112/etc/bashrc
# [WM_COMPILER_TYPE] - Compiler location:
# = system | ThirdParty
export WM_COMPILER_TYPE=system
# [WM_COMPILER] - Compiler:
# = Gcc | Clang | Icc | Icx | Amd | Arm | Cray | Fujitsu |
# Gcc<digits> | Clang<digits>
# [Not well tested: Pgi | Nvidia]
export WM_COMPILER=Gcc
# [WM_MPLIB] - MPI implementation:
# = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPI | MPICH | MPICH-GM |
# HPMPI | CRAY-MPICH | FJMPI | QSMPI | SGIMPI | INTELMPI | USERMPI
# Specify SYSTEMOPENMPI1, SYSTEMOPENMPI2 for internal tracking (if desired)
# Can also use INTELMPI-xyz etc and define your own wmake rule
export WM_MPLIB=SYSTEMOPENMPI
ThirdPartyのコンパイル
初めにThirdPartyをコンパイルします。
$ source ~/src/OpenFOAM-v2112/etc/bashrc
$ cd ~/src/ThirdParty-v2112
$ ./Allwmake -j
METISとADIOS2でエラーが出ました。
METISのエラー
========================================
METIS decomposition
Did not find ... metis-5.1.0
Missing sources: 'metis-5.1.0'
Possible download locations for metis :
http://glaros.dtc.umn.edu/gkhome/metis/metis/overview
http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/metis-5.1.0.tar.gz
---------------------------------------------------
Optional component (METIS) had build issues
OpenFOAM will nonetheless remain largely functional
---------------------------------------------------
ADIOS2のエラー
========================================
ADIOS2
Did not find ... ADIOS2-2.6.0
========================================
Build adios library ADIOS2-2.6.0 for sys-openmpi
Starting build: ADIOS2-2.6.0 (using cmake)
./makeAdios2: 191: cd: can't cd to $HOME/src/ThirdParty-v2112/ADIOS2-2.6.0
Error building: ADIOS2-2.6.0
---------------------------------------------------
Optional component (ADIOS2) had build issues
OpenFOAM will nonetheless remain largely functional
---------------------------------------------------
どちらもソースがない模様。
METISのソースをダウンロードして、ThirdParty-v2112/sourcesに展開。
$ wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/metis-5.1.0.tar.gz
$ tar zxvf metis-5.1.0.tar.gz -C sources
ADIOS2は2.6.0ではなく2.7.1があったので、そちらを使うことにします。解凍するとADIOS2-2.7.1というディレクトリができます。
$ wget https://github.com/ornladios/ADIOS2/archive/refs/tags/v2.7.1.tar.gz
$ tar zxvf v2.7.1.tar.gz
使用するADIOS2のバージョンをデフォルトから変更するので、OpenFOAM-v2112/etc/config.sh/adios2を編集します。
OpenFOAM-v2112/etc/config.sh/adios2
--- OpenFOAM-v2112/etc/config.sh/adios2,orig 2021-12-21 03:57:43.000000000 +0900
+++ OpenFOAM-v2112/etc/config.sh/adios2 2022-03-13 21:49:25.779751600 +0900
@@ -21,7 +21,7 @@
#------------------------------------------------------------------------------
# USER EDITABLE PART: Changes made here may be lost with the next upgrade
-adios2_version=ADIOS2-2.6.0
+adios2_version=ADIOS2-2.7.1
export ADIOS2_ARCH_PATH="$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$adios2_version"
# END OF (NORMAL) USER EDITABLE PART
OpenFOAM-v2112/etc/bashrcの再読み込みは必要ないかもしれないけど、いちおう。
$ source ~/src/OpenFOAM-v2112/etc/bashrc
$ ./Allwmake -j
で、コンパイルできました。
OpenFOAMのコンパイル
ThirdPartyのコンパイルが終わったので、次にOpenFOAMをコンパイルします。
$ cd ~/src/OpenFOAM-v2112
$ ./Allwmake -j -s -l
CfdOFで動作確認
OpenFOAMのコンパイルも終わったら、CfdOFで参照するOpenFOAMのインストールディレクトリの設定を修正して、動作確認をしてみます。
む、ダメです。並列計算の実行でエラーとなりました。
CfdOFから並列計算の実行でのエラー
--> FOAM FATAL ERROR: (openfoam-2112)
attempt to run parallel on 1 processor
From static bool Foam::UPstream::init(int&, char**&, bool)
in file UPstream.C at line 297.
FOAM aborting
#0 Foam::error::printStack(Foam::Ostream&)
--> FOAM FATAL ERROR: (openfoam-2112)
attempt to run parallel on 1 processor
From static bool Foam::UPstream::init(int&, char**&, bool)
in file UPstream.C at line 297.
FOAM aborting
at ??:?
#1 Foam::error::simpleExit(int, bool)#0 Foam::error::printStack(Foam::Ostream&)
--> FOAM FATAL ERROR: (openfoam-2112)
attempt to run parallel on 1 processor
From static bool Foam::UPstream::init(int&, char**&, bool)
in file UPstream.C at line 297.
FOAM aborting
#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#2 Foam::error::exiting(int, bool) at ??:?
#1 Foam::error::simpleExit(int, bool)
--> FOAM FATAL ERROR: (openfoam-2112)
attempt to run parallel on 1 processor
From static bool Foam::UPstream::init(int&, char**&, bool)
in file UPstream.C at line 297.
FOAM aborting
#0 Foam::error::printStack(Foam::Ostream&) at ??:?
at ??:?
#1 Foam::error::simpleExit(int, bool)#2 Foam::error::exiting(int, bool) at ??:?
#3 Foam::UPstream::init(int&, char**&, bool) at ??:?
#4 Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#2 Foam::error::exiting(int, bool) at ??:?
#3 Foam::UPstream::init(int&, char**&, bool) at ??:?
#5 at ??:?
#4 Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#3 Foam::UPstream::init(int&, char**&, bool) at ??:?
#4 Foam::argList::argList(int&, char**&, bool, bool, bool) at ??:?
#1 Foam::error::simpleExit(int, bool)? at ??:?
#5 in ~/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/bin/potentialFoam
#6 __libc_start_main at ??:?
#5 in /lib/x86_64-linux-gnu/libc.so.6
#7 ?? at ??:?
#2 Foam::error::exiting(int, bool)? in ~/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/bin/potentialFoam
#6 __libc_start_main in ~/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/bin/potentialFoam
[OPTIPLEX:00691] *** Process received signal ***
[OPTIPLEX:00691] Signal: Aborted (6)
[OPTIPLEX:00691] Signal code: (-6)
[OPTIPLEX:00691] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0)[0x7f82150b83c0]
[OPTIPLEX:00691] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f8214ef503b]
[OPTIPLEX:00691] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f8214ed4859]
[OPTIPLEX:00691] [ 3] /home/zzzzz/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam5error10simpleExitEib+0x17a)[0x7f8215823d2a]
[OPTIPLEX:00691] [ 4] /home/zzzzz/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam5error7exitingEib+0x4a)[0x7f821582732a]
[OPTIPLEX:00691] [ 5] /home/zzzzz/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/sys-openmpi/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x1ad9)[0x7f8214ea2dc9]
[OPTIPLEX:00691] [ 6] /home/zzzzz/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x7e2)[0x7f8215853f42]
[OPTIPLEX:00691] [ 7] potentialFoam(+0x31ed3)[0x56249e35aed3]
[OPTIPLEX:00691] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f8214ed60b3]
[OPTIPLEX:00691] [ 9] potentialFoam(+0x347ee)[0x56249e35d7ee]
[OPTIPLEX:00691] *** End of error message ***
in ~/src/OpenFOAM-v2112/platforms/linux64GccDPInt32Opt/bin/potentialFoam
#6 __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 690 RUNNING AT OPTIPLEX
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestion
使用するOpenMPIを変更
OpenMPIをコンパイルして使うようにしてみます。
OpenFOAM-v2112/etc/bashrc
--- OpenFOAM-v2112/etc/bashrc,orig 2021-12-21 03:57:43.000000000 +0900
+++ OpenFOAM-v2112/etc/bashrc 2022-03-14 21:55:31.189751600 +0900
@@ -95,7 +95,7 @@
# HPMPI | CRAY-MPICH | FJMPI | QSMPI | SGIMPI | INTELMPI | USERMPI
# Specify SYSTEMOPENMPI1, SYSTEMOPENMPI2 for internal tracking (if desired)
# Can also use INTELMPI-xyz etc and define your own wmake rule
-export WM_MPLIB=SYSTEMOPENMPI
+export WM_MPLIB=OPENMPI
#------------------------------------------------------------------------------
再度、ThirdPartyのコンパイル
$ source ~/src/OpenFOAM-v2112/etc/bashrc
$ cd ~/src/ThirdParty-v2112
$ ./Allwmake -j
なんか、SCOTCHでエラーがあったみたいです。
SCOTCHのエラー
make \
CC="/home/zzzzz/src/ThirdParty-v2112/platforms/linux64Gcc/openmpi-4.1.2/bin/mpicc" \
CFLAGS="-m64 -DOPENFOAM=2112 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -O3 -fPIC -fPIC -O3 -UCOMMON_FILE_COMPRESS -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_DETERMINISTIC -DSCOTCH_RENAME -DIDXSIZE64 -Drestrict=__restrict -DSCOTCH_PTSCOTCH" \
CCDFLAGS="-m64 -DOPENFOAM=2112 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -O3 -fPIC -fPIC -O3 -UCOMMON_FILE_COMPRESS -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_DETERMINISTIC -DSCOTCH_RENAME -DIDXSIZE64 -Drestrict=__restrict -DSCOTCH_PTSCOTCH" \
ptscotch.h \
ptscotchf.h \
libptscotch.so \
libptscotcherr.so \
libptscotcherrexit.so
make[2]: ディレクトリ '/home/zzzzz/src/ThirdParty-v2112/sources/scotch/scotch_6.1.0/src/libscotch' に入ります
/home/zzzzz/src/ThirdParty-v2112/platforms/linux64Gcc/openmpi-4.1.2/bin/mpicc -m64 -DOPENFOAM=2112 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -O3 -fPIC -fPIC -O3 -UCOMMON_FILE_COMPRESS -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_DETERMINISTIC -DSCOTCH_RENAME -DIDXSIZE64 -Drestrict=__restrict -DSCOTCH_PTSCOTCH -DSCOTCH_VERSION_NUM=6 -DSCOTCH_RELEASE_NUM=1 -DSCOTCH_PATCHLEVEL_NUM=0 dummysizes.c -o ptdummysizes -Xlinker --no-as-needed -m64 -lm -lrt
In file included from dummysizes.c:92:
common.h:123:21: fatal error: mpi.h: そのようなファイルやディレクトリはありません
123 | #include <mpi.h>
| ^~~~~~~
compilation terminated.
ThirdParty-v2112に入っているSCOTCHはバージョン6.1.0ですが、バージョン7.0.1と6.1.3が出ているようなので、6.1.3を使ってみます。
$ wget https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.3/scotch-v6.1.3.tar.gz
$ tar zxvf scotch-v6.1.3.tar.gz -C sources/scotch
$ mv sources/scotch/scotch-v6.1.3 sources/scotch/scotch_6.1.3
使用するSCOTCHのバージョンを変えるので、OpenFOAM-v2112/etc/config.sh/scothを修正します。
OpenFOAM-v2112/etc/config.sh/scotch
--- OpenFOAM-v2112/etc/config.sh/scotch,orig 2021-12-21 03:57:43.000000000 +0900
+++ OpenFOAM-v2112/etc/config.sh/scotch 2022-03-14 23:09:30.169751600 +0900
@@ -33,7 +33,7 @@
#------------------------------------------------------------------------------
# USER EDITABLE PART: Changes made here may be lost with the next upgrade
-SCOTCH_VERSION=scotch_6.1.0
+SCOTCH_VERSION=scotch_6.1.3
export SCOTCH_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER$WM_PRECISION_OPTION$WM_LABEL_OPTION/$SCOTCH_VERSION
# END OF (NORMAL) USER EDITABLE PART
で、再度、コンパイル。
$ ./Allwmake -j
再度、OpenFOAMのコンパイル
ThirdPartyのコンパイルが終わったら、OpenFOAMを再度コンパイルします。
$ cd ~/src/OpenFOAM-v2112
$ ./Allwmake -j -s -l
CfdOFで動作確認
今度はうまく並列計算を計算実行できました！
Discussion