容器的md5值: d3c01bcfd03bdc93952ad87f75dd1939

使用方法：
(注：“/path/to”表示举例路径，现网需要根据实际情况调整，建议部署在共享路径中，后续章节凡是遇到安装路径的命令，都以现网实际规划的安装路径为准进行替换，不再单独说明。)

一、安装singularity
yum -y install libatomic libstdc++ libstdc++-devel libseccomp-devel glib2-devel gcc squashfs-tools tar
cd /path/to/go_build 
wget https://go.dev/dl/go1.19.linux-arm64.tar.gz --no-ch 
tar -C /path/to/go_install -xf go1.19.linux-arm64.tar.gz 
export PATH=/path/to/go_install/go/bin:$PATH
source ~/.bashrc

cd /path/to/singularity_build 
wget https://github.com/sylabs/singularity/releases/download/v3.10.2/singularity-ce-3.10.2.tar.gz --no-ch 
tar -xf singularity-ce-3.10.2.tar.gz 
cd singularity-ce-3.10.2 
./mconfig --prefix=/path/to/singularity_install
make -C ./builddir 
make -C ./builddir install 
export PATH=/path/to/singularity_install/bin:$PATH

二、安装hmpi
cd /path/to/bs_build
wget https://mirrors.huaweicloud.com/kunpeng/archive/compiler/bisheng_compiler/BiShengCompiler-3.2.0-aarch64-linux-mini.tar.gz
tar xzvf BiShengCompiler-3.2.0-aarch64-linux-mini.tar.gz -C /path/to/bs_install --strip-components=1
export PATH=/path/to/bs_install/bin:$PATH
export LD_LIBRARY_PATH=/path/to/bs_install/lib:/path/to/bs_install/lib/clang/15.0.4/lib:$LD_LIBRARY_PATH
export INCLUDE=/path/to/bs_install/include:/path/to/bs_install/lib/clang/15.0.4/include:$INCLUDE
export CC=clang CXX=clang++ FC=flang

yum install -y perl-Data-Dumper autoconf automake libtool binutils flex cmake
cd /path/to/hucx_build
wget https://gitee.com/kunpengcompute/hucx/repository/archive/v2.3.0-huawei.tar.gz -f hucx-2.3.0-huawei.tar.gz
tar xf hucx-2.3.0-huawei.tar.gz
cd hucx-v2.3.0-huawei
./autogen.sh
./contrib/configure-opt --prefix=/path/to/hucx_install --disable-numa --enable-mt CFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" CXXFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion"
make -j
make -j install
export PATH=/path/to/hucx_install/bin:$PATH
export LD_LIBRARY_PATH=/path/to/hucx_install/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=/path/to/hucx_install/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/path/to/hucx_install/include:$CPLUS_INCLUDE_PATH
export CFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" CXXFLAGS="-Wno-unused-but-set-variable -Wno-error=int-conversion" FFLAGS="-Wno-error=int-conversion"

cd /path/to/xucg_build
wget https://gitee.com/kunpengcompute/xucg/repository/archive/v2.3.0-huawei.tar.gz -f xucg-2.3.0-huawei.tar.gz
tar xf xucg-2.3.0-huawei.tar.gz
cd xucg-v2.3.0-huawei
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/xucg_install -DCMAKE_BUILD_TYPE=Release -DUCG_BUILD_WITH_UCX=/path/to/hucx_install -DUCG_ENABLE_MT=ON -DUCG_BUILD_TESTS=OFF -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_Fortran_COMPILER=$FC
make -j
make -j install
export PATH=/path/to/xucg_install/bin:$PATH
export LD_LIBRARY_PATH=/path/to/xucg_install/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=/path/to/xucg_install/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/path/to/xucg_install/include:$CPLUS_INCLUDE_PATH

cd /path/to/hmpi_build
wget https://gitee.com/kunpengcompute/hmpi/repository/archive/v2.3.0-huawei.tar.gz -f hmpi-2.3.0-huawei.tar.gz
tar xf hmpi-2.3.0-huawei.tar.gz
cd hmpi-v2.3.0-huawei/
./autogen.pl
./configure --prefix=/path/to/hmpi_install --with-platform=contrib/platform/mellanox/optimized --enable-mpi1-compatibility --with-ucx=/path/to/hucx_install --with-ucg=/path/to/xucg_install
make -j
make -j install
export PATH=/path/to/hmpi_install/bin:$PATH
export LD_LIBRARY_PATH=/path/to/hmpi_install/lib:$LD_LIBRARY_PATH
export C_INCLUDE_PATH=/path/to/hmpi_install/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/path/to/hmpi_install/include:$CPLUS_INCLUDE_PATH

三、获取容器sif
在本页面下载relion_3.1.2_SVE_v1.sif文件，并上传到服务器目录/path/to/sif

四、获取测试算例relion_benchmark.tar.gz
wget ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion_benchmark.tar.gz 
算例目录解压后在 /path/to/TEST/ 目录层运行二进制


五、运行容器
1、单节点运行：
cd /path/to/TEST/relion_benchmark
mkdir class3d
可选命令1（使用外部mpi运行）：
time -p mpirun --allow-run-as-root -x LD_LIBRARY_PATH -x PATH -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -mca btl ^vader,tcp,openib,uct -x UCX_TLS=self,sm,dc -N 32 numactl --localalloc   --cpunodebind=0,1,2,3  singularity exec -B /path/to/TEST/relion_benchmark  /path/to/sif/relion_3.1.2_SVE_v1.sif    relion_refine_mpi --j 4 --cpu --pool 100 --dont_combine_weights_via_disc --i Particles/shiny_2sets.star --ref emd_2660.map:mrc --firstiter_cc --ini_high 60 --ctf --ctf_corrected_ref --iter 2 --tau2_fudge 4 --particle_diameter 360 --K 6 --flatten_solvent --zero_mask --oversampling 1 --healpix_order 2 --offset_range 5 --offset_step 2 --sym C1 --norm --scale --random_seed 0 --maxsig 500  --o class3d/test02


可选命令2（使用容器内部mpi运行）：
singularity exec -B /path/to/TEST/relion_benchmark  /path/to/sif/relion_3.1.2_SVE_v1.sif mpirun --allow-run-as-root -x LD_LIBRARY_PATH -x PATH $CCS_MPI_OPTIONS -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -mca btl ^vader,tcp,openib,uct -x UCX_TLS=self,sm,dc -N 32 numactl --localalloc   --cpunodebind=0,1,2,3  relion_refine_mpi --j 4 --cpu --pool 100 --dont_combine_weights_via_disc --i Particles/shiny_2sets.star --ref emd_2660.map:mrc --firstiter_cc --ini_high 60 --ctf --ctf_corrected_ref --iter 2 --tau2_fudge 4 --particle_diameter 360 --K 6 --flatten_solvent --zero_mask --oversampling 1 --healpix_order 2 --offset_range 5 --offset_step 2 --sym C1 --norm --scale --random_seed 0 --maxsig 500  --o class3d/test02


（说明：-B：绑定本地算例目录，exec命令进入容器为只读模式，不可在容器内对文件进行读写操作。np为MPI进程数量。程序运行结束时，在当前目录生成roms_examples.log，单节点可不依赖物理机上的mpi直接运行，对不需>要进行多节点运行的用户可选择命令2，即不需要额外安装mpi）

2、多节点运行：
time -p mpirun --allow-run-as-root -x LD_LIBRARY_PATH -x PATH -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -mca btl ^vader,tcp,openib,uct -x UCX_TLS=self,sm,dc --hostfile hostfile -N 32 numactl --localalloc   --cpunodebind=0,1,2,3  singularity exec -B /path/to/TEST/relion_benchmark  /path/to/sif/relion_3.1.2_SVE_v1.sif    relion_refine_mpi --j 4 --cpu --pool 100 --dont_combine_weights_via_disc --i Particles/shiny_2sets.star --ref emd_2660.map:mrc --firstiter_cc --ini_high 60 --ctf --ctf_corrected_ref --iter 2 --tau2_fudge 4 --particle_diameter 360 --K 6 --flatten_solvent --zero_mask --oversampling 1 --healpix_order 2 --offset_range 5 --offset_step 2 --sym C1 --norm --scale --random_seed 0 --maxsig 500  --o class3d/test02