Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/queue.csv
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
queue,submit_cmd,template
SLURM_RIKYU,sbatch,"-p ${queue_group} -t ${elapse} -N ${nodes} --ntasks-per-node=${numproc_node} --cpus-per-task=${nthreads} --gpus-per-node=${numproc_node}"
SLURM_RIKYU,sbatch,"-p ${queue_group} --account=${account} -t ${elapse} -N ${nodes} --ntasks-per-node=${numproc_node} --cpus-per-task=${nthreads} --gpus=${proc}"
FJ,pjsub,"-L rscunit=rscunit_ft01,rscgrp=${queue_group},elapse=${elapse},node=${nodes} --mpi max-proc-per-node=${numproc_node} -x PJM_LLIO_GFSCACHE=/vol0002:/vol0003:/vol0004:/vol0005"
PJM_GENKAI,pjsub,"-L rscgrp=${queue_group},elapse=${elapse},node=${nodes} --mpi proc=${proc}"
SLURM_RC,sbatch,"-p ${queue_group} -t ${elapse} -N ${nodes} --ntasks-per-node=${numproc_node} --cpus-per-task=${nthreads}"
Expand Down
6 changes: 3 additions & 3 deletions config/system.csv
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
system,mode,tag_build,tag_run,queue,queue_group
RIKYU,cross,ai4ss_login,ai4ss_jacamar,SLURM_RIKYU,1n1gpu
system,mode,tag_build,tag_run,queue,queue_group,account
RIKYU,cross,ai4ss_login,ai4ss_jacamar,SLURM_RIKYU,gpu,rkp00012
Fugaku,cross,fugaku_login1,fugaku_jacamar,FJ,small
FugakuCN,native,,fugaku_jacamar,FJ,small
RC_GH200,native,,cloud_jacamar,SLURM_RC,qc-gh200
RC_DGXSP,native,,cloud_jacamar,SLURM_RC,ng-dgx-3h
RC_DGXSP,native,,cloud_jacamar,SLURM_RC,ng-dgx-m1
RC_GENOA,native,,cloud_jacamar,SLURM_RC,genoa
RC_FX700,native,,cloud_jacamar,SLURM_RC,fx700
MiyabiG,cross,miyabi_g_login,miyabi_g_jacamar,PBS_Miyabi,debug-g
Expand Down
5 changes: 3 additions & 2 deletions docs/cx/BENCHKIT_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ BenchKit の system / queue 定義層である。
- `queue.csv` に scheduler template を持つ
- app 側の `list.csv` から system 固有事情を切り離す

`system.csv` は system 名、mode、runner tag、queue、queue_group などの正本である。
`system.csv` は system 名、mode、runner tag、queue、queue_group、scheduler account などの正本である。

This is the system and queue definition layer.

Expand All @@ -261,7 +261,8 @@ Main roles:
- `queue.csv` holds scheduler templates
- system-specific concerns are separated from app-side `list.csv`

`system.csv` is the source of truth for items such as system name, mode, runner tag, queue, and queue_group.
`system.csv` is the source of truth for items such as system name, mode,
runner tag, queue, queue_group, and scheduler account.

### 6.4 ポータル層 / Portal Layer

Expand Down
9 changes: 5 additions & 4 deletions docs/guides/add-site.md
Original file line number Diff line number Diff line change
Expand Up @@ -547,12 +547,12 @@ command_delay = "30s"
### `config/system.csv` にシステムを追加

```csv
system,mode,tag_build,tag_run,queue,queue_group
system,mode,tag_build,tag_run,queue,queue_group,account
# 既存エントリ...
# cross モード(ビルドと実行が別ノード)
NewSystem,cross,newsystem_login,newsystem_jacamar,PBS_NewSystem,default
NewSystem,cross,newsystem_login,newsystem_jacamar,PBS_NewSystem,default,
# native モード(同一ノードでビルドと実行)
NewSystemLN,native,,newsystem_login,none,default
NewSystemLN,native,,newsystem_login,none,default,
```

- `system`: システム名(アプリの `list.csv` から参照される)
Expand All @@ -561,6 +561,7 @@ NewSystemLN,native,,newsystem_login,none,default
- `tag_run`: 実行用GitLab Runnerタグ(`native`の場合はbuild_runジョブ用)
- `queue`: `config/queue.csv` のキュー名(ログインノードは `none`)
- `queue_group`: キューグループ名
- `account`: scheduler account/project。不要な拠点では空欄

### `config/queue.csv` にキューシステムを追加(必要な場合)

Expand All @@ -571,7 +572,7 @@ queue,submit_cmd,template
PBS_NewSystem,qsub,"-q ${queue_group} -l select=${nodes} -l walltime=${elapse} -W group_list=your_group"
```

テンプレート内で使える変数:`${queue_group}`, `${nodes}`, `${numproc_node}`, `${nthreads}`, `${elapse}`, `${proc}`(`nodes * numproc_node`), `${cpu_per_node}`, `${gpu_per_node}`, `${cpu_sockets}`(`nodes * cpu_per_node`), `${gpu_cards}`(`nodes * gpu_per_node`)
テンプレート内で使える変数:`${queue_group}`, `${account}`, `${nodes}`, `${numproc_node}`, `${nthreads}`, `${elapse}`, `${proc}`(`nodes * numproc_node`), `${cpu_per_node}`, `${gpu_per_node}`, `${cpu_sockets}`(`nodes * cpu_per_node`), `${gpu_cards}`(`nodes * gpu_per_node`)

`${cpu_per_node}` と `${gpu_per_node}` は `config/system_info.csv` から取得します。CPU socket 数や GPU card 数を scheduler に明示するサイトでは、`system_info.csv` の値も投入条件に使われます。

Expand Down
78 changes: 78 additions & 0 deletions programs/sbd/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# SBD benchmark

This BenchKit application runs the NVIDIA Thrust TPB selected-basis
diagonalization benchmark from `github.com/r-ccs-cms/sbd`.

The RIKYU recipe uses the H2O cc-pVDZ FCIDUMP with the `1em7` selected alpha
determinant file (about 628 million product determinants), one MPI rank per
B200 GPU, and the rank-distributed/index-reordered/NCCL build configuration
validated in the SubWG2 benchmark study. The inputs are included under
`programs/sbd/data/h2o/` and can be overridden with `BK_SBD_INPUT_DIR` when a
project-storage copy is preferred.

The DGX Spark route uses the H2O `1em5` input (about 30.4 million product
determinants), the Thrust backend with `cc120`, NVHPC/HPC-X CUDA 13 `26.3`,
and one MPI rank on the single GB10 GPU. The larger `1em7` case reaches about
98 GB before its first Davidson iteration and is killed by the 128 GB node's
memory limit. `1em5` retains a substantial GPU workload while leaving enough
memory headroom and runtime margin for continuous benchmarking. NCCL is disabled
for the single-GPU case, and all three explicit communicator sizes are one. The
Rikyu rows retain the `1em7` input and validated `1 x 2 x 2` explicit layout;
their remaining rank factor becomes SBD's implicit Hamiltonian communicator.

The R-CCS Cloud GH200 route uses the H2O `1em6` input (about 191 million
product determinants), the Thrust backend with `cc90`, NVHPC/HPC-X CUDA 13
`26.3`, and one MPI rank on its unified Grace Hopper superchip. No Slurm GPU
request is made because `qc-gh200` exposes its single GPU as part of the node.
NCCL is disabled for the single-rank case.

The R-CCS Cloud FX700 route uses the H2O `1em4` input (about 2.38 million
product determinants) with four MPI ranks and 12 OpenMP threads per rank,
bound one rank per A64FX NUMA/CMG domain. SBD requires C++17 features missing
from Fujitsu compiler 4.11.1's bundled libc++, so this route uses the system
GCC 8.5/MPICH stack with 512-bit SVE enabled. It links Fujitsu's optimized
LAPACK and its required runtime libraries by absolute path. The absolute paths
are intentional: adding the Fujitsu library directory with `-L` causes
MPICH's trailing `-lmpi` to resolve to Fujitsu MPI instead, mixing two MPI
implementations in one executable.

The input files are copied unchanged from SBD's tracked `data/h2o` directory;
`data/h2o/README.md` records the upstream revision, checksums, and Apache-2.0
provenance.

The emitted FOM is SBD's internal Davidson time in seconds. Shell wall time is
not used because MPI startup and scheduler overhead are substantial on RIKYU.
The run is accepted only when its energy matches the established reference for
its selected input within a combined `1e-12 + 1e-11 |E|` tolerance.

The DGX Spark route was validated on `ng-dgx-m2` with NVHPC 26.3. The exact
energy was `-76.24373504205295 Ha`; the internal Davidson FOM was
`228.928853 s`, and the complete Slurm job took 4 minutes 9 seconds.

The R-CCS Cloud GH200 route was directly validated on `qc-gh200-01` with
NVHPC 26.3. The exact energy was `-76.24377593489788 Ha`; the internal
Davidson FOM was `520.734720 s`, the multiply section was `22.730310 s`, and
the complete Slurm job took 9 minutes 34 seconds.

The R-CCS Cloud FX700 route was directly validated on `fx29` with GCC 8.5,
MPICH 4.0, 512-bit SVE, and Fujitsu LAPACK. The exact energy was
`-76.2429584823075 Ha`; the internal Davidson FOM was `747.636165 s`, the
multiply section was `33.897903 s`, and the complete Slurm job took 13 minutes
3 seconds.

The Rikyu route was validated with NVHPC 26.3 on project `rkp00012`. All
three rows converged to `-76.243776776861 Ha`:

| B200 GPUs | Nodes | Internal Davidson FOM (s) | 4-GPU-relative speedup |
|---:|---:|---:|---:|
| 4 | 1 | 565.510 | 1.00x |
| 8 | 2 | 290.222 | 1.95x |
| 16 | 4 | 152.144 | 3.72x |

A fresh BenchKit validation rebuilt upstream SBD commit
`9481f290c2f49d4f8e5df9b0c9c87ea0f7937c2c` through `build.sh`, confirmed the
effective `cc100` rank-distributed/index-reordered/NCCL configuration, and
submitted the eight-GPU row through `scripts/test_submit.sh`. The job converged
to `-76.2437767768609 Ha` with a `290.281226 s` internal Davidson FOM, and
`scripts/result.sh` produced valid result JSON with source provenance and the
`mult` timing section.
94 changes: 94 additions & 0 deletions programs/sbd/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/bin/bash
set -euo pipefail

system="$1"
source scripts/bk_functions.sh

REPO_URL="https://github.com/r-ccs-cms/sbd.git"
REPO_DIR="sbd"
BUILD_DIR=""
ARTIFACT_DIR="${PWD}/artifacts"

mkdir -p "${ARTIFACT_DIR}"
bk_fetch_source "${REPO_URL}" "${REPO_DIR}" "main"
cd "${REPO_DIR}"

case "${system}" in
RIKYU)
module purge
module load nvhpc/26.3
BUILD_DIR="build-rikyu-nvhpc-thrust-rankdist-nccl"
nccl_root="/shared/software/hpc_sdk/Linux_aarch64/26.3/comm_libs/13.1/nccl"
cmake -S . -B "${BUILD_DIR}" \
-DCMAKE_BUILD_TYPE=Release \
-DSBD_GPU_BACKEND=thrust \
-DSBD_GPU_ARCH=cc100 \
-DSBD_THRUST_SAFE_MPI_ALLREDUCE=ON \
-DSBD_USE_RANK_DISTRIBUTION=ON \
-DSBD_USE_BLOCK_RANK_DISTRIBUTION=ON \
-DSBD_REORDER_INDEX_ARRAY=ON \
-DSBD_USE_NCCL=ON \
-DCMAKE_CXX_FLAGS="-I${nccl_root}/include" \
-DCMAKE_EXE_LINKER_FLAGS="-L${nccl_root}/lib -lnccl"
cmake --build "${BUILD_DIR}" --parallel
;;
RC_DGXSP)
source /etc/profile.d/modules.sh
module purge
module load system/ng-dgx nvhpc-hpcx-cuda13/26.3
BUILD_DIR="build-dgxsp-nvhpc-thrust-rankdist"
cmake -S . -B "${BUILD_DIR}" \
-DCMAKE_BUILD_TYPE=Release \
-DSBD_GPU_BACKEND=thrust \
-DSBD_GPU_ARCH=cc120 \
-DSBD_THRUST_SAFE_MPI_ALLREDUCE=ON \
-DSBD_USE_RANK_DISTRIBUTION=ON \
-DSBD_USE_BLOCK_RANK_DISTRIBUTION=ON \
-DSBD_REORDER_INDEX_ARRAY=ON \
-DSBD_USE_NCCL=OFF
cmake --build "${BUILD_DIR}" --parallel
;;
RC_GH200)
module purge
module load system/qc-gh200 nvhpc-hpcx-cuda13/26.3
BUILD_DIR="build-gh200-nvhpc-thrust-rankdist"
cmake -S . -B "${BUILD_DIR}" \
-DCMAKE_BUILD_TYPE=Release \
-DSBD_GPU_BACKEND=thrust \
-DSBD_GPU_ARCH=cc90 \
-DSBD_THRUST_SAFE_MPI_ALLREDUCE=ON \
-DSBD_USE_RANK_DISTRIBUTION=ON \
-DSBD_USE_BLOCK_RANK_DISTRIBUTION=ON \
-DSBD_REORDER_INDEX_ARRAY=ON \
-DSBD_USE_NCCL=OFF
cmake --build "${BUILD_DIR}" --parallel
;;
RC_FX700)
module purge
module load system/fx700 mpi/mpich-aarch64
BUILD_DIR="build-fx700-gcc-mpich-rankdist"
fjlib="/opt/FJSVstclanga/cp-1.0.30.01/lib64"
cmake -S . -B "${BUILD_DIR}" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_CXX_FLAGS="-march=armv8.2-a+sve -msve-vector-bits=512" \
-DBLAS_LIBRARIES="${fjlib}/libfjlapack.so" \
-DLAPACK_LIBRARIES="${fjlib}/libfjlapack.so" \
-DCMAKE_CXX_STANDARD_LIBRARIES="${fjlib}/libfj90i.so ${fjlib}/libfj90f.so ${fjlib}/libfjsrcinfo.so -lelf" \
-DSBD_USE_RANK_DISTRIBUTION=ON \
-DSBD_USE_BLOCK_RANK_DISTRIBUTION=ON \
-DSBD_REORDER_INDEX_ARRAY=ON
cmake --build "${BUILD_DIR}" --parallel
;;
*)
echo "Unknown system: ${system}" >&2
exit 1
;;
esac

binary="${BUILD_DIR}/apps/chemistry_tpb_selected_basis_diagonalization/diag"
if [[ ! -x "${binary}" ]]; then
echo "SBD executable not found: ${binary}" >&2
exit 1
fi
cp "${binary}" "${ARTIFACT_DIR}/diag"
24 changes: 24 additions & 0 deletions programs/sbd/data/h2o/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# SBD H₂O benchmark inputs

These files are copied unchanged from the SBD repository's tracked
`data/h2o/` directory:

- Repository: https://github.com/r-ccs-cms/sbd
- Upstream revision: `02324eee32a49f3203522d230bcbc34ef032a6a6`
- Files: `fcidump.txt`, `h2o-1em4-alpha.txt`, `h2o-1em5-alpha.txt`,
`h2o-1em6-alpha.txt`, `h2o-1em7-alpha.txt`

SHA-256 checksums:

```text
a3c2302834a33dce7260e8050a3f5180e05dbba1bb748f3e2f6410a7eacbd94d fcidump.txt
858c1ef9d430aafe0c325281c7257c2e6cb0e77310a707c1109c55747db02139 h2o-1em4-alpha.txt
40b29271726daa2be5a5b181535647164e915646710c2641e46dd1ed078a2ee6 h2o-1em5-alpha.txt
c36920e57507d27dd49d2b729662cbb5a92afce26d61fb0e044e2691599eb804 h2o-1em6-alpha.txt
1f7af972b56143a9ae6862c02288668656a5bd3fb561337c430e0e478f7cb716 h2o-1em7-alpha.txt
```

The upstream SBD repository distributes these files under its Apache-2.0
license (`LICENSE.txt`). They are retained here so the BenchKit application
can run without private project storage. Set `BK_SBD_INPUT_DIR` to use an
equivalent project-storage copy instead.
Loading
Loading