Skip to content

Fix Rikyu and DGX Spark scheduler configuration - #74

Open
william-dawson wants to merge 2 commits into
RIKEN-RCCS:developfrom
william-dawson:fix-rikyu-cloud-site
Open

Fix Rikyu and DGX Spark scheduler configuration#74
william-dawson wants to merge 2 commits into
RIKEN-RCCS:developfrom
william-dawson:fix-rikyu-cloud-site

Conversation

@william-dawson

@william-dawson william-dawson commented Jul 31, 2026

Copy link
Copy Markdown

Problem

BenchKit's shared scheduler configuration no longer matched the live Rikyu
and R-CCS Cloud DGX Spark queues:

  • Rikyu used the retired 1n1gpu partition and did not pass a project account.
  • DGX Spark used the retired ng-dgx-3h partition.

Changes

  • Add an optional account field to config/system.csv and expose it to
    scheduler templates.
  • Configure Rikyu to use partition gpu and account rkp00012.
  • Configure DGX Spark to use partition ng-dgx-m1.
  • Pass the Rikyu account through matrix generation and test_submit.sh.
  • Request Rikyu GPUs as the scheduler-required total --gpus=N, rather than
    the unsupported per-node form.
  • Document the optional site-level scheduler account.

Systems that do not require an explicit account leave the field empty and
retain their current behavior.

Validation

  • bash -n scripts/job_functions.sh scripts/matrix_generate.sh scripts/test_submit.sh
  • python3 result_server/tests/check_site_config.py
  • git diff --check
  • Generated the matrix under Linux Bash on Rikyu and confirmed:
    • Rikyu: -p gpu --account=rkp00012
    • DGX Spark: -p ng-dgx-m1
  • Used scripts/test_submit.sh sbd 2 from the dependent PR checkout to submit
    a real two-node/eight-GPU Rikyu job. Slurm accepted
    --gpus=8, the job completed successfully, and SBD converged to
    -76.2437767768609 Ha.

The SBD application PR depends on this shared site fix.

Signed-off-by: William Dawson <william.dawson@riken.jp>
Signed-off-by: William Dawson <william.dawson@riken.jp>
@william-dawson william-dawson mentioned this pull request Jul 31, 2026
4 tasks
william-dawson added a commit to william-dawson/benchkit that referenced this pull request Jul 31, 2026
Build: nvhpc/26.3 + OpenACC + cc100 (B200/Blackwell), with
FORTRAN_COMPILER_HAS_MPI_VERSION3=OFF to work around the
MPI_Iallreduce hang at >1 rank.

Run: per-rank GPU binding via wrapper.sh (CUDA_VISIBLE_DEVICES),
UCX_IB_GPU_DIRECT_RDMA=no for 2+ nodes, and runtime patching of
the &parallel section to nproc_ob=N (orbital-only decomposition,
no real-space grid split) which is the verified GPU-optimal layout.

Depends on RIKEN-RCCS#74 for RIKYU system/queue config (gpu partition,
account=rkp00012).

Signed-off-by: William Dawson <william.dawson@riken.jp>

💘 Generated with Crush

Assisted-by: glm-5.2 via Crush <crush@charm.land>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant