Differences between revisions 25 and 26
⇤ ← Revision 25 as of 2012-12-23 07:21:43 →
Size: 2040
Editor: AndreasKloeckner
Comment:
← Revision 26 as of 2012-12-23 07:35:57 → ⇥
Size: 2542
Editor: AndreasKloeckner
Comment:
Deletions are marked like this. Additions are marked like this.
Line 33: Line 33:
{{{
setfacl -m g:gpu:rw /dev/nvidiactl
}}}
and
{{{
setfacl -m g:gpu:rw /dev/nvidia$i
}}}

Also `NVIDIA_CARDS=N` must be correct in `/etc/default/nvidia-kernel`.

For AMD cards, we use a snippet in `/etc/gdm3/Init/Default` to call a script `/etc/enable-amd-compute`, which contains:
Line 35: Line 46:
setfacl -m g:gpu:rw /dev/nvidia$i #! /bin/sh

xauth extract /tmp/x11-auth-file "$DISPLAY"
chmod 660 /tmp/x11-auth-file
chgrp amdgpu /tmp/x11-auth-file

cat > /tmp/enable-amd-compute <<EOF
export COMPUTE=$DISPLAY
unset DISPLAY
export XAUTHORITY=/tmp/x11-auth-file
EOF

Admin Notes

User Database

A shared user database is maintained in /mnt/nfs-main/etc/extrausers. The passwd, group, shadow files from this directory must be copied into /var/lib/extrausers for the changes to take effect. This directory is then read by the nss-extrausers package.

Creating a user works as follows:

  1. cd /mnt/nfs-main/etc/extrausers

  2. cat passwd to find a free UID

  3. Create it locally on some machine: adduser --uid NNN USERNAME

  4. ./relocate-user USERNAME

  5. vi passwd (in extrausers) to change home directory to /exthome/USERNAME

  6. vi group (in extrausers) to add user to relevant gpu groups

  7. mv /home/USERNAME /exthome/USERNAME

  8. Run /mnt/nfs-main/etc/extrausers/update-system on each participating machine. (done automatically at boot) This will perform the copy to /var/lib/extrausers described above.

  9. Don't forget to add SSH keys to /exthome/USERNAME/.ssh/authorized_keys.

The user db should have home directories point to /exthome/<username>, which (for now) symlinks to /mnt/nfs-main/home, which is where home directories physically live.

UIDs and GIDs in the shared user DB should be 2000 or greater to avoid clashes with locally-created users.

Access to haamster

Simply drop the new user's SSH key into /home/extuser/.ssh/authorized_keys on haamster.

GPU Access

GPU access is by the group gpu for Nvidia, and amdgpu for the (unnecessarily) more sensitive access to the entire X11 server and thereby the AMD GPU.

How's this implemented? Check /etc/init.d/nvidia-kernel where there's an extra 

setfacl -m g:gpu:rw /dev/nvidiactl

and 

setfacl -m g:gpu:rw /dev/nvidia$i

Also NVIDIA_CARDS=N must be correct in /etc/default/nvidia-kernel.

For AMD cards, we use a snippet in /etc/gdm3/Init/Default to call a script /etc/enable-amd-compute, which contains: 

xauth extract /tmp/x11-auth-file "$DISPLAY"
chmod 660 /tmp/x11-auth-file
chgrp amdgpu /tmp/x11-auth-file

cat > /tmp/enable-amd-compute <<EOF
export COMPUTE=$DISPLAY
unset DISPLAY
export XAUTHORITY=/tmp/x11-auth-file
EOF

OpenCL

The AMD and Nvidia OpenCL implementations (and the GPU drivers) are available as packages from Debian. Also install opencl-headers. The Intel implementation must be installed by hand.

WarburtonCluster/AdminNotes (last edited 2012-12-23 07:35:57 by AndreasKloeckner)