...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

NOTE THESE ARE NOT TO BE APPLIED TO 2.0 TESTING, USE THE QUICKSTARTS IN THE 2.0 ON-LINE DOCUMENTATION

Table of Contents

...

• sudo access configured
• password-less ssh configured
• pdsh installed (or some other means of running multiple remote commands in parallel)

In addition the server nodes should also have:

...

For the use of the commands outlined on this page the following shell variables will need to be defined:

...

• CLIENT_NODES
• SERVER_NODES
• ALL_NODES

For example, if one wanted to use node-1 as their admin node, node-2 and node-3 as client nodes, and node-[4-6] as their server nodes then these variables would be defined as:

ADMIN_NODE=node-1
CLIENT_NODES=node-2,node-3
SERVER_NODES=node-4,node-5,node-6
ALL_NODES=$ADMIN_NODE,$CLIENT_NODES,$SERVER_NODES

ALL_NODES=$CLIENT_NODES,$SERVER_NODES

RPM Installation

In this section the required RPMs will be installed on the each of nodes based upon their role.  Admin and client nodes require the installation of the daos-client RPM and server nodes require the installation of the daos-server RPM.

...

1.    Prepare the pmem devices on Server nodes

   Code Block
   language bash
   pdsh -w $SERVER_NODES -x $SERVER_NODES daos_server storage prepare --scm-only Preparing locally-attached SCM... Memory allocation goals for SCM will be changed and namespaces modified, this will be a destructive operation. Please ensure namespaces are unmounted and locally attached SCM & NVMe devices are not in use. Please be patient as it may take several minutes and subsequent reboot maybe required. Are you sure you want to continue? (yes/no) yes A reboot is required to process new SCM memory allocation goals.

   
   

2. Reboot the server node

3. Re run the prepare cmdline again

   Code Block
   language bash
   pdsh -w $SERVER_NODES -x $SERVER_NODES daos_server storage prepare --scm-only Preparing locally-attached SCM... SCM namespaces: SCM Namespace Socket ID Capacity ------------- --------- -------- pmem0 0 3.2 TB pmem1 0 3.2 TB

   
   

4. Prepare the NVME devices on Server nodes

   Code Block
   language bash
   pdsh -w $SERVER_NODES -x $SERVER_NODES daos_server storage prepare --nvme-only -u root Preparing locally-attached NVMe storage...

   
   

5. Scan the available storage on the Server nodes

   Code Block
   language bash
   pdsh -w $SERVER_NODES -x $SERVER_NODES daos_server storage scan Scanning locally-attached storage... NVMe PCI Model FW Revision Socket ID Capacity -------- ----- ----------- --------- -------- 0000:5e:00.0 INTEL SSDPE2KE016T8 VDV10170 0 1.6 TB 0000:5f:00.0 INTEL SSDPE2KE016T8 VDV10170 0 1.6 TB 0000:81:00.0 INTEL SSDPED1K750GA E2010475 1 750 GB 0000:da:00.0 INTEL SSDPED1K750GA E2010475 1 750 GB SCM Namespace Socket ID Capacity ------------- --------- -------- pmem0 0 3.2 TB pmem1 1 3.2 TB

   
   

Generate certificates

In this section certificates will be generated and installed for encrypting DAOS control plane communications.

Administrative nodes require the following certificate files:

• CA root certificate (daosCA.crt) owned by the current user
• Admin certificate (admin.crt) owned by the current user
• Admin key (admin.key) owned by the current user

...

certificates

In this section certificates will be generated and installed for encrypting DAOS control plane communications.

Administrative nodes require the following certificate files:

• CA root certificate (daosCadaosCA.crt) owned by the current user
• Agent Admin certificate (agentadmin.crt) owned by the daos_agent current userAgent
• Admin key (agentadmin.key) owned by the daos_agent current user

Server Client nodes require the following certificate files:

• CA root certificate (daosCA.crt) owned by the daos_server user
• Server certificate (server.crt) owned by the daos_server user
• Server key (server.keydaosCa.crt) owned by the daos_server current user
• A copy of the Client Agent certificate (clientagent.crt) owned by the daos_server agent user

See https://daos-stack.github.io/admin/deployment/#certificate-configuration for more inforamation.

Generate a new set of certificates.

cd /tmp
/usr/lib64/daos/certgen/gen_certificates.sh

...

Copy the certificates to a common location on each node in order to be able to move them to the final location

pdsh -S -w $ALL_NODES -x $(hostname -s) scp -r $(hostname -s):/tmp/daosCA /tmp

Copy the certificates to their default location (/etc/daos) on each admin node

...

...

• Agent key (agent.key) owned by the daos_agent user

Server nodes require the following certificate files:

• CA root certificate (daosCA.crt) owned by the daos_server user
• Server certificate (server.crt) owned by the daos_server user
• Server key (server.key) owned by the daos_server user
• A copy of the Client certificate (client.crt) owned by the daos_server user

See https://daos-stack.github.io/admin/deployment/#certificate-configuration for more inforamation.

1. Generate a new set of certificates.
   

   Code Block
   language bash
   cd /tmp /usr/lib64/daos/certgen/gen_certificates.sh

   
   

   Note
   These files should be protected from unauthorized access and preserved for future use.

   
   

2. Copy the certificates to a common location on each node in order to be able to move them to the final location
   

   Code Block
   language bash
   pdsh -S -w $ADMIN_NODE sudo cp /tmp/daosCA/certs/admin.key /etc/daos/certs/.
   Note
   If the /etc/daos/certs directory does not exist on the admin nodes then use the following command to create it: pdsh -S -w $ADMIN_NODES sudo mkdir /etc/daos/certs $ALL_NODES -x $(hostname -s) scp -r $(hostname -s):/tmp/daosCA /tmp

   
   

3. Copy the certificates to their default location (/etc/daos) on each client node
   

   Code Block
   language bash
   pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/daosCA.crt /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/agent.crt /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/agent.key /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/admin.crt /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/agentadmin.key /etc/daos/certs/.

   
   

   Note
   If the /etc/daos/certs directory does not exist on the client nodes then use the following command to create it: pdsh -S -w $CLIENT_NODES sudo mkdir /etc/daos/certs

   
   

4. Copy the certificates to their default location (/etc/daos) on each server node
   

   Code Block
   language bash
   pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/daosCA.crt /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/server.crt /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/server.key /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/agent.crt /etc/daos/certs/clients/agent.crt

   
   

5. Set the ownership of the client and admin certificates on each admin client node
   

   Code Block
   language bash
   pdsh -S -w $ADMIN$CLIENT_NODENODES sudo chown $USER:$USER /etc/daos/certs/daosCA.crt pdsh -S -w $ADMIN$CLIENT_NODENODES sudo chown $USER:$USERdaos_agent:daos_agent /etc/daos/certs/adminagent.*

   Set the ownership of the client certificates on each client node
   

   Code Block
   language bash
   pdsh -S -w $CLIENT_NODES sudo chown $USER:$USER /etc/daos/certs/daosCA.crt pdsh -S -w $CLIENT_NODES sudo chown daos_agent:daos_agent /etc/daos/certs/agentadmin.*

   
   

6. Set the ownership of the server certificates on each server node
   

   Code Block
   language bash
   pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/daosCA.crt pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/server.* pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/clients/agent.crt pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/clients

   
   

...