Table of Contents

...

• sudo access configured
• password-less ssh configured
• pdsh installed (or some other means of running multiple remote commands in parallel)

In addition the server nodes should also have:

...

In this section certificates will be generated and installed for encrypting DAOS control plane communications.

Administrative nodes require the following certificate files:

...

1. Generate a new set of certificates.
   

   Code Block
   language bash
   cd /tmp /usr/lib64/daos/certgen/gen_certificates.sh

   Note
   These files should be protected from unauthorized access and preserved for future use.

   
   

2. Copy the certificates to a common location on each node in order to be able to move them to the final location
   

   Code Block
   language bash
   pdsh -S -w $ALL_NODES -x $(hostname -s) scp -r $(hostname -s):/tmp/daosCA /tmp

3. Copy the certificates to their default location (/etc/daos) on each admin node
   

   Code Block
   language bash
   pdsh -S -w $ADMIN_NODE sudo cp /tmp/daosCA/certs/daosCA.crt /etc/daos/certs/. pdsh -S -w $ADMIN_NODE sudo cp /tmp/daosCA/certs/admin.crt /etc/daos/certs/. pdsh -S -w $ADMIN_NODE sudo cp /tmp/daosCA/certs/admin.key /etc/daos/certs/.

   Note
   If the /etc/daos/certs directory does not exist on the admin nodes then use the following command to create it: pdsh -S -w $ADMIN_NODES sudo mkdir /etc/daos/certs

4. Copy the certificates to their default location (/etc/daos) on each client node
   

   Code Block
   language bash
   pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/daosCA.crt /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/agent.crt /etc/daos/certs/. pdsh -S -w $CLIENT_NODES sudo cp /tmp/daosCA/certs/agent.key /etc/daos/certs/.

   Note
   If the /etc/daos/certs directory does not exist on the client nodes then use the following command to create it: pdsh -S -w $CLIENT_NODES sudo mkdir /etc/daos/certs

   
   

5. Copy the certificates to their default location (/etc/daos) on each server node
   

   Code Block
   language bash
   pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/daosCA.crt /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/server.crt /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/server.key /etc/daos/certs/. pdsh -S -w $SERVER_NODES sudo cp /tmp/daosCA/certs/agent.crt /etc/daos/certs/clients/agent.crt

   
   

6. Set the ownership of the admin certificates on each admin node
   

   Code Block
   language bash
   pdsh -S -w $ADMIN_NODE sudo chown $USER:$USER /etc/daos/certs/daosCA.crt pdsh -S -w $ADMIN_NODE sudo chown $USER:$USER /etc/daos/certs/admin.*

   
   

7. Set the ownership of the client certificates on each client node
   

   Code Block
   language bash
   pdsh -S -w $CLIENT_NODES sudo chown $USER:$USER /etc/daos/certs/daosCA.crt pdsh -S -w $CLIENT_NODES sudo chown daos_agent:daos_agent /etc/daos/certs/agent.*

   
   

8. Set the ownership of the server certificates on each server node
   

   Code Block
   language bash
   pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/daosCA.crt pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/server.* pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/clients/agent.crt pdsh -S -w $SERVER_NODES sudo chown daos_server:daos_server /etc/daos/certs/clients

   
   

...

1. First determine the addresses for the NVMe devices on the server nodes

   Code Block
   language bash
   pdsh -S -w $SERVER_NODES sudo lspci | grep -i nvme

   Info
   Save the addresses of the NVMe devices to use with each DAOS server, e.g. "81:00.0", from each server node.  This information will be used to populate the "bdev_list" server configuration parameter below.

2. Create a server configuration file by modifying the default /etc/daos/daos_server.yml file on the server nodes.  Below is an example daos_server.yml.  Copy modified server yaml file to all the server nodes at `/etc/daos/daos_server.yml. 

   More details on configuring the daos_server.yml file are available at Server configuration file details

   
   

   Code Block
   language yml
   name: daos_server access_points: ['node-4'] port: 10001 transport_config: allow_insecure: false client_cert_dir: /etc/daos/certs/clients ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/server.crt key: /etc/daos/certs/server.key provider: ofi+verbs;ofi_rxm socket_dir: /var/run/daos_server nr_hugepages: 4096 control_log_mask: DEBUG control_log_file: /tmp/daos_server.log helper_log_file: /tmp/daos_admin.log engines: - targets: 8 nr_xs_helpers: 0 fabric_iface: ib0 fabric_iface_port: 3141631316 log_mask: INFO log_file: /tmp/daos_engine_0.log env_vars: - CRT_TIMEOUT=30 scm_mount: /mnt/daos0 scm_class: dcpm scm_list: [/dev/pmem0] bdev_class: nvme bdev_list: ["0000:81:00.0"] # generate regular nvme.conf - targets: 8 nr_xs_helpers: 0 fabric_iface: ib1 fabric_iface_port: 31416 log_mask: INFO log_file: /tmp/daos_engine_1.log env_vars: - CRT_TIMEOUT=30 scm_mount: /mnt/daos1 scm_class: dcpm scm_list: [/dev/pmem1] bdev_class: nvme bdev_list: ["0000:83:00.0"] # generate regular nvme.conf Copy modified server yaml file to all the server nodes at `/etc/daos/daos_server.yml`

   
   

   
   

3. Create an agent configuration file by modifying the default /etc/daos/daos_agent.yml file on the client nodes.  Below is an example daos_agent.yml. Copy modified agent yaml file to all the client nodes at `/etc/daos/daos_agent.yml. More details on configuring the daos_agent.yml file are available at Agent configuration file details

   
   

   Code Block
   language yml
   name: daos_server access_points: ['node-4'] port: 10001 transport_config: allow_insecure: false ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/agent.crt key: /etc/daos/certs/agent.key runtime_dir: /var/run/daos_agent log_file: /tmp/daos_agent.log

4. Create a dmg configuration file by modifying the default /etc/daos/daos_control.yml file on the admin node. Below is an example daos_control.yml. More details on configuring the daos_control.yml file are available at DMG configuration file details

   
   

   Code Block
   language yml
   name: daos_server port: 10001 hostlist: ['node-4', 'node-5', 'node-6'] transport_config: allow_insecure: false ca_cert: /etc/daos/certs/daosCA.crt cert: /etc/daos/certs/admin.crt key: /etc/daos/certs/admin.key

...