Deploy a 3-Node MongoDB 3.6 Replica Set with X.509 Authentication +...

This article will guide you through the process of setting up a MongoDB cluster that will utilise X.509 authentication with self signed certificates. Our cluster will consist of 3 nodes that will be deployed as a replica set. I will also include tips along the way to help your workflow.

An overview of the walkthrough:

Installation of MongoDB on your VPSs. This article documents Red Hat CentOS7 installation, but I will outline the official * MongoDB resources you can refer to for installation on your OS.
Adding a admin user to your nodes.
Attaching an XFS formatted volume that will serve as your means of data storage.
Adding hostnames for each of your cluster nodes and configuring your firewall.
Creating a X.509 certificate for each node using a self signed CA.
Configuring MongoDB for a X.509 Authentication replica set.
Initialising a replica set.

Why Choose MongoDB?

Before you undertake the task of deploying MongoDB as your means of data storage, make sure it is the right solution for you. MongoDB is a document oriented database offering a flexible scaling and accessibility solution.
Installation

The first stage is to install MongoDB on your 3 nodes. For this article I will use 3 Digital Ocean VPS Droplets.

Note: This article is for production purposes where each node is run on a separate VPS, but if you wish to install MongoDB in your development enviornment, you can still run 3 or more nodes on the same machine. You simply have to run 3 mongod processes on different ports.

Installing MongoDB on CentOS7 is very easy, and is achieved via rpm. We are interested in installing the MongoDB Community Edition, the free open-source version of MongoDB. Visit this page for the official Red Hat installation instructions, or run the following commands:

#create a mongodb repo
sudo vi /etc/yum.repos.d/mongodb-org-3.6.repo

#paste the following into the file and save
[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

#install via yum
sudo yum install -y mongodb-org

If you are using SELinux in enforcing mode, be sure to enable the ports MongoDB will be using. Port 27017 is used by default, and we will also be using this port. Run the following to make the changes.

semanage port -a -t mongod_port_t -p tcp 27017

You can also edit the SELinux configuration file and disable it completely, or change the mode to permissive - although I would not recommend lowering our security for the sake of enabling ports.

Note: You can configure SELinux to be permissive to certain programs. A great example of this is when running NginX and PHP-FPM. By Default SELinux blocks communication between these 2 processes. This is a great article that documents how to configure SELinux to be permissive to this communication whilst maintaining its enforcing policy systemwide.
Lastly, launch MongoDB by starting the mongod process:

sudo service mongod start

Repeat this process for all your nodes.

Deploying XFS formatted block storage volumes

In this step we will deploy some XFS formatted storage devices, attach them to our droplets, and change our mongod.conf configuration file to point to this directory for our database storage.

Note: When we installed mongodb, a configuration file was created at /etc/mongod.conf. This file configures our data location as well as other important factors like authentication, SSL configuration, replica set name etc. Take a look inside to familairise yourself:

less /etc/mongod.conf

Note 2: XFS is a file system designed to handle large scale data, and is recommended by MongoDB to be used as it's means of data storage.

Within your Digital Ocean management panel, go to your Droplets followed by Volumes tab, then Create Volume. Select a volume size depending on how much storage you envisage needing. It is perfectly acceptable to start at 1GB, at a $0.10 monthly cost, and resize your volumes as and when you need to.

Choose your Droplet to attach the volume to. Give it a name. E.g. mongo1, mongo2 and mongo3.

In some regions you can automatically format the volume into an XFS formatted device. If not, stick with Manual and Create Volume. Digital Ocean will then give you the commands needed to format and mount the volume to your Droplet with 1 exception. The first command formats into EXT4, not XFS. Simply change mkfs.ext4 to mkfs.xfs, so you get something like this: (do not copy, your volume name is different).

sudo mkfs.xfs -F /dev/disk/by-id/scsi-0DO_Volume_XXX

Go through the provided commands to format, mount and auto mount at reboot, and we are done.

Now, open your mongodb configuration file and navigate to the storage.dbPath field. Change this to your mounted device location. Feel free to create a subfolder on the mounted device if you do not wish to store your database files in the root directory. A folder called data perhaps.

Lastly, change the permissions of our mounted device, giving mongod ownership. Change the following path to your volume path and execute:

sudo chown -R mongod:mongod /path-to-your-mounted-volume

In the previous step we started our mongod instances. Since we changed the data path we are now going to restart them. The new configuration will then take effect:

sudo service mongod restart

Carry out this same process for all 3 nodes.

Adding an admin user to your nodes

User roles and permissions in MongoDB are rather verbose, but flexible - we can also configure our own roles. To get a feel of the roles available, visit MongoDB's built-in roles page.

To add our admin user we need to run the mongo shell, the interactive environment for managing your mongod instances. We will tell it we are using the admin database, and then insert our new user with our desired priviledges. Let's call our user adminUser, with the password adminTest (change to a long random string in your case). We do this for each of our 3 nodes:

#load the mongo shell
mongo --port 27017

#switch to the admin database
use admin

#create our new user
db.createUser({ user: "adminUser", pwd:"adminTest", roles: [ \  
    { role: "userAdminAnyDatabase", db: "admin" }, \
    { role: "dbAdminAnyDatabase", db: "admin" }, \
    { role: "readWriteAnyDatabase", db:"admin" }, \
    { role: "clusterAdmin",  db: "admin" } \
] })

#quit the shell
quit()

So we have given 4 roles to our user that allow administration of all databases, users, and cluster management. Refer to the built-in roles page for a thorough breakdown of these roles.

Repeat this process for all your nodes

Hostnames in /etc/hosts and firewall

In this quick step we will set up hostnames for our 3 nodes. Our X.509 generation and replica set members will refer to these values. mongoN is a desirable naming convention, starting with mongo1 .

#open your hosts file
sudo vi /etc/hosts

# Mongo cluster nodes
X.X.X.X mongo1 #ip address of node 1 (the one you are logged into now)
X.X.X.X mongo2 #ip address of node 2
X.X.X.X mongo3 #ip address of node 2

Also open the port 27017 to your firewall. Here is an example of how to do that with firewalld:

firewall-cmd --zone=public --add-port=27017/tcp --permanent
sudo firewall-cmd --reload

Refer to your firewall documentation to open the port, or limit port access to your mongo hostnames only for added security.

Repeat this process for all your nodes.

Generating the X.509 Certificates

An X.509 certificate needs to be generated for each of our nodes. You will act as the CA so we will sign them ourselves. To do this we firstly create a private key, issue a CA certificate and thereafter issue 3 more certificates for each MongoDB node.
On the mongo1 Droplet, use this command to create the private key:

openssl genrsa -out mongoTest.key -aes256 8192

You will be asked for a passphrase. Go ahead and create a secure passphrase for an additional level of security.

Next, create your CA certificate. The details of this certificate should differ to those we will give to the mongo certificates. The content in general does not matter. A company name, address and website will suffice.

openssl req -x509 -new -extensions v3_ca -key mongoCA.key -days 365 -out mongoCA.crt

Next, create 3 certificates, 1 for each mongo node.

To do this we can create a bash script to automate this creation process and ensure there are no errors. A prompt at the begining will ask for your CN if you do not provide it at the command level. Copy the following code, changing the bolded details before running it.

As your hostname, use what you configured earlier from your hosts file; mongo1 in this case.

Important Note: All 3 certificate details have to be identical, apart from the host name. This is the reason hardcoding an automated script is a more preferred and quicker way of generating our certificates.

#!/bin/bash
if [ "$1" = "" ]; then
echo 'Please enter your hostname (CN):'
exit 1
fi
HOST_NAME="$1"
SUBJECT="/C=GB/ST=London/L=London/O=OrganisationName/OU=OnlineServices/CN=$HOST_NAME"
openssl req -new -nodes -newkey rsa:4096 -subj "$SUBJECT" -keyout $HOST_NAME.key -out $HOST_NAME.csr
openssl x509 -CA mongoCA.crt -CAkey mongoCA.key -CAcreateserial -req -days 365 -in $HOST_NAME.csr -out $HOST_NAME.crt
rm $HOST_NAME.csr
cat $HOST_NAME.key $HOST_NAME.crt > $HOST_NAME.pem
rm $HOST_NAME.key
rm $HOST_NAME.crt

Run this script 3 times for mongo1, mongo2 and mongo3. The result should be 3 new .pem files coinciding with your hostnames. Remember, non-identical details or the incorrect CN value will result in our replica set not authenticating.

Now, for mongo1, let's move mongo1.pem and mongoCA.crt to an appropriate directory and configure the appropriate permissions and ownership.

#create ssl directory in mongodb folder
sudo mkdir -p /etc/mongodb/ssl

#move mongo1.pem and copy mongoCA.crt into it
sudo mv mongo1.pem /etc/mongodb/ssl/ 
sudo cp mongoCA.crt /etc/mongodb/ssl/

#chmod to 700 and change permissions of the folder to mongo.
sudo chmod 700 /etc/mongodb/ssl  
sudo chown -R mongod:mongod /etc/mongodb

Repeat this process for all your nodes. Move mongo2 and mongo3 as well as mongoCA.crt onto node 2 and node 3 in the same directory.

Configuring MongoDB for a X.509 Authentication replica set

Back to your first node, we will now configure the mongod.conf file correctly so our secure replica set can start running.

sudo vi /etc/mongod.conf

Ensure you have the folllowing configurations, ensuring the values are correct based on your filenames:

storage:  
    dbPath: "/data/mongo1/data"
systemLog:  
    destination: file
    path: "/var/log/mongodb/mongod.log"
    logAppend: true
    timeStampFormat: iso8601-utc
replication:  
    replSetName: "rs0"
net:  
    port: 27017
    bindIp: 0.0.0.0
    ssl:
        mode: preferSSL
        PEMKeyFile: /etc/mongodb/ssl/mongo1.pem
        CAFile: /etc/mongodb/ssl/mongoCA.crt
        clusterFile: /etc/mongodb/ssl/mongo.pem
        PEMKeyPassword: <your-ssl-passphrase>
        clusterPassword: <your-ssl-passphrase>
security:
    authorization: enabled
    clusterAuthMode: x509

Some notes on this file:

Remember dbPath is your database storage path - your XFS volume
Ensure bindIp is 0.0.0.0
replicaSetName has followed official MongoDB naming conventions, using rs0
The ssl.mode is preferSSL.This setting will still allow non-encrypted connections. Change this to requireSSL if you want to restrict access to encrypted only
Security simply enables authentication and applies x509 as its means of authentication

Restart mongod for these changes to take effect, and we are done with configurations:

sudo service mongod restart

Repeat this process for all of your nodes. Be sure to change your .pem file name, or data volume directory accordingly.

Initialise the Replica Set

We are ready to initialise the replica set. To do this we will access the mongo shell on mongo1 to initialise the replica set. Run the mongo command with the following arguments and flags: This will open the mongo shell.

sudo mongo --authenticationDatabase 'admin' --ssl --sslCAFile '/etc/mongodb/ssl/mongoCA.crt' --sslPEMKeyFile '/etc/mongodb/ssl/mongo1.pem' -u 'adminUser' -p 'adminTest' --host 'mongo1'

Let's quickly review this command:

-authenticationDatabase 'admin' tells mongo to use the admin database to authenticate your user, adminUser.
The - ssl flag means we require SSL configuration. - sslCAFile and - sslPEMKeyFile provide our certificates we created earlier.
-u and -p will give our username and password.
- host as mongo1 will connect to our mongo node running on mongo1. You could change this to mongo2 or 3 and the shell will still connect, just as long as you provide the correct SSL certificates.

So we are now in the mongo shell. Initiate the replica set with the following function:

rs.initiate(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "mongo1" },
      { _id : 1, host : "mongo2" },
      { _id : 2, host : "mongo3" }
    ]
  }
)

Now, if all is successful your replica set should be active. We can make changes using the primary node. You will now either have PRIMARY> or SECONDARY> as your shell prompt. Regardless of this, you may want to always want mongo1 to be primary, as long as it is reachable. To do this, let's change the priorities of the other nodes, and submit our reconfiguration. The following functions will do this:

cfg = rs.conf()
cfg.members[0].priority = 1
cfg.members[1].priority = 0.5
cfg.members[2].priority = 0.5
rs.reconfig(cfg)

We take the replica set configuration - rs.conf() - with a variable cfg, before changing the priorities of each member of the set. We then take this configuration and apply it to our replica set.

Great! To make sure the secondary nodes are syncing correctly, run the following function to check if they are syncing:

rs.printSlaveReplicationInfo()

If you see something similar to the following for each secondary, we know everything is running smoothly:

0 secs (0 hrs) behind the primary

Conclusion

This concludes the setting up of our MongoDB 3.6 replica set. I will be providing more articles in the future, including how to interact with your newly created replica set using the PHP Driver and MongoDB Composer package. I hope this guide was useful to you.

Deploy a 3-Node MongoDB 3.6 Replica Set with X.509 Authentication + Self Signed Certificates