Installing Mattermost on Kubernetes

This document describes installing and deploying a production-ready Mattermost system on a Kubernetes cluster using the Mattermost Kubernetes operator.

An operator is a set of product- or application-specific instructions packaged into its own program. It tells an application what to run, where to find it, and how to configure it. An operator can automate complex application deployment and operation activities, such as installation, configuration changes, software updates, failure recovery, and more. The Mattermost operator does this by interacting with MySQL and MinIO operators.

Deploying the Mattermost Operator on Kubernetes is the recommended installation process. It provides a single common installation and management method that can be implemented in practically any environment with less IT overhead and more automation.

Tasks such as zero-downtime upgrades, database restore, blue/green deployments, canary builds, and failure recovery are all managed by the Mattermost operator.

Setting Up a Kubernetes Cluster

Prerequisites

If you don’t have a production-ready Kubernetes cluster, version 1.12 or higher, you will need to set one up. If you already have a Kubernetes cluster you can skip to Confirm Resource Requirements. Note that the Kubernetes-specific CLI tool, kubectl, also needs to be installed and configured.

It’s recommended that you have an understanding of basic Kubernetes concepts (such as deployments, pods) and actions (such as applying manifests, viewing pod logs). It’s also advisable to consult the official Kubernetes setup documentation on how to set up a cluster in your environment. If you’re unsure about which environment you want to use for your Kubernetes cluster, we suggest using a managed service such as as Amazon EKS, Azure Kubernetes Service, Google Kubernetes Engine, or DigitalOcean Kubernetes.

Confirm Resource Requirements

Running Mattermost in Kubernetes requires different resources based on your total number of users. The table below details the minimum Kubernetes cluster resources that Mattermost requires at different scales.

Note: These are minimum requirements and yours may differ significantly.

User Count Node Count Memory per Node vCPU per Node
5,000 6 8 GB 4
10,000 8 16 GB 4
25,000 14 16 GB 4

Note:

  • These resources take into account all components required for Mattermost, including proxy, database and file storage.
  • Resource requirements may vary depending on user usage and bot activity.
  • For larger installations, it may be beneficial to use nodes for the databases that have more memory and/or are optimized for memory.
  • For installations of more than 25,000 users please contact us for sizing guidelines.

Installing the Operators

The required operators are installed using kubectl and each operator is created in its own namespace. Shorter namespace names are recommended. You can install and run multiple Mattermost installations in the same cluster using different namespaces.

The steps provided below include database and storage operators. To install only the Mattermost operator, see Using a Custom Configuration

1. Install the MySQL Operator

$ kubectl create ns mysql-operator
$ kubectl apply -n mysql-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mysql-operator/mysql-operator.yaml

2. Install the MinIO Operator

$ kubectl create ns minio-operator
$ kubectl apply -n minio-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/minio-operator/minio-operator.yaml

3. Install NGINX Ingress Controller

Follow the instructions here.

4. Install the Mattermost Operator

$ kubectl create ns mattermost-operator
$ kubectl apply -n mattermost-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mattermost-operator/mattermost-operator.yaml

Once completed, the installed components are listed in the configuration of your cluster.

Deploying a Mattermost Installation

This guide describes deploying a complete Mattermost installation in Kubernetes including a database. In most cases you’ll create and apply a single manifest. In the case of an Enterprise installation, two manifests will be applied - one for the license and one for the cluster installation.

Manifest files contain the configurations needed for the operator to perform tasks and communicate with Kubernetes. Create the manifest file locally in a text editor, copy and paste the contents, and save the file. Recommended file names are provided, but your naming conventions may differ.

1. (Enterprise only) Create a Mattermost License Secret

Open a text editor and create a text file with the following details.

Note: Replace %LICENSE_FILE_CONTENTS% with the contents of your Mattermost license file.

apiVersion: v1
kind: Secret
metadata:
  name: mattermost-license
type: Opqaue
stringData:
  license: %LICENSE_FILE_CONTENTS%

Save the file as mattermost-license-secret.yaml. Apply the file, specifying the correct path, using:

$ kubectl apply -f /path/to/mattermost-license-secret.yaml

2. Create an Installation Manifest File

The Mattermost installation manifest contains fields which must be edited in line with your configuration and environment requirements.

Field Description Must Edit
metadata.name The name of your Mattermost as it will be shown in Kubernetes. The shorter the better. Yes
spec.size The size of your installation. This can be ‘100users’, ‘1000users, ‘5000users’, ‘10000users’, or ‘25000users’. Yes
spec.ingressName The DNS for your Mattermost installation. Yes
spec.version The Mattermost version. No
spec.mattermostLicenseSecret The name of the Kubernetes secret containing your license (e.g. mattermost-license-secret). Required for Enterprise deployments. No
spec.database.storageSize The storage size for your database. Your Kubernetes cluster must have persistent volumes this size or larger. No
spec.minio.storageSize The storage size for your file storage. Your Kubernetes cluster must have persistent volumes this size or larger. No
spec.elasticSearch The section for Elasticsearch settings. Remove this section if you will not be using Elasticsearch. No
spec.elasticSearch.host The hostname for your Elasticsearch instance. Remove this section if you will not be using Elasticsearch No
spec.elasticSearch.username The username for your Elasticsearch instance.Remove this section if you will not be using Elasticsearch No
spec.elasticSearch.password The password for your Elasticsearch instance. Remove this section if you will not be using Elasticsearch No

There are more advanced fields documented here.

Open a text editor and create a text file with the following details.

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mm-example-full
spec:
  size: 5000users
  ingressName: example.mattermost-example.com
  ingressAnnotations:
    kubernetes.io/ingress.class: nginx
  version: 5.14.0
  mattermostLicenseSecret: ""
  database:
    storageSize: 50Gi
  minio:
    storageSize: 50Gi
  elasticSearch:
    host: ""
    username: ""
    password: ""

Save the file as mattermost-installation.yaml.

3. Apply the Installation Manifest File

To initiate deployment, apply the file, specifying the correct path, using:

$ kubectl create ns mattermost
$ kubectl apply -n mattermost -f /path/to/mattermost-installation.yaml

The deployment process can be monitored in the Kubernetes user interface.

4. Configure DNS and Use Mattermost

When the deployment is complete, obtain the hostname or IP address of your Mattermost deployment using the following command:

$ kubectl -n mattermost get ingress

Copy the resulting hostname or IP address from the ADDRESS column, open your browser, and connect to Mattermost.

Use your domain registration service to create a canonical name or IP address record for the ingressName in your manifest, pointing to the address you just copied. For example, on AWS you would do this within a hosted zone in Route53.

Navigate to the ingressName URL in your browser and use Mattermost.

Restoring an Existing Mattermost MySQL Database

The Mattermost Operator can be used in a backup and restore scenario to apply an existing Mattermost MySQL database to a new Mattermost installation, in its own namespace. This can also be helpful in the event that you need to revert your Mattermost instance’s database to the most recent backup point, on your existing installation. In both cases, you will need a backup of your database.

The steps you follow to create and upload your backup depends on the provider you’re using and your use case. It’s recommended that you consult the relevant documentation or, if your deployment is managed in a different way, consult your Administrator.

It is important to note that this process requires the creation of a new Mattermost installation - editing the existing .yaml files is not recommended and can result in data loss.

The process described below needs to be completed prior to proceeding with the Mattermost deployment.

  1. Create a backup of your database (e.g. using mysqldump).
  2. Deploy a new server (e.g. an AWS instance).
  3. Install a backup program and back up the database on the new server/instance.
  4. Upload the backed up database to your cloud storage provider (e.g. Amazon S3).
  5. Create a secret.yaml file:

Open a text editor and create a text file containing your credentials which will be used to access the uploaded database.

Save the file as secret.yaml. The example below is for AWS/S3.

apiVersion: v1
kind: Secret
metadata:
 name: test-restore
type: Opaque
stringData:
 AWS_ACCESS_KEY_ID: XXXXXXXXXXXX
 AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXX/XXXXXXXXXXXX
 AWS_REGION: us-east-1
 S3_PROVIDER: AWS

Parameters

  • name. The name of this manifest which is referenced in the installation manifest.
  1. Create a Mattermost cluster installation manifest.

Open a text editor and create a text file with the following details. Save the file as mattermost-installation.yaml:

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mm-example-full
spec:
  size: ""
  ingressName: example.mattermost-example.com
  ingressAnnotations:
    kubernetes.io/ingress.class: nginx
  version: 5.14.0
  mattermostLicenseSecret: ""
  database:
    storageSize: 50Gi
  minio:
    storageSize: 50Gi
  elasticSearch:
    host: ""
    username: ""
    password: ""

The Mattermost installation manifest contains fields which must be edited in line with your configuration and environment requirements.

  1. Create a restore manifest:

Open a text editor and create a text file with the following details. Save the file as restore.yaml:

apiVersion: mattermost.com/v1alpha1
kind: MattermostRestoreDB
metadata:
  name: example-mattermostrestoredb
spec:
  initBucketURL: s3://my-sample/my-backup.gz
  mattermostClusterName: example-clusterinstallation
  mattermostDBName: mattermostdb
  mattermostDBPassword: supersecure
  mattermostDBUser: ""
  restoreSecret: ""

Parameters

  • mattermostClusterName. The ClusterInstallation file name.
  • restoreSecret. The location of the backup file.
  • mattermostDBPassword. The password used to access the database.
  • mattermostDBUser. The username required to access the database.
  • initBucketURL. The URL of the storage instance/server where the backed up DB is stored.
  1. To initiate deployment, apply the file and specify the path where the newly-created files have been saved:
$ kubectl create ns mattermost
$ kubectl apply -n mattermost -f /path/to/secret.yaml
$ kubectl apply -n mattermost -f /path/to/mattermost-installation.yaml
$ kubectl apply -n mattermost -f /path/to/restore.yaml

The deployment process can be monitored in the Kubernetes user interface. If errors or issues are experienced, review the Mattermost, Operator, and MySQL logs for guidance including error messages. If remediation is not successful, contact Mattermost customer support for assistance.

Once complete, access your Mattermost instance and confirm that the database has been restored.

Using Mattermost Operator Functionality

Rolling Upgrades

The Mattermost Kubernetes Operator supports rolling upgrades so you can upgrade your Mattermost deployment with zero downtime. This process requires at least two replicas as a rolling upgrade cannot be performed if there is only one pod. Replicas are created when a user count is selected and exceeds 100.

New Mattermost releases are announced via our community server, as well as social media and email.

Performing rolling upgrades

  1. Log in to your Kubernetes instance.
  2. Open the mattermost-installation.yaml manifest (the one created during installation).
  3. Update the spec.version value to the new version.
  4. Save the changes.

Apply the changes with kubectl using

$ kubectl apply -n mattermost -f /path/to/mattermost-installation.yaml

The operator initiates a job in the Kubernetes cluster and once migration is complete the pods are restarted. If necessary, a database migration is also performed.

To view information about the running job, use

$ kubectl -n mattermost get jobs

At least one pod is available at all times and once all pods are restarted with the new version the upgrade is complete.

To view the status of the pods and to confirm their state, use

$ kubectl -n mattermost get pods

The STATUS of the pods should be running/ready, with an AGE of 10-15 seconds.

Blue-green Deployments

Blue-green deployments can reduce downtime and increase stability during automated tasks in a production environment. This technique entails running two identical production environments in tandem, with one acting as a live environment and one as an idle environment.

An upgrade can be rolled out to the idle environment (“blue”) while the other (“green”) is the production environment. Once the upgrade has been successfully rolled out, all traffic can be switched from “green” to “blue”.

Note: Blue-green can be run on a permanent basis but utilizes more resources in the Kubernetes cluster than normal deployments.

Configuring Blue-green Deployments

Open the mattermost-installation.yaml file and add blueGreen under spec.

blueGreen:
enable: true
productionDeployment: blue
blue:
  version: <version number>
  image: mattermost/mattermost-enterprise-edition
green:
  version: <version number>
  image: mattermost/mattermost-enterprise-edition

Parameters

  • enable. When enabled it ignores the regular spec.version and spec.image. When set to false it is ignored.
  • productionDeployment. Set to blue or green to route all users to the specified deployment.
  • image. Optional. Select the image used.

When the manifest is updated, two new ingresses (proxies) are created at blue.yourmattermosturl.com and green.yourmattermosturl.com.

To access the new ingresses, create CNAME or IP address records in your DNS registration service for the ingressName in your manifest, pointing to the address you just copied.

For example, on AWS you would do this within a hosted zone in Route 53. Use the required ingressName URL in your browser to directly access blue or green at any time.

To update the version of blue or green, change the version in the manifest to match the current version or the version you’d like to deploy. This change (regardless of which is the productionDeployment) initiates a database migration. The schema is backwards and forwards compatible across minor versions (from 5.9 onwards) and will not disrupt the production deployment. However, it will auto-upgrade the database.

Canary Builds

A canary build is used to test an experimental or untested build. It’s similar to a blue-green deployment in that multiple environments are run simultaneously. However, where blue-green deployments have different URLs, canary builds are set up to direct a random segment of users to the test environment. Users are not explicitly aware that they’re on the canary build environment.

The redirect is managed with a cookie, which is valid for 24 hours.

The Mattermost Operator currently allows segmenting by percentage (i.e., splitting the user pool between production and the canary build). In future releases segmentation options will include teams and individual users.

Configuring canary builds requires an update to the mattermost-installation.yaml file and the addition of a plugin via System Console. Before proceeding, first download the Mattermost Plugin for Canary Deployments.

Configuring Canary Builds

Open the mattermost-installation.yaml file and add the following under spec.

canary:
enable: true
Deployment:
  version: 5.15.0
  image: mattermost/mattermost-enterprise-edition

Next, navigate to System Console > Plugin Management, enable plugins, and upload the Mattermost Canary Plugin. Once uploaded, refresh your page and then select Settings from the Canary plugin modal. Enter the percentage of users you’d like to direct to the canary build.

Once complete, navigate to your Mattermost instance and open the Developer Tools menu in your browser. The entry for the Mattermost instance will display always or never depending on the segment you’ve been allocated to.

You can disable canary builds in the mattermost-installation.yaml file by changing the enable field to false.

Frequently Asked Questions

What is the difference between the Mattermost Operator and Helm Charts?

The Mattermost Operater is a self-contained set of application/product-specific instructions that runs inside Kubernetes and facilitates application management and deployment.

Helm is a tool used to deploy Kubernetes manifests to a cluster, but does not facilitate application management.

Does the Mattermost Operator replace the Helm Chart?

No. Helm is a good mechanism to deploy operators and other applications but doesn’t facilitate application management.

Is minIO the only available storage option?

No, other options and operators can be added.

Do I have to install a separate SQL server to use the Mattermost Operator?

No, you can use the operator on top of your existing deployment without setting up another database. We will shortly be providing steps for this configuration.

Can you use blue-green deployments with different database schemas?

Currently this is not supported as it introduces the possiblilty of missing a data entry in the database.