Go to End

Note

26/2/2023: Trash folder added.

28/7/2022: Correct the positions of two statements. "gedit" changed to "nano" in case non gui interface is used.

7/5/2019: Slight adjustments. "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

25/12/2014: First created.

Intro

Samba file server enables specified directories to be accessible by Windows computers on the same network.

Install

Install the package​s:​

$ sudo apt update 
$ sudo apt install samba
$ sudo apt install samba-vfs-modules

samba-vfs-modules is used to enable Trash folder.

(vsf-modules added, 26/2/2023)

Define a workgroup

Edit the config file:

$ sudo nano /etc/samba/smb.conf

​​​​​Define workgroup name as "kctcl" in the "[global]" section:

workgroup = kctcl

Uncomment to restrict access to server users only:

security = user

Add the following if OpenVPN used:

hosts allow = 192.168.0. 10.8.0. 127.0.0.

(added. 7/5/2019):

Put the last two statements under the "[global]" section after "Networking" instead of at the end, otherwise "security = user" will apply to the last shared directory causing it to be non-assessible. 

(added, 28/7/2022)

Define directories to be shared

Add a section at the end to share directories:

[<sub-directory name or other short name>] 
pth = /<directory name>/<sub-directory name> 
browseable = yes 
guest ok = yes 
read only = no 
create mask = 0775
directory mask = 0775 
# do not include the next two lines if access is restricted to the owning user (added 7 May 2019)
force user = nobody 
force group = nogroup
# Enable Trash folder, KCTang 18/1/2023
vfs object = recycle
# Specify a folder relative to the path above
# Do not specify absolute path unless the path is outside the path above
# The folder will be created automatically upon first deletion
# Hidden folder (prefixed with '.') not used
# Use %U if want to record the user name, not used KCTang 26/2/2023
#recycle:repository = Trash/%U
recycle:repository = Trash
# Mode permits all users to delete
recycle:directory_mode = 0775
# Change last accessed time when moved to the Trash folder
recycle:touch = yes
# Keep modified time
recycle:touch_mtime = no
# Keep folder tree
recycle:keeptree = yes
# Files of the same name deleted will be kept with newer deleted file named as "Copy # of ..."
recycle:versions = yes
# Exclusions
recycle.exclude = *.tmp, ~*, thumb.db

(Trash folder added, 26/2/2023)

Create the directory to be shared, if not already existing:

$ sudo mkdir -p /<directory name>/<sub-directory name>

Change ownership of the directory:

$ sudo chown nobody:nogroup /<directory name>/<sub-directory name>

Start service

Start or restart Samba service whenever the config file is changed:

$ sudo systemctl restart smbd nmbd

or, if ".service" is not automatically appended when executing the above command:

$ sudo systemctl restart smbd.service nmbd.service

Define Crontab

Define crontab to delete Samba Trash files older than 30 days:

$ crontab -e

Edit to include:

#Delete Samba Trash files folder than 30 days
0 0 * * * /usr/bin/find /kctcl/Trash -type f -atime +30 -delete

(added, 26/2/2023)

 

End of Page

Go to End

Note

• 22 Aug 2022: Client mount setting revised.
• 17 Aug 2022: Client mount setting revised.

​​​​​​​Intro

NFS network file system enables sharing of directories on a Ubuntu server to another Ubuntu client computer.

Install on server computer

Install network file system:

$ sudo apt-get install nfs-kernel-server

Configure to specify the directory to be exported:

$ sudo gedit /etc/exports

Specify:

/<directory exported> <ip address of computer to export to> (rw, sync, no_root_squash)

where "rw" = read and write.

Start the server, required after any rebooting:

$ sudo systemctl start nfs-kernel-server
or
$ sudo service nfs-kernel-server start

Set up client computer

Install nfs-common on the client side:

$ sudo apt install nfs-common

Create a local directory:

$ sudo mkdir -p /media/<local directory>

Mount the remote directory temporarily:

$ sudo mount <ip address of NFS server>:/<directory exported from the NFS server> /media/<local directory>

but remounting required after every reboot.

To keep the mounting permanently unaffected by reboot, edit "fstab" file: 

$ sudo gedit /etc/fstab

Add a line:

<ip address of NFS server>:/<directory exported from the NFS server> /media/<local directory> nfs auto,vers=4.0 0 0

(revised 22 Aug 2022, "vers=4.0" added)

(revised 17 Aug 2018, "defaults" changed to "auto")

Mount again all devices as defined in the "fstab" file after changes:

$ sudo mount -a

Check devices actually mounted (this would show more than those defined in "fstab"):

$ sudo mount -l

End of Page

Go to End

Note

31 Jan 2023: Updated.

12 May 2020: Updated.

30 May 2019: Created.

Intro

backintime enables incremental backup of files.

Listing it as the first application software to be installed after installing Ubuntu is an indication of the importance of backing up files.

Install

Execute:

$ sudo apt install backintime*
$ sudo apt install backintime-qt

(updated 12 May 2020)

Bring up

Select Activities.

Type "ba" to bring up some application icons:

Select Back in Time (root) to show the front page:

Alternatively, execute

$ sudo i backintime
or
$ pkexec backintime

However, the terminal commands are not reliable to bring up backintime. Using the graphical desktop is more reliable.

(updated 12 May 2020)

Config

Select the setting button:

Select the General page:

Specify:

• where to save snapshots
• host computer name, user name "root" to have widest permission, profile starting with 1
• scheduled intervals and hours to save snapshots

Select the Include page > Add folder to select the folders to include in backup :

Add file can only include files not a whole folder.

Select Exclude page.

Generally accept the default exclusions.

Select Add folder to select sub-folders to be excluded from the backup when their parent folders have been included:

Select Auto-remove to set time criteria to automatically remove old spanshots:

Select Options page.

Generally accept the default settings.

Select Expert Options page.

Generally accept the default settings.

Select OK to save.

Exit the software. It will run at the specified times.

Take snapshot any time

Bring up the front page:

Select the take snapshot button: 

It would take some time to take a snapshot depending on the extent of file changes since the last snapshot.

Restore

Restore backup files when the present files have been lost or corrupted.

Bring up the front page:

 

Select:

• the desired snapshot in the left window
• the desired backup folder in the middle window
• one or more folders or files in the right window

Select the restore button:

Select whether to backup local files with a trailing suffix before restoring the old files:

Select Yes to proceed if for sure.

Folders and files will be restored.

It would take some time depending on the size to be restored.

Let the process finish with the use permissions set back to the original state.

If only the folders in the middle window is selected before the restore button is selected, this message will appear to indicate that the "/" root folder will be restored:

It is important not to just select the folders in the middle window and select the restore button to proceed, because it will easily restore files to the "/" root directory and overwrite the still valid system and programme files in the root directory to cause problems.

Repair

Sometimes, backintime cannot be brought up to run, even after a removal and re-installation.

This can be caused by the removal of some other programmes which backintime depends on but which have been removed when some other application software is removed. Re-installation of backintime may not bring them back.

The following shows the programmes which backintime depends on:

Try to re-install the missing programmes:

$ sudo apt install <name of programme>

rsync, python3, openssh-client are likely missing programmes.

 

In case of the following error:

Traceback (most recent call last):
  File "/usr/share/backintime/common/backintime.py", line 28, in <module>
    import config
  File "/usr/share/backintime/common/config.py", line 45, in <module>
    import tools
  File "/usr/share/backintime/common/tools.py", line 37, in <module>
    from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'

 

Execute:

$ sudo apt install python3-packaging

(last 2 paragraphs added, 31 Jan 2023)

 

End of Page

 

Go to End

/Note8

8/12/2024: moved to "install VNC Server" page.

26/5/2020: systemctl file command revised.

27/5/2019: Security settings added.

11/4/2018: Page added.

Intro

OpenSSH enables remote client computers and smartphones to access the server computer's text based terminal shell in a secured manner. "SSH" stands for secured shell.

Install OpenSSH server

Execute:

$ sudo apt install openssh-server

The software will be installed at /etc/ssh.

In case of complaint of no directory, execute  to make directory first:

$ sudo mkdir /etc/ssh

Edit config file:

$ cd /etc/ssh
$ sudo gedit sshd_config

Specify:

# Port 22                             (which is the default port)
Port 2nnn                             (change to some other 4-digit port, 2nnn)
# PermitRootLogin prohibit-password   (meaning no password required)
PermitRootLogin no                    (meaning no root login)

(security settings added, 27 May 2019)

Restart the service:

$ sudo systemctl restart sshd.service
or
$ sudo systemctl restart ssh.service

Check status:

$ sudo systemctl status sshd.service

If found disabled:

$ sudo systemctl enable ssh

(status check added, 26 May 2020)

Change the internet router to permit the use of port 2nnn.

The above is already sufficient for use. Read https://help.ubuntu.com/lts/serverguide/openssh-server.html for more configurations, if desired.

Install SSH client on Windows

Download Putty from https://www.putty.org/ and install.

Enter the Host Name, change the Port to 2nnn, highlight Default Settings and press Save:

"Only on clean exit" is the default. When the server's terminal window is exited with "exit" or "logoff", the PuTTY screen and connection would only close if other processes using the PuTTY connection have all been closed.

Press Open.

Accept the next screen to confirm the server's security key shown, if trusted. This would be necessary for the first time only.

Log in as the usual command terminal. No graphical interface is provided.

Use PuTTYgen that comes installed with PuTTY to generate key pairs, only if required. Read its Help.

End of Page

Go to End

Note

8/12/2024: Created.

Intro

VNC server enables remote access to the graphical desktop of Ubuntu server.

There are different VNC server packages: x11vnc, Tight VNC server, Tiger VNC server, etc.

x11vnc is described on a different page.

Set router

Set the internet router to permit access to the server through ports 5900, 5901, 5902, etc.

Disable Wayland

Ubuntu-desktop uses gdm3 (Wayland) display manager. Wayland is not compatible with VNC servers.

Execute to disable Wayland:

$ sudo gedit /etc/gdm3/custom.conf

Remove "#" to uncomment the following line:

#WaylandEnable=false

Reboot the computer.

Install VNC server

Log in an account on the server directly or via a SSH client such as Putty installed as https://kctang.com.hk/web/install-openssh-services.

Execute:

$ sudo apt install tigervnc-standalone-server
or
$ sudo apt install tightvncserver
$ vncpasswd
Password:
Verify:
Would you like to enter a view-only password (y/n)? n

The password will be stored in the file ~/.vnc/passwd, i.e. home/<current user>/.vnc/passwd.

Execute to create a file:

$ nano startvnc.sh

to contain:

#!/bin/bash
vncserver -kill :2
vncserver :2 -xstartup /usr/bin/gnome-session-classic -geometry 1600x900 -localhost -alwaysshared 

vncserver -kill :2 = kill any existing VNC server at port 5902

vncserver :2 = start a new VNC server at port 5902

-xstartup /usr/bin/gnome-session-classic = use gnome-session-classic as the server desktop

-geometry 1600x900 = screen size 1600x900

-localhost = permit access via a SSH tunnel from a remote client computer or mobile phone that the user is using, and not permit access via internet browser

-alwaysshared = permit sharing the server desktop with other remote computers or mobile phones

The server desktop will not be seen on the server itself. It is a virtual desktop accessible to other remote computers or mobile phones. This is a called "headless" server.

The VNC server will stay until it is killed by command: vncserver -kill :2

Execute to start the vncserver:

$ ./startvnc.sh

Start again after server reboot.

Set up SSH tunnel via Putty

Putty has been used for OpenSSH as https://kctang.com.hk/web/install-openssh-services.

Configure a SSH tunnel by entering the Source port and Destination as follows, then press Add to move the setting to the upper window:

With "Local" selected, the Source port means the port of the client computer. It can be "5900" or any free port, e.g. 5902 in this case. "Localhost:5902" at the Destination means the server computer, not the client computer. "5902" refers to the port number on the server computer providing VNC server service.

Go back to the first screen, highlight Default Settings and press Save again.

Log in to the same user account used to start the VNC server.

Install VNC viewer

Download and install a VNC viewer app on the local computer.

There are many VNC viewer apps, such as:

RealVNC at: https://www.realvnc.com/en/connect/download/vnc/

UltraVNC at: https://uvnc.com/downloads/ultravnc.html

There are also mobile phone apps. The setting is similar.

For RealVNC Viewer, after installation, select File > New connection to show:

For VNC Server, enter:

kctang.com.hk: 5902 

for access via an internet browser where the server port is set to 5902, or

localhost: 5902

for use when "-localhost" is included in the start server file, for access via a SSH tunnel from the remote device to the server where the server port is set to 5902. This is more secure.

Press OK to save as an icon:

Click the icon to open the password screen, and enter the password previously set for the server:

The server screen will appear.

End of Page

Go to End

Note

28 May 2019: Disabling Wayland added.

18 May 2019: Page added.

Intro

x11vnc enables remote access to the graphical desktop of Ubuntu server.

Set router

Set the internet router to permit access to the server through ports 5900, 5901, etc.

Disable Wayland

(section added, 28 May 2019)

Ubuntu-desktop uses gdm3 (Wayland) display manager. Wayland is not compatible with Xorg dispaly server used by x11vnc.

Execute to disable Wayland:

$ sudo gedit /etc/gdm3/custom.conf

Remove "#" to uncomment the following line:

#WaylandEnable=false

Reboot the computer.

Install

Execute:

$ sudo apt install x11vnc
$ sudo apt install xvfb

xvfb provides a virtual X window, which is required for the -create option below.

Start

Execute:

$ x11vnc                    (not requiring a password to connect to port 5900)
or
$ x11vnc -rfbport <port number>
or
$ x11vnc -usepw             (requiring a password to connect)
or
$ x11vnc -create -usepw     (create a virtual desktop)

The default port to connect from client computers is 5900.

Specify 5901 for the <port number> if connection is to be permitted at port 5901.

Connection with or without password is possible.

Execute to set up password:

$ x11vnc -storepasswd

The password will be stored in the file ~/.vnc/passwd, i.e. home/<current user>/.vnc/passwd.

Connection is possible if there is already a graphical desktop logged on at the server. The screen movement at the server and the client computer will be synchronised. This would be good for monitoring the screen movement at the server.

If there is no graphical desktop logged on at the server, then:

• use the "-create" option
• connect remotely which should show a terminal window at the server
• execute at the terminal window to bring up other software, such as:

• $ firefox
  $ nautilus

• or the graphical session:

• $ gnome-session
  

• click Activities to access other software.

The remote screen movement will not be seen at the server.

Ubuntu-desktop with gdm3 is used above.

x11vnc does not work well with gdm3 (Wayland) desktop. Therefore, just use gdm3 desktop.

Gnome (Wayland) desktop also does not permit the starting of backintime

Stop

Press Ctrl-C to terminate the connection.

End of Page

Go to End

Note

26 Apr 2022: "gedit" changed to "nano". Minor error corrected.

5 Sep 2019: "0755" changed to "0775" for "Ftp" directory.

7 May 2019: "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

25 Dec 2014: Created.

Intro

FTP server enables directories to be accessible for downloading or uploading by users outside the local network.

Install

Install the package:

$ sudo apt-get install vsftpd

Edit config file:

$ sudo nano /etc/vsftpd.conf

Uncomment the following line to enable uploading:

write_enable=YES

Define as the following line to change the default directory permissions to 775 (drwxrwxr-x) and default file permissions to 664 (-rw-rw-r--):

local_umask=002

Uncomment the following lines to restrict users to their home except for those listed in the file represented by "chroot_list_file":

chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd.chroot_list

("vsftpd.choot_list" corrected as "vsftpd.chroot_list", 7 May 2019)

Save file after uncommenting.

Specify users who can go outside their home by inserting their user login names one on each line in the file represented by "chroot_list_file":

$ sudo nano /etc/vsftpd.chroot_list

("vsftpd/chroot_list" corrected as "vsftpd.chroot_list", 7 May 2019)

Restart ftp service whenever the config files are changed:

$ sudo systemctl restart vsftpd
or
$ sudo service vsftpd restart

Set the internet router to re-direct ftp connections to server port 21.

Set up a root FTP Directory to contain all FTP job folders

Change directory to the top directory assessible for use by Windows network through Samba:

$ cd /<full directory path from root>​

Make a directory specially for FTP storage, called "Ftp" in this example:

$ sudo mkdir Ftp

Change its ownership so that it can be accessed by Windows network:

$ sudo chown nobody:nogroup Ftp

Change its permissions to "read only" for other users:

$ sudo chmod 0775 Ftp

("0755" changed to "0775" because for unknown reasons sub-directory cannot be created under "Ftp", 5 Sep 2019)

Check setting:

$ ls -ls

should show "drwxrwxr-x" and "nobody nogroup" against the "Ftp" item.

(drwxr-xr-x corrected as drwxrwxr-x, 26 Apr 2022)

Create a ftp user for specific job

Create a new user with authority to download and upload the job ftp directory:

$ sudo adduser <ftp user name>

Change the new user's root directory from /home/<ftp user name> to the job ftp directory:

​$ sudo usermod -d /<full directory path from root>/Ftp/<job name> <ftp user name>

• ​<ftp user name> and <job name> can be the same or different
• <job name> will become ftp users' root directory, they will be restricted to see only files at or below the root directory, they will not see the name of <job name> or the directory structure outside the root directory
• ​Instead of <job name>, a further sub-directory such as <job name>/<sub job name> may be defined as the root​ directory
• ​The directory /<full directory path from root>/Ftp/<job name> will still exist but not be used for ftp

Set up a ftp directory for specific job for downloading

Create a job ftp directory under the Ftp directory:

• using Windows Explorer:

​\\<server name>\<full folder path from server>\Ftp\<job name>

• or at the server terminal:

$ cd /<full directory path from root>​/Ftp
$ sudo mkdir <job name>
$ sudo chown nobody:nogroup <job name>
$ ls -ls

should show "drwxr-xr-x" or "drwxrwxr-x" and "nobody nogroup" against the <job name> item.

Further sub-directories may be created similarly for downloading purposes.

Set up a ftp directory for specific job for uploading

Create an "upload" sub-directory under the job ftp directory:

• using Windows Explorer:

​\\<server name>\<full folder path from server>\Ftp\<job name>\upload

• or at the server terminal:

$ cd /<full directory path from root>​/Ftp/<job name>
$ sudo mkdir upload
$ sudo chown nobody:nogroup upload

Change its permissions on the server to enable "write" for all:

$ cd /<full directory path from root>/Ftp/<job name>
$ sudo chmod a+w upload

Check settings:

$ ls -ls

should show "drwxrwxrwx" and "nobody nogroup" against the "upload" item.

Upload or download

​Internally, use Windows file explorer to copy or move files between the Windows networked computers to the ftp directories:

• ​​copy files to ​\\< server name>\<full folder path from server >\Ftp\<job name> for downloading
• copy files from \\<server name>\<full folder path from server >\Ftp\<job name>\upload after uploading by others

Externally, inform external users the ftp user login name i.e. <ftp user name> and password for downloading or uploading.

End of Page

Go to End

Note

• 30 Jul 2024: Updated for PEM pass phrase. "$" prompts changed to "#" when in root mode.
• 15 Jul 2023: Windows folder to contain config files revised.
• 26 Apr 2022: Cipher added.
• 18 Jan 2022: Installation procedures updated.
• 11 Apr 2018: Re-direct function added.
• 2 Sep 2018: Revised to suit Ubuntu 18.04 which requires a change of the network card device name.

Intro

OpenVPN enables remote client computers and smartphones to access VPN server's files and structure, and optionally re-direct clients' IP traffic through the VPN server.

Install VPN server for accessing file server

Switch to root:

$ sudo -s

Install openvpn and easy-rsa:

# apt-get install openvpn easy-rsa

Set up public key infrastructure:

# mkdir /etc/openvpn/easy-rsa/
# cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/
# nano /etc/openvpn/easy-rsa/vars

Define in vars:

export KEY_COUNTRY="CN"
export KEY_PROVINCE="HK"
export KEY_CITY="HongKong"
export KEY_ORG="K C Tang Consultants Ltd"
export KEY_EMAIL="kctang@kctang.com.hk"
export KEY_OU=kctclVPN
export KEY_NAME=kctclVPN
# next line added to avoid error when building the certificate and key 
export KEY_ALTNAMES=kctclVPN

Generate master Certificate Authority (CA) certificate and key:

# cd /etc/openvpn/easy-rsa/
# source vars
# ./easyrsa init-pki
# ./easyrsa build-ca

Enter New CA Key Passphrase: <>

Re-enter New CA Key Passphrase: <>

Enter PEM pass phrase (if prompted): <>

Verifying - Enter PEM pass phrase (if prompted): <>

(last two lines added, 30/7/2024)

Enter Common Name: kctclVPN

Generate a key pair for the server:

# ./easyrsa gen-req kctclVPN nopass

Accept Common Name default [kctclVPN]: <enter key>

Generate Diffie Hellman parameters and generate certificate for the server

# ./easyrsa gen-dh
# ./easyrsa sign-req server kctclVPN

Check and confirm the Common Name: yes

Enter pass phrase (i.e. the PEM pass phrase if entered above, otherwise the CA Key Passphrase): <>

(last line revise to mention PEM pass phrase, 30/7/2024)

Copy certificates and keys generated in subdirectory pki/ to /etc/openvpn/:

# cp    pki/dh.pem    pki/ca.crt    pki/issued/kctclVPN.crt    pki/private/kctclVPN.key    /etc/openvpn/

Config server.conf:

# cd /
# cp    /usr/share/doc/openvpn/examples/sample-config-files/server.conf    /etc/openvpn/
# nano    /etc/openvpn/server.conf

Define as follows:

port 1194
proto udp
dev tun
# Enter the paths if files not in the same directory as the server.conf file
ca </path to file/>ca.crt
cert </path to file/>kctclVPN.crt
key </path to file/>kctclVPN.key
dh </path to file/>dh.pem # not dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
keepalive 10 120
tls-auth ta.key 0
cipher AES-256-GCM
user nobody
group nogroup
# last two lines added for linux system
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
verb 3
explicit-exit-notify 1

(cipher added, 26 Apr 2022)

Generate ta.key for tls-auth:

# cd /etc/openvpn
# openvpn    --genkey    tls-auth    ta.key
$ openvpn --genkey secret ta.key # not --secret

Config sysctl.conf:

# nano /etc/sysctl.conf

Uncomment the following line to enable IP forwarding:

net.ipv4.ip_forward=1

Reload sysctl.conf:

# sysctl    -p    /etc/sysctl.conf

Start the server:

# systemctl    start    openvpn@server
# systemctl    status    openvpn@server

Check if OpenVPN created a tun0 interface:

# ifconfig tun0

Check syslog if tun0 does not appear:

# grep -i vpn /var/log/syslog

Exit from root:

# exit
$

Set the internet router to re-direct OpenVPN connections to server port 1194.

Extend to re-direct clients' IP traffic through VPN server

(section added 5 April 2018)

Define optionally in server.conf to re-direct clients' IP traffic such as web browsing and DNS lookups to go through the VPN server, i.e. the clients will appear to use the IP of the VPN server instead of the actual IP of the clients for internet traffic:

Config server.conf:

$ sudo nano /etc/openvpn/server.conf

Define by uncommenting the following line:

push "redirect-gateway def1 bypass-dhcp"

Some guide suggests to add the following, but this results in email server not working: (10 April 2018)

push "dhcp-option DNS 10.8.0.1"

Some other guides suggest to uncomment the following, this works: (10 April 2018)

push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

However, it is found that it still works when the above two lines are left commented. Therefore, the only line needing change is the 'redirect-gateway' line. (2 Sep 2018)

Execute to restart the service:

$ sudo systemctl restart openvpn@server

Execute to see the network card device names:

$ ip route

Find the output line beginning with "default", e.g.:

default via 192.168.0.1 dev enp4s0 proto static metro 100

The name "enp4s0" after the word "dev" is the default network card device name. Previously, the default name is "eth0", but this has been changed after Ubuntu 16.04.

(2 Sep 2018)

Execute with the default name inserted after "-o":

$ sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o enp4s0 -j MASQUERADE

Note that the iptables configuration will be lost after reboot.

Store the current iptables configurations:

$ sudo sh -c "iptables-save > /etc/iptables.up.rules"

View and remove any configurations no longer applicable:

$ sudo nano /etc/iptables.up.rules

Do the same whenever the iptables configurations have been changed.

Config file for use on reboot:

$ sudo nano /etc/network/interfaces

Define to reuse the stored configurations:

auto lo
iface lo inet loopback
post-up iptables-restore < /etc/iptables.up.rules

Generate files for each Windows client

Switch to root:

$ sudo -s

Generate a certificate and private key for each client user of <username>:

# cd /etc/openvpn/easy-rsa/
# ./easyrsa gen-req <username> nopass
# ./easyrsa sign-req client <username>

Check and confirm the Common Name: yes

Enter Passphrase as previously defined: <>

Copy or move client's certificate and key to a Samba directory, which is for temporary use only: to enable emailing:

# cd    /etc/openvpn/easy-rsa/
# cp    pki/ca.crt    pki/issued/<username>.crt    pki/private/<username>.key    /<Samba directory>/
# cd    /etc/openvpn/
# cp    ta.key    /<Samba directory>/ 

Change the owners of the files:

# cd    /<Samba directory>
# chown    nobody:nogroup    ca.crt ta.key    <username>.crt    <username>.key
# chmod    644    ca.crt    ta.key    <username>.crt    <username>.key

Create a <username>.txt under the Samba directory and define it to contain:

client
dev tun
proto udp
remote kctang.com.hk 1194
resolv-retry infinite
nobind
persist-key
persist-tun
mute-replay-warnings
ca ca.crt
cert <\\path\\><username>.crt
key <\\path\\><username>.key
remote-cert-tls server
tls-auth ta.key 1
cipher AES-256-GCM
auth-nocache
verb 3

(cipher added, 26 Apr 2022)

Specify path in Windows format if ca.crt, ta.key, <username>.crt, and <username>.key are to be saved in a folder different from <username>.txt.

Change filename from <username>.txt to <username>.ovpn. Creating as a txt file first permits editing by the usual text processors. Watch out that the line breaks are using Windows' linebreak code.

E-mail ca.crt, ta.key, <username>.crt, <username>.key, and <username>.ovpn files to the client computer.

Remove the files:

# rm    ca.crt    ta.key <username>.crt    <username>.key    <username>.ovpn

Exit from root:

# exit
$

Install on Windows client computer

Download and install the latest OpenVPN Windows Installer.

An OpenVPN GUI icon should appear at the bottom system tray, with no connection yet.

Save ca.crt, ta.key, <username>.crt, <username>.key, and <username>.ovpn files (emailed from the server) under:

• C:\Program Files\OpenVPN\config-auto\ (for use as a service)

or

• C:\Program Files\OpenVPN\config\ (for use with GUI).

(revised 15 Jul 2023)

Run on Windows client as a service

• Right-click OpenVPN GUI icon on Desktop.
• Click Run this program as an administrator > Yes.

Or

• Click the OpenVPN GUI icon on the bottom system tray to connect or right-click the icon and click Connect.

Map drive for quick access

• Open file manager and enter \\10.8.0.1 to access the vpn server.
• Right-click the desired folder.
• Click Map network drive.
• Choose a drive name to represent the folder.
• Click Finish.

End of Page

Go to End

Note

• 7/10/2024: "mail.kctang.com.hk" also added as email domain name.
• 19/9/2024: Delete separate configuration instructions for spamassassin. Change micro to nano as non-graphical editor.
• 13/2/2023: Use Letsencrypt ssl. Change gedit to micro as non-graphical editor.
• 11/2/2023: Add full config files. Correct opendkim socket.
• 1/2/2023: Correct typos.
• 10/10/2022: Add DKIM setting.
• 18/6/2021: Adjust TLS setting.
• 5/1/2021: Increase imap-login process limit. List Dovecot full custom settings.
• 9/12/2020: Define cron job to delete filtered mails.
• 20/9/2020: Stop using mail-stack-delivery.
• 8/5/2020: Correct typo errors.
• 30/5/2019: Add anti-virus and spam mail filtering.
• 17/5/2019: Add copying emails to external accounts.
• 7/5/2019: Change "gksudo gedit" to "sudo gedit" as Ubuntu 18.04 dropped "gksudo". Add header_checks. Add auto creation of Trash folders. Add webmaster.
• 29/9/2018: Add "body_checks" for spam control.
• 20/9/2018: Increase message_size_limit to 20 times the default.
• 5/4/2018: Increase message_size_limit to 10 times the default.
• 25/12/2018: Publish on web.
• 12/4/2014: Specify maximal_queue_lifetime to notify unsuccessful delivery immediately.
• 2/4/2014: Specify fully qualified domain name.

Intro

Postfix is a mail transfer agent (MTA) responsible for sending out and receiving emails between servers.

Dovecot is a mail delivery agent (MDA) responsible for sending out and receiving emails between a server and its users.

Mail-stack-delivery was a combined package containing both Postfix and Dovecot. It is now no longer supported.

Re-direct to server ports​

Set the internet router to re-direct the following connections to server ports:

• SMTP = port 25 (for receiving or sending emails)
• secure SMTP = port 465 (for receiving or sending emails securely)
• IMAP = port 143 (for retrieving emails)
• secure IMAP = port 993 (for retrieving emails securely)
• POP3 = port 110 (for retrieving emails)

Install both Postfix and Dovecot​

Ubuntu 20.04 does not support the combined package "mail-stack-delivery" anymore.

Install Dovecot and Postfix individually:

$ sudo apt install dovecot-imapd dovecot-pop3d
$ sudo apt install postfix

However, the previous config file "/etc/dovecot/conf.d/99-mail-stack-delivery.conf" is still retained for use, because "99" represents the last and overriding config file. This eliminates the need to change the individual config files.

(revised to install individually, 20/9/2020)

Reconfigure Postfix

Reconfig:

$ sudo dpkg-reconfigure postfix

Use Tab key to change selection.

Select "Internet Site".

Enter the following information:

System mail name: <fully qualified domain name, such as "kctang.com.hk">
Root and postmaster mail recipient: <such as "kctang">
Other destinations to accept mail: <fully qualified domain name, such as "mail.kctang.com.hk" and "kctang.com.hk">, <server name such as "server">, localhost.localdomain, localhost
Force synchronous updates on mail queue: No
Local networks: <leave it blank to accept the default>
Use procmail for local delivery: No
Mailbox size limit (bytes): 0
Local address extension character: +
Internet protocols to use: all

("mail.kctang.com.hk" also added, 7/10/2024)

Edit "main.cf" settings:

$ sudo nano /etc/postfix/main.cf

Specify in full:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on
# fresh installs.
compatibility_level = 2

# the following automatically set by dpkg-reconfigure postfix
myhostname = kctang.com.hk
# fully qualified domain name instead of server name used,
# otherwise some servers would not accept e-mails sent without fully qualified domain name, 2/4/2014

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = mail.kctang.com.hk, kctang.com.hk, server3, localhost.localdomain, localhost
# "mail.kctang.com.hk" also added, 7/10/2024
relayhost = 
mynetworks = 127.0.0.1/32 10.8.0.1/32 [::1]/128
# something similar to the above line, not the same
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

# ALL the following added to default
inet_protocols = all
home_mailbox = Maildir/
mailbox_command = /usr/lib/dovecot/deliver -c /etc/dovecot/dovecot.conf -m "${EXTENSION}"

message_size_limit = 204800000
# last line added to increase the default to 10 times, KCTang 5/4/2014
# increased to 20 times, KCTang 20/9/2018

maximal_queue_lifetime = 0
# last line added to report unsuccessful delivery immediately instead of after the default of 5 days, KCTang 12/4/2014

bounce_queue_lifetime = 0
# last line added, this should not be bigger than maximal_queue_lifetime, KCTang 20/5/2019

# smtpd setting
#smtpd_proxy_timeout = 240s
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/dovecot-auth
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_security_options = noanonymous,noplaintext
# noplaintext in last line added to prevent unencrypted credentials, KCTang 13/2/2023
smtpd_sasl_tls_security_options = noanonymous
# last line added, KCTang 13/2/2023
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_recipient_restrictions = 
    reject_unknown_sender_domain
    reject_unknown_recipient_domain 
    reject_unauth_pipelining
    permit_mynetworks
    permit_sasl_authenticated 
    reject_unauth_destination
    check_policy_service unix:private/policyd-spf
# last line added to enable spf, KCTang 19/2/2022
smtpd_relay_restrictions = 
    permit_mynetworks
    permit_sasl_authenticated
    defer_unauth_destination
smtpd_sender_restrictions = reject_unknown_sender_domain

# TLS parameters
smtp_use_tls = yes
smtp_tls_security_level=may
smtpd_tls_security_level=may
smtp_tls_note_starttls_offer=yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes

# use the following two as provided as default
#smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
#smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

# use the following two if self define key and certificate
#smtpd_tls_key_file=/etc/ssl/private/server.key
#smtpd_tls_cert_file=/etc/ssl/certs/server.crt

# use the following two if using letsencrypt , KCTang 13/2/2023
# using those set up when setting up Apache2 web server, actually they are soft-linked to the following files, KCTang 19/9/2024
smtpd_tls_cert_file=/etc/letsencrypt/live/kctang.com.hk/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/kctang.com.hk/privkey.pem

# use the following if using self certification authority
#smtp_tls_CApath=/etc/ssl/certs/

smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_auth_only = yes
smtpd_tls_mandatory_ciphers = medium
# smtpd_tls_mandatory_protocols = SSLv3, TLSv1 # commented, replaced below, KCTang 18/6/2021
# next 2 lines added to enhance security, KCTang 1/7/2021
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
tls_random_source = dev:/dev/urandom

#header_checks = regexp:/etc/postfix/header_checks
# last line added to check headers, KCTang 30/9/2018, disabled after using amavis 1/6/2019

#body_checks = regexp:/etc/postfix/body_checks
# last line added to refer to another file to check contents of email bodies, KCTang 29/9/2018, disabled after using amavis 1/6/2019

# virtual_alias_domains = kctang.com.hk
# last line added on 17/5/2019, but disabled since same domain name already defined as destination above, KCTang 20/5/2019

virtual_alias_maps = hash:/etc/postfix/virtual
# last line added to forward emails to another server, KCTang 17/5/2019

content_filter = smtp-amavis:[127.0.0.1]:10024
# last line added for amavis, KCTang 1/6/2019

# the following line added to enable spf, KCTang 19/2/2022
policyd-spf_time_limit = 3600

# Milter configuration, added but disabled, KCTang 2/6/2019, enabled again, KCTang 19/2/2022
milter_default_action = accept
milter_protocol = 6
smtpd_milters = local:opendkim/opendkim.sock
non_smtpd_milters = $smtpd_milters

(full file given, 11/2/2023)

(updated, 13/2/2023)

(note regaridng letsencrypt added, 19/9/2024)

Edit "master.cf" settings:

$ sudo nano /etc/postfix/master.cf

Specify​ in full:

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd
#smtp      inet  n       -       y       -       1       postscreen
#smtpd     pass  -       -       y       -       -       smtpd
#dnsblog   unix  -       -       y       -       0       dnsblog
#tlsproxy  unix  -       -       y       -       0       tlsproxy
#submission inet n       -       y       -       -       smtpd
#  -o syslog_name=postfix/submission
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_tls_auth_only=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
# last line uncommented, KCTang 2/4/2014
#  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
# last line added to force use of TLS, KCTang 2/4/2014
  -o smtpd_sasl_auth_enable=yes
# last line uncommented to enable STARTTLS authentication, KCTang 2/4/2014
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# last line uncommented to reject if not authenticated, KCTang 2/4/2014
  -o milter_macro_daemon_name=ORIGINATING
# last line uncommented, KCTang 2/4/2014
#628       inet  n       -       y       -       -       qmqpd
pickup    unix  n       -       y       60      1       pickup
  -o content_filter=
  -o receive_override_options=no_header_body_checks
# last two lines added for anti-spam, KCTang 29/5/2019
cleanup   unix  n       -       y       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       y       1000?   1       tlsmgr
rewrite   unix  -       -       y       -       -       trivial-rewrite
bounce    unix  -       -       y       -       0       bounce
defer     unix  -       -       y       -       0       bounce
trace     unix  -       -       y       -       0       bounce
verify    unix  -       -       y       -       1       verify
flush     unix  n       -       y       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       y       -       -       smtp
relay     unix  -       -       y       -       -       smtp
  -o syslog_name=postfix/$service_name
#  -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
  -o smtp_connect_timeout=60s
# last line added, KCTang 10/3/2021
showq     unix  n       -       y       -       -       showq
error     unix  -       -       y       -       -       error
retry     unix  -       -       y       -       -       error
discard   unix  -       -       y       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       y       -       -       lmtp
anvil     unix  -       -       y       -       1       anvil
scache    unix  -       -       y       -       1       scache
postlog   unix-dgram n  -       n       -       1       postlogd
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix    -    n    n    -    2    pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}

# the following added, KCTang 29/5/2019
smtp-amavis     unix    -       -       -       -       2       smtp
  -o smtp_data_done_timeout=1200
  -o smtp_send_xforward_command=yes
  -o disable_dns_lookups=yes
  -o max_use=20
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
  -o content_filter=
  -o local_recipient_maps=
  -o relay_recipient_maps=
  -o smtpd_restriction_classes=
  -o smtpd_delay_reject=no
 -o smtpd_client_restrictions=permit_mynetworks,reject
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o smtpd_data_restrictions=reject_unauth_pipelining
  -o smtpd_end_of_data_restrictions=
  -o mynetworks=127.0.0.0/8
  -o smtpd_error_sleep_time=0
  -o smtpd_soft_error_limit=1001
  -o smtpd_hard_error_limit=1000
  -o smtpd_client_connection_count_limit=0
  -o smtpd_client_connection_rate_limit=0
  -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    
# the following added for spf, KCTang 1/6/2019, 
# then disabled as considered not necessary
# then re-enabled 19/2/2022 to overcome rejection by gmail
policyd-spf   unix  -       n       n       -       0       spawn
  user=policyd-spf argv=/usr/bin/policyd-spf

(full file given, 11/2/2023)

Create "body_checks" file, if required:

$ sudo nano /etc/postfix/body_checks

Specify​ one or more lines of texts within //:

/unique text contained in email you do not want to receive/ DISCARD

"DISCARD" means delete from the server.

Create "header_checks" file, if required:

$ sudo nano /etc/postfix/header_checks

Specify similarly.

(header checks added, 7/5/2019)

Change Dovecot settings

Edit config file:

$ sudo nano /etc/dovecot/conf.d/99-mail-stack-delivery.conf

Note that a number of default config files are contained in sub-directory "conf.d". To override them, create a config file beginning with "99" so that it is read the latest to override the others. The name is based on the previous "mail-stack-delivery" config file, but can be any.

Specify in full:

# Some general options
# Installed protocols are now auto-included by /etc/dovecot/dovecot.conf
# Since mail-stack-delivery depends on them it is more flexible to not
# explicitly list them here, but achieves the same.
# protocols = imap pop3 sieve
disable_plaintext_auth = yes
# Since 18.04 basic SSL enablement is set up by dovecot-core and configured
# in /etc/dovecot/conf.d/10-ssl.conf.
# So by default basic enablement is no more done here. The old section is kept
# as comment for reference to the old defaults.
#
# ssl = yes
# ssl_cert = </etc/dovecot/dovecot.pem
# ssl_key = </etc/dovecot/private/dovecot.pem
#
# If you keep a formerly used custom SSL enablement in this file it will (as
# before) continue to overwrite the new defaults in 10-ssl.conf as this file is
# sorted later being 99-*.conf
#
# If you choose to take the new defaults (no ssl config in this file) please
# make sure you have also chosen the package defaults for 10-ssl.conf (to enable
# it there) when dovecot-core configures. Also check that the links for cert/key
# set up there got created correctly (they would not be created if they conflict with your
# old keys done by mail-stack-delivery).
#
# use letsencrypt ssl, KCTang 13/2/2023 
ssl = yes
ssl_cert = </etc/letsencrypt/live/kctang.com.hk/fullchain.pem
ssl_key = </etc/letsencrypt/live/kctang.com.hk/privkey.pem
ssl_client_ca_dir =
#ssl_protocols = !SSLv2 !SSLv3
mail_location = maildir:~/Maildir:LAYOUT=fs
# LAYOUT=fs added to last line, to use "/" instead of "." to denote sub-folders, KCTang 25/5/2019
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
# IMAP configuration
protocol imap {
        mail_max_userip_connections = 1000
        # 10 in last line increased to 1000, KCTang 25/5/2019
        imap_client_workarounds = delay-newmail
}
# POP3 configuration
protocol pop3 {
        mail_max_userip_connections = 50
        # 10 in last line increased to 50, KCTang 25/5/2019 
        pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
# LDA configuration
protocol lda {
        postmaster_address = postmaster
        mail_plugins = sieve
        quota_full_tempfail = yes
        deliver_log_format = msgid=%m: %$
        rejection_reason = Your message to <%t> was automatically rejected:%n%r
}
# Plugins configuration
plugin {
        sieve=~/.dovecot.sieve
        sieve_dir=~/sieve
}
# Authentication configuration
auth_mechanisms = plain login
service auth {
  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/dovecot-auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}
# The following are additional to those in 15-mailboxes.conf.
# They are to auto create Trash folders.
# Trash folders would not be backed up with back-in-time,
# and therefore would need to be re-created after email recovery is done from back-in-time.
namespace inbox {
  mailbox Trash {
    auto = subscribe
  }
}
# process_limit increased, KCTang 4/1/2021
service imap-login {
  process_limit = 200
}
# The following added to resolve stats-writer failure, KCTang 3/6/2021
service stats {
  unix_listener stats-reader {
    user = root
    group = root
    mode = 0660
  }
  unix_listener stats-writer {
    user = root
    group = dovecot
    mode = 0660
  }
}

(full file given, 11/2/2023)

(updated, 13/2/2023)

Activate the changes:

$ sudo systemctl reload postfix
or
$ sudo service postfix reload
and
$ sudo systemctl restart dovecot
or
$ sudo service dovecot reload

Verify success

See whether the Postfix server is running:

$ telnet localhost 25

should display:

220 kctang.com.hk ESMTP Postfix (Ubuntu)

ehlo localhost

should display the following:

250-kctang.com.hk

250-PIPELINING

250-SIZE 102400000

250-VRFY

250-ETRN

250-STARTTLS

250-ENHANCEDSTATUSCODES

250-8BITMIME

250 DSN

Ctrl-]

to exit to "telnet >" prompt.

quit

to exit telnet.

Try also:

$ telnet localhost 993

("993", not "995", 1/2/2023)

similarly:

$ telnet localhost 465

should display either one:

Connected to localhost

Connected to kctang.com.hk

"Ctrl-]"

to exit to "telnet >" prompt.

quit

to exit telnet.

Specify internal email forwarding

Edit "aliases" file:

$ sudo nano /etc/aliases

Specify​:

postmaster: kctang
webmaster: kctang
kctcl: kctcl, kctclpop

meaning:

• forwarding e-mails sent to postmaster@localhost and webmaster@localhost to kctang@localhost, no email will be left at postmaster or webmaster

(webmaster added, 7/5/2019)

• forwarding e-mails sent to kctcl@localhost to kctcl@localhost (itself) and to kctclpop@localhst, i.e. making a copy

"localhost" means "kctang.com.hk" in our case. For the email user name before "@", there is no need to create a file system user account for it if all emails addressed to it are forwarded elsewhere. The name serves as an alias only of the email account forwarded to.

(added, 17/5/2019)

 

Activate setting everytime the "aliases" file has been changed:

$ sudo newaliases

 

Copy to external email accounts

(section added, 17/5/2019)

Execute:

$ sudo nano /etc/postfix/main.cf

Specify​ at the end of the file (already shown above):

virtual_alias_domains = kctang.com.hk
virtual_alias_maps = hash:/etc/postfix/virtual

Execute:

$ sudo nano /etc/postfix/virtual

Specify​ to make a copy to itself and a copy to the external email account:

# from               to one or more addresses, separated by a space 
kctang@kctang.com.hk kctang@kctang.com.hk name1@external.account.name
kctcl@kctang.com.hk kctcl@kctang.com.hk name2@external.account.name

Omit making a copy to itself if only email forwarding is required.

Save and exit.

Execute after the "virtual" file is created or changed:

$ sudo postmap /etc/postfix/virtual

Execute:

$ sudo systemctl restart postfix

Test by sending emails.

Install anti-virus Clamav-Daemon

(section added, 30/5/2019)

Install clamav-daemon:

$ sudo apt update
$ sudo apt install clamav-daemon            (clamav-freshclam also automatically installed)
$ sudo dpkg-reconfigure clamav-daemon   (must be done, generally accept all defaults, set yes to scan emails)
$ sudo systemctl start clamav-freshclam
$ sudo systemctl start clamav-daemon
$ sudo systemctl status clamav-freshclam    (check if running)
$ sudo systemctl status clamav-daemon       (check if running, OK if reported "/bin/mkdir /run/clamav (code=exited, status=1/FAILURE" because directory already created)
$ sudo tail -f /var/log/clamav/clamav.log        (see running progress, Ctrl-Z to exit)

(typo corrected, 8/5/2020)

(use sudo, 11/2/2023)

Filter spam mails

(section added, 30/5/2019)

Execute to install various software:

$ sudo apt update
$ sudo apt install amavisd-new spamassassin
$ sudo apt install postfix-policyd-spf-python                    (optionally required for spf) 
$ sudo apt install opendkim                                                   (optionally required for opendkim)
$ sudo apt install pyzor razor                                                (optional extras)
$ sudo apt install arj cabextract cpio lhasa nomarch pax rar unrar unzip zip  ("lhasa", not "lha")
$ sudo razor-admin -create                                                     (added 19/9/2024)
$ sudo razor-admin -register                                                  (added 19/9/2024)

Configure ClamAV:

$ sudo adduser clamav amavis
$ sudo adduser amavis clamav

Configure SpamAssassin:

$ sudo micro /etc/default/spamassassin

Change "ENABLED=0" to:

(no need to configure because amavisd-new handles it, 19/9/2024)

ENABLED=1

(no longer set here, 8/5/2020)

Start the service:

$ sudo systemctl start spamassassin.service

(no need to configure because amavisd-new handles it, 19/9/2024)

Configure Amavisd-new:

$ sudo nano /etc/amavis/conf.d/15-content_filter_mode

Specify:

use strict;
# Uncomment the two lines below to enable antivirus checking mode
@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
# Uncomment the two lines below to enable SPAM checking mode
@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
1;  # insure a defined return

Execute:

$ sudo nano /etc/amavis/conf.d/20-debian_defaults

Change "D_PASS" to:

$final_spam_destiny = D_DISCARD

(deleted, 8/5/2020, reinstated, 26/7/2021)

Adjust the following values only if desired to flag more messages as spam:

$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 6.31; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.31; # triggers spam evasive actions
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent

Execute:

$ sudo nano /etc/amavis/conf.d/50-user

Specify:

$myhostname = 'kctang.com.hk';

Re-start the service:

$ sudo systemctl restart amavis.service

Edit the following file to specify domains to be whitelisted if necessary:

$ sudo nano /etc/amavis/conf.d/40-policy_banks

Test that the Amavisd-new SMTP is listening:

telnet localhost 10024
Trying 127.0.0.1...
Connected to kctang.com.hk.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready

Press Ctrl-] and enter "quit" to exit.

Configure Postfix master.cf:

$ sudo nano /etc/postfix/master.cf

Add the following to the end of the file (already done above):

smtp-amavis     unix    -       -       -       -       2       smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters

Add the following immediately below the "pickup" transport service (already done above):

     -o content_filter=
     -o receive_override_options=no_header_body_checks

Configure Postfix main.cf:

$ sudo nano /etc/postfix/main.cf

Add the following to the end of the file (already done above):

content_filter = smtp-amavis:[127.0.0.1]:10024

Comment it with "#" at the start of the line if desired to stop using amavis.

Restart service:

$ sudo systemctl restart postfix

Check the hidden header of email received after the above to see the presence of one or more of the following:

X-Spam-Level: 
X-Virus-Scanned: Debian amavisd-new at kctang.com.hk
X-Spam-Status:
X-Spam-Level: 

If present, the spam filter is working.

See https://help.ubuntu.com/lts/serverguide/mail-filtering.html.en for a full explanation of the above.

(outdated, 19/9/2024)

Delete filtered mails

(section added, 9/12/2020)

Filtered files are stored in /var/lib/amavis/virusmails and should be deleted regularly.

Edit to define a scheduled job to delete filtered files:

$ sudo crontab -e -u amavis 

where:

-e = edit

-u amavis = user amavis

Define as follows:

0 0 * * * find /var/lib/amavis/virusmails/ -type f -mtime +15 -delete

which means at

0 = 0 minute

0 = 0 hour

* = every day of month

* = every month

* = every day of week

find /var/lib/amavis/virusmails = find in that directory

-type f = regular files

-mtime +15 = file modified more than 15 days ago

-delete = delete the file

Set up DKIM

(implemented, 19/2/2022

section added, 10/10/2022)

Set up DomainKeys Identified Mail (DKIM) to prevent email phishing by others by putting a digital key in the outgoing email headers so that the receiving email server can verify the digital key by looking up the public key in the DNS record of the sender domain.

Install OpenDKIM:

$ sudo apt install opendkim opendkim-tools

Add postfix user to opendkim group:

$ sudo gpasswd -a postfix opendkim

Configure:

$ sudo nano /etc/opendkim.conf

Specify:

# keep to log in /var/log/mail.log
Syslog             yes
SyslogSuccess      yes
# change to "yes" if desired to log more details
Logwhy             no
# keep
Canonicalization   relaxed/simple
# uncomment
Mode               sv
SubDomains         no
# keep
OversignHeaders    from
# add
AutoRestart         yes
AutoRestartRate     10/1M
Background          yes
DNSTimeout          5
SignatureAlgorithm  rsa-sha256
# keep
UserID              opendkim
UMask               007
# comment
#Socket             local:/run/opendkim/opendkim.sock
# add
Socket              local:/var/spool/postfix/opendkim/opendkim.sock
# keep
PidFile             /run/opendkim/opendkim.pid
TrustAnchorFile     /usr/share/dns/root.key
# keep
KeyTable            refile:/etc/opendkim/key.table
SigningTable        refile:/etc/opendkim/signing.table
ExternalIgnoreList  /etc/opendkim/trusted.hosts
InternalHosts       /etc/opendkim/trusted.hosts

Create a directory structure:

$ sudo mkdir /etc/opendkim
$ sudo mkdir /etc/opendkim/keys
$ sudo chown -R opendkim:opendkim /etc/opendkim
$ sudo chmod go-rw /etc/opendkim/keys

Create the signing table:

$ sudo nano /etc/opendkim/signing.table

Specify:

*@kctang.com.hk        default._domainkey.kctang.com.hk
# if sub-domain used
*@*.kctang.com.hk      default._domainkey.kctang.com.hk    

Create the key table:

$ sudo nano /etc/opendkim/key.table

Specify:

default._domainkey.kctang.com.hk     kctang.com.hk:default:/etc/opendkim/keys/kctang.com.hk/default.private

Create the trusted hosts file:

$ sudo nano /etc/opendkim/trusted.hosts

Specify to sign but not to verify emails from localhost or the following domain:

127.0.0.1
localhost
*.kctang.com.hk

("*" added, 1/2/2023)

Create separate folder for the domain and generate Private/Public Keypair:

$ sudo mkdir /etc/opendkim/keys/kctang.com.hk
$ sudo opendkim-genkey -b 1024 -d kctang.com.hk -D /etc/opendkim/keys/kctang.com.hk -s default -v

Use -b 2048 instead of -b 1024 for bits of key if the domain name server permits.
-d for domain name

-D for directory to store the keys

-s for selector

The private key will be written to default.private file and the public key will be written to default.txt file.

Change owner and permission:

$ sudo chown opendkim:opendkim /etc/opendkim/keys/kctang.com.hk/default.private
$ sudo chmod 600 /etc/opendkim/keys/kctang.com.hk/default.private

Display the public key:

$ sudo cat /etc/opendkim/keys/kctang.com.hk/default.txt

The string after the p parameter is the public key:

default._domainkey      IN      TXT     ( "v=DKIM1; h=sha256; k=rsa; "
          "p=..." )  ; ----- DKIM key default for kctang.com.hk

Define in the Domain Name Record on the domain name host:

Name = default._domainkey

Address or value = the text in ( ) above, but deleting double quotes and spaces

Type = SPF (txt)

Test DKIM Key:

$ sudo opendkim-testkey -d kctang.com.hk -s default -vvv

Successful if an opendkim-testkey output is key OK.

No problem if an opendkim-testkey output is key not secure because DNSSEC may not have been enabled on the domain name.

The DKIM record may take up to 24 hours to propagate to the Internet.

Can also go to https://www.dmarcanalyzer.com/dkim/dkim-check/, enter "kctang.com.hk" as the domain name and "default" as the selector to check DKIM record propagation.

In case of the query timed out error, comment out the following line in /etc/opendkim.conf file and restart opendkim.service.

TrustAnchorFile       /usr/share/dns/root.key

Create a directory to hold the OpenDKIM socket file, and allow only opendkim user and postfix group to access it:

$ sudo mkdir /var/spool/postfix/opendkim
$ sudo chown opendkim:postfix /var/spool/postfix/opendkim

Edit:

$ sudo nano /etc/default/opendkim 

Comment out:

# SOCKET=local:$RUNDIR/opendkim.sock

Add to suit Ubuntu:

SOCKET=local:/var/spool/postfix/opendkim/opendkim.sock

Create the file

$ sudo mkdir /var/spool/postfix/opendkim 
$ sudo chown -R opendkim:opendkim /var/spool/postfix/opendkim

(file creation added, 11/2/2023)

Edit Postfix main.cf to configure Milter as shown above.

Restart opendkim and postfix service and check status to see any errors:

$ sudo systemctl restart opendkim postfix
$ sudo systemctl status opendkim postfix

(status check added, 11/2/2023)

Set up Domain Name record

Set up the Domain Name record at the domain hosting company as follows:

v=DMARC1; p=quarantine; pct=10
v=spf1 ip4:202.69.68.114 include:_spf.google.com include:spf.protection.outlook.com ~all

The ip4 is our company's IP address. This is not a secret.

(added, 21/2/2023)

 

End of Page

Go to End

Note

20/9/2024: Certbot updated.

30/7/2020: Updated to use Python3.

6/1/2020: Updated.

8/5/2019: More explanation on default configuration given. "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

11/4/2018:  "apache2" changed to "apache2.service" when used in conjunction with systemctl.

25/12/2014: First created.

Intro

Apache2 web server provides web page services.

Prepare

Define hosts:

$ sudo gedit /etc/hosts

Specify:

127.0.0.1 kctang.com.hk <computer name> localhost
127.0.1.1 <computer name>

Define hostname:

$ sudo gedit /etc/hostname

Specify a line to contain:

<computer name>

Install

Install:

$ sudo apt install apache2

or before Ubuntu 16.04:

$ sudo apt-get install apache2

Start service:

$ sudo systemctl start apache2.service

or:

$ sudo service apache2 start

Set the internet router to re-direct http connections to server port 80.

(The following added, 8/5/2019)

Edit the enabled configuration file:

$ sudo gedit /etc/apache2/sites-enabled/000-default.conf

("ls -ls" changed to "gedit", 6/1/2020)

The file is symbolic linked to the actual location at /etc/apache2/sites-available/000-default.conf.

The file includes the following:

#ServerName www.example.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html

Leave the ServerName to the HTTPS setting below.

Change the ServerAdmin email address to the correct address, or add that address when setting up the email server.

The DocumentRoot tells that the website directories and files will be stored under /var/www/html.

Note that previously the DocumentRoot was at /var/www. The change from /var/www/html would affect the installation of Drupal as explaned on that web page.

Serf to https://www.kctang.com.hk or https://kctang.com.hk on web browser, the following page (var/www/html/index.html) will be displayed to indicate successful installation:

(end of add)

Configure to use HTTPS

This is optional. Starting to use on 6 April 2018.

When the Apache2 server is configured to use HTTPS, and when "https://" is used as the prefix to the website address URL (Uniform Resource Locator) in the web browser navigation bar, encrypted communications will be used with the Apache2 server. This will enhance security.

To do this, enable the mod_ssl module:

$ sudo a2enmod ssl

In order for Apache2 to use HTTPS service, a certificate and a key file are needed. Use EFF's Certbot to automatically deploy Let's Encrypt certificates and enable HTTPS.

Certbot is downloadable at:

https://certbot.eff.org/

No need to download from there.

(revised 8/5/2019)

Install certbot and configure Apache2:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot              (outdated and not necessary)
$ sudo apt update
$ sudo apt install python3-certbot-apache                        (may not be required)
$ sudo certbot --apache
$ sudo certbot renew                                                                (for renewal only)

(python changed to python3, 30/7/2020)

When answering questions

• enter "kctang.com.hk" for name to activate HTTPS
• decide whether to re-install certificate or renew and replace
• select to redirect HTTP traffic to HTTPS.

(added, 8/5/2019).

The following will happen:

• /etc/letsencrypt directory created to contain certificate and key obtained from Let's Encrypt:
  • /etc/letsencrypt/live/kctang.com.hk/fullchain.pem
  • /etc/letsencrypt/live/kctang.com.hk/privkey.pem
  • (these two files can also be used by the email server)
• A file /etc/apache2/sites-available/000-default-le-ssl.conf added and enabled.

(revised, 8/5/2019)

(rfile names stated, 20/9/2024)

• The following lines inserted in /etc/apache2/sites-available/000-default.conf to enforce the use of "https://":

RewriteEngine on
RewriteCond %{SERVER_NAME} =kctang.com.hk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

• A cron job created to renew the certificate which lasts for 90 days before expiry.

Enable new module and disable default module if not already automatically done:

$ sudo a2ensite 000-default-le-ssl
$ sudo a2dissite default-ssl
$ sudo systemctl restart apache2.service

 

End of Page

Go to End

Note

• 28 Apr 2024: Updated. "Manipulate whole database" section added.
• 7 Nov 2023: Changing transaction_isolation.
• 8 Jan 2020: Configuring MySQL-server updated. Removing completely MySQL added.
• 6 Jan 2020: Updated.
• 8 May 2019: References to PHP5 deleted. Secure installation added. Root user authentication added. automysqlbackup added.
• 25 Dec 2014: First created.

Intro

MySQL is a database server.

PHP (PHP5 before Ubuntu 16.04) is a web page programming language.

phpMyAdmin is a web interface to administer the MySQL server.

Replace "[text]" with "actual text without [ or ] and without spaces" in the following commands.

Install MySQL-server

Execute:

$ sudo apt update
$ sudo apt install mysql-server
$ sudo systemctl start mysql.service
$ sudo systemctl status mysql.service

Config MySQL-server

(section added, 8 May 2019)

Execute:

$ sudo mysql_secure_installation

Enter sudo user password, and new mysql root password for the first time or the existing mysql root password.

When answering the secure installation questions:

• validate password component - yes
• set password validation policy - 2 for HIGH

(corrected, "2" is for HIGH, 28 April 2023)

• set password strength - high
• change the password for root - yes or no as appropriate
• set new password - as appropriate
• re-enter new password -
• continue with the password - yes or no as appropriate
• remove anonymous users - yes
• disallow root login remotely - no
• remove test database - yes
• reload privilege table now - yes

(steps updated, 8 Jan 2022)

(suggested answers added, 6 Jan 2020)

Adjust authentication for use with phpMyAdmin

(section added, 8 May 2019)

If not done, the phpMyAdmin menu would not provide a choice to add users.

Execute

$ mysql -u root -p

enter the mysql root user's password when prompted.

(revised to remove unnecessary "sudo", 28 Apr 2024)

Execute:

mysql> SELECT user,plugin FROM mysql.user;

It will show that the root user's authentication plugin is "auth_socket".

Execute:

mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY [mysql_root_user_password];
mysql> FLUSH PRIVILEGES;
mysql> SELECT user,plugin FROM mysql.user;
mysql> quit

(semi colon added after "PRIVILEGES", 6 Jan 2020)

It should show that the root user's authentication plugin has been changed to "mysql_native_password".

Install PHP and related apache2 module

Execute:

$ sudo apt install php libapache2-mod-php

Install phpMyAdmin

Execute:

$ sudo apt install phpmyadmin

Set its own password.

Choose whether to keep the existing database whenever phpmyadmin is removed and re-installed.

Enable Apache2 config:

$ sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/conf-available/phpmyadmin.conf
$ sudo a2enconf phpmyadmin.conf

Whenever Apache2 has been purged and re-installed, this enabling must be done again.

Restart Apache2 service:

Execute:

$ sudo systemctl restart apache2.service

Create a database user and a database

Execute:

 https://localhost/phpmyadmin
 (or)
 https://www.kctang.com.hk/phpmyadmin

Enter login name and password.

Create a user called [name, e.g. Drupal] together with a database also called [database_name]:

• click "Users" at the top menu bar
• click "Add user" at the page middle
• enter [database_name] at the User name entry
• select "Local" at the Host entry
• enter and re-type the password
• click "Create database with same name and grant all privileges"
• click "Go" at the bottom

Install automysqlbackup

(section added, 8 May 2019)

The software will backup mySQL databases daily, weekly and monthly.

Execute to install:

$ sudo apt install automysqlbackup

Edit the configuration file:

$ sudo gedit /etc/default/automysqlbackup

Specify:

BACKUPDIR="/var/lib/automysqlbackup" as the backup directory

MAILCONTENT="log" to send log email

MAILADDR="root" to send the email to root user, which has been set under Postfix Aliases to re-direct to the appropriate user.

Execute to run for the first time:

$ sudo automysqlbackup

Inspect daily backups:

$ sudo ls /var/lib/automysqlbackup/daily

Set transaction_isolation to READ COMMITTED

(section added, 6 Nov 2023)

To suit Drupal 10, change the transaction_isolation from "REPEATABLE READ" to "READ COMMITTED".

Execute:

$ mysql -u root -p               (enter mysql root user's password)

(revised to remove "sudo", 28 Apr 2024)

Execute:

mysql> SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
mysql> SELECT @@GLOBAL.transaction_isolation;
mysql> quit

The first command is to set. The second command is to verify.

Manipulate whole database

(section added, 28 Apr 2024)

Execute:

$ mysql -u root -p               (enter mysql root user's password when prompted)
or
$ mysql -u [database_username] -p[database_user_password]   (no space after "-p")

Show existing databases:

mysql> SHOW DATABASES;

Create database:

mysql> CREATE DATABASE [new_database_name];
(or)
$ mysqladmin -u [database_username] -p[database_user_password] create [new_.database_name]

Delete database:

mysql> DROP DATABASE [database_name];
(or)
$ mysqladmin -u [database_username] -p[database_user_password] drop [database_name]

Export database:

$ mysqldump -u [database_username] -p[database_user_password] -R [database_name] > [filename].sql

No space right after "-p".

"-R" to include stored procedures and functions.

Import exported database:

$ mysql -u [database_username] -p[database_user_password] [empty_database_name] < [filename].sql

Rename database:

• Export as above.
• Create a blank empty database of a new name as above
• Import to the new empty database as above.
• Check if restored database useable as intended.

Restore backup database:

• Find a backup .sql or .sql.gz file, copy to the current directory, and, if gz file, unzip to sql file:

$ sudo ls -ls /var/lib/automysqlbackup/daily/
(or)
$ sudo ls -ls /var/lib/automysqlbackup/weekly/
(or)
$ sudo ls -ls /var/lib/automysqlbackup/monthly/
$ sudo cp [path_to_backup_file]/[backup_filename] [temp_filename].sql.gz
$ sudo gunzip [temp_filename].sql.gz
$ sudo chmod 777 [temp_filename].sql

• Backup the existing database as above.
• Delete the existing database as above.
• Create a blank empty database of the same name or a new name as above.
• Import to the new empty database of the desired name as above.
• Check if restored database useable as intended.
• Delete the temporary file if no longer to be used:

$ rm [temp_filename].sql

Remove mySQL completely

(section added, 8 Jan 2022)

sudo systemctl stop mysql.service
sudo apt purge mysql*
sudo apt autoremove                    (optional)
sudo apt autoclean                     (optional)
sudo apt remove dbconfig-mysql
sudo rm -r /var/lib/mysql
sudo rm -r /log/mysql
sudo rm -r /etc/mysql
sudo deluser mysql

End of Page

Go to End

Note

• 22/11/2024: "print.css" added to the list in append-to-bartik-css.sh
• 21/9/2024: Highlight mark settings added to layout.css.
• 11/9/2024: php uploading capacity increased.
• 26/6/2024: Setting of "state_cache" in settings.php added.
• 28/4/2024: Sections on restoring and relocating Drupal revised.
• 24/4/2024: Sections re-arranged to describe fresh installation procedures followed by upgrading procedures, instead of upgrading procedures followed by fresh installation procedures. "print.css" for printer-friendly version added. List of modules added.
• 24/1/2024: "php/8.3" used.
• 8/11/2023: TOP API and Convert Bundles installations added. Adding buttons to CKEditor revised.
• 5/11/2023: Upgrading to Drupal 10 added. Drupal 9 and before deleted.
• 21/12/2021: Bartik theme used instead of Mayo.
• 11/11/2021: Errors in updating core manually corrected.
• 5/1/2020: Problems when updating to Drupal 8.8 under php7.4 described. Some text updating.
• 9/9/2019: Composer files also copied when relocating system.ttp
• 18/7/2019: Composer configuration actions added.
• 7/7/2019: Original user-defined files and attributes kept when updating manually.
• 26/5/2019: Composer and Drush installations added. Manual core installation added. Deleting configuration file added.
• 16/5/2019: Simplified installation step added. Using new shell to restore added.
• 8/5/2019: Page on Drupal 8 withdrawn. All "/var/www/web" changed to "/var/www/html/web" because Apache2 prefers to put website files underneath "/www/web/html", and it is easier that way. Relocating Drupal 8 added. Print page stylesheet re-written.
• 18/1/2019: Updating Drupal 8 core using composer added.
• 30/9/2018: Setting CKEditor to use Mayo theme stylesheet added. Setting print page to also use Mayo theme stylesheet added. Adding custom styles added.
• 23/9/2018: "php/7.0" changed to "php/7.2".
• 3/9/2018: First created from the page on Drupal 7 after major upgrading to Drupal 8. Resolution of CKEditor table border added.

Intro

(section revised, 5/11/2023)

Drupal is a web site content management system serving web pages to the internet.

Upgrading from Drupal 9 to 10 has been deferred because Drupal 10 has changed the default theme Bartik to Olivero front-end theme and Claro administration theme and some of the modules do not have compatible upgrades. We have been using Bartik and prefer it over those new themes.

However, Drupal 9 reached end of life on 1 Nov 2023. Our Drupal 9.5.11 was therefore upgraded to Drupal 10.1.6 on 4 November 2023. 

This page documents the procedures for a fresh Drupal 10 installation while upgrading procedures are described later. Most of the previous text about installing Drupal 8 and upgrading to Drupal 9 has been deleted because they are not useful anymore. Some old bits and pieces have been retained in case they may be useful.

Prepare

(section revised, 5/11/2023)

(section moved to the front, 22/4/2024)

Install Apache2 web server if not already installed.

Enable Apache2 rewrite module (should have been automatically installed when installing HTTPS):

$ sudo a2enmod rewrite

Configure Apache2:

$ sudo gedit /etc/apache2/apache2.conf

Specify, keeping "<" and ">":

<Directory /var/www/>
    Options Indexes FollowSymLinks
    # AllowOverride None # replaced with next line
    AllowOverride All
    Require all granted
</Directory>

Restart Apache2 service:

$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Install MySQL server + PHP + phpMyAdmin, and create a user with database both called "drupal8". (This old name has not been changed since its creation.)

Install Composer

(updated, 22/4/2024)

Install Composer from any where, and go to the directory under which all Drupal files will be installed. "/var/www/html" is the top one possible

$ sudo apt install composer

Install Drupal 10

(section added, 5/11/2023)

(updated merging others, 22/4/2024)

Go to Drupal's website https://www.drupal.org/ for general reference.

Go to https://www.drupal.org/project/drupal/releases to find the latest release.

Assuming the latest is "x.y.z", execute

$ cd /var/www/html
$ composer create-project drupal/recommended-project:x.y.z "install-dir"

Change "install-dir" to a new name such as "drupal".

The folder structure will be:

/var/www/html/drupal/web
/var/www/html/drupal/web/core
(etc.)
/var/www/html/drupal/vendor

Because our website was upgraded from Drupal 9 to 10, our website used "drupal/legacy-project" structure instead of "drupal/recommended-project" structure. "drupal/legacy-project" has now been removed as an installation choice, but we keep it.

The folder structure is:

/var/www/html/web
/var/www/html/web/core
(etc.)
/var/www/html/web/vendor

"/web" in the drupal/legacy_project" structure would mean "/drupal/web" in the "drupal/recommended-project" structure.

Here we use "/web", and work from there.

Create a location for site specific files:

$ cd /var/www/html/web
$ chmod a+W sites/default
$ mkdir sites/default/files
$ sudo chown -R www-data:www-data sites/default/files

Create the initial configuration file:

$ sudo cp sites/default/default.settings.php sites/default/settings.php

Change ownership:

$ chmod a+W sites/default/settings.php

Disable a hidden file to disable re-direction of webpages:

$ cd /var/www/html/
$ mv .htaccess .htaccessx

Restart Apache2 service:

$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Open the web browser.

Go to https://localhost/web/core/install.php.

Follow the steps there:

• Choose language: English language
• Choose profile: Standard
• Setup database: MySQL
• MySQL database name, user name and password as defined above
• Advanced options
  • Host: localhost
  • Port number: 3306
  • Transaction isolation level: READ COMMITTED
• Site name: www.kctang.com.hk
• Site e-mail address: <>@kctang.com.hk
• Site maintenance username: <name of the administrator>
• Site maintenance user e-mail address: <>@kctang.com.hk
• Default country: Hong Kong S.A.R., China
• Default time zone: Asia/Hong Kong
• Check for updates automatically: Checked
• Receive e-mail notifications: Checked
• Save and continue and the website will be up and running.

Enable the hidden file:

$ mv .htaccessx .htaccess
$ cd web

Change directory and file permissions to restrict:

$ cd web
$ chmod go-w sites/default/settings.php
$ chmod go-w sites/default

Increase upload file size limit (php 8.3 used now as of 22/1/2024):

$ sudo gedit /etc/php/8.3/apache2/php.ini

Change existing capacity to:

; increased from 8M to 200M to 1000M
post_max_size = 1000M
; increased from 2M to 50M to 1000M
upload_max_filesize = 1000M
; increased from 20
max_file_uploads = 100

(capacity increased, 11/9/2024)

Restart Apache2 service after every change:

$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Edit "settings.php" file:

$ sudo nano sites/default/settings.php

Specify trusted hosts:

$settings['trusted_host_patterns'] = array(
   '^www\.kctang\.com\.hk$',
   '^kctang\.com\.hk$',
   '^localhost$',
);

Set state_cache:

$settings['state_cache'] = True;

(added, 20/6/2024)

Disable "install.php" file:

$ sudo mv core/install.php core/<some new name>

Re-direct http path to sub-directory "web"

The default DocumentRoot in the default enabled Apache2 configuration file /etc/apache2/sites-enabled/000-default.conf is "/var/www/html/".

Web browers can access any permissible directories or files underneath the DocumentRoot. Access outside the Document Root will not be possible.

When the Drupal root directory /web is placed underneath /var/www/html such as /var/www/html/web, then https://kctang.com.hk will access the default index.html file underneath /var/www/html, but in the case of https://kctang.com.hk/web, access to those underneath /web is possible.

To re-direct https://kctang.com.hk to access /var/www/html/web directly, create a new ".htaccess" file under "/var/www/html":

$ sudo gedit /var/www/html/.htaccess

Specify:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteBase /
  RewriteRule ^(.*)$ web/$1 [R]
</IfModule>

"^(.*)$" represents the full text from start to end after the domain name, e.g. "abc/def" in "kctang.com.hk/abc/def".

"web/$1" represents the substitution text where "$1" represents the text represented by "(.*)" with "web/" inserted before it, i.e. "web/abc/def" based on the above example. This would redirect the path to "web/abc/def".

"[R]" is to redirect the path and will show "web/" as part of the redirected path. When using "[R]", "RewriteBase /" is required to add "/" before "web/$1". Without this, "/var/www/html" will be added before "web/$1" and will cause unexpected results. This effect was discovered after many days of error discovery using "[L]".

Using [L] as suggested by many people may hide "web/" from the displayed path, and does not require the use of "RewriteBase", but some of the web pages will still show "web/" unavoidably. A mixed use with or without "web/" displayed will cause denial of access rights or redirection to external URL in some cases. Therefore, it is better to force to display "web/" using "[R]".

The following has the same effect:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteBase /web
  RewriteRule ^(.*)$ $1 [R]
</IfModule>

Remove "www." prefix from URL

This is not essential for using Drupal, but is adopted only to simplify.

Insert the following in /var/www/html/.htaccess after the RewriteBase line:

# Remove "www." prefix from URL
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http%{ENV:protossl}://%1%{REQUEST_URI} [L,R=301]

Install drupal/bartik as a contributed theme:

$ composer require drupal/bartik

Go to Manage > Appearance to make it the default theme.

Install Drush

(updated, 22/4/2024)

Drush is a tool to handle installation and removal of modules and themes.

Install Drush:

$ cd var/www/html/web
$ composer require drush/drush

Drush is installed under /var/www/html/web/vendor/drush/drush.

Define Bartik theme color scheme

(section revised, 5/11/2023)

Log in as an administrator if not already in.

Choose Manage > Appearance > Settings, and define Bartik color scheme as follows:

 

Configure Bartik's css files as described below to match the preferences of this website.

Add custom styles to Bartik theme

(section added, 21/12/2021)

(section revised, 6/11/2023)

Log in the server.

Create a "misc" directory under the Drupal root directory, and edit to create 4 new files to re-define webpage styles:

$ cd /var/www/html/web
$ mkdir misc
$ cd misc
$ nano <new file name>

Create "append-to-elements.css" file to contain:

/* the following settings added, KCTang 18/12/2021 */
body {
 margin: 0;
 color: blue;
 font-family: Arial;
}
p {
 margin: 0.2em 0 0.2em;
}
pre{
 border: 1px solid green;
 font-family: monospace;
 margin: 0.15em 0.2em 0.15em;
 padding: 0.2em;
 background-color: #f8f9fa;
 white-space: pre-wrap;
}
.widget-toc{
 display: table;
 border: 1px solid green;
 background-color: #f8f9fa;
 padding: 0.5em;
 font-size: 0.9em;
}
.hangtwice {
 margin-left: 80px;
 text-indent: -80px;
}
.hang1 {
 margin-left: 40px;
 text-indent: -40px;
}
.hang2 {
 margin-left: 80px;
 text-indent: -40px;
}
.hang3 {
 margin-left: 120px;
 text-indent: -40px;
}
.hang4 {
 margin-left: 160px;
 text-indent: -40px;
}
.hang5 {
 margin-left: 200px;
 text-indent: -40px;
}
.indent1 {
 margin-left: 40px;
}
.indent2 {
 margin-left: 80px;
}
.indent3 {
 margin-left: 120px;
}
.indent4 {
 margin-left: 160px;
}
h1,
.heading-a {
 color: blue;
 margin: 0 0 0.5em;
}
h2 {
 border: 1px solid;
#  border-bottom: 1px solid;
 color: red;
 text-align: center
}
h2.hang {
 border: 1px solid;
 margin-left: 40px;
 text-indent: -40px;
 color: red;
}
h3 {
 border-bottom: 1px solid;
 font-weight: bold;
 font-weight: normal;
 color: magenta;
}
h3.hang {
 margin-left: 40px;
 text-indent: -40px;
 font-weight: normal;
 color: magenta;
}
h4.hang {
 margin-left: 40px;
 text-indent: -40px;
}
a,
a.link {
 border-bottom: none;
}
.site-branding__name {
 font-family: "Times New Roman";
}
img {
 margin: 15px;
}
/* the following settings added, KCTang 11/2/2023 */
blockquote {
 border-left: 2px solid #bbb; /* LTR */
 background: #F7F84D;
}
blockquote:before {
 margin-right: 0;
 content: "";
}
[dir="rtl"] blockquote:before {
 content: "";
}
blockquote:after {
 content: "";
}
[dir="rtl"] blockquote:after {
 content: "";
}
/* the following settings added, KCTang 22/3/2023 */
pre{
 font-family: math;

Create "append-to-layout.css" file to contain:

/* the following settings added, KCTang 18/12/2021 */
@media all and (min-width: 851px) {
 .layout-container {
   max-width: 1920px;
 }
}

/* the following settings added, KCTang 21/9/2024 */
mark.marker-blue {
  background-color: hsl(201, 97%, 72%);
}
mark.marker-green {
  background-color: hsl(120, 93%, 68%);
}
mark.marker-pink {
  background-color: hsl(345, 96%, 73%);
}
mark.marker-yellow {
  background-color: hsl(60, 97%, 73%);
}
mark.pen-blue {
  background-color: inherit;
  color: hsl(201, 97%, 72%);
}
mark.pen-green {
  background-color: inherit;
  color: hsl(112, 100%, 27%);
}
mark.pen-red {
  background-color: inherit;
  color: hsl(0, 85%, 49%);
}

(highlight mark settings added, 21/9/2024)

Create "append-to-main-content.css" file to contain:

/* the following settings added, KCTang 18/12/2021*/
.main-content h2 {
 margin: 0.5em 0;
 font-size: 1.429em;
}

Create "append-to-text-formatted.css" file to contain:

/* the following settings added, KCTang 18/12/2021*/
.text-formatted ul,
.text-formatted ol {
 margin: 0;
 padding: 0 0 0.25em 15px; /* LTR */
}

Create "print.css" file to contain:

<!-- Format the printer-friendly print layout -->
<!-- created, KCTang 29/3/2024 -->
a,
a.link {
  text-decoration: none;
  border-bottom: 1px dotted;
}
a:hover,
a:active,
a:focus,
.link:hover,
.link:active,
.link:focus {
  text-decoration: none;
  border-bottom-style: solid;
}
a[href*="EndOfPage"] {
	display: none;
}
body {
  color: black;
  font-family: Arial;
  font-size: 87.5%;
  line-height: 1.5;
  margin: 50px;
  min-height: 100%;
  word-wrap: break-word;
}
del {
  text-decoration: line-through;
}
blockquote {
  border-left: 2px solid #bbb; /* LTR */
  background: #F7F84D;
  font-style: italic;
  margin: 1.5em 10px;
  padding: 0.5em 10px;
}
[dir="rtl"] blockquote {
  border-right: 1px solid #bbb;
  border-left: none;
}
blockquote:before {
  color: #bbb;
  content: "" ;
  font-size: 3em;
  line-height: 0.1em;
  margin-right: 0; /* LTR */
  vertical-align: -0.4em;
}
[dir="rtl"] blockquote:before {
  margin-right: 0;
  margin-left: 0.2em;
  content: "";
}
blockquote:after {
  color: #bbb;
  content: "";
  font-size: 3em;
  line-height: 0.1em;
  vertical-align: -0.45em;
}
[dir="rtl"] blockquote:after {
  content: "";
}
blockquote > p:first-child {
  display: inline;
}
html {
  height: 100%;
}
h1, heading-a {
  border-bottom: none;
  color: blue;
  font-size: 2em;
  font-weight: inherit;
  line-height: 1em;
  margin: 1em 0 0.5em;
  page-break-before: always;
  text-align: center;
}
h2, heading-b {
  border: 1px solid;
#  border-bottom: 1px solid;
  color: red;
  display: block;
  font-size: 1.143em;
  font-weight: inherit;
  line-height: 1.4;
  margin: 0.5em 0;
#  margin-block-start: 0.83em;
#  margin-block-end: 0.83em;
#  margin-inline-start: 0px;
#  margin-inline-end: 0px;
  text-align: center
}
h3, heading--c {
  border-bottom: 1px solid;
  color: magenta;
  font-size: 1.092em;
  font-weight: inherit;
  margin: 1em 0 0.5em;
}
h4, heading-d {
  font-size: 1.05em;
  font-weight: inherit;
  margin: 0.5em 0;
}
h5, heading-e {
  letter-spacing: 0.1em;
  text-transform: uppercase;
  font-size: 0.889em;
  font-weight: inherit;
  margin: 0.5em 0;
}
h6, heading-f {
  letter-spacing: 0.1em;
  text-transform: uppercase;
  font-size: 0.67em;
  font-weight: inherit;
  margin: 0.5em 0;
}
h1 a,
h2 a {
  border-bottom: none;
}
img {
  margin: 15px;
  max-width: 100%;
  height: auto;
}
li {
  display: list-item;
}
ol {
  list-style-type: decimal;
}
ul {
  list-style-type: disc;
}
ul,
ol,
ol ol,
ul ul {
  margin: 0;
  padding: 0 0 0.25em 1em; /* LTR */
}
[dir="rtl"] ol,
[dir="rtl"] ul,
[dir="rtl"] ol ol,
[dir="rtl"] ul ul {
  padding: 0 1em 0.25em 0;
}
mark.marker-blue {
  background-color: hsl(201, 97%, 72%);
}
mark.marker-green {
  background-color: hsl(120, 93%, 68%);
}
mark.marker-pink {
  background-color: hsl(345, 96%, 73%);
}
mark.marker-yellow {
  background-color: hsl(60, 97%, 73%);
}
mark.pen-blue {
  background-color: inherit;
  color: hsl(201, 97%, 72%);
}
mark.pen-green {
  background-color: inherit;
  color: hsl(112, 100%, 27%);
}
mark.pen-red {
  background-color: inherit;
  color: hsl(0, 85%, 49%);
}
p {
  display: block;
  margin: 0.2em 0 0.2em;
#  margin-block-start: 1em;
#  margin-block-end: 1em;
#  margin-inline-start: 0px;
#  margin-inline-end: 0px;
}
pre {
  background-color: #f8f9fa;
  border: 1px solid green;
  font-family: monospace;
  font-family: math;
  margin: 0.15em 0.2em 0.15em;
  padding: 0.2em;
  white-space: pre-wrap;
}
.align-center {
  display: block;
  margin-right: auto;
  margin-left: auto;
}
.align-left {
  float: left;
}
.align-right {
  float: right;
}
.block {
  margin: 10px;
}
.content {
  margin: 10px;
  padding: 15px;
}
.feed-icon {
  display: block;
  margin: 25px 0 0 0;
}
.field.field--name-created {
	display: none;
}
.field.field--name-title {
	display: none;
}
.field.field--name-uid {
	display: none;
}
.hangtwice {
  margin-left: 80px;
  text-indent: -80px;
}
.hang {
  margin-left: 40px;
  text-indent: -40px;
}
.hang1 {
  margin-left: 40px;
  text-indent: -40px;
}
.hang2 {
  margin-left: 80px;
  text-indent: -40px;
}
.hang3 {
  margin-left: 120px;
  text-indent: -40px;
}
.hang4 {
  margin-left: 160px;
  text-indent: -40px;
}
.hang5 {
  margin-left: 200px;
  text-indent: -40px;
}
.footerL {
  position: fixed;
  top: 0;
}
.headerL {
  position: fixed;
  bottom: 0;
}
.indent1 {
  margin-left: 40px;
}
.indent2 {
  margin-left: 80px;
}
.indent3 {
  margin-left: 120px;
}
.indent4 {
  margin-left: 160px;
}
.link {
  font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
}
.main-content {
  margin: 10px
  padding: 15px;
}
.page-title {
  font-size: 2em;
  line-height: 1em;
}
.section {
  margin: 10px;
  padding: 15px;
}
.site-branding__name {
  font-family: "Times New Roman";
}
.tabledrag-handle:hover,
.tabledrag-handle {
  border: none;
}
.text-align-center {
  text-align: center;
}
.text-align-left {
  text-align: left;
}
.text-align-right {
  text-align: right;
}
.text-align-justify {
  text-align: justify;
}
.text-formatted ul,
.text-formatted ol {
  padding: 0 0 0.25em 15px; /* LTR */
  margin: 0;
}
[dir="rtl"] .textfmedia-formatted ul,
[dir="rtl"] .text-formatted ol {
  padding: 0 15px 0.25em 0;
}
.widget-toc {
  display: table;
  border: 1px solid green;
  background-color: #f8f9fa;
  padding: 0.5em;
  font-size: 0.9em;
}
@media screen and (max-width: 37.5em) { /* 600px */
  th.priority-low,
  td.priority-low,
  th.priority-medium,
  td.priority-medium {
	display: none;
  }
}
@media screen and (max-width: 60em) { /* 920px */
  th.priority-low,
  td.priority-low {
	display: none;
  }
}
#EndOfPage {
	display: none;
}

Change to /var/www/html/web directory.

Create "append-to-bartik-css.sh" file to contain:

#!/bin/bash
cat /var/www/html/web/misc/append-to-elements.css >> /var/www/html/web/themes/contrib/bartik/css/base/elements.css
cat /var/www/html/web/misc/append-to-layout.css >> /var/www/html/web/themes/contrib/bartik/css/layout.css
cat /var/www/html/web/misc/append-to-main-content.css >> /var/www/html/web/themes/contrib/bartik/css/components/main-content.css
cat /var/www/html/web/misc/append-to-text-formatted.css >> /var/www/html/web/themes/contrib/bartik/css/components/text-formatted.css
cp /var/www/html/web/misc/print.css /var/www/html/web/themes/contrib/bartik/css
ls -ls /var/www/html/web/themes/contrib/bartik/css

This file is to append the settings in the last 5 files to Bartik's css files to override Bartik's settings.

(print.css file added to the list, 22/11/2024)

Execute to append whenever there is an update to Bartik:

$ cd /var/www/html/web
$ ./append-to-bartik-css.sh

Clear cache to reveal the effects of the new settings:

$ vendor/drush/drush/drush cr

Add buttons to CKEditor 5

(revised, 8/11/2023)

Log in the website as an administrator.

Choose Manage > Configuration > Text formats and editors > Configure Full HTML.

Move the required buttons down to a suitable position in the active toolbar:

Include moving the Style button.

Enter a list of classes and titles as follows:

p.hangtwice|Hang Twice
p.hang1|Hang 1 
p.hang2|Hang 2
p.hang3|Hang 3
p.hang4|Hang 4
p.hang5|Hang 5
p.indent1|Indent 1
p.indent2|Indent 2
p.indent3|Indent 3
p.indent4|Indent 4
h2.hang|Head2Hang
h3.hang|Head3Hang
h4.hang|Head4Hang

"p" stands for paragraph.

".hangtwice" stands for the name of class as defined in the "css/style.css" file.

"Hang Twice" stands for the title seen when the Styles button is pressed to give a drop down menu.

Drush is installed under /var/www/html/web/vendor/drush/drush.

Add TOC API for table of contents

(section added, 8/11/2023)

Unlike CKEditor 4, CKEditor 5 does not have table of contents for free. The previous tables of contents have to be removed page by page.

Use TOC API instead.

Go to Drupal's top folder and install drupal/toc_api:

$ cd /var/www/html/web
$ composer require drupal/toc_api

TOP API Example will automatically be installed also. TOP API is the back end. The Example actually adds the table of contents.

Enable the two modules at Manage > Extend > List.

Go to Manage > Structure > Table of contents types to see a list of example tables.

The default is the one that will be used. The other are examples.

Edit the default. Use responsive to suit mobile phones, which will show on mobile phones the title only with a drop down menu.

Change back to top maximum level to "h2". Change "Back to top" to "-> Top".

No change here.

Change numbering suffix from ") " to ". " both ending with a space.

The numbering suffice follows the numbers to the headings in the body. The numbering separator follows the numbers in the table of contents.

Install other modules

(added, 22/4/2024)

Install the following modules using "composer require <module name as below>":

• drupal/back_to_top_with_pi
• drupal/bartik (already mentioned)
• drupal/book_link_weight
• drupal/book_tree_menu
• drupal/ckeditor
• drupal/ckeditor5_findandreplace
• drupal/ckeditor5_font
• drupal/ckeditor5_fullscreen
• drupal/ckeditor5_highlight
• drupal/ckeditor5_media_embed
• drupal/ckeditor5_show_block
• drupal/ckeditor_find
• drupal/ckeditor_font
• drupal/classy
• drupal/color
• drupal/color-color
• drupal/colorbox
• drupal/colorbutton
• drupal/convert_bundles
• drupal/core (if not already present)
• drupal/core-composer-scaffold (if not already present)
• drupal/core-project-message (if not already present)
• drupal/core-recommended (if not already present)
• drupal/core-vendor-hardening (if not already present)
• drupal/ctools_entity_mask
• drupal/entity_reference_revisions
• drupal/etc
• drupal/font_resize
• drupal/gdoc_field
• drupal/imce
• drupal/insert
• drupal/jquery_ui
• drupal/jquery_ui_effects
• drupal/pathauto
• drupal/pdf
• drupal/pdf_reader
• drupal/quick_node_clone
• drupal/toc_api
• drupal/token
• drupal/upgrade_status
• drupal/view_unpublished
• drush/drush (already mentioned)
• roave/better-reflection

Go to Manage > Entend > relevant module, and click to activate and click to configure as necessary.

Use "composer update <module name>" to update.

Use "composer remove <module name>" to remove.

If "composer remove" cannot remove the module entirely, execute:

vendor/drush/drush/drush pmu -y <module name>
vendor/drush/drush/drush cex 
(to export the settings)

Install Convert Bundles

(section added, 8/11/2023)

When testing various table of contents modules, it was found that some would apply to the Basic Pages but not the Book Pages. This website previously used mainly Book Pages. The drupal/convert_bundles module can be used to convert a type of contents pages to another type. Generally, follow the default settings. After many trials, TOC API has been chosen for the table of contents. By that time all the Book Pages have been changed to Basic Pages. It has not been tested whether the conversion is really required in order to use TOC API.

Update Drupal using Composer

(section added, 18/1/2019)

(simplified, 5/11/2023)

(updated, 22/4/2024)

Put site into maintenance mode:

• Select Admin > Manage > Configuration: Maintenance mode > Put site into maintenance mode > Save configuration.  

Go to the root directory of the website:

$ cd /var/www/html/web

Check for outdated modules:

$ composer show --outdated drupal/* -vvv

"-vvv" is to display what is going on.

Update Drupal core:

$ cd /var/www/html/web
$ composer update "drupal/core-*" --with-all-dependencies
(or)
$ composer update "drupal/core-*" --W

Update all:

$ composer update drupal ---vvv

Update database:

$ vendor/drush/drush/drush updatedb -vvv

Answer "yes" when asked to run specified post-update changes.

Clear cache:

$ vendor/drush/drush/drush cr -vvv

Browse the website to see that the webpages can show up successfully.

Set to release the website from maintenance mode:

• Select Admin > Manage > Configuration: Maintenance mode > unselect Put site into maintenance mode > Save configuration.

Restore Drupal site

(section added, 8/5/2019)

(revised, 16/5/2019)

(revised, 28/4/2024)

Follow Manipulate whole database to find an old database backup of the desired time.

From Back In Time backups, find the web files backed up at a slightly later time.

The restored webpages will be based on the database while newer web files may be ignored.

Follow Manipulate whole database to:

• Back up the existing database
• Delete the existing database.

Move existing web files to another location (e.g. "oldweb"):

$ cd /var/www/html
$ sudo mv web oldweb

Follow Manipulate whole database to:

• Create a new empty database of the same name
• Import the chosen database backup to the newly created empty database.

Restore the chosen web files using Back in Time to the original directory (e.g. /var/www/html/web).

Execute

$ sudo gedit /var/www/new/web/sites/default/settings.php

For

$database['default']['default'] = array (
    'database' = '<text>'
    'username' = '<text>'
    'password' = '<text>'
    ....
);

check the various <text> to match the existing, particularly the password.

Use web browser to see whether web pages are displayed properly.

If web pages are displayed properly, the pair of database and web files match.

Execute to update database and clear caches:

$ cd /var/www/html/web
$ composer update
$ vendor/drush/drush/drush updatedb
$ vendor/drush/drush/drush cr

Resolve problems arising.

Delete "oldweb" if not to be used.

$ cd /var/www/html
$ sudo rm -r oldweb

Relocate Drupal site

(section added, 28/4/2024)

Relocation can mean:

• Renaming the database and moving the web files.
• Keeping the database name but moving the web files.

Use phpMyAdmin to rename the database. This will be the simplest.

Alternatively:

Follow Manipulate whole database to:

• Back up the existing database
• Delete the existing database.
• Create an empty database of new name
• Import the existing database backup to the newly created empty database.

Move existing web files to another location, say "web2" under the same DocumentRoot directory "/var/www/html":

$ cd /var/www/html
$ sudo mv web web2

If a different DocumentRoot directory is used:

• Execute:

$ cd /var/www
$ sudo mkdir <new_DocumentRoot>
$ sudo mv html/web <new_DocumentRoot>            (if "web" name not changed)
(or)
$ sudo mv html/web <new_DocumentRoot>/web2       (if "web" name changed to "web2")
$ sudo cp html/.htaccess <new_DocumentRoot>

• Edit config file to change "/var/www/html" to "var/www/<new_DocumentRoot>":

$ sudo gedit /etc/apache2/sites-enabled/000-default.conf

Change the ".htaccess" file under "/var/www/html" or "var/www/<new_DocumentRoot>" as the case may be to re-direct web to access "web2" (if changed from web) directly:

$ sudo gedit /var/www/html/.htaccess
(or)
$ sudo gedit /var/www/<new_DocumentRoot>/.htaccess

Specify "web2" instead of "web":

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteBase /
  RewriteRule ^(.*)$ web2/$1 [R]
</IfModule>

Execute:

$ sudo gedit /var/www/new/web/sites/default/settings.php

For:

$database['default']['default'] = array (
    'database' = '<text>'
    'username' = '<text>'
    'password' = '<text>'
    ....
);

check the various <text> to match the new, particularly the database name and password.

Use web browser to see whether web pages are displayed properly.

If web pages are displayed properly, the relocation is successful.

Execute to update database and clear caches:

$ cd /var/www/html/web2
(or)
$ cd /var/www/<new_DocumentRoot>/<web or web2>
$ composer update
$ vendor/drush/drush/drush updatedb
$ vendor/drush/drush/drush cr

Resolve problems arising.

Delete left-over module configuration files

(section added, 26/5/2019)

If a left-over configuration setting file is reported when a module is re-installed, execute to delete it:

$ cd /var/www/html/web
$ vendor/drush/drush/drush config:delete '<name of configuration file to delete>'

Upgrade Drupal 9 to 10

(section added, 5 Nov 2023)

The upgrading command as given above is very simple, but the reality is not that simple because there can be many errors due to incompatible modules and themes as well as locking to some previous releases, resulting in the inability to upgrade. There are many other websites describing how to solve the upgrading problems, but the solutions mostly cannot work. After about 12 hours of trials and errors over two days, the upgrading was finally successful. The following records the procedures used (based on memory).

Go to Drupal's top folder and install drupal/upgrade_status:

$ cd /var/www/html/web
$ composer require drupal/upgrade_status

Go to Manage > Reports > Upgrade status.

Get the list of modules and themes which should be updated for Drupal 10.

Get the list of modules and themes which are incompatible with Drupal 10 or no longer used.

Go to Manage > Extend or > Appearance to update or remove modules and themes.

Alternatively, use composer to update or remove:

$ composer require <group name>/<name>
or
$ composer require <group name>/<name>:<add version number as necessary>
or
$ composer remove <group name>/<name>

Go to Manage > Configuration > Text formats and editors to change the text editor from CKEditor to CKEditor 5.

Change back to CKEditor if errors occur when saving the change. Remove those not acceptable menu icons. Change back to CKEditor 5 until the change is successfully saved.

Remove drupal/upgrade_status because the installed version is not compatible with Drupal 10.

$ composer remove drupal/upgrade_status

Update all files and settings to the current version:

$ composer update --with-all-dependencies
$ vendor/drush/drush/drush updatedb

"--with-all-dependencies" can be abbreviated as "-W".

Enable write access (not tested whether this is really necessary):

chmod 777 web/sites/default
chmod 666 web/sites/default/*settings.php
chmod 666 web/sites/default/*services.yml

Install Drupal 10 but without updating, and edit composer.json (see alternative below):

$ composer require drupal/core-recommended:10.1.6 drupal/core-composer-scaffold:10.1.6 drupal/core-project-message:10.1.6 --no-update
$ nano composer.json

Change the following line to state the new release number (it is an important step in this order, otherwise, upgrading will report incompatible new release or locked old releases):

"drupal/core": "10.1.6",

Upgrade now based on the newly specified release:

$ composer update --with-all-dependencies

Alternatively, the following command may work instead of the above command with "--no-update" option and the update command:

$ composer require "drupal/core:9.5.11 as 10.1.16" --no-update && composer update

However, the following line in composer.json still needs to be changed as such afterward.

"drupal/core": "10.1.6",

Update drush and update the database

$ composer require drush/drush
$ vendor/drush/drush/drush updatedb

The updatedb command may report post-update changes. Generally answer "Yes" to accept the changes.

The upgrading should be successful.

Restore read-only access:

chmod 755 web/sites/default
chmod 644 web/sites/default/*settings.php
chmod 644 web/sites/default/*services.yml

Install drupal/bartik as a contributed theme:

$ composer require drupal/bartik

Go to Manage > Appearance to make it the default theme.

Add back any compatible modules previously deleted due to incompatibility.

Go to Manage > Reports > Status report to see what errors arise.

It may remind to download colorbox-master. Download and extract it to a different name as /var/www/html/web/libraries/colorbox.

It may also remind to download DOMPurify-main. Download and extract it. Move its "dist" directory to become /var/www/html/web/libraries/dompurify/dist.

Some modules are related to CKEditor, they have to be removed before CKEditor can be removed. Some of them can be re-installed after the removal of CKEditor and automatically linked to CKEditor 5.

End of Page

Go to End

Note

11/10/2024: smtp setting revised.

23/1/2024: Minor changes.

26/3/2023: smtp setting revised.

16/12/2021: Updated installation command.

27/5/2019: Changed to use extracting directly to web directory. Protection measures added.

8/5/2019: All "/var/www/web" changed to "/var/www/html/web". "<version number>" used instead of a specific number. More descriptions on settings added.

25/12/2014: Created.

Intro

Roundcube is a webmail client.

Install Roundcube using apt

Execute:

$ sudo apt install roundcube

(Deleted, not working, 11/10/2024)

Install Roundcube from roundcube.net

If the apt installation does not work, execute to remove:

$ sudo apt remove roundcube

Then download roundcubemail-<version number>-complete.tar.gz by selecting "Complete: <version number>" at https://roundcube.net/download/, usually to own "Downloads" directory.

Extract the compressed file, move the extracted directory and files to underneath "/var/www/html", and rename the directory as "roundcube".

Ensure that the hidden ".htaccess" file is also moved.

A quicker way is to extract directly to "/var/www/html" and rename the directory there.

Install Roundcube under "/var/www/html":

$ cd /var/www/html
$ sudo tar -xzfv /home/<own account name>/Downloads/roundcubemail-<version number>-complete.tar.gz  (extract to current directory)
$ tar -xzf /home/<own account name>/Downloads/roundcubemail-<version number>-complete.tar.gz (extract to current directory) 
$ ls      (see the existence of the extracted directory)​
$ sudo mv roundcubemail-<version number> roundcube    (change directory name)
$ cd roundcube
$ sudo chown www-data:www-data logs
$ sudo chown www-data:www-data temp

"-xzfv" can be remembered as extract zipped file verbose. Use "-xzf" if "v" not working. 

(changed to use "-xzf", added cd roundcube, 16/12/2021)

(changed to use "tar" directly to web directory,  27/5/2019)

Set up Roundcube

Install MySQL database + PHP + phpMyAdmin, if not already installed.

Login phpmyadmin at web browser:

https://localhost/phpmyadmin

Create a database user "roundcube" with database "roundcube":

• click "Users" at the top menu bar
• click "Add user" at the page middle
• enter "roundcube" at the User name entry
• select "Local" for "localhost" at the Host entry

(Select "Local", not enter "local", 11/10/2024)

• enter and re-type the password
• click "Create database with same name and grant all privileges"
• click "Go" at the bottom

Enter at web browser:

https://localhost/roundcube/installer

Configure Roundcube:

• accept all of the defaults
• enter database user "roundcube", database "roundcube" and password under Database setup
• click: "CREATE CONFIG"
• download the configuration file generated and save it as config.inc.php
• move the downloaded file:

$ mv /home/<own account name>/Downloads/config.inc.php /var/www/html/roundcube/config

• click: "CONTINUE" on the webpage
• click: "Initialize database"
• set to use secure SMTP port 465 (i.e. set "localhost:465" as SMTP Host)

• (last line not working, changed to next line, 26/3/2023)

• set "ssl://kctang.com.hk" as SMTP Host

• (revised again, 11/10/2024)

• enter username and password, and sender and recipient full email addresses to test SMTP config
• accept "localhost" as IMAP Host
• enter username and password to test IMAP config

If installed from roundcube.net, remove or rename installer directory, and protect logs and temp directories:

$ sudo cd /var/www/html/roundcube
$ sudo rm -R installer                   (remove)
$ sudo mv installer installer.original   (or rename)
$ sudo chown -R $USER:$USER logs
$ sudo chown -R $USER:$USER temp

(revised to add protection, 27/5/2019)

Configure PHP5 only (not necessary for PHP7 and later):

$ sudo gedit /etc/php5/apache2/php.ini

• define:

date.timezone = Asia/Hong_Kong

otherwise the date column of the web mail would be blank. 

Restart Apache2 service:

$ sudo systemctl reload apache2
or
$ sudo service apache2 reload

Log in at web browser:

https://localhost/roundcube

or

https://www.kctang.com.hk/roundcube

Click Settings > Identity to define some settings:
 ​

Click Settings > Preferences > User Interface

> Time zone > Asia/Hong Kong

> Time format > 07:30

> Date format > 24/7/2023

> Save.

Click Settings > Preferences > Composing Messages

> Compose HTML messages > always

> When replying > start new message above the quote

> Force standard separator in signatures > turn off

> Save.

Click Settings > Contacts > Import to import contacts files, e.g. previously exported from Google.

(more descriptions on settings added, 8/5/2019)

(further added, 26/3/2023)

Upgrade if installed from roundcube.net

Download and extract the subdirectory of the new version to "Downloads" directory as "roundcubemail-<new version number>".

Upgrade existing directory using the installto.sh script:

$ cd /home/<own account name>/Downloads/roundcubemail-<new version number>/bin
$ ./installto.sh /var/www/html/roundcube

In case of big trouble

Uninstall Roundcube:

$ sudo apt-get remove roundcube

Remove "/var/www/html/roundcube":

$ cd /var/www/html
$ sudo rm -r roundcube

Delete mySQL user "roundcube" and database "roundcube" using phpMyAdmin.

Intall Roundcube as new.

End of Page

Go to End

Note

MoinMoin wiki engine serves wiki web pages.

The following installation instructions are outdated. Check https://help.ubuntu.com/lts/serverguide/moinmoin.html for updated instructions.

Prepare

Install Apache2 web server if not already installed.

Download MoinMoin (filename moin-1.9.7.tar.gz) from https://moinmo.in/MoinMoinDownload, usually to own "Downloads" directory.

Use the file manager to extract the contents of the compressed file under the "Downloads" directory as "moin-1.9.7".

Install

Install MoinMoin:

$ cd /home/< own account name >/Downloads/moin-1.9.7
$ sudo python setup.py install --force --prefix /usr/local --record=install.log

Install "wsgi" module and test:

$ sudo apt-get install libapache2-mod-wsgi
$ cd /usr/local/share/moin/server 
$ sudo python test.wsgi

Point the web browser to the address shown on the terminal, such as, https://localhost:8000/.

Setup wiki sites

Setup files for a wiki called "qswiki":

$ cd /usr/local/share/moin
$ sudo mkdir qswiki
$ sudo cp -R data qswiki
$ sudo cp -R underlay qswiki
$ sudo cp server/moin.wsgi qswiki
$ sudo cp config/wikiconfig.py qswiki

Do similar for "dscwiki".

Configure Apache2 for MoinMoin site:

$ sudo gedit /etc/apache2/sites-available/moinmoin.conf

Specify, keeping "<" and ">" in the following codes:

# MoinMoin WSGI configuration
# invoke moin wiki at the root url, like https://servername/mywiki/FrontPage:
# servername can be www.kctang.com.hk or localhost
# qswiki and dscwiki use FrontScreen instead of FrontPage to avoid changes from being overridden when a new FrontPage is installed with new software
WSGIScriptAlias /qswiki   /usr/local/share/moin/qswiki/moin.wsgi
WSGIScriptAlias /dscwiki   /usr/local/share/moin/dscwiki/moin.wsgi
<Directory /usr/local/share/moin/qswiki>
    AllowOverride None
    Require all granted
</Directory>
<Directory /usr/local/share/moin/dscwiki>
     AllowOverride None
     Require all granted
</Directory>
# create some wsgi daemons - use these parameters for a simple setup
WSGIDaemonProcess moin user=www-data group=www-data processes=5 threads=10 maximum-requests=1000 umask=0007
# use the daemons we defined above to process requests!
WSGIProcessGroup moin

Enable the site:

$ sudo a2ensite moinmoin

which will create a linked file in "/etc/apache2/sites-enabled"

If required to disable:

$ sudo a2dissite moinmoin

Configure wsgi:

$ sudo gedit /usr/local/share/moin/qswiki/moin.wsgi

Add at the end of the a2) paragraph:

sys.path.insert(0, '/usr/local/share/moin/qswiki')

Do the same for "dscwiki".

Enable web access, specify access rights and restart Apache2 service:

$ cd /usr/local/share 
$ sudo chown -R www-data:www-data moin 
$ sudo chmod -R ug+rwX moin 
$ sudo chmod -R o-rwx moin
$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Configure wiki:

$ sudo gedit /usr/local/share/moin/qswiki/wikiconfig.py

Specify:

# -*- coding: utf-8 -*-
## -*- coding: iso-8859-1 -*-

#url_prefix_static = '/mywiki' + url_prefix_static
url_prefix_static = '/qswiki' + url_prefix_static

# sitename = u'Untitled Wiki'
sitename = u'QS Wiki'

#page_front_page = u"FrontPage"
page_front_page = u"FrontScreen"

# superuser = [u"YourName", ]
#    (note: remove "#" at the front, and replace "YourName" with user id within quotation marks)

#acl_rights_before = u"YourName:read,write,delete,revert,admin"
#    (note: remove "#" at the front, and replace "YourName" with user id within quotation marks)

# the following added at the end of the Security section

textchas = {
    'en': {
        u"Please enter antispam password (email to kctang@kctang.com.hk for password):": ur"[password]",
        },
    }
textchas_disabled_group = u"TrustedGroup"

#    (note: replace [password] with actual password)

# end of addition

#mail_smarthost = ""
mail_smarthost = "kctang.com.hk"

# The return address, e.g u"Jgen Wiki <noreply@mywiki.org>" [Unicode]
#mail_from = u""
mail_from = u"K C Tang <kctang@kctang.com.hk>"

# "user pwd" if you need to use SMTP AUTH
#mail_login = ""
mail_login = u"kctang [email password]"

navi_bar = [
    # If you want to show your page_front_page here:
    #u'%(page_front_page)s',
    u'RecentChanges',
    u'FindPage',
    u'HelpContents',
    u'FrontScreen',
]

# The default theme anonymous or new users get
# theme_default = 'modernized'
theme_default = 'modernized_mobile'

Note MoinMoin Config files are:

• /etc/apache2/sites-available/moinmoin.conf
• /etc/apache2/sites-enabled/moinmoin.conf
• /usr/local/share/moin/qswiki/moin.wsgi
• /usr/local/share/moin/qswiki/wikiconfig.py
• /usr/local/share/moin/dscwiki/moin.wsgi
• /usr/local/share/moin/dscwiki/wikiconfig.py

End of Page