Computer Application 電腦應用

Computer Application 電腦應用 KCTang

Commonly used Ubuntu Linux commands 常用的Ubuntu Linux指令

Commonly used Ubuntu Linux commands 常用的Ubuntu Linux指令 KCTang

Go to End

Note

29/7/2022: Setting up SSH keys added.

18/4/2022: cp -Rav added.

14/4/2022: Command to delete multiple empty sub-directories added.

18/12/2021: Updated.

6/10/2018: Updated.

27/9/2018: Updated.

21/3/2018: Updated.

27/8/2017: Created.

Intro

$ = the terminal command prompt against which commands are entered.

sudo = superuser.

<text> = information to be entered, angle brackets themselves are not to be entered.

[abc] = options a, b and c, brackets are not necessary.

Capitals and small letters behave differently.

View system information

Show who am I:

$ whoami

Show current date and time:

$ date

Show this month's calendar:

$ cal

List open ports and their processes id:

$ sudo lsof -i

or

$ sudo netstat -lptu

or

$ sudo netstat -lptun

Display disk space:

$ df

Display disk space in a more readable format:

$ df -h

Display file contents:

$ cat <directory>/<sub-directory>/<filename>

Use path and file names

<directory path> = <directory>/<sub-directory> referring from the current directory, without filename stated.

<filename path> = <directory>/<sub-directory>/<filename> referring from the current directory.

Precede with "/" if referring the path from the root directory. Deleting files under a sub-directory but incorrectly referring to the root directory is dangerous. Better avoid referring from the root directory.

Use "*" as a wildcard to represent texts before or after directory or file name, e.g. "*name", "name*" or "*" for any name.

Use "." alone to represent the current directory.

Use "../" to represent the directory immediately above the current directory.

Use "/." at the end to represent all under the stated <directory path>, e.g. "<sub-directory>/."

Search things

Find files with filenames containing <filename> in <directory path>:

$ find <directory path> -name "<filename>"

Use "-iname" to ignore upper or lower case differences.

Find the files and open for editing:

$ find <directory path> -iname "<filename>" -exec nano '{}' \;

Find text in files:

$ grep -[options] <text> <filename path>

where option:

  • -i = ignore upper or lower case differences
  • -r = search sub-directories recursively
  • -h = hide names of files preceding lines of output
  • -w = search for exact text
  • -c = count the number of matches
  • -n = precede lines of output with the numbers of lines containing the text
  • -v = show only lines not containing the text
  • -l = list filenames only
  • --colour = display the output in colour

Adjust disk space

Find duplicate files across several directories and replace duplicate files with hard links to save space:

$ sudo rdfind -makehardlinks true <directory path 1> <directory path 2>

note: rdfind downloadable at http://rdfind.pauldreik.se/rdfind.1.html#lbAG.

Recover such harddisk space of deleted files not reported by "$ df":

$ cd /<directory name of the harddisk> 
​​$ sudo dd if=/dev/zero of=tempfile 
$ sudo rm tempfile

Edit files

$ sudo nano <filename path>

or

$ gksudo gedit <filename path>

Note that the following can work sometimes but may have problem under some desktop environment:

$ sudo gedit <filename path>

Shutdown and start up

Shutdown the system:

$ sudo shutdown

Shutdown and reboot the system:

$ sudo reboot

Start a service:

$ sudo systemctl start <service name>

or older method:

$ sudo service <service name> start

or even older method:

$ sudo /etc/init.d/<service name> start

Restart a service:

$ sudo systemctl restart <service name>

or older method:

$ sudo service <service name> restart

or even older method:

$ sudo /etc/init.d/<service name> restart

Stop a service:

$ sudo systemctl stop <service name>

or older method:

$ sudo service <service name> stop

or even older method:

$ sudo /etc/init.d/<service name> stop

Mount devices

Mount a single device "/media/newbackupdrive":

$ sudo mount /media/newbackupdrive

Remounting required after reboot.

Umount single device "/media/newbackupdrive":

$ sudo umount /media/newbackupdrive

Check devices defined in the filesystem table file "fstab" to be mounted upon booting:

$ cat /etc/fstab

Check devices actually mounted (this would show more than those defined in "fstab"):

$ sudo mount -l

Set up SSH keys to access remote computer

Generate public/private rsa key pair:

$ ssh-keygen -t rsa

Enter file in which to save the key (/home/<your user name>/.ssh/id_rsa): <accept or change>

Skip entering passphrase if do not want to use it upon logging in with ssh.

Using $ sudo ssh-keygen -t rsa will prompt to save the key at /root/.ssh/id_rsa.

If the file already exists, answer overwrite or not.

Copy the public key to the remote computer:

$ ssh-copy-id -p <port number> <login name>@<remote computer address>

Enter password to the remote login name.

It will check whether the key has already existed.

Use this if desired to force copy:

$ ssh-copy-id -f -p <port number> <login name>@<remote computer address>

Verify by logging in after successful copying:

ssh -p <port number> <login name>@<remote computer address>

If login successful, you are at the command prompt of the remote computer and can execute commands there.

Exit after use.

(added, 29/7/2022)

Add or delete groups and users

Create new group:

$ sudo addgroup <new group name>

Create new user, with a group and a home directory of the same name created if not already existing:

$ sudo adduser <new user name>

To verify:

$ ls -lh /home

shows that the new directory has "drwxr-xr-x" permissions, i.e. "d" for directory with "rwx" owner permissions but "r-x" group and others' permissions.

Add a user to a group:

$ sudo adduser <user name> <group name>

Delete user, and group of the same name, keeping the home directory:

$ sudo deluser <user name>

Delete group:

$ sudo delgroup <group name>

Change ownership

Change file or directory ownership:

$ chown -R <owner name>:<group name> <filename path>

where:

  • -R = recursively from and below if <filename> is a sub-directory

Change permissions

Change file or directory permissions:

$ chmod -R [ugoa][-+=][rwxXst] <filename path>

where:

  • -R = recursively from and below if <filename> is a <sub-directory>
  • u=owner, g=group, o=others, a=all
  • -+= mean minus, add or equal permissions
  • r=read, w=write and delete, x=execute file or change directory into; Xst=for more special choices
  • e.g. "u+rw" = add read and write permission to owner

An alternative form is:

$ chmod -R <ugo> <filename path>

where:

  • <ugo> is a 3 digit number where u=owner, g=group, o=others
  • each digit can be:
    • 0 for nothing
    • 1 for execute "x"
    • 2 for write "w"
    • 3 = 1 + 2 = "wx"
    • 4 for read "r"
    • 5 = 4 + 1 = "rx"
    • 6 = 4 + 2 = "rw"
    • 7 = 4 + 2 + 1 = "rwx"

therefore:

  • 666 = read and write permissions to all
  • 777 = read, write and execute permissions to all

Change password

Change own password:

$ passwd

Change other user 's password:

$ sudo passwd <other user's name>

Create directories

Make new directory:

$ mkdir <directory>

Make new directory and sub-directory in one go, "-p" means making parent directory also:

$ mkdir -p <directory>/<sub-directory>

Remove files and directories

Remove empty sub-directory:
$ rmdir <directory path>

Remove one empty sub-directory and its parent directory in one go:

$ rmdir /<parent directory>/<sub-directory>

Remove all empty sub-directories and their parent directories in one go:

$ cd <directory>
$ find . -type d -empty -print -delete

If directory is empty, print name then delete.

(added, 14/4/2022)

Remove file:

$ rm <filename path>

Remove files and directories starting from and below sub-directory, even for empty sub-directory:

$ rm -r <directory path>

List and change directories

List names of current directory contents, hiding entries starting with ".":

$ ls

List current directory contents, with more detailed information:

$ ls -[options]

where options

  • l = list also permissions, owners, date and size
  • a = list also entries starting with "."
  • h = to be used in conjunction with "l", show file sizes in "K" or "M"

List other directory contents:

$ ls -[options] <directory path>

Change working directory:

$ cd <directory path>

Copy files and directories

Copy a file within the same directory:

$ cp <source filename> <new filename>

Copy a file across different directories, keeping the same filename:

$ cp <source directory>/<sub-directory>/<filename> <target directory>/<sub-directory>/.​

to give:

<target directory>/<sub-directory>/<filename>

​Copy a file across different directories, to a new filename:

$ cp <source directory>/<sub-directory>/<filename> <target directory>/<sub-directory>/<new filename>

to give:

<target directory>/<sub-directory>/<new filename>

Omit:

<source directory>/<sub-directory>/

if copying files in the current directory.

Copy directories recursively, keeping the directory name:

$ cp -R <source directory>/<sub-directory A> <target directory>/.

to give:

<target directory>/<sub-directory A>

Copy directories recursively:

$ cp -R <source directory>/<sub-directory A> <target directory>/<sub-directory B>

If sub-directory B exists, all files and directories under sub-directory A will be copied under sub-directory B.

If sub-directory B does not exist, it will be created, and all files and directories under sub-directory A will be copied under it.

Copy directories recursively (-R) keeping original attributes (-a) and showing verbose progress (-v):

$ cp -Rav <source directory>/<sub-directory A> <target directory>/<sub-directory B>

(added, 18/4/2022)

Move files and directories

Use "mv" instead of "cp" for the above copy commands. No need to use "-R". Recursive move is the default.

Backup a directory of files

Archive (-a) all files under a sub-directory (A) and all sub-sub-directories underneath to the same sub-directory name under another sub-directory (B) keeping all the file attributes, symbolic links and time-stamps unchanged, preserving hard-links (-H) and displaying the progress verbosely (-v) and the numbers in human-readable format (-h):

$ sudo rsync -aHvh <source directory>/<sub-directory A> <target directory>/<sub-directory B>/​

to give:

<target directory>/<sub-directory B>/<sub-directory A>

The command can be used repeatedly to update the files in sub-directory A in the new location. If the source files have not been changed, no over-writing copying will be done. This would save time and is better than the cp command.

End of Page

Set Up Ubuntu Server 架設Ubuntu伺服器

Set Up Ubuntu Server 架設Ubuntu伺服器 KCTang

Go to End

Note

28 May 2019: Notes regarding unsuccessful installation added. Showing boot messages added.

7 May 2019: Contents added.

25 Dec 2014: First created as a flysheet without its own contents.

Intro

Ubuntu is a linux server software.

It has a desktop version and a server version, amongst other products.

The kernels of the desktop version and the server version are the same. The installation procedures are slightly different.

The desktop version provides a graphical user interface and the server version has a text based user interface, though a graphical user interface can subsequently be installed on the server version. Once installed with the graphical user interface, the two versions would not look much different to the users.

For small system, the desktop version should be easier to use.

Download

Download free of charge here

Install

Read installation guides:

Read fuller user guide here.

Set up storage drives

When allocating drive spaces, choose "Something else" with the desktop version and "manual" with the server version if one does not want to adopt the default settings, e.g. if one wants to have more options, such as setting one drive for "/boot" partition and one for "/" root system partition.

Set the computer bios to boot from the selected boot up disk. Remember to save the setting.

It is said that the boot partition needs only be about 300 Mb big. However, experience tells that it gets full easily because of frequent updating of the kernels and retainage of the last few kernels in the boot partition.

To remove kernels no longer needed to be retained, execute:

$ sudo apt autoremove

However, sometimes, the 300Mb boot partition has gone up to 100% full and insufficient to contain the required last few kernels such that there are no unused kernels to be removed to free up space.

Therefore, it is recommended to set up a boot partition of 1Gb.

Our company set-up:

  • boot partition on the smaller SSD disk sold with the computer
  • root system partition on the bigger hard disk relocated from old computer
  • additional hard disks mounted on the system for storage of data

To see disks mounted, execute:

$ df -h

The mounting configuration file is contained in /etc/fstab.

To mount disk using graphical user interface permanently, click "Activities" , enter "disk" and select the "Disks":

Mounting using "Disks" would change /etc/fstab permanently, without the need to change the /etc/fstab file manually.

Install application software packages

Update the software repository:

$ sudo apt update

Install software package:

$ sudo apt install <name of software>

Uninstall software package, keeping its configuration files:

$ sudo apt remove <name of software>

Uninstall software package and its configuration files:

$ sudo apt purge <name of software>

Alternatively, to bring up the graphical software package installation manager, execute:

$ sudo synaptic

or, on the graphical desktop, click "Activities" , enter "synaptic".

Upgrade Ubuntu release

Upgrade Ubuntu release:

$ sudo do-release-upgrade

Re-install

If it is necessary to re-install Ubuntu, back up everything first. 

Ubuntu usually can recognize the existing boot and root systems and data disks when Ubuntu is re-installed.

Try to keep the new co-existing with the old and change later after the system is running.

Keep the existing configurations and re-use the existing configuration files as much as possible.

Most of the configuration files are stored in the /etc directory.

If Ubuntu is freshly installed, copy back the files and directories of the application software to the new system before re-installing the application software, e.g. the following files or directories:

  • /etc/aliases
  • /etc/aliases.db
  • /etc/anacrontab
  • /etc/apache2
  • /etc/ca-certificates
  • /etc/ca-certificates.conf
  • /etc/dovecot
  • /etc/hostname
  • /etc/hosts
  • /etc/openvpn
  • /etc/phpmyadmin
  • /etc/postfix
  • /etc/samba
  • /etc/vsftpd.conf
  • /etc/vsftpd.chroot_list

Do not copy back the /etc/fstab file because the file system configuration there is no longer applicable to the new system and copying back will cause the new system not re-bootable.

If copied back, use "Disk" software mentioned above to make some changes so that the /etc/fstab file reflects the latest configuration, before re-booting.

The information of the previous user names, groups and passwords are contained in the following files:

  • /etc/passwd
  • /etc/group
  • /etc/gshadow
  • /etc/shadow

Use text editor and spreadsheet software to pick out old information in the old files and not superseded by the new information and copy it cover to the new files to enable the users to access their previous data directories, otherwise, re-redefine all the user names and passwords one by one.

Re-install the additional application software after coping back as described above.

The software might have updated the default configuration files over time but usually would tolerate using the old configuration files. Therefore, it is preferred to keep using the old configuration files to ensure that the software is successfully re-installed before making changes. There may be very slight changes to new configuration files. The installation process will usually prompt to ask which file to keep and give an opportunity to see a comparison. It may be better to record the changes first and change later. Using the new configuration files in conjunction with the old physical configurations may cause the software not workable. Carefully check for any slight changes to the new configuration files if using the new configuration files to add the old configuration settings.

If re-installation is not successful after :

$ sudo apt install <name of software>

try:

$ sudo apt remove <name of software>

or even:

$ sudo apt purge <name of software>

or even remove the software directory before:

$ sudo apt install <name of software>

Usually, it is the configuration which is causing problems.

(The following added, 28 May 2019)

The "remove" option will leave behind the user modified configuration files.

Rename them.

Re-install the application software.

Use the new configuration file if see if the application software can run successfully.

If yes, copy the user modifications in the old configuration file to the new so far as compatible, and see whether the software application can run successfully.

The "purge" option will remove all configuration files and other related software. Care should be exercised to review that the related software would not be required by other software.

Removing the software directory is the extreme option with little additional effect.

Show messages when booting

(section added, 28 May 2019)

Execute:

$ sudo gedit /etc/default/grub

Specify:

#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"    (uncomment the default)
GRUB_CMDLINE_LINUX_DEFAULT=""                 (add)

Execute:

$ sudo update-grub

 

End of Page

Install backintime

Install backintime KCTang

Go to End

Note

31 Jan 2023: Updated.

12 May 2020: Updated.

30 May 2019: Created.

Intro

backintime enables incremental backup of files.

Listing it as the first application software to be installed after installing Ubuntu is an indication of the importance of backing up files.

Install

Execute:

$ sudo apt install backintime*
$ sudo apt install backintime-qt

(updated 12 May 2020)

Bring up

Select Activities.

Type "ba" to bring up some application icons:

Select Back in Time (root) to show the front page:

Alternatively, execute

$ sudo i backintime
or
$ pkexec backintime

However, the terminal commands are not reliable to bring up backintime. Using the graphical desktop is more reliable.

(updated 12 May 2020)

Config

Select the setting button:

Select the General page:

Specify:

  • where to save snapshots
  • host computer name, user name "root" to have widest permission, profile starting with 1
  • scheduled intervals and hours to save snapshots

Select the Include page > Add folder to select the folders to include in backup :

Add file can only include files not a whole folder.

Select Exclude page.

Generally accept the default exclusions.

Select Add folder to select sub-folders to be excluded from the backup when their parent folders have been included:

Select Auto-remove to set time criteria to automatically remove old spanshots:

Select Options page.

Generally accept the default settings.

Select Expert Options page.

Generally accept the default settings.

Select OK to save.

Exit the software. It will run at the specified times.

Take snapshot any time

Bring up the front page:

Select the take snapshot button: 

It would take some time to take a snapshot depending on the extent of file changes since the last snapshot.

Restore

Restore backup files when the present files have been lost or corrupted.

Bring up the front page:

 

Select:

  • the desired snapshot in the left window
  • the desired backup folder in the middle window
  • one or more folders or files in the right window

Select the restore button:

Select whether to backup local files with a trailing suffix before restoring the old files:

Select Yes to proceed if for sure.

Folders and files will be restored.

It would take some time depending on the size to be restored.

Let the process finish with the use permissions set back to the original state.

If only the folders in the middle window is selected before the restore button is selected, this message will appear to indicate that the "/" root folder will be restored:

It is important not to just select the folders in the middle window and select the restore button to proceed, because it will easily restore files to the "/" root directory and overwrite the still valid system and programme files in the root directory to cause problems.

Repair

Sometimes, backintime cannot be brought up to run, even after a removal and re-installation.

This can be caused by the removal of some other programmes which backintime depends on but which have been removed when some other application software is removed. Re-installation of backintime may not bring them back.

The following shows the programmes which backintime depends on:

Try to re-install the missing programmes:

$ sudo apt install <name of programme>

rsync, python3, openssh-client are likely missing programmes.

 

In case of the following error:

Traceback (most recent call last):
  File "/usr/share/backintime/common/backintime.py", line 28, in <module>
    import config
  File "/usr/share/backintime/common/config.py", line 45, in <module>
    import tools
  File "/usr/share/backintime/common/tools.py", line 37, in <module>
    from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'

 

Execute:

$ sudo apt install python3-packaging

(last 2 paragraphs added, 31 Jan 2023)

 

End of Page

 

Install Samba file server for Windows

Install Samba file server for Windows KCTang

Go to End

Note

26/2/2023: Trash folder added.

28/7/2022: Correct the positions of two statements. "gedit" changed to "nano" in case non gui interface is used.

7/5/2019: Slight adjustments. "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

25/12/2014: First created.

Intro

Samba file server enables specified directories to be accessible by Windows computers on the same network.

Install

Install the package​s:​

$ sudo apt update 
$ sudo apt install samba
$ sudo apt install samba-vfs-modules

samba-vfs-modules is used to enable Trash folder.

(vsf-modules added, 26/2/2023)

Define a workgroup

Edit the config file:

$ sudo nano /etc/samba/smb.conf

​​​​​Define workgroup name as "kctcl" in the "[global]" section:

workgroup = kctcl

Uncomment to restrict access to server users only:

security = user

Add the following if OpenVPN used:

hosts allow = 192.168.0. 10.8.0. 127.0.0.

(added. 7/5/2019):

Put the last two statements under the "[global]" section after "Networking" instead of at the end, otherwise "security = user" will apply to the last shared directory causing it to be non-assessible. 

(added, 28/7/2022)

Define directories to be shared

Add a section at the end to share directories:

[<sub-directory name or other short name>] 
pth = /<directory name>/<sub-directory name> 
browseable = yes 
guest ok = yes 
read only = no 
create mask = 0775
directory mask = 0775 
# do not include the next two lines if access is restricted to the owning user (added 7 May 2019)
force user = nobody 
force group = nogroup
# Enable Trash folder, KCTang 18/1/2023
vfs object = recycle
# Specify a folder relative to the path above
# Do not specify absolute path unless the path is outside the path above
# The folder will be created automatically upon first deletion
# Hidden folder (prefixed with '.') not used
# Use %U if want to record the user name, not used KCTang 26/2/2023
#recycle:repository = Trash/%U
recycle:repository = Trash
# Mode permits all users to delete
recycle:directory_mode = 0775
# Change last accessed time when moved to the Trash folder
recycle:touch = yes
# Keep modified time
recycle:touch_mtime = no
# Keep folder tree
recycle:keeptree = yes
# Files of the same name deleted will be kept with newer deleted file named as "Copy # of ..."
recycle:versions = yes
# Exclusions
recycle.exclude = *.tmp, ~*, thumb.db

(Trash folder added, 26/2/2023)

Create the directory to be shared, if not already existing:

$ sudo mkdir -p /<directory name>/<sub-directory name>

Change ownership of the directory:

$ sudo chown nobody:nogroup /<directory name>/<sub-directory name>

Start service

Start or restart Samba service whenever the config file is changed:

$ sudo systemctl restart smbd nmbd

or, if ".service" is not automatically appended when executing the above command:

$ sudo systemctl restart smbd.service nmbd.service

Define Crontab

Define crontab to delete Samba Trash files older than 30 days:

$ crontab -e

Edit to include:

#Delete Samba Trash files folder than 30 days
0 0 * * * /usr/bin/find /kctcl/Trash -type f -atime +30 -delete

(added, 26/2/2023)

 

End of Page

Install Postfix + Dovecot e-mail server

Install Postfix + Dovecot e-mail server KCTang

Go to End

Note

  • 13 Feb 2023: Use Letsencrypt ssl. Change gedit to micro as non-graphical editor.
  • 11 Feb 2023: Add full config files. Correct opendkim socket.
  • 1 Feb 2023: Correct typos.
  • 10 Oct 2022: Add DKIM setting.
  • 18 Jun 2021: Adjust TLS setting.
  • 5 Jan 2021: Increase imap-login process limit. List Dovecot full custom settings.
  • 9 Dec 2020: Define cron job to delete filtered mails.
  • 20 Sep 2020: Stop using mail-stack-delivery.
  • 8 May 2020: Correct typo errors.
  • 30 May 2019: Add anti-virus and spam mail filtering.
  • 17 May 2019: Add copying emails to external accounts.
  • 7 May 2019: Change "gksudo gedit" to "sudo gedit" as Ubuntu 18.04 dropped "gksudo". Add header_checks. Add auto creation of Trash folders. Add webmaster.
  • 29 Sep 2018: Add "body_checks" for spam control.
  • 20 Sep 2018: Increase message_size_limit to 20 times the default.
  • 5 Apr 2018: Increase message_size_limit to 10 times the default.
  • 25 Dec 2018: Publish on web.
  • 12 Apr 2014: Specify maximal_queue_lifetime to notify unsuccessful delivery immediately.
  • 2 Apr 2014: Specify fully qualified domain name.

Intro

Postfix is a mail transfer agent (MTA) responsible for sending out and receiving emails between servers.

Dovecot is a mail delivery agent (MDA) responsible for sending out and receiving emails between a server and its users.

Mail-stack-delivery was a combined package containing both Postfix and Dovecot. It is now no longer supported.

Re-direct to server ports​

Set the internet router to re-direct the following connections to server ports:

  • SMTP = port 25 (for receiving or sending emails)
  • secure SMTP = port 465 (for receiving or sending emails securely)
  • IMAP = port 143 (for retrieving emails)
  • secure IMAP = port 993 (for retrieving emails securely)
  • POP3 = port 110 (for retrieving emails)

Install both Postfix and Dovecot​

Ubuntu 20.04 does not support the combined package "mail-stack-delivery" anymore.

Install Dovecot and Postfix individually:

$ sudo apt install dovecot-imapd dovecot-pop3d
$ sudo apt install postfix

However, the previous config file "/etc/dovecot/conf.d/99-mail-stack-delivery.conf" is still retained for use, because "99" represents the last and overriding config file. This eliminates the need to change the individual config files.

(revised to install individually, 20/9/2020)

Reconfigure Postfix

Reconfig:

$ sudo dpkg-reconfigure postfix

Use Tab key to change selection.

Select "Internet Site".

Enter the following information:

System mail name: <fully qualified domain name, such as "kctang.com.hk">
Root and postmaster mail recipient: <such as "kctang">
Other destinations to accept mail: <fully qualified domain name, such as "kctang.com.hk">, <server name such as "server">, localhost.localdomain, localhost
Force synchronous updates on mail queue: No
Local networks: <leave it blank to accept the default>
Use procmail for local delivery: No
Mailbox size limit (bytes): 0
Local address extension character: +
Internet protocols to use: all

Edit "main.cf" settings:

$ sudo micro /etc/postfix/main.cf

Specify in full:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on
# fresh installs.
compatibility_level = 2
# the following automatically set by dpkg-reconfigure postfix
myhostname = kctang.com.hk
# fully qualified domain name instead of server name used,
# otherwise some servers would not accept e-mails sent without fully qualified domain name, 2/4/2014
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = kctang.com.hk, server3, localhost.localdomain, localhost
relayhost = 
mynetworks = 127.0.0.1/32 10.8.0.1/32 [::1]/128
# something similar to the above line
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
# ALL the following added to default
inet_protocols = all
home_mailbox = Maildir/
mailbox_command = /usr/lib/dovecot/deliver -c /etc/dovecot/dovecot.conf -m "${EXTENSION}"
message_size_limit = 204800000
# last line added to increase the default to 10 times, KCTang 5/4/2014
# increased to 20 times, KCTang 20/9/2018
maximal_queue_lifetime = 0
# last line added to report unsuccessful delivery immediately instead of after the default of 5 days, KCTang 12/4/2014
bounce_queue_lifetime = 0
# last line added, this should not be bigger than maximal_queue_lifetime, KCTang 20/5/2019
# smtpd setting
#smtpd_proxy_timeout = 240s
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/dovecot-auth
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_security_options = noanonymous,noplaintext
# noplaintext in last line added to prevent unencrypted credentials, KCTang 13/2/2023
smtpd_sasl_tls_security_options = noanonymous
# last line added, KCTang 13/2/2023
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_recipient_restrictions = 
    reject_unknown_sender_domain
    reject_unknown_recipient_domain 
    reject_unauth_pipelining
    permit_mynetworks
    permit_sasl_authenticated 
    reject_unauth_destination
    check_policy_service unix:private/policyd-spf
# last line added to enable spf, KCTang 19/2/2022
smtpd_relay_restrictions = 
    permit_mynetworks
    permit_sasl_authenticated
    defer_unauth_destination
smtpd_sender_restrictions = reject_unknown_sender_domain
# TLS parameters
smtp_use_tls = yes
smtp_tls_security_level=may
smtpd_tls_security_level=may
smtp_tls_note_starttls_offer=yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
# use the following two as provided as default
#smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
#smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
# use the following two if self define key and certificate
#smtpd_tls_key_file=/etc/ssl/private/server.key
#smtpd_tls_cert_file=/etc/ssl/certs/server.crt
# use the following two if using letsencrypt, KCTang 13/2/2023
smtpd_tls_cert_file=/etc/letsencrypt/live/kctang.com.hk/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/kctang.com.hk/privkey.pem
# use the following if using self certification authority
#smtp_tls_CApath=/etc/ssl/certs/
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_tls_auth_only = yes
smtpd_tls_mandatory_ciphers = medium
# smtpd_tls_mandatory_protocols = SSLv3, TLSv1 # commented, replaced below, KCTang 18/6/2021
# next 2 lines added to enhance security, KCTang 1/7/2021
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
tls_random_source = dev:/dev/urandom
#header_checks = regexp:/etc/postfix/header_checks
# last line added to check headers, KCTang 30/9/2018, disabled after using amavis 1/6/2019
#body_checks = regexp:/etc/postfix/body_checks
# last line added to refer to another file to check contents of email bodies, KCTang 29/9/2018, disabled after using amavis 1/6/2019
# virtual_alias_domains = kctang.com.hk
# last line added on 17/5/2019, but disabled since same domain name already defined as destination above, KCTang 20/5/2019
virtual_alias_maps = hash:/etc/postfix/virtual
# last line added to forward emails to another server, KCTang 17/5/2019
content_filter = smtp-amavis:[127.0.0.1]:10024
# last line added for amavis, KCTang 1/6/2019
# the following line added to enable spf, KCTang 19/2/2022
policyd-spf_time_limit = 3600
# Milter configuration, added but disabled, KCTang 2/6/2019, enabled again, KCTang 19/2/2022
milter_default_action = accept
milter_protocol = 6
smtpd_milters = local:opendkim/opendkim.sock
non_smtpd_milters = $smtpd_milters

(full file given, 11/2/2023)

(updated, 13/2/2023)

Edit "master.cf" settings:

$ sudo micro /etc/postfix/master.cf

Specify​ in full:

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd
#smtp      inet  n       -       y       -       1       postscreen
#smtpd     pass  -       -       y       -       -       smtpd
#dnsblog   unix  -       -       y       -       0       dnsblog
#tlsproxy  unix  -       -       y       -       0       tlsproxy
#submission inet n       -       y       -       -       smtpd
#  -o syslog_name=postfix/submission
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_tls_auth_only=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
# last line uncommented, KCTang 2/4/2014
#  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
# last line added to force use of TLS, KCTang 2/4/2014
  -o smtpd_sasl_auth_enable=yes
# last line uncommented to enable STARTTLS authentication, KCTang 2/4/2014
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# last line uncommented to reject if not authenticated, KCTang 2/4/2014
  -o milter_macro_daemon_name=ORIGINATING
# last line uncommented, KCTang 2/4/2014
#628       inet  n       -       y       -       -       qmqpd
pickup    unix  n       -       y       60      1       pickup
  -o content_filter=
  -o receive_override_options=no_header_body_checks
# last two lines added for anti-spam, KCTang 29/5/2019
cleanup   unix  n       -       y       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       y       1000?   1       tlsmgr
rewrite   unix  -       -       y       -       -       trivial-rewrite
bounce    unix  -       -       y       -       0       bounce
defer     unix  -       -       y       -       0       bounce
trace     unix  -       -       y       -       0       bounce
verify    unix  -       -       y       -       1       verify
flush     unix  n       -       y       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       y       -       -       smtp
relay     unix  -       -       y       -       -       smtp
        -o syslog_name=postfix/$service_name
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
     -o smtp_connect_timeout=60s
# last line added, KCTang 10/3/2021
showq     unix  n       -       y       -       -       showq
error     unix  -       -       y       -       -       error
retry     unix  -       -       y       -       -       error
discard   unix  -       -       y       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       y       -       -       lmtp
anvil     unix  -       -       y       -       1       anvil
scache    unix  -       -       y       -       1       scache
postlog   unix-dgram n  -       n       -       1       postlogd
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix    -    n    n    -    2    pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}
# the following added, KCTang 29/5/2019
smtp-amavis     unix    -       -       -       -       2       smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
    
# the following added for spf, KCTang 1/6/2019, 
# then disabled as considered not necessary
# then re-enabled 19/2/2022 to overcome rejection by gmail
policyd-spf   unix  -       n       n       -       0       spawn
    user=policyd-spf argv=/usr/bin/policyd-spf

(full file given, 11/2/2023)

Create "body_checks" file, if required:

$ sudo micro /etc/postfix/body_checks

Specify​ one or more lines of texts within //:

/unique text contained in email you do not want to receive/ DISCARD

"DISCARD" means delete from the server.

Create "header_checks" file, if required:

$ sudo micro /etc/postfix/header_checks

Specify similarly.

(header checks added, 7/5/2019)

Change Dovecot settings

Edit config file:

$ sudo micro /etc/dovecot/conf.d/99-mail-stack-delivery.conf

Note that a number of default config files are contained in sub-directory "conf.d". To override them, create a config file beginning with "99" so that it is read the latest to override the others. The name is based on the previous "mail-stack-delivery" config file, but can be any.

Specify in full:

# Some general options
# Installed protocols are now auto-included by /etc/dovecot/dovecot.conf
# Since mail-stack-delivery depends on them it is more flexible to not
# explicitly list them here, but achieves the same.
# protocols = imap pop3 sieve
disable_plaintext_auth = yes
# Since 18.04 basic SSL enablement is set up by dovecot-core and configured
# in /etc/dovecot/conf.d/10-ssl.conf.
# So by default basic enablement is no more done here. The old section is kept
# as comment for reference to the old defaults.
#
# ssl = yes
# ssl_cert = </etc/dovecot/dovecot.pem
# ssl_key = </etc/dovecot/private/dovecot.pem
#
# If you keep a formerly used custom SSL enablement in this file it will (as
# before) continue to overwrite the new defaults in 10-ssl.conf as this file is
# sorted later being 99-*.conf
#
# If you choose to take the new defaults (no ssl config in this file) please
# make sure you have also chosen the package defaults for 10-ssl.conf (to enable
# it there) when dovecot-core configures. Also check that the links for cert/key
# set up there got created correctly (they would not be created if they conflict with your
# old keys done by mail-stack-delivery).
#
# use letsencrypt ssl, KCTang 13/2/2023 
ssl = yes
ssl_cert = </etc/letsencrypt/live/kctang.com.hk/fullchain.pem
ssl_key = </etc/letsencrypt/live/kctang.com.hk/privkey.pem
ssl_client_ca_dir =
#ssl_protocols = !SSLv2 !SSLv3
mail_location = maildir:~/Maildir:LAYOUT=fs
# LAYOUT=fs added to last line, to use "/" instead of "." to denote sub-folders, KCTang 25/5/2019
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
# IMAP configuration
protocol imap {
        mail_max_userip_connections = 1000
        # 10 in last line increased to 1000, KCTang 25/5/2019
        imap_client_workarounds = delay-newmail
}
# POP3 configuration
protocol pop3 {
        mail_max_userip_connections = 50
        # 10 in last line increased to 50, KCTang 25/5/2019 
        pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
}
# LDA configuration
protocol lda {
        postmaster_address = postmaster
        mail_plugins = sieve
        quota_full_tempfail = yes
        deliver_log_format = msgid=%m: %$
        rejection_reason = Your message to <%t> was automatically rejected:%n%r
}
# Plugins configuration
plugin {
        sieve=~/.dovecot.sieve
        sieve_dir=~/sieve
}
# Authentication configuration
auth_mechanisms = plain login
service auth {
  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/dovecot-auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}
# The following are additional to those in 15-mailboxes.conf.
# They are to auto create Trash folders.
# Trash folders would not be backed up with back-in-time,
# and therefore would need to be re-created after email recovery is done from back-in-time.
namespace inbox {
  mailbox Trash {
    auto = subscribe
  }
}
# process_limit increased, KCTang 4/1/2021
service imap-login {
  process_limit = 200
}
# The following added to resolve stats-writer failure, KCTang 3/6/2021
service stats {
  unix_listener stats-reader {
    user = root
    group = root
    mode = 0660
  }
  unix_listener stats-writer {
    user = root
    group = dovecot
    mode = 0660
  }
}

(full file given, 11/2/2023)

(updated, 13/2/2023)

Activate the changes:

$ sudo systemctl reload postfix
or
$ sudo service postfix reload
and
$ sudo systemctl restart dovecot
or
$ sudo service dovecot reload

Verify success

See whether the Postfix server is running:

$ telnet localhost 25

should display:

220 kctang.com.hk ESMTP Postfix (Ubuntu)

ehlo localhost

should display the following:

250-kctang.com.hk

250-PIPELINING

250-SIZE 102400000

250-VRFY

250-ETRN

250-STARTTLS

250-ENHANCEDSTATUSCODES

250-8BITMIME

250 DSN

Ctrl-]

to exit to "telnet >" prompt.

quit

to exit telnet.

Try also:

$ telnet localhost 993

("993", not "995", 1/2/2023)

similarly:

$ telnet localhost 465

should display either one:

Connected to localhost

Connected to kctang.com.hk

"Ctrl-]"

to exit to "telnet >" prompt.

quit

to exit telnet.

Specify internal email forwarding

Edit "aliases" file:

$ sudo micro /etc/aliases

Specify​:

postmaster: kctang
webmaster: kctang
kctcl: kctcl, kctclpop

meaning:

  • forwarding e-mails sent to postmaster@localhost and webmaster@localhost to kctang@localhost, no email will be left at postmaster or webmaster

(webmaster added, 7/5/2019)

  • forwarding e-mails sent to kctcl@localhost to kctcl@localhost (itself) and to kctclpop@localhst, i.e. making a copy

"localhost" means "kctang.com.hk" in our case. For the email user name before "@", there is no need to create a file system user account for it if all emails addressed to it are forwarded elsewhere. The name serves as an alias only of the email account forwarded to.

(added, 17/5/2019)

 

Activate setting everytime the "aliases" file has been changed:

$ sudo newaliases

 

Copy to external email accounts

(section added, 17/5/2019)

Execute:

$ sudo micro /etc/postfix/main.cf

Specify​ at the end of the file (already shown above):

virtual_alias_domains = kctang.com.hk
virtual_alias_maps = hash:/etc/postfix/virtual

Execute:

$ sudo micro /etc/postfix/virtual

Specify​ to make a copy to itself and a copy to the external email account:

# from               to one or more addresses, separated by a space 
kctang@kctang.com.hk kctang@kctang.com.hk name1@external.account.name
kctcl@kctang.com.hk kctcl@kctang.com.hk name2@external.account.name

Omit making a copy to itself if only email forwarding is required.

Save and exit.

Execute after the "virtual" file is created or changed:

$ sudo postmap /etc/postfix/virtual

Execute:

$ sudo systemctl restart postfix

Test by sending emails.

Install anti-virus Clamav-Daemon

(section added, 30/5/2019)

Install clamav-daemon:

$ sudo apt update
$ sudo apt install clamav-daemon            (clamav-freshclam also automatically installed)
$ sudo dpkg-reconfigure clamav-daemon   (must be done, generally accept all defaults, set yes to scan emails)
$ sudo systemctl start clamav-freshclam
$ sudo systemctl start clamav-daemon
$ sudo systemctl status clamav-freshclam    (check if running)
$ sudo systemctl status clamav-daemon       (check if running, OK if reported "/bin/mkdir /run/clamav (code=exited, status=1/FAILURE" because directory already created)
$ sudo tail -f /var/log/clamav/clamav.log        (see running progress, Ctrl-Z to exit)

(typo corrected, 8/5/2020)

(use sudo, 11/2/2023)

Filter spam mails

(section added, 30/5/2019)

Execute to install various software:

$ sudo apt update
$ sudo apt install amavisd-new spamassassin
$ sudo apt install postfix-policyd-spf-python                                 (optionally required for spf) 
$ sudo apt install opendkim                                                   (optionally required for opendkim)
$ sudo apt install pyzor razor                                                (optional extras)
$ sudo apt install arj cabextract cpio lhasa nomarch pax rar unrar unzip zip  ("lhasa", not "lha")

Configure ClamAV:

$ sudo adduser clamav amavis
$ sudo adduser amavis clamav

Configure SpamAssassin:

$ sudo micro /etc/default/spamassassin

Change "ENABLED=0" to:

ENABLED=1

(no longer set here, 8/5/2020)

Start the service:

$ sudo systemctl start spamassassin.service

Configure Amavisd-new:

$ sudo micro /etc/amavis/conf.d/15-content_filter_mode

Specify:

use strict;
# Uncomment the two lines below to enable antivirus checking mode
@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
# Uncomment the two lines below to enable SPAM checking mode
@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
1;  # insure a defined return

Execute:

$ sudo micro /etc/amavis/conf.d/20-debian_defaults

Change "D_PASS" to:

$final_spam_destiny = D_DISCARD

(deleted, 8/5/2020, reinstated, 26/7/2021)

Adjust the following values only if desired to flag more messages as spam:

$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 6.31; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.31; # triggers spam evasive actions
$sa_dsn_cutoff_level = 10; # spam level beyond which a DSN is not sent

Execute:

$ sudo micro /etc/amavis/conf.d/50-user

Specify:

$myhostname = 'kctang.com.hk';

Re-start the service:

$ sudo systemctl restart amavis.service

Edit the following file to specify domains to be whitelisted if necessary:

$ sudo micro /etc/amavis/conf.d/40-policy_banks

Test that the Amavisd-new SMTP is listening:

telnet localhost 10024
Trying 127.0.0.1...
Connected to kctang.com.hk.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready

Press Ctrl-] and enter "quit" to exit.

Configure Postfix master.cf:

$ sudo micro /etc/postfix/master.cf

Add the following to the end of the file (already done above):

smtp-amavis     unix    -       -       -       -       2       smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters

Add the following immediately below the "pickup" transport service (already done above):

     -o content_filter=
     -o receive_override_options=no_header_body_checks

Configure Postfix main.cf:

$ sudo micro /etc/postfix/main.cf

Add the following to the end of the file (already done above):

content_filter = smtp-amavis:[127.0.0.1]:10024

Comment it with "#" at the start of the line if desired to stop using amavis.

Restart service:

$ sudo systemctl restart postfix

Check the hidden header of email received after the above to see the presence of one or more of the following:

X-Spam-Level: 
X-Virus-Scanned: Debian amavisd-new at kctang.com.hk
X-Spam-Status:
X-Spam-Level: 

If present, the spam filter is working.

See https://help.ubuntu.com/lts/serverguide/mail-filtering.html.en for a full explanation of the above.

Delete filtered mails

(section added, 9/12/2020)

Filtered files are stored in /var/lib/amavis/virusmails and should be deleted regularly.

Edit to define a scheduled job to delete filtered files:

$ sudo crontab -e -u amavis 

where:

-e = edit

-u amavis = user amavis

Define as follows:

0 0 * * * find /var/lib/amavis/virusmails/ -type f -mtime +15 -delete

which means at

0 = 0 minute

0 = 0 hour

* = every day of month

* = every month

* = every day of week

find /var/lib/amavis/virusmails = find in that directory

-type f = regular files

-mtime +15 = file modified more than 15 days ago

-delete = delete the file

Set up DKIM

(implemented, 19/2/2022

section added, 10/10/2022)

Set up DomainKeys Identified Mail (DKIM) to prevent email phishing by others by putting a digital key in the outgoing email headers so that the receiving email server can verify the digital key by looking up the public key in the DNS record of the sender domain.

Install OpenDKIM:

$ sudo apt install opendkim opendkim-tools

Add postfix user to opendkim group:

$ sudo gpasswd -a postfix opendkim

Configure:

$ sudo micro /etc/opendkim.conf

Specify:

# keep to log in /var/log/mail.log
Syslog             yes
SyslogSuccess      yes
# change to "yes" if desired to log more details
Logwhy             no
# keep
Canonicalization   relaxed/simple
# uncomment
Mode               sv
SubDomains         no
# keep
OversignHeaders    from
# add
AutoRestart         yes
AutoRestartRate     10/1M
Background          yes
DNSTimeout          5
SignatureAlgorithm  rsa-sha256
# keep
UserID              opendkim
UMask               007
# comment
#Socket             local:/run/opendkim/opendkim.sock
# add
Socket              local:/var/spool/postfix/opendkim/opendkim.sock
# keep
PidFile             /run/opendkim/opendkim.pid
TrustAnchorFile     /usr/share/dns/root.key
# keep
KeyTable            refile:/etc/opendkim/key.table
SigningTable        refile:/etc/opendkim/signing.table
ExternalIgnoreList  /etc/opendkim/trusted.hosts
InternalHosts       /etc/opendkim/trusted.hosts

Create a directory structure:

$ sudo mkdir /etc/opendkim
$ sudo mkdir /etc/opendkim/keys
$ sudo chown -R opendkim:opendkim /etc/opendkim
$ sudo chmod go-rw /etc/opendkim/keys

Create the signing table:

$ sudo micro /etc/opendkim/signing.table

Specify:

*@kctang.com.hk        default._domainkey.kctang.com.hk
# if sub-domain used
*@*.kctang.com.hk      default._domainkey.kctang.com.hk    

Create the key table:

$ sudo micro /etc/opendkim/key.table

Specify:

default._domainkey.kctang.com.hk     kctang.com.hk:default:/etc/opendkim/keys/kctang.com.hk/default.private

Create the trusted hosts file:

$ sudo micro /etc/opendkim/trusted.hosts

Specify to sign but not to verify emails from localhost or the following domain:

127.0.0.1
localhost
*.kctang.com.hk

("*" added, 1/2/2023)

Create separate folder for the domain and generate Private/Public Keypair:

$ sudo mkdir /etc/opendkim/keys/kctang.com.hk
$ sudo opendkim-genkey -b 1024 -d kctang.com.hk -D /etc/opendkim/keys/kctang.com.hk -s default -v

Use -b 2048 instead of -b 1024 for bits of key if the domain name server permits.
-d for domain name

-D for directory to store the keys

-s for selector

The private key will be written to default.private file and the public key will be written to default.txt file.

Change owner and permission:

$ sudo chown opendkim:opendkim /etc/opendkim/keys/kctang.com.hk/default.private
$ sudo chmod 600 /etc/opendkim/keys/kctang.com.hk/default.private

Display the public key:

$ sudo cat /etc/opendkim/keys/kctang.com.hk/default.txt

The string after the p parameter is the public key:

default._domainkey      IN      TXT     ( "v=DKIM1; h=sha256; k=rsa; "
          "p=..." )  ; ----- DKIM key default for kctang.com.hk

Define in the Domain Name Record on the domain name host:

Name = default._domainkey

Address or value = the text in ( ) above, but deleting double quotes and spaces

Type = SPF (txt)

Test DKIM Key:

$ sudo opendkim-testkey -d kctang.com.hk -s default -vvv

Successful if an opendkim-testkey output is key OK.

No problem if an opendkim-testkey output is key not secure because DNSSEC may not have been enabled on the domain name.

The DKIM record may take up to 24 hours to propagate to the Internet.

Can also go to https://www.dmarcanalyzer.com/dkim/dkim-check/, enter "kctang.com.hk" as the domain name and "default" as the selector to check DKIM record propagation.

In case of the query timed out error, comment out the following line in /etc/opendkim.conf file and restart opendkim.service.

TrustAnchorFile       /usr/share/dns/root.key

Create a directory to hold the OpenDKIM socket file, and allow only opendkim user and postfix group to access it:

$ sudo mkdir /var/spool/postfix/opendkim
$ sudo chown opendkim:postfix /var/spool/postfix/opendkim

Edit:

$ sudo micro /etc/default/opendkim 

Comment out:

# SOCKET=local:$RUNDIR/opendkim.sock

Add to suit Ubuntu:

SOCKET=local:/var/spool/postfix/opendkim/opendkim.sock

Create the file

$ sudo mkdir /var/spool/postfix/opendkim 
$ sudo chown -R opendkim:opendkim /var/spool/postfix/opendkim

(file creation added, 11/2/2023)

Edit Postfix main.cf to configure Milter as shown above.

Restart opendkim and postfix service and check status to see any errors:

$ sudo systemctl restart opendkim postfix
$ sudo systemctl status opendkim postfix

(status check added, 11/2/2023)

Set up Domain Name record

Set up the Domain Name record at the domain hosting company as follows:

v=DMARC1; p=quarantine; pct=10
v=spf1 ip4:202.69.68.114 include:_spf.google.com include:spf.protection.outlook.com ~all

The ip4 is our company's IP address. This is not a secret.

(added, 21/2/2023)

 

End of Page

Install Ftp server

Install Ftp server KCTang

Go to End

Note

26 Apr 2022: "gedit" changed to "nano". Minor error corrected.

5 Sep 2019: "0755" changed to "0775" for "Ftp" directory.

7 May 2019: "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

25 Dec 2014: Created.

Intro

FTP server enables directories to be accessible for downloading or uploading by users outside the local network.

Install

Install the package:

$ sudo apt-get install vsftpd

Edit config file:

$ sudo nano /etc/vsftpd.conf

Uncomment the following line to enable uploading:

write_enable=YES

Define as the following line to change the default directory permissions to 775 (drwxrwxr-x) and default file permissions to 664 (-rw-rw-r--):

local_umask=002

Uncomment the following lines to restrict users to their home except for those listed in the file represented by "chroot_list_file":

chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd.chroot_list

("vsftpd.choot_list" corrected as "vsftpd.chroot_list", 7 May 2019)

Save file after uncommenting.

Specify users who can go outside their home by inserting their user login names one on each line in the file represented by "chroot_list_file":

$ sudo nano /etc/vsftpd.chroot_list

("vsftpd/chroot_list" corrected as "vsftpd.chroot_list", 7 May 2019)

Restart ftp service whenever the config files are changed:

$ sudo systemctl restart vsftpd
or
$ sudo service vsftpd restart

Set the internet router to re-direct ftp connections to server port 21.

Set up a root FTP Directory to contain all FTP job folders

Change directory to the top directory assessible for use by Windows network through Samba:

$ cd /<full directory path from root>​

Make a directory specially for FTP storage, called "Ftp" in this example:

$ sudo mkdir Ftp

Change its ownership so that it can be accessed by Windows network:

$ sudo chown nobody:nogroup Ftp

Change its permissions to "read only" for other users:

$ sudo chmod 0775 Ftp

("0755" changed to "0775" because for unknown reasons sub-directory cannot be created under "Ftp", 5 Sep 2019)

Check setting:

$ ls -ls

should show "drwxrwxr-x" and "nobody nogroup" against the "Ftp" item.

(drwxr-xr-x corrected as drwxrwxr-x, 26 Apr 2022)

Create a ftp user for specific job

Create a new user with authority to download and upload the job ftp directory:

$ sudo adduser <ftp user name>

Change the new user's root directory from /home/<ftp user name> to the job ftp directory:

​$ sudo usermod -d /<full directory path from root>/Ftp/<job name> <ftp user name>
  • ​<ftp user name> and <job name> can be the same or different
  • <job name> will become ftp users' root directory, they will be restricted to see only files at or below the root directory, they will not see the name of <job name> or the directory structure outside the root directory
  • ​Instead of <job name>, a further sub-directory such as <job name>/<sub job name> may be defined as the root​ directory
  • ​The directory /<full directory path from root>/Ftp/<job name> will still exist but not be used for ftp

Set up a ftp directory for specific job for downloading

Create a job ftp directory under the Ftp directory:

  • using Windows Explorer:
​\\<server name>\<full folder path from server>\Ftp\<job name>
  • or at the server terminal:
$ cd /<full directory path from root>​/Ftp
$ sudo mkdir <job name>
$ sudo chown nobody:nogroup <job name>
$ ls -ls

should show "drwxr-xr-x" or "drwxrwxr-x" and "nobody nogroup" against the <job name> item.

Further sub-directories may be created similarly for downloading purposes.

Set up a ftp directory for specific job for uploading

Create an "upload" sub-directory under the job ftp directory:

  • using Windows Explorer:
​\\<server name>\<full folder path from server>\Ftp\<job name>\upload
  • or at the server terminal:
$ cd /<full directory path from root>​/Ftp/<job name>
$ sudo mkdir upload
$ sudo chown nobody:nogroup upload

Change its permissions on the server to enable "write" for all:

$ cd /<full directory path from root>/Ftp/<job name>
$ sudo chmod a+w upload

Check settings:

$ ls -ls

should show "drwxrwxrwx" and "nobody nogroup" against the "upload" item.

Upload or download

​Internally, use Windows file explorer to copy or move files between the Windows networked computers to the ftp directories:

  • ​​copy files to ​\\< server name>\<full folder path from server >\Ftp\<job name> for downloading
  • copy files from \\<server name>\<full folder path from server >\Ftp\<job name>\upload after uploading by others

Externally, inform external users the ftp user login name i.e. <ftp user name> and password for downloading or uploading.

End of Page

Install Apache2 web server

Install Apache2 web server KCTang

Go to End

Note

30 Jul 2020: Updated to use Python3.

6 Jan 2020: Updated.

8 May 2019: More explanation on default configuration given. "gksudo gedit" changed to "sudo gedit" as Ubuntu 18.04 dropped "gksudo".

11 Apr 2018:  "apache2" changed to "apache2.service" when used in conjunction with systemctl.

25 Dec 2014: First created.

Intro

Apache2 web server provides web page services.

Prepare

Define hosts:

$ sudo gedit /etc/hosts

Specify:

127.0.0.1 kctang.com.hk <computer name> localhost
127.0.1.1 <computer name>

Define hostname:

$ sudo gedit /etc/hostname

Specify a line to contain:

<computer name>

Install

Install:

$ sudo apt install apache2

or before Ubuntu 16.04:

$ sudo apt-get install apache2

Start service:

$ sudo systemctl start apache2.service

or:

$ sudo service apache2 start

Set the internet router to re-direct http connections to server port 80.

(The following added on 8 May 2019)

Edit the enabled configuration file:

$ sudo gedit /etc/apache2/sites-enabled/000-default.conf

("ls -ls" changed to "gedit", 6 Jan 2020)

The file is symbolic linked to the actual location at /etc/apache2/sites-available/000-default.conf.

The file includes the following:

#ServerName www.example.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html

Leave the ServerName to the HTTPS setting below.

Change the ServerAdmin email address to the correct address, or add that address when setting up the email server.

The DocumentRoot tells that the website directories and files will be stored under /var/www/html.

Note that previously the DocumentRoot was at /var/www. The change from /var/www/html would affect the installation of Drupal as explaned on that web page.

Serf to http://www.kctang.com.hk or http://kctang.com.hk on web browser, the following page (var/www/html/index.html) will be displayed to indicate successful installation:

(end of add)

Configure to use HTTPS

This is optional. Starting to use on 6 April 2018.

When the Apache2 server is configured to use HTTPS, and when "https://" is used as the prefix to the website address URL (Uniform Resource Locator) in the web browser navigation bar, encrypted communications will be used with the Apache2 server. This will enhance security.

To do this, enable the mod_ssl module:

$ sudo a2enmod ssl

In order for Apache2 to use HTTPS service, a certificate and a key file are needed. Use EFF's Certbot to automatically deploy Let's Encrypt certificates and enable HTTPS.

Certbot is downloadable at:

https://certbot.eff.org/

No need to download from there.

(revised 8 May 2019)

Install certbot and configure Apache2:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install python3-certbot-apache
$ sudo certbot --apache
$ sudo certbot renew

(python changed to python3, 30 Jul 2020)

When answering questions

  • enter "kctang.com.hk" for name to activate HTTPS
  • decide whether to re-install certificate or renew and replace
  • select to redirect HTTP traffic to HTTPS.

(added on 8 May 2019).

The following will happen:

  • /etc/letsencrypt directory created to contain certificate obtained from Let's Encrypt.
  • A file /etc/apache2/sites-available/000-default-le-ssl.conf added and enabled.

(revised 8 May 2019)

  • The following lines inserted in /etc/apache2/sites-available/000-default.conf to enforce the use of "https://":
RewriteEngine on
RewriteCond %{SERVER_NAME} =kctang.com.hk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
  • A cron job created to renew the certificate which lasts for 90 days before expiry.

Enable new module and disable default module if not already automatically done:

$ sudo a2ensite 000-default-le-ssl
$ sudo a2dissite default-ssl
$ sudo systemctl restart apache2.service

 

End of Page

Install MySQL server + PHP + phpMyAdmin

Install MySQL server + PHP + phpMyAdmin KCTang

Go to End

Note

  • 7 Nov 2023: Changing transaction_isolation.
  • 8 Jan 2020: Configuring MySQL-server updated. Removing completely MySQL added.
  • 6 Jan 2020: Updated.
  • 8 May 2019: References to PHP5 deleted. Secure installation added. Root user authentication added. automysqlbackup added.
  • 25 Dec 2014: First created.

Intro

MySQL is a database server.

PHP (PHP5 before Ubuntu 16.04) is a web page programming language.

phpMyAdmin is a web interface to administer the MySQL server.

Install MySQL-server

Execute:

$ sudo apt update
$ sudo apt install mysql-server
$ sudo systemctl start mysql.service
$ sudo systemctl status mysql.service

Config MySQL-server

(section added, 8 May 2019)

Execute:

$ sudo mysql_secure_installation

Enter new root password for the first time or the existing root password.

When answering the secure installation questions:

  • validate password component - yes
  • set password validation policy - 2 for MEDIUM
  • set password strength - high
  • change the password for root - yes or no as appropriate
  • set new password - as appropriate
  • re-enter new password -
  • continue with the password - yes or no as appropriate
  • remove anonymous users - yes
  • disallow root login remotely - no
  • remove test database - yes
  • reload privilege table now - yes

(steps updated, 8 Jan 2022)

(suggested answers added, 6 Jan 2020)

Adjust authentication for use with phpMyAdmin

(section added, 8 May 2019)

If not done, the phpMyAdmin menu would not provide a choice to add users.

Execute:

$ sudo mysql -u root -p

enter the root user's password when prompted.

Execute:

mysql> SELECT user,plugin FROM mysql.user;

It will show that the root user's authentication plugin is "auth_socket".

Execute:

mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<root user password>';
mysql> FLUSH PRIVILEGES;
mysql> SELECT user,plugin FROM mysql.user;

(semi colon added after "PRIVILEGES", 6 Jan 2020)

It should show that the root user's authentication plugin has been changed to "mysql_nature_password".

Install PHP and related apache2 module

Execute:

$ sudo apt install php libapache2-mod-php

Install phpMyAdmin

Execute:

$ sudo apt install phpmyadmin

Set its own password.

Choose whether to keep the existing database whenever phpmyadmin is removed and re-installed.

Enable Apache2 config:

$ sudo ln -s /etc/phpmyadmin/apache.conf /etc/apache2/conf-available/phpmyadmin.conf
$ sudo a2enconf phpmyadmin.conf

Whenever Apache2 has been purged and re-installed, this enabling must be done again.

Restart Apache2 service:

Execute:

$ sudo systemctl restart apache2.service

Create a database user and a database

Execute:

 http://localhost/phpmyadmin
  or
 http://www.kctang.com.hk/phpmyadmin

Enter login name and password.

Create a user called <name, e.g. Drupal8> together with a database also called <name>:

  • click "Users" at the top menu bar
  • click "Add user" at the page middle
  • enter <name> at the User name entry
  • select "Local" at the Host entry
  • enter and re-type the password
  • click "Create database with same name and grant all privileges"
  • click "Go" at the bottom

Install automysqlbackup

(section added, 8 May 2019)

The software will backup mySQL databases daily, weekly and monthly.

Execute to install:

$ sudo apt install automysqlbackup

Edit the configuration file:

$ sudo gedit /etc/default/automysqlbackup

Specify:

BACKUPDIR="/var/lib/automysqlbackup" as the backup directory

MAILCONTENT="log" to send log email

MAILADDR="root" to send the email to root user, which has been set under Postfix Aliases to re-direct to the appropriate user.

Execute to run for the first time:

$ sudo automysqlbackup

Inspect daily backups:

$ sudo ls /var/lib/automysqlbackup/daily

Set transaction_isolation to READ COMMITTED

(section added, 6 Nov 2023)

To suit Drupal 10, change the transaction_isolation from "REPEATABLE READ" to "READ COMMITTED".

Execute:

$ sudo mysql -u root -p

enter the root user's password when prompted.

Execute:

mysql> SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
mysql> SELECT @@GLOBAL.transaction_isolation;

The first command is to set. The second command is to verify.

Remove mySQL completely

(section added, 8 Jan 2022)

sudo systemctl stop mysql.service
sudo apt purge mysql*
sudo apt autoremove                    (optional)
sudo apt autoclean                     (optional)
sudo apt remove dbconfig-mysql
sudo rm -r /var/lib/mysql
sudo rm -r /log/mysql
sudo rm -r /etc/mysql
sudo deluser mysql

End of Page

Install Drupal 10 content management system

Install Drupal 10 content management system KCTang

Go to End

Note

  • 24/1/2024: "php/8.3" used.
  • 8/11/2023: TOP API and Convert Bundles installations added. Adding buttons to CKEditor revised.
  • 5/11/2023: Upgrading to Drupal 10 added. Drupal 9 and before deleted.
  • 21/12/2021: Bartik theme used instead of Mayo.
  • 11/11/2021: Errors in updating core manually corrected.
  • 5/1/2020: Problems when updating to Drupal 8.8 under php7.4 described. Some text updating.
  • 9/9/2019: Composer files also copied when relocating system.
  • 18/7/2019: Composer configuration actions added.
  • 7/7/2019: Original user-defined files and attributes kept when updating manually.
  • 26/5/2019: Composer and Drush installations added. Manual core installation added. Deleting configuration file added.
  • 16/5/2019: Simplified installation step added. Using new shell to restore added.
  • 8/5/2019: Page on Drupal 8 withdrawn. All "/var/www/web" changed to "/var/www/html/web" because Apache2 prefers to put website files underneath "/www/web/html", and it is easier that way. Relocating Drupal 8 added. Print page stylesheet re-written.
  • 18/1/2019: Updating Drupal 8 core using composer added.
  • 30/9/2018: Setting CKEditor to use Mayo theme stylesheet added. Setting print page to also use Mayo theme stylesheet added. Adding custom styles added.
  • 23/9/2018: "php/7.0" changed to "php/7.2".
  • 3/9/2018: First created from the page on Drupal 7 after major upgrading to Drupal 8. Resolution of CKEditor table border added.

Intro

(section revised, 5/11/2023)

Drupal is a web site content management system serving web pages to the internet.

Upgrading from Drupal 9 to 10 has been deferred because Drupal 10 has changed the default theme Bartik to Olivero front-end theme and Claro administration theme and some of the modules do not have compatible upgrades. We have been using Bartik and prefer it over those new themes.

However, Drupal 9 reached end of life on 1 Nov 2023. Our Drupal 9.5.11 was therefore upgraded to Drupal 10.1.6 on 4 November 2023. This page documents the upgrading procedures. Most of the previous text about installing Drupal 8 and upgrading to Drupal 9 has been deleted because they are not useful anymore. Some old bits and pieces have been retained in case they may be useful.

Install Drupal 10

(section added, 5/11/2023)

Go to Drupal's website https://www.drupal.org/ and go to the web page of the latest release.

The webpage for Drupal 10.1.6 is https://www.drupal.org/project/drupal/releases/10.1.6.

According to it, to start a new Drupal project:

$ cd /var/www/html/
$ composer create-project drupal/recommended-project:10.1.6 "install-dir"

Change "install-dir" to a new name such as "web".

To update an existing Drupal site and all dependencies to the latest release of Drupal:

$ cd /var/www/html/web
$ composer update "drupal/core-*" --with-all-dependencies

To update an existing Drupal site to this specific release:

$ cd /var/www/html/web
$ composer require drupal/core-recommended:10.1.6 drupal/core-composer-scaffold:10.1.6 drupal/core-project-message:10.1.6 --update-with-all-dependencies

Upgrade Drupal 9 to 10

(section added, 5 Nov 2023)

The upgrading command as given above is very simple, but the reality is not that simple because there can be many errors due to incompatible modules and themes as well as locking to some previous releases, resulting in the inability to upgrade. There are many other websites describing how to solve the upgrading problems, but the solutions mostly cannot work. After about 12 hours of trials and errors over two days, the upgrading was finally successful. The following records the procedures used (based on memory).

Go to Drupal's top folder and install drupal/upgrade_status:

$ cd /var/www/html/web
$ composer require drupal/upgrade_status

Go to Manage > Reports > Upgrade status.

Get the list of modules and themes which should be updated for Drupal 10.

Get the list of modules and themes which are incompatible with Drupal 10 or no longer used.

Go to Manage > Extend or > Appearance to update or remove modules and themes.

Alternatively, use composer to update or remove:

$ composer require <group name>/<name>
or
$ composer require <group name>/<name>:<add version number as necessary>
or
$ composer remove <group name>/<name>

Go to Manage > Configuration > Text formats and editors to change the text editor from CKEditor to CKEditor 5.

Change back to CKEditor if errors occur when saving the change. Remove those not acceptable menu icons. Change back to CKEditor 5 until the change is successfully saved.

Remove drupal/upgrade_status because the installed version is not compatible with Drupal 10.

$ composer remove drupal/upgrade_status

Update all files and settings to the current version:

$ composer update --with-all-dependencies
$ vendor/drush/drush/drush updatedb

"--with-all-dependencies" can be abbreviated as "-W".

Enable write access (not tested whether this is really necessary):

chmod 777 web/sites/default
chmod 666 web/sites/default/*settings.php
chmod 666 web/sites/default/*services.yml

Install Drupal 10 but without updating, and edit composer.json (see alternative below):

$ composer require drupal/core-recommended:10.1.6 drupal/core-composer-scaffold:10.1.6 drupal/core-project-message:10.1.6 --no-update
$ nano composer.json

Change the following line to state the new release number (it is an important step in this order, otherwise, upgrading will report incompatible new release or locked old releases):

"drupal/core": "10.1.6",

Upgrade now based on the newly specified release:

$ composer update --with-all-dependencies

Alternatively, the following command may work instead of the above command with "--no-update" option and the update command:

$ composer require "drupal/core:9.5.11 as 10.1.16" --no-update && composer update

However, the following line in composer.json still needs to be changed as such afterward.

"drupal/core": "10.1.6",

Update drush and update the database

$ composer require drush/drush
$ vendor/drush/drush/drush updatedb

The updatedb command may report post-update changes. Generally answer "Yes" to accept the changes.

The upgrading should be successful.

Restore read-only access:

chmod 755 web/sites/default
chmod 644 web/sites/default/*settings.php
chmod 644 web/sites/default/*services.yml

Install drupal/bartik as a contributed theme:

$ composer require drupal/bartik

Go to Manage > Appearance to make it the default theme.

Add back any compatible modules previously deleted due to incompatibility.

Go to Manage > Reports > Status report to see what errors arise.

It may remind to download colorbox-master. Download and extract it to a different name as /var/www/html/web/libraries/colorbox.

It may also remind to download DOMPurify-main. Download and extract it. Move its "dist" directory to become /var/www/html/web/libraries/dompurify/dist.

Some modules are related to CKEditor, they have to be removed before CKEditor can be removed. Some of them can be re-installed after the removal of CKEditor and automatically linked to CKEditor 5.

Add TOC API for table of contents

(section added, 8/11/2023)

Unlike CKEditor 4, CKEditor 5 does not have table of contents for free. The previous tables of contents have to be removed page by page.

Use TOC API instead.

Go to Drupal's top folder and install drupal/toc_api:

$ cd /var/www/html/web
$ composer require drupal/toc_api


TOP API Example will automatically be installed also. TOP API is the back end. The Example actually adds the table of contents.

Enable the two modules at Manage > Extend > List.

Go to Manage > Structure > Table of contents types to see a list of example tables.

The default is the one that will be used. The other are examples.

Table of contents types

Edit the default. Use responsive to suit mobile phones, which will show on mobile phones the title only with a drop down menu.

General settings

Change back to top maximum level to "h2". Change "Back to top" to "-> Top".

Back to top settings

No change here.

Header settings

Change numbering suffix from ") " to ". " both ending with a space.

The numbering suffice follows the numbers to the headings in the body. The numbering separator follows the numbers in the table of contents.

Numbering settings

Install Convert Bundles

(section added, 8/11/2023)

When testing various table of contents modules, it was found that some would apply to the Basic Pages but not the Book Pages. This website previously used mainly Book Pages. The drupal/convert_bundles module can be used to convert a type of contents pages to another type. Generally, follow the default settings. After many trials, TOC API has been chosen for the table of contents. By that time all the Book Pages have been changed to Basic Pages. It has not been tested whether the conversion is really required in order to use TOC API.

Prepare for a newly installed Drupal

(section revised, 5/11/2023)

Install Apache2 web server if not already installed.

Enable Apache2 rewrite module (should have been automatically installed when installing HTTPS):

$ sudo a2enmod rewrite

Configure Apache2:

$ sudo gedit /etc/apache2/apache2.conf

Specify, keeping "<" and ">":

<Directory /var/www/>
    Options Indexes FollowSymLinks
    # AllowOverride None # replaced with next line
    AllowOverride All
    Require all granted
</Directory>

Restart Apache2 service:

$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Install MySQL server + PHP + phpMyAdmin, and create a user with database both called "drupal8". (This old name has not been changed since its creation.)

Follow the instructions on Drupal webpages to set up the rest.

The following bits and pieces were done for Drupal 9. Not sure whether they are the same now since new installation has been done.

  • Change ownership:
$ sudo chown www-data:www-data sites/default/settings.php
  • Choose language: English language
  • Choose profile: Standard
  • Setup database: MySQL
  • MySQL database name, user name and password as defined above
  • Site name: www.kctang.com.hk
  • Site e-mail address: <>@kctang.com.hk
  • Site maintenance username: <name of the administrator>
  • Site maintenance user e-mail address: <>@kctang.com.hk
  • Default country: Hong Kong S.A.R., China
  • Default time zone: Asia/Hong Kong
  • Check for updates automatically
  • Receive e-mail notifications
  • Change directory permissions:
$ sudo chmod 555 sites/default
$ sudo chmod 444 sites/default/settings.php
  • Permit uploading of files:
$ sudo chown www-data:www-data -R sites/default/files
  • Increase upload file size limit (php 8.3 used now as of 22/1/2024):
$ sudo gedit /etc/php/8.3/apache2/php.ini
  • Change existing to:
post_max_size = 200M
upload_max_filesize = 50M
max_file_uploads = 100
  • Restart Apache2 service after every change:
$ sudo systemctl restart apache2
or
$ sudo service apache2 restart
  • Edit "settings.php" file:
$ sudo nano /var/www/html/web/sites/default/settings.php
  • Specify trusted hosts:
$settings['trusted_host_patterns'] = array(
   '^www\.kctang\.com\.hk$',
   '^kctang\.com\.hk$',
   '^localhost$',
);
  • Disable "update.php" file:
$ sudo mv /var/www/html/web/update.php /var/www/web/<some new name>

Re-direct http path to sub-directory "web"

The default DocumentRoot in the default enabled Apache2 configuration file /etc/apache2/sites-enabled/000-default.conf is "/var/www/html/".

Web browers can access any permissible directories or files underneath the DocumentRoot. Access outside the Document Root will not be possible.

When the Drupal root directory /web is placed underneath /var/www/html such as /var/www/html/web, then http://kctang.com.hk will access the default index.html file underneath /var/www/html, but in the case of http://kctang.com.hk/web,access to those underneath /web is possible.

To re-direct http://kctang.com.hk to access /var/www/html/web directly, create a new ".htaccess" file under "/var/www/html":

$ sudo gedit /var/www/html/.htaccess

Specify:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteBase /
  RewriteRule ^(.*)$ web/$1 [R]
</IfModule>

"^(.*)$" represents the full text from start to end after the domain name, e.g. "abc/def" in "kctang.com.hk/abc/def".

"web/$1" represents the substitution text where "$1" represents the text represented by "(.*)" with "web/" inserted before it, i.e. "web/abc/def" based on the above example. This would redirect the path to "web/abc/def".

"[R]" is to redirect the path and will show "web/" as part of the redirected path. When using "[R]", "RewriteBase /" is required to add "/" before "web/$1". Without this, "/var/www/html" will be added before "web/$1" and will cause unexpected results. This effect was discovered after many days of error discovery using "[L]".

Using [L] as suggested by many people may hide "web/" from the displayed path, and does not require the use of "RewriteBase", but some of the web pages will still show "web/" unavoidably. A mixed use with or without "web/" displayed will cause denial of access rights or redirection to external URL in some cases. Therefore, it is better to force to display "web/" using "[R]".

The following has the same effect:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteBase /web
  RewriteRule ^(.*)$ $1 [R]
</IfModule>

Remove "www." prefix from URL

This is not essential for using Drupal, but is adopted only to simplify.

Insert the following in /var/www/html/.htaccess after the RewriteBase line:

# Remove "www." prefix from URL
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http%{ENV:protossl}://%1%{REQUEST_URI} [L,R=301]

Define Bartik theme color scheme

(section revised, 5/11/2023)

Log in as an administrator if not already in.

Choose Manage > Appearance > Settings, and define Bartik color scheme as follows:

 

Configure Bartik's css files as described below to match the preferences of this website.

Add custom styles to Bartik theme

(section added, 21/12/2021)

(section revised, 6/11/2023)

Log in the server.

Create a "misc" directory under the Drupal root directory, and edit to create 4 new files to re-define webpage styles:

$ cd /var/www/html/web
$ mkdir misc
$ cd misc
$ nano <new file name>

Create "append-to-elements.css" file to contain:

/* the following settings added, KCTang 18/12/2021 */
body {
 margin: 0;
 color: blue;
 font-family: Arial;
}
p {
 margin: 0.2em 0 0.2em;
}
pre{
 border: 1px solid green;
 font-family: monospace;
 margin: 0.15em 0.2em 0.15em;
 padding: 0.2em;
 background-color: #f8f9fa;
 white-space: pre-wrap;
}
.widget-toc{
 display: table;
 border: 1px solid green;
 background-color: #f8f9fa;
 padding: 0.5em;
 font-size: 0.9em;
}
.hangtwice {
 margin-left: 80px;
 text-indent: -80px;
}
.hang1 {
 margin-left: 40px;
 text-indent: -40px;
}
.hang2 {
 margin-left: 80px;
 text-indent: -40px;
}
.hang3 {
 margin-left: 120px;
 text-indent: -40px;
}
.hang4 {
 margin-left: 160px;
 text-indent: -40px;
}
.hang5 {
 margin-left: 200px;
 text-indent: -40px;
}
.indent1 {
 margin-left: 40px;
}
.indent2 {
 margin-left: 80px;
}
.indent3 {
 margin-left: 120px;
}
.indent4 {
 margin-left: 160px;
}
h1,
.heading-a {
 color: blue;
 margin: 0 0 0.5em;
}
h2 {
 border: 1px solid;
#  border-bottom: 1px solid;
 color: red;
 text-align: center
}
h2.hang {
 border: 1px solid;
 margin-left: 40px;
 text-indent: -40px;
 color: red;
}
h3 {
 border-bottom: 1px solid;
 font-weight: bold;
 font-weight: normal;
 color: magenta;
}
h3.hang {
 margin-left: 40px;
 text-indent: -40px;
 font-weight: normal;
 color: magenta;
}
h4.hang {
 margin-left: 40px;
 text-indent: -40px;
}
a,
a.link {
 border-bottom: none;
}
.site-branding__name {
 font-family: "Times New Roman";
}
img {
 margin: 15px;
}
/* the following settings added, KCTang 11/2/2023 */
blockquote {
 border-left: 2px solid #bbb; /* LTR */
 background: #F7F84D;
}
blockquote:before {
 margin-right: 0;
 content: "";
}
[dir="rtl"] blockquote:before {
 content: "";
}
blockquote:after {
 content: "";
}
[dir="rtl"] blockquote:after {
 content: "";
}
/* the following settings added, KCTang 22/3/2023 */
pre{
 font-family: math;

Create "append-to-layout.css" file to contain:

/* the following settings added, KCTang 18/12/2021 */
@media all and (min-width: 851px) {
 .layout-container {
   max-width: 1920px;
 }
}

Create "append-to-main-content.css" file to contain:

/* the following settings added, KCTang 18/12/2021*/
.main-content h2 {
 margin: 0.5em 0;
 font-size: 1.429em;
}

Create "append-to-text-formatted.css" file to contain:

/* the following settings added, KCTang 18/12/2021*/
.text-formatted ul,
.text-formatted ol {
 margin: 0;
 padding: 0 0 0.25em 15px; /* LTR */
}

Change to /var/www/html/web directory.

Create "append-to-bartik-css.sh" file to contain:

#!/bin/bash
cat /var/www/html/web/misc/append-to-elements.css >> /var/www/html/web/themes/contrib/bartik/css/base/elements.css
cat /var/www/html/web/misc/append-to-layout.css >> /var/www/html/web/themes/contrib/bartik/css/layout.css
cat /var/www/html/web/misc/append-to-main-content.css >> /var/www/html/web/themes/contrib/bartik/css/components/main-content.css
cat /var/www/html/web/misc/append-to-text-formatted.css >> /var/www/html/web/themes/contrib/bartik/css/components/text-formatted.css
ls -ls /var/www/html/web/themes/contrib/bartik/css

This file is to append the settings in the last 4 files to Bartik's css files to override Bartik's settings.

Execute to append whenever there is an update to Bartik:

$ cd /var/www/html/web
$ ./append-to-bartik-css.sh

Clear cache to reveal the effects of the new settings:

$ vendor/drush/drush/drush cr

Add buttons to CKEditor 5

(revised, 8/11/2023)

Log in the website as an administrator.

Choose Manage > Configuration > Text formats and editors > Configure Full HTML.

Move the required buttons down to a suitable position in the active toolbar:

Active toolbar

Include moving the Style button.

Enter a list of classes and titles as follows:

p.hangtwice|Hang Twice
p.hang1|Hang 1 
p.hang2|Hang 2
p.hang3|Hang 3
p.hang4|Hang 4
p.hang5|Hang 5
p.indent1|Indent 1
p.indent2|Indent 2
p.indent3|Indent 3
p.indent4|Indent 4
h2.hang|Head2Hang
h3.hang|Head3Hang
h4.hang|Head4Hang

"p" stands for paragraph.

".hangtwice" stands for the name of class as defined in the "css/style.css" file.

"Hang Twice" stands for the title seen when the Styles button is pressed to give a drop down menu.

Install Composer and Drush

(section added, 26/5/2019)

Drupal.org website describes various methods of installing Composer and Drush, generally referring to other websites for complicated steps which are difficult to follow.

The following simple steps do work.

Execute:

$ sudo apt install composer
$ cd /var/www/html/web
$ composer require drush/drush

Drush is installed under /var/www/html/web/vendor/drush/drush.

Update Drupal using Composer

(section added, 18/1/2019)

(simplified, 5/11/2023)

Go to the root directory of the website:

$ cd /var/www/html/web

Check for outdated modules:

$ composer show --outdated drupal/* -vvv

"-vvv" is to display what is going on.

Update all:

$ composer update drupal ---vvv

Update database:

$ vendor/drush/drush/drush updatedb -vvv

Answer "yes" when asked to run specified post-update changes.

Clear cache:

$ vendor/drush/drush/drush cr -vvv

Browse the website to see that the webpages can show up successfully.

Set to release the website from maintenance mode.

To copy sub-directories from one directory to another while keeping the original attributes:

$ sudo cp -Rav <old directory>/<name> <new directory>

Relocate or restore Drupal 8

(section added, 8/5/2019)

(revised, 16/5/2019)

Export the relevant MySQL database using phpMyAdmin as a backup.

Backup the whole Drupal directories and files. For example, if Drupal's root directory is /web in /var/www/html/web, then everything from and below /web.

Use old pair of backups if fresh pair not available.

Create a new user and blank database only if new computer or new MySQL-server is used.

Import the database backup.

Move the whole Drupal directories and files to the new directory, e.g. /var/www/new/web.

Keep "new" as the original name in case of no change, e.g. /var/www/html/web.

Keep the name "/web" unchanged since this is hard-coded in the beginning of links in the form of "/web/..." to media and files embedded in the web pages.

Execute

$ sudo gedit /var/www/new/web/sites/default/settings.php

For

$database['default']['default'] = array (
    'database' = '<text>'
    'username' = '<text>'
    'password' = '<text>'
    ....
);

change the <text> to match the new database.

Change the DocumentRoot in the new site's currently used Apache2 site configuration file, e.g. /etc/apache2/sites-enabled/000-default.conf, from "/var/www/html/" to /var/www/new, still one level above /web. Links in the form of "/web/..." to media and files will be interpreted as "/var/www/new/web/...", still accessible.

Web browsing to http://kctang.com.hk/web should be able to access Drupal at the new location.

Go to the next section in case web browsing fails to display.

If the DocumentRoot is set directly to the Drupal root directory "/var/www/new/web, web browsing of Drupal web pages is still possible, but links to media and files will be interpreted as "/var/www/new/web/web/... ", and the media and files will not be displayed.

Move the ".htaccess" file from /var/www/html to /var/www/new to re-direct web browsing to http://kctang.com.hk/web:

$ sudo mv /var/www/html/.htaccess /var/www/new

Execute as necessary:

$ vendor/drush/drush/drush updatedb -vvv
$ vendor/drush/drush/drush cr -vvv

("vendor/drush/drush/" added, 26/5/2019)

Use new shell to relocate and restore

(section added, 16/5/2019)

(revised, 26/5/2019)

If the relocation or restoration is not successful, non-display may happen because of some corruptions in the file settings.

Follow the Drupal core manual installation steps described above, but substitute "/var/www/html" described there with "/var/www/new".

Setting the website to maintenance mode would not be possible.

Before changing the settings file, move the ".htaccess" file from /var/www/html to /var/www/new:

$ sudo mv /var/www/html/.htaccess /var/www/new

Delete left-over module configuration files

(section added, 26/5/2019)

If a left-over configuration setting file is reported when a module is re-installed, execute to delete it:

$ cd /var/www/html/web
$ vendor/drush/drush/drush config:delete '<name of configuration file to delete>'

 

End of Page

Install MoinMoin wiki engine

Install MoinMoin wiki engine KCTang

Go to End

Note

MoinMoin wiki engine serves wiki web pages.

The following installation instructions are outdated. Check https://help.ubuntu.com/lts/serverguide/moinmoin.html for updated instructions.

Prepare

Install Apache2 web server if not already installed.

Download MoinMoin (filename moin-1.9.7.tar.gz) from http://moinmo.in/MoinMoinDownload, usually to own "Downloads" directory.

Use the file manager to extract the contents of the compressed file under the "Downloads" directory as "moin-1.9.7".

Install

Install MoinMoin:

$ cd /home/< own account name >/Downloads/moin-1.9.7
$ sudo python setup.py install --force --prefix /usr/local --record=install.log

Install "wsgi" module and test:

$ sudo apt-get install libapache2-mod-wsgi
$ cd /usr/local/share/moin/server 
$ sudo python test.wsgi

Point the web browser to the address shown on the terminal, such as, http://localhost:8000/.

Setup wiki sites

Setup files for a wiki called "qswiki":

$ cd /usr/local/share/moin
$ sudo mkdir qswiki
$ sudo cp -R data qswiki
$ sudo cp -R underlay qswiki
$ sudo cp server/moin.wsgi qswiki
$ sudo cp config/wikiconfig.py qswiki

Do similar for "dscwiki".

Configure Apache2 for MoinMoin site:

$ sudo gedit /etc/apache2/sites-available/moinmoin.conf

Specify, keeping "<" and ">" in the following codes:

# MoinMoin WSGI configuration
# invoke moin wiki at the root url, like http://servername/mywiki/FrontPage:
# servername can be www.kctang.com.hk or localhost
# qswiki and dscwiki use FrontScreen instead of FrontPage to avoid changes from being overridden when a new FrontPage is installed with new software
WSGIScriptAlias /qswiki   /usr/local/share/moin/qswiki/moin.wsgi
WSGIScriptAlias /dscwiki   /usr/local/share/moin/dscwiki/moin.wsgi
<Directory /usr/local/share/moin/qswiki>
    AllowOverride None
    Require all granted
</Directory>
<Directory /usr/local/share/moin/dscwiki>
     AllowOverride None
     Require all granted
</Directory>
# create some wsgi daemons - use these parameters for a simple setup
WSGIDaemonProcess moin user=www-data group=www-data processes=5 threads=10 maximum-requests=1000 umask=0007
# use the daemons we defined above to process requests!
WSGIProcessGroup moin

Enable the site:

$ sudo a2ensite moinmoin

which will create a linked file in "/etc/apache2/sites-enabled"

If required to disable:

$ sudo a2dissite moinmoin

Configure wsgi:

$ sudo gedit /usr/local/share/moin/qswiki/moin.wsgi

Add at the end of the a2) paragraph:

sys.path.insert(0, '/usr/local/share/moin/qswiki')

Do the same for "dscwiki".

Enable web access, specify access rights and restart Apache2 service:

$ cd /usr/local/share 
$ sudo chown -R www-data:www-data moin 
$ sudo chmod -R ug+rwX moin 
$ sudo chmod -R o-rwx moin
$ sudo systemctl restart apache2
or
$ sudo service apache2 restart

Configure wiki:

$ sudo gedit /usr/local/share/moin/qswiki/wikiconfig.py

Specify:

# -*- coding: utf-8 -*-
## -*- coding: iso-8859-1 -*-

#url_prefix_static = '/mywiki' + url_prefix_static
url_prefix_static = '/qswiki' + url_prefix_static

# sitename = u'Untitled Wiki'
sitename = u'QS Wiki'

#page_front_page = u"FrontPage"
page_front_page = u"FrontScreen"

# superuser = [u"YourName", ]
#    (note: remove "#" at the front, and replace "YourName" with user id within quotation marks)

#acl_rights_before = u"YourName:read,write,delete,revert,admin"
#    (note: remove "#" at the front, and replace "YourName" with user id within quotation marks)

# the following added at the end of the Security section

textchas = {
    'en': {
        u"Please enter antispam password (email to kctang@kctang.com.hk for password):": ur"[password]",
        },
    }
textchas_disabled_group = u"TrustedGroup"

#    (note: replace [password] with actual password)

# end of addition

#mail_smarthost = ""
mail_smarthost = "kctang.com.hk"

# The return address, e.g u"Jgen Wiki <noreply@mywiki.org>" [Unicode]
#mail_from = u""
mail_from = u"K C Tang <kctang@kctang.com.hk>"

# "user pwd" if you need to use SMTP AUTH
#mail_login = ""
mail_login = u"kctang [email password]"

navi_bar = [
    # If you want to show your page_front_page here:
    #u'%(page_front_page)s',
    u'RecentChanges',
    u'FindPage',
    u'HelpContents',
    u'FrontScreen',
]

# The default theme anonymous or new users get
# theme_default = 'modernized'
theme_default = 'modernized_mobile'

Note MoinMoin Config files are:

  • /etc/apache2/sites-available/moinmoin.conf
  • /etc/apache2/sites-enabled/moinmoin.conf
  • /usr/local/share/moin/qswiki/moin.wsgi
  • /usr/local/share/moin/qswiki/wikiconfig.py
  • /usr/local/share/moin/dscwiki/moin.wsgi
  • /usr/local/share/moin/dscwiki/wikiconfig.py

End of Page

Install Roundcube webmail client

Install Roundcube webmail client KCTang

Go to End

Note

23 Jan 2024: Minor changes.

26 Mar 2023: smtp setting revised.

16 Dec 2021: Updated installation command.

27 May 2019: Changed to use extracting directly to web directory. Protection measures added.

8 May 2019: All "/var/www/web" changed to "/var/www/html/web". "<version number>" used instead of a specific number. More descriptions on settings added.

25 Dec 2014: Created.

Intro

Roundcube is a webmail client.

Install Roundcube using apt

Execute:

$ sudo apt install roundcube

Install Roundcube from roundcube.net

If the apt installation does not work, execute to remove:

$ sudo apt remove roundcube

Then download roundcubemail-<version number>-complete.tar.gz by selecting "Complete: <version number>" at http://roundcube.net/download/, usually to own "Downloads" directory.

Extract the compressed file, move the extracted directory and files to underneath "/var/www/html", and rename the directory as "roundcube".

Ensure that the hidden ".htaccess" file is also moved.

A quicker way is to extract directly to "/var/www/html" and rename the directory there.

Install Roundcube under "/var/www/html":

$ cd /var/www/html
$ sudo tar -xzfv /home/<own account name>/Downloads/roundcubemail-<version number>-complete.tar.gz  (extract to current directory)
$ tar -xzf /home/<own account name>/Downloads/roundcubemail-<version number>-complete.tar.gz (extract to current directory) 
$ ls      (see the existence of the extracted directory)​
$ sudo mv roundcubemail-<version number> roundcube    (change directory name)
$ cd roundcube
$ sudo chown www-data:www-data logs
$ sudo chown www-data:www-data temp

"-xzfv" can be remembered as extract zipped file verbose. Use "-xzf" if "v" not working. 

(changed to use "-xzf", added cd roundcube, 16 Dec 2021)

(changed to use "tar" directly to web directory,  27 May 2019)

Set up Roundcube

Install MySQL database + PHP + phpMyAdmin, if not already installed.

Login phpmyadmin at web browser:

http://localhost/phpmyadmin

Create a database user "roundcube" with database "roundcube":

  • click "Users" at the top menu bar
  • click "Add user" at the page middle
  • enter "roundcube" at the User name entry
  • select "Local" at the Host entry
  • enter and re-type the password
  • click "Create database with same name and grant all privileges"
  • click "Go" at the bottom

Enter at web browser:

http://localhost/roundcube/installer

Configure Roundcube:

  • accept all of the defaults
  • enter database user "roundcube", database "roundcube" and password under Database setup
  • click: "CREATE CONFIG"
  • download the configuration file generated and save it as config.inc.php
  • move the downloaded file:
$ mv /home/<own account name>/Downloads/config.inc.php /var/www/html/roundcube/config
  • click: "CONTINUE" on the webpage
  • click: "Initialize database"
  • set to use secure SMTP port 465 (i.e. set "localhost:465" as SMTP Host)
  • (last line not working, changed to next line, 26 Mar 2023)

  • set "ssl:kctang.com.hk" as SMTP Host
  • enter username and password, and sender and recipient full email addresses to test SMTP config
  • accept "localhost" as IMAP Host
  • enter username and password to test IMAP config

If installed from roundcube.net, remove or rename installer directory, and protect logs and temp directories:

$ sudo cd /var/www/html/roundcube
$ sudo rm -R installer                   (remove)
$ sudo mv installer installer.original   (or rename)
$ sudo chown -R $USER:$USER logs
$ sudo chown -R $USER:$USER temp

(revised to add protection, 27 May 2019)

Configure PHP5 only (not necessary for PHP7 and later):

$ sudo gedit /etc/php5/apache2/php.ini
  • define:
date.timezone = Asia/Hong_Kong

otherwise the date column of the web mail would be blank. 

Restart Apache2 service:

$ sudo systemctl reload apache2
or
$ sudo service apache2 reload

Log in at web browser:

http://localhost/roundcube

or

http://www.kctang.com.hk/roundcube

Click Settings > Identity to define some settings:
 

Click Settings > Preferences > User Interface

> Time zone > Asia/Hong Kong

> Time format > 07:30

> Date format > 24/7/2023

> Save.

Click Settings > Preferences > Composing Messages

> Compose HTML messages > always

> When replying > start new message above the quote

> Force standard separator in signatures > turn off

> Save.

Click Settings > Contacts > Import to import contacts files, e.g. previously exported from Google.

(more descriptions on settings added, 8 May 2019)

(further added, 26 Mar 2023)

Upgrade if installed from roundcube.net

Download and extract the subdirectory of the new version to "Downloads" directory as "roundcubemail-<new version number>".

Upgrade existing directory using the installto.sh script:

$ cd /home/<own account name>/Downloads/roundcubemail-<new version number>/bin
$ ./installto.sh /var/www/html/roundcube

In case of big trouble

Uninstall Roundcube:

$ sudo apt-get remove roundcube

Remove "/var/www/html/roundcube":

$ cd /var/www/html
$ sudo rm -r roundcube

Delete mySQL user "roundcube" and database "roundcube" using phpMyAdmin.

Intall Roundcube as new.

End of Page

Install network file system

Install network file system KCTang

Go to End

Note

  • 22 Aug 2022: Client mount setting revised.
  • 17 Aug 2022: Client mount setting revised.

​​​​​​​Intro

NFS network file system enables sharing of directories on a Ubuntu server to another Ubuntu client computer.

Install on server computer

Install network file system:

$ sudo apt-get install nfs-kernel-server

Configure to specify the directory to be exported:

$ sudo gedit /etc/exports

Specify:

/<directory exported> <ip address of computer to export to> (rw, sync, no_root_squash)

where "rw" = read and write.

Start the server, required after any rebooting:

$ sudo systemctl start nfs-kernel-server
or
$ sudo service nfs-kernel-server start

Setup client computer

Install nfs-common on the client side:

$ sudo apt install nfs-common

Create a local directory:

$ sudo mkdir -p /media/<local directory>

Mount the remote directory temporarily:

$ sudo mount <ip address of NFS server>:/<directory exported from the NFS server> /media/<local directory>

but remounting required after every reboot.

To keep the mounting permanently unaffected by reboot, edit "fstab" file: 

$ sudo gedit /etc/fstab

Add a line:

<ip address of NFS server>:/<directory exported from the NFS server> /media/<local directory> nfs auto,vers=4.0 0 0

(revised 22 Aug 2022, "vers=4.0" added)

(revised 17 Aug 2018, "defaults" changed to "auto")

Mount again all devices as defined in the "fstab" file after changes:

$ sudo mount -a

Check devices actually mounted (this would show more than those defined in "fstab"):

$ sudo mount -l

End of Page

Install OpenVPN services

Install OpenVPN services KCTang

Go to End

Note

  • 15 Jul 2023: Windows folder to contain config files revised.
  • 26 Apr 2022: Cipher added.
  • 18 Jan 2022: Installation procedures updated.
  • 11 Apr 2018: Re-direct function added.
  • 2 Sep 2018: Revised to suit Ubuntu 18.04 which requires a change of the network card device name.

Intro

OpenVPN enables remote client computers and smartphones to access VPN server's files and structure, and optionally re-direct clients' IP traffic through the VPN server.

Install VPN server for accessing file server

Switch to root:

$ sudo -s

Install openvpn and easy-rsa:

$ apt-get install openvpn easy-rsa

Set up public key infrastructure:

$ mkdir /etc/openvpn/easy-rsa/
$ cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/
$ nano /etc/openvpn/easy-rsa/vars

Define in vars:

export KEY_COUNTRY="CN"
export KEY_PROVINCE="HK"
export KEY_CITY="HongKong"
export KEY_ORG="K C Tang Consultants Ltd"
export KEY_EMAIL="kctang@kctang.com.hk"
export KEY_OU=kctclVPN
export KEY_NAME=kctclVPN
# next line added to avoid error when building the certificate and key 
export KEY_ALTNAMES=kctclVPN

Generate master Certificate Authority (CA) certificate and key:

$ cd /etc/openvpn/easy-rsa/
$ source vars
# ./easyrsa init-pki
$ ./easyrsa build-ca

Enter New CA Key Passphrase: <>

Re-enter New CA Key Passphrase: <>

Enter Common Name: kctclVPN

Generate a key pair for the server:

$ ./easyrsa gen-req kctclVPN nopass

Accept Common Name default [kctclVPN]: <enter key>

Generate Diffie Hellman parameters and generate certificate for the server

$ ./easyrsa gen-dh
$ ./easyrsa sign-req server kctclVPN

Check and confirm the Common Name: yes

Enter Passphrase as previously defined: <>

Copy certificates and keys generated in subdirectory pki/ to /etc/openvpn/:

$ cp pki/dh.pem pki/ca.crt pki/issued/kctclVPN.crt pki/private/kctclVPN.key /etc/openvpn/

Config server.conf:

$ cd /
$ cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf /etc/openvpn/
$ nano /etc/openvpn/server.conf

Define as follows:

port 1194
proto udp
dev tun
# Enter the paths if files not in the same directory as the server.conf file
ca </path to file/>ca.crt
cert </path to file/>kctclVPN.crt
key </path to file/>kctclVPN.key
dh </path to file/>dh.pem # not dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
keepalive 10 120
tls-auth ta.key 0
cipher AES-256-GCM
user nobody
group nogroup
# last two lines uncommented for linux system
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
verb 3
explicit-exit-notify 1

(cipher added, 26 Apr 2022)

Generate ta.key for tls-auth:

$ cd /etc/openvpn
$ openvpn --genkey tls-auth ta.key
$ openvpn --genkey secret ta.key # not --secret

Config sysctl.conf:

$ nano /etc/sysctl.conf

Uncomment the following line to enable IP forwarding:

net.ipv4.ip_forward=1

Reload sysctl.conf:

$ sysctl -p /etc/sysctl.conf

Start the server:

$ systemctl start openvpn@server

Check if OpenVPN created a tun0 interface:

$ ifconfig tun0

Check syslog if tun0 does not appear:

$ grep -i vpn /var/log/syslog

Exit from root:

$ exit

Set the internet router to re-direct OpenVPN connections to server port 1194.

Extend to re-direct clients' IP traffic through VPN server

(section added 5 April 2018)

Define optionally in server.conf to re-direct clients' IP traffic such as web browsing and DNS lookups to go through the VPN server, i.e. the clients will appear to use the IP of the VPN server instead of the actual IP of the clients for internet traffic:

Config server.conf:

$ sudo nano /etc/openvpn/server.conf

Define by uncommenting the following line:

push "redirect-gateway def1 bypass-dhcp"

Some guide suggests to add the following, but this results in email server not working: (10 April 2018)

push "dhcp-option DNS 10.8.0.1"

Some other guides suggest to uncomment the following, this works: (10 April 2018)

push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

However, it is found that it still works when the above two lines are left commented. Therefore, the only line needing change is the 'redirect-gateway' line. (2 Sep 2018)

Execute to restart the service:

$ sudo systemctl restart openvpn@server

Execute to see the network card device names:

$ ip route

Find the output line beginning with "default", e.g.:

default via 192.168.0.1 dev enp4s0 proto static metro 100

The name "enp4s0" after the word "dev" is the default network card device name. Previously, the default name is "eth0", but this has been changed after Ubuntu 16.04.

(2 Sep 2018)

Execute with the default name inserted after "-o":

$ sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o enp4s0 -j MASQUERADE

Note that the iptables configuration will be lost after reboot.

Store the current iptables configurations:

$ sudo sh -c "iptables-save > /etc/iptables.up.rules"

View and remove any configurations no longer applicable:

$ sudo nano /etc/iptables.up.rules

Do the same whenever the iptables configurations have been changed.

Config file for use on reboot:

$ sudo nano /etc/network/interfaces

Define to reuse the stored configurations:

auto lo
iface lo inet loopback
post-up iptables-restore < /etc/iptables.up.rules

Generate files for each Windows client

Switch to root:

$ sudo -s

Generate a certificate and private key for each client user of <username>:

$ cd /etc/openvpn/easy-rsa/
$ ./easyrsa gen-req <username> nopass
$ ./easyrsa sign-req client <username>

Check and confirm the Common Name: yes

Enter Passphrase as previously defined: <>

Copy or move client's certificate and key to a Samba directory, which is for temporary use only: to enable emailing:

$ cd /etc/openvpn/easy-rsa/
$ cp pki/ca.crt pki/issued/<username>.crt pki/private/<username>.key /<Samba directory>/
$ cd /etc/openvpn/
$ cp ta.key /<Samba directory>/ 

Change the owners of the files:

$ cd /<Samba directory>
$ chown nobody:nogroup ca.crt ta.key <username>.crt <username>.key
$ chmod 644 ca.crt ta.key <username>.crt <username>.key

Create a <username>.txt under the Samba directory and define it to contain:

client
dev tun
proto udp
remote kctang.com.hk 1194
resolv-retry infinite
nobind
persist-key
persist-tun
mute-replay-warnings
ca ca.crt
cert <\\path\\><username>.crt
key <\\path\\><username>.key
remote-cert-tls server
tls-auth ta.key 1
cipher AES-256-GCM
auth-nocache
verb 3

(cipher added, 26 Apr 2022)

Specify path in Windows format if ca.crt, ta.key, <username>.crt, and <username>.key are to be saved in a folder different from <username>.txt.

Change filename from <username>.txt to <username>.ovpn. Creating as a txt file first permits editing by the usual text processors. Watch out that the line breaks are using Windows' linebreak code.

E-mail ca.crt, ta.key, <username>.crt, <username>.key, and <username>.ovpn files to the client computer.

Remove the files:

$ rm ca.crt ta.key <username>.crt <username>.key <username>.ovpn

Exit from root:

$ exit

Install on Windows client computer

Download and install the latest OpenVPN Windows Installer.

An OpenVPN GUI icon should appear at the bottom system tray, with no connection yet.

Save ca.crt, ta.key, <username>.crt, <username>.key, and <username>.ovpn files (emailed from the server) under:

  • C:\Program Files\OpenVPN\config-auto\ (for use as a service)

or

  • C:\Program Files\OpenVPN\config\ (for use with GUI).

(revised 15 Jul 2023)

Run on Windows client as a service

To start OpenVPN automatically as a service every time after rebooting:

  • Click Windows Start > search for Services.
  • Right-click OpenVPN Interactive Service > Properties > Start or Restart.
  • Change Startup type to Automatic.
  • Click OK.

Open file manager and enter \\10.8.0.1 to access the vpn server. All clients will use the same \\10.8.0.1 to map the actual different ip addresses assigned by the vpn server.

If connection is not successful, restart the computer.

Run on Windows client using GUI

To start OpenVPN manually every time after rebooting, or re-connect after loss of connection after sleep or hibernation::

  • Right-click OpenVPN GUI icon on Desktop.
  • Click Run this program as an administrator > Yes.

Or

  • Click the OpenVPN GUI icon on the bottom system tray to connect or right-click the icon and click Connect.

Map drive for quick access

Define a drive to be listed in the file manager directory to represent the server:
  • Open file manager and enter \\10.8.0.1 to access the vpn server.
  • Right-click the desired folder.
  • Click Map network drive.
  • Choose a drive name to represent the folder.
  • Click Finish.

End of Page

Install x11vnc

Install x11vnc KCTang

Go to End

Note

28 May 2019: Disabling Wayland added.

18 May 2019: Page added.

Intro

x11vnc enables remote access to the graphical desktop of Ubuntu server.

Set router

Set the internet router to permit access to the server through ports 5900, 5901, etc.

Disable Wayland

(section added, 28 May 2019)

Ubuntu-desktop uses gdm3 (Wayland) display manager. Wayland is not compatible with Xorg dispaly server used by x11vnc.

Execute to disable Wayland:

$ sudo gedit /etc/gdm3/custom.conf

Remove "#" to uncomment the following line:

#WaylandEnable=false

Reboot the computer.

Install

Execute:

$ sudo apt install x11vnc
$ sudo apt install xvfb

xvfb provides a virtual X window, which is required for the -create option below.

Start

Execute:

$ x11vnc                    (not requiring a password to connect to port 5900)
or
$ x11vnc -rfbport <port number>
or
$ x11vnc -usepw             (requiring a password to connect)
or
$ x11vnc -create -usepw     (create a virtual desktop)

The default port to connect from client computers is 5900.

Specify 5901 for the <port number> if connection is to be permitted at port 5901.

Connection with or without password is possible.

Execute to set up password:

$ x11vnc -storepasswd

The password will be stored in the file ~/.vnc/passwd, i.e. home/<current user>/.vnc/passwd.

Connection is possible if there is already a graphical desktop logged on at the server. The screen movement at the server and the client computer will be synchronised. This would be good for monitoring the screen movement at the server.

If there is no graphical desktop logged on at the server, then:

  • use the "-create" option
  • connect remotely which should show a terminal window at the server
  • execute at the terminal window to bring up other software, such as:
  • $ firefox
    $ nautilus
  • or the graphical session:
  • $ gnome-session
    
  • click Activities to access other software.

The remote screen movement will not be seen at the server.

Ubuntu-desktop with gdm3 is used above.

x11vnc does not work well with gdm3 (Wayland) desktop. Therefore, just use gdm3 desktop.

Gnome (Wayland) desktop also does not permit the starting of backintime

Stop

Press Ctrl-C to terminate the connection.

End of Page

Install OpenSSH services

Install OpenSSH services KCTang

Go to End

Note

26 May 2020: systemctl file command revised.

27 May 2019: Security settings added.

11 Apr 2018: Page added.

Intro

OpenSSH enables remote client computers and smartphones to access the server computer's text based terminal shell in a secured manner. "SSH" stands for secured shell.

Install OpenSSH server

Execute:

$ sudo apt install openssh-server

The software will be installed at /etc/ssh.

In case of complaint of no directory, execute  to make directory first:

$ sudo mkdir /etc/ssh

Edit config file:

$ cd /etc/ssh
$ sudo gedit sshd_config

Specify:

# Port 22                             (which is the default port)
Port 2nnn                             (change to some other 4-digit port, 2nnn)
# PermitRootLogin prohibit-password   (meaning no password required)
PermitRootLogin no                    (meaning no root login)

(security settings added, 27 May 2019)

Restart the service:

$ sudo systemctl restart sshd.service
or
$ sudo systemctl restart ssh.service

Check status:

$ sudo systemctl status sshd.service

If found disabled:

$ sudo systemctl enable ssh

(status check added, 26 May 2020)

Change the internet router to permit the use of port 2nnn.

The above is already sufficient for use. Read https://help.ubuntu.com/lts/serverguide/openssh-server.html for more configurations, if desired.

Install SSH client on Windows

Download Putty from https://www.putty.org/ and install.

Enter the Host Name, change the Port to 2nnn, highlight Default Settings and press Save:

"Only on clean exit" is the default. When the server's terminal window is exited with "exit" or "logoff", the PuTTY screen and connection would only close if other processes using the PuTTY connection have all been closed.

Press Open.

Accept the next screen to confirm the server's security key shown, if trusted. This would be necessary for the first time only.

Log in as the usual command terminal. No graphical interface is provided.

Use PuTTYgen that comes installed with PuTTY to generate key pairs, only if required. Read its Help.

Configure for VNC

If PuTTY is used for VNC connection, config the tunnel by entering the Source port and Destination as follows, then press Add to move the setting to the upper window:

With "Local" selected, the Source port means the port of the client computer. It can be "5900" or any free port. "Localhost:5900" at the Destination means the host computer, not the client computer. "5900" refers to the port number on the host computer providing VNC server service.

Go back to the first screen, highlight Default Settings and press Save again.

End of Page

Use Excel

Use Excel KCTang

Go to End

Note

11 Mar 2020: Rounding up to 2 significant digits added

17 Jan 2020: Using macros and Add-ins added.

01 Feb 2019: Bracketing negative numbers and hiding zeros added.

26 Sep 2018: Showing one-digit as two-digits added.

19 Sep 2018: First created.

Insert page number and total number of pages

Select Page Layout > Print Titles > Header/Footer > Custom Footer:

(By the way, click Scale with document and Align with page margins to enable the Header and Footer not to encroach into the margins.)

Select icon to insert Page number:

Enter " / ".

Select icon to insert Number of Pages:

Select OK.

Select Page:

Change First page number from "Auto" to "1".

If it is "Auto", the page number of all Worksheets selected to print will run consecutively.

If it is "1", each Worksheet will start at page 1.

However, the total number of pages will be the total of all Worksheets selected printed, not the total number of the respective Worksheet. This may not be desirable.

The next is a solution.

Show total number of pages of a Worksheet

Select Formulas > Name Manager > New.

Define:

  • Name: NumberOfPages
  • Refers to: GETDOCUMENT(50)

Enter in a cell a formula =NumberOfPages

The displayed value will be the total number of pages of the worksheet.

Enter in a cell a formula ="Total " & NumberOfPages & " Pages" (note spaces) will display a meaningful text.

This cannot be used in the Header or Footer.

Show single-digit whole number as two-digit whole number

Press Ctrl-1 > Number > Custom,

Enter "00" against Type. Select OK.

To make the double-digit number actually become a text for other uses, use the TEXT() formula, where "00" represents the Format Type used above, as follows:

Show negative numbers in brackets

(added 01 Feb 2019)

Use: 

#,##0.00;(#,##0.00)

The part before ";" defines the format of positive numbers.

The part after ";" defines the format of positive numbers.

Show negative numbers in brackets

Right-align positive numbers with negative numbers before the ")"

(added 01 Feb 2019)

Append "_-" to the first part:

#,##0.00_-;(#,##0.00)

 

Show negative numbers in brackets and align

Hide zeros for individual cells

(added 01 Feb 2019)

Append ";;@" to the end:

#,##0.00;(#,##0.00);;@

Hide zeros for individual cells

Hide zeros for the whole worksheet

(added 01 Feb 2019)

Select File > Options > Advanced.

De-select Show a zero in cells that have zero value.

All zeros will be hidden.

Excel-Worksheet-HideZeroForWhole

Round up numerical values to 2 significant digits

(added 11 Mar 2020)

Use formula in the cell to display:

"=IF(INT(Q)>0,ROUNDUP(Q,2-LEN(INT(Q))),ROUNDUP(Q,2))"

where Q = referenced cell address (or formula)

It can be read as: if integer of Q > 0, then round it to 2 significant digits, else round up Q to 2 decimal places.

If Q = 1831.03, then

  • INT(Q) > 0.
  • LEN(INT(Q)) = 4
  • 2 - LEN(INT(Q)) = 2 - 4 = -2 (meaning, reduce length from 4 to 2 by -2)
  • ROUNDUP(Q, -2) = 1900.

If Q = 0.1234, then

  • ROUNDUP(Q, 2) = 0.13.

Application:

  • The unit rates used in Cost Estimates are approximate rates. 
  • It would not make sense to show them in many digits and with decimal places to make them look like very accurate.
  • 2 significant digits should be sufficient.
  • Estimates should include buffers. 
  • Rounding up is appropriate.

Use macros

(added 17 Jan 2020)

Macros can be used to execute keyboard or mouse commands automatically. This will simplify the tasks if the same set of keyboard or mouse commands are repeated frequently.

Record macros: 

  • Select View > Macros > Record Macro
  • Enter <Macro Name>
  • Select OK
  • Do an example of the intended task with keyboards and mouse
  • Select View > Macros > Stop Recording

View macros: 

  • Select View > Macros > View Macros
  • Select the desired <Macro Name>
  • Select Run to run the macro to execute the recorded commands
  • Select Step into to run the macro step by step
  • Select Edit to edit the macro
  • Select Option to assign a shortkey to run the macro

Note:

  • The macro will record based on the cells used when recording the macro.
  • Editing is required when the same set of commands is to be run starting from a cell at a different location.
  • Change the relative addresses of the cells as recorded in the macro, and test run to get the expected results.
  • Once successful, the macro can be used for similar cases when the cells used are of the same relative layout.
  • More complex macro would need to be written if more varying circumstances are to be handled as well.
  • The recorded macro can serve as the base or give code segments for use.

Use macros on different files

To run macros on a different file:

  • Save the file with macros as an macro file with ".xlsm" extension to enable it to be run, otherwise the macros must be enabled each time when they are run
  • Open the macro file
  • Open the target file and display the target worksheet as the active worksheet
  • View macro as described above, and select and run the desired macro

Use Add-in to run macros (standard method)

The above process would need a number of steps before the macros can be run.

A simpler way is to add the macros as commands in the ribbon at the top.

Add macros to a ribbon

  • Save the macro file with ".xlsm" extension as an Add-in file with ".xlam" extension, which will save it by default to the following folder:
C:\Users\<Admin or user name>\AppData\Roaming\Microsoft\AddIns
  • Right-click any empty space on any of the ribbons at the top
  • Select Customize the Ribbon > Add-ins 
  • Select Go against Manage:  Excel Add-ins
  •  Select the desired ".xlas" Add-in file > OK
  • Right-click any empty space on any of the ribbons at the top
  • Select Customize the Ribbon > Customize Ribbon
  • Select Macros under Choose commands from to show the available macros
  • Select New Tab to create a New Tab with a New Group (Custom)
  •  
  • Select the desired macro > Add to add the macro under the New Group (Custom)
  • Select the existing added macro > Remove if removal is desired
  • Select OK 
  • A New Tab with menu items for the macros should be available for use
  • Close Excel without the need to save

Use Add-in to run macros (quick method)

  • Copy the required Add-in file with ".xlam" extension to the folder:
C:\Users\<Admin or user name\AppData\Roaming\Microsoft\AddIns
  • Right-click any empty space on any of the ribbons at the top
  • Select Customize the Ribbon > Developer > OK to add the Developer tab
  • Select Developer > Excel-Add-ins
  • Select the desired ".xlam" Add-in file
  • De-select the superseded Add-in file in case of updating
  • Select OK, which should add the Add-in tab
  • Select Add-ins tab to access the menu items for the macros for use.
  • Close and re-open Excel to see that the Add-in ribbon stays for use.

Note:

  • The Add-in file with ".xlam" extension is previously created by saving the macro file with ".xlsm" extension.
  • The macros in the Add-in files will not be seen in the usual View Macro menu.
  • The Add-in file can be accessed by pressing Alt-F11.
  • The VBA macro codes are contained under the Modules folder of the Add-in file directory structure.
  • Two extra macros are contained the "ThisWorkbook" item under "Microsoft Excel Objects" item of the Add-in file directory structure.
  • One of the two macros is called "Private Sub Workbook_Open()" which will remove all existing Add-in menu items and re-add new Add-in menu items when Excel is opened or when the Add-in file is added through Excel-Add-ins.
  • The other of the two macros is called "Private Sub Workbook_BeforeClose()" to remove all Add-in menu items upon closing Excel.
  • It has been suggested that the two macros should be named as "Private Sub Workbook_AddinInstall()"  and "Private Sub Workbook AddinUnintall()" to be effected upon adding or removing the Add-in file, but the AddinInstall is not very effective because the Add-in tab will disappear after closing Excel.

End of Page

Install Thunderbird

Install Thunderbird KCTang

Go to End

Note

  • 11 May 2023: More CardBook installation steps added.
  • 10 May 2023: PrintingTools NG Options revised. Use CardBook instead of TbSync. Delivery Status Notification added.
  • 2 Aug 2022: PrintingTools NG Options revised for version 2.1.4.
  • 22 Mar 2022: Print page setup revised for version 91. Printing Tools NG Options revised.
  • 9 Dec 2021: Exporting email procedures added.
  • 29 Jun 2021: Advice against upgrading to version 78 removed because more add-ons become compatible. Some notes added.
  • 16 Jan 2021: TbSync manual configuration for Google added.
  • 15 Jan 2021: Google Calendar Plugin added. PrintingTools attachments per line option added. Header fixed font size revised. CardBook add-on removed because TbSync supports Google again. Lists of add-ons updated. 
  • 19 Nov 2020: Option set to show all folders without subscribing individually. CardBook add-on added because TbSync no longer supports Google. Lists of add-ons updated. 
  • 8 Sep 2020: 'Manually sort folders' add-on added.
  • 12 Aug 2020: Revised to fix font for print headers only.
  • 11 Aug 2020: Created.

Intro

Thunderbird is an email client.

Install Thunderbird

Download the latest release from here and install:

https://www.thunderbird.net/en-US/download/

Download old releases from here and install:

https://archive.mozilla.org/pub/thunderbird/releases

Set up the first email account.

For Gmail account, enter the Gmail address and use automatic setup.

Use the following settings to manually set up email account accessing company's email server:

  • Account Name: kctcl
  • Your Name: K C Tang Consultants Ltd.
  • Email Address: kctcl@kctang.com.hk
  • Incoming Server Type: IMAP Mail Server
  • Server Name: kctang.com.hk
  • Port: 993
  • Connection security: SSL/TLS
  • Authentication method: Normal password
  • Outgoing Server Name: kctang.com.hk
  • Port: 465
  • Connection security: SSL/TLS
  • Authentication method: Normal password

Select Tools > Accounts Setting to further define or adjust the settings:

Uncheck "Show only subscribed folders" to show all folders instead of subscribing individually:

(Added, 11 Nov 2020)

The default is to keep all messages on the local computer as shown below, but it is suggested not to do so to avoid using up local computer's storage space. Subject headings will always be downloaded. The email body and attachments will be downloaded only when being read, and may take a bit longer time, but this should be tolerable.

(Note added, 29 Jun 2021)

​​​​​​​​​​​​​​​​​​​

Select Tools > Accounts Setting > Account Actions > Add Mail Account to add other accounts.

Leave the options under Tools > Options as the default except in the following case (not required now as the Postfix configuration can be raised).

When the tls security level has to be lowered if the email server is not of the level expected by Thunderbird:

Select Tools > Options > General > Config Editor > I accept the risk > Search "security.tls" > security.tls.version.min

Change the default value "3" to "1" and select OK and exit.

(Note to adjust tls security level added, 29 Jun 2021)

Set Delivery Status Notification

Select Options > Delivery Status Notification if desired to have record of delivery:

Selecting Return Receipt will require the recipient to confirm receipt. This is not quite useful.

(Session added, 10 May 2023)

Install Add-ons​​​​​​​

Select Tools > Add-ons.

Search at "Find more extensions" for the following add-ons, install them and restart Thunderbird:

  1. AttachmentExtractor Continued - to download attachments (emails must have been fully downloaded before the full attachments can be extracted)
  2. CardBook - to sync Google contacts
  3. Google Calendar Plugin - to open Google calendar
  4. ImportExport Tools NG - to import or export emails or folders, good for exporting for archive 
  5. Lightning - to use calendars (better use Google calendars directly)
  6. LookOut (fix version) - to open Outlook attachments
  7. Manually sort folders - to re-sequence order of accounts and folders
  8. PrintingTools NG - to set print page layout
  9. Provider for CalDAD & CardDAV - required for TbSync below (not used)
  10. Quick Folder Move - to move or copy emails between folders
  11. Remove Duplicate Messages - to remove duplicate emails
  12. SmartTemplate4 - to set templates for composition, reply and forwarding
  13. TbSync - to synchronize contact, task and calendar information (not used because of log in problem).

(List updated, 15 Jan 2021)

(Note against AttachmentExtractor added, 29 Jun 2021)

(List updated 10 May 2023)

Use Quick Folder Move

Select some messages.

Press the following keys to open a context menu, enter a few characters for the name of the destination folder to display possible choices, move down to select the folder, and press Enter.

Press:

Shift+M to move the currently selected messages to the destination folder.

Ctrl+Shift+M to repeat moving to the previously selected folder.

Shift+Y to copy the currently selected messages to the destination folder.

Ctrl+Shift+Y to repeat copying to the previously selected folder.

Shift+G to jump to the destination folder (without the need to pre-select come messages).

Set up Remove Duplicate Messages

Select Tools > Add-on Options > Remove Duplicate Options:

Select Move to Trash instead of Delete permanently if not sure.

Select Folder if comparison is not across different folder.

Set up Print Page (Thunderbird version 91)

Select Tools > Preferences > General > Config Editor (at bottom of page).

Search "headerleft" and press the pencil icon to edit to enter "&T" to represent email subject.

​​​​​​​

Press the refuse bin icon to remove similar but outdated setup for named printers (similarly for other settings described below.

Search "headercenter" and edit to remove the value.

Search "headeright" and edit to remove the value.

Search "footer" and edit to set "&PT" for page number for footerleft, and "&D" for date for footerright.

Press Ctrl-P to pop up the print menu to define the paper size, scale, margins, and options:

​​​​​​​

Set up Print Page (Thunderbird version 78)

Select File > Page Setup:

Set up PrintingTools NG Options

Select File > Printing Tools NG Options and set as follows:

Copy and paste this Custom date format: %a,  %e %B %Y %t %H:%M:%S [GMT %z]

Skip PDF Output. Functions to be understood.

Right-click Thunderbird toolbar to customise to add the following icon:

(Section updated, 10 May 2023)​​​​​​​​​​​​​​​​​

Set up SmartTemplate4

Select Tools > Add-on Options > SmartTemplate4:

Copy and paste the following latest Code in the Quote Header of the Reply and Forward options:

<a blank line first>
Regards,
Mr. K C Tang <replace with your name>
for
K C Tang Consultants Ltd.
T:(852)2866-6451 F:(852)2865-4751 W:www.kctang.com.hk
<hr/>
<SPAN style="FONT-SIZE: 10pt">
<b>Sent: </b>%date% ​​​​​​​<Use this to show: Sent: Mon, 14 Dec 2020 10:47:45 +0800>
<b>Sent: </b>%X:=sent% %datelocal% %date_tz% ​​​​​​​<Or use this to show: Sent:  Monday, Dec 14, 2020, 10:47 +0800>
<b>From: </b>%from(name,bracketMail())%
<b>To: </b>
<SPAN style="FONT-SIZE: 8pt">
%to(name,bracketMail())%</SPAN>
[[<br><b>Cc: </b>
<SPAN style="FONT-SIZE: 8pt">
%cc(name,bracketMail())%</SPAN>]][[<br><b>Bcc: </b>
<SPAN style="FONT-SIZE: 8pt">
%bcc(name,bracketMail())%</SPAN>]]
<b>Subject: </b>%subject%
</SPAN>
<hr/>

(Note against date formats added, 29 Jun 2021)

Insert license key in the next menu.

 

Set up CardBook

Select Tools > Add-ons and Themes > Find more add-ons > Enter CardBook to find and add the App.

Select Next when the following appears:

Select Validate if remote address book found (password to be entered in the next validation screen) or select Next:

The CardBook icon will appear on the right of the Toolbar.

Right-click the Toolbar to move the icon position to the left as appropriate:​​​​​​​

Click CardBook icon or select Tools > CardBook:

The following will show up:

 

Right-click within this side window space to bring up the following and select New Address Book:

Select Remote and Next:

Select Google, enter Username and select Validate:

Enter password in the next Google screen and permit access to Google contacts.

The Google contacts will be synchronized for use.

Add additional address book as necessary.

Select Edit Address Book to further customise:

Select General to change Address Book colour:

Select Sync to set synchronization interval to 15 minutes:

Remove the original Address Book icon from the toolbar because it will not be used anymore.

CardBook has many settings and entries for the contact but just use the simplest entries.

For the name of the contact, enter in the format of "Short company name - Full personal name" as both the Display Name and Family Name. Do not split to use Given or Other Names.

Press Synchronize in the CardBook toolbar as necessary.

(Section added, 10 May 2023)

(Updated, 11 May 2023)

Set up TbSync

Select Tools > Synchronization Settings (TbSync):

Or select Tools > Add-on Options to go to the following, then select Add new account > CalDAV & CardDAV:

Select Install Provider > download link:

Add CalDAV & CardDAV:

After adding, and re-starting Thunderbird as necessary, go back and select Manual Configuration > Next:

Enter

Account name: friendly account name, e.g. kctangcl

User name: gmail.com account name

CalDAV server address: https://apidata.googleusercontent.com/caldav/v2

CardDAV server address: https://www.googleapis.com/.well-known/carddav

Select Next:

(Manual configuration added, 15 Jan 2021)

The following is an older version of TbSync which provided automatic Google support. In that case, select Google > Next:

Enter a friendly account name and select Next:

After selecting Next in the manual case or automatic case, enter Google account email address and its password (may require doing it twice):

Select Finish:

Select Enable and synchronize this account:

Available resources like Google address book and calendars will be displayed.

Select those to be synchronized > Synchronize now:

The address book and calendars will be available for use.

Change Periodic synchronization (in minutes) to 30, which is the minimum interval. 

Export emails

Use ImportExportTools NG to export emails for archive.

To ensure that all email attachments have been downloaded. The email folders should be synchronised.

Select Tools > Account Settings > Advanced:

It may not be necessary to change to "Synchronise all messages locally regardless of age". Change only if the final results show that not all messages with attachments have been synchronised. Change back after use.

Select Download against folders intended to be exported:

Wait a while for the messages to be fully downloaded before exporting.

Alternatively, select relevant folder > Properties > Download Now to download all messages:

Select Tools > ImportExportTools NG > Options or alternatively select relevant folder > ImportExportTools NG > Options to set options:

Accept default options as follows:

Change the "format for filenames", "cut complete file path length" and "Charset for filenames" as follows:

Note the Custom Date format:

Accept default options as follows:

Accept default options as follows:

Select the relevant folder name and the following options to export to HTML format (with attachment):

Select the destination folder under which the exported emails will be saved.

After exporting, the following sub-folder for the selected source folder "XYZ" with the date and time of exporting appended will be created:

The following will appear under the sub-folder. The index.html will give an index page linking to all the emails and attachments under the Messages sub-sub-folder:

The emails under the Messages sub-sub-folder are in html format. Their filenames follows the filename format defined above. Their attachments are in the Attachments folders, one for each email with attachments.

Check whether the attachment files have been fully downloaded. If the file size is only 1KB, it means not fully downloaded in most of the cases.

The html files cannot be imported back to the email server. Export again to eml format as a double measure:

The exported sub-folder will have the same pair of index.html and Messages sub-sub-folder, but the index.html cannot link to email attachments. Each email has its own embedded attachments. There will be no separate Attachments folders.

End of Page

Set Up QNAP Network Attached Storage

Set Up QNAP Network Attached Storage KCTang

Go to End

Note

  • 28 Apr 2022: OpenVPN connection added.
  • 25 Apr 2022: Created.

Intro

A network attached storage is self complete with hard disks, ram and CPUs but without monitor screen to store files. It can be networked to computers and smartphones for viewing and file management.

Connect from Windows on the same local area network

Open File Explorer on Windows computer connected to the same local area network of a properly set up NAS (setting up described later below)..

Click "Network" oOpen File Explorer on Windows con the folder tree to find that the NAS appears as a connected device.

Click its icon.

Enter the username, password and confirm remembering the credential for subsequent connections.

The shared folders should appear for use.

Map the top shared folder to a drive number for easy access.

Connect from Windows from internet using OpenVPN

Install OpenVPN services if not already installed.

Copy the OpenVPN configuration file (obtained from the NAS after setting up as described later below) to Windows' C:\Program Files\OpenVPN\config\.

Unlike others, this configuration file connection needs manual connect.

To start this manually every time after rebooting, or re-connect after loss of connection after sleep or hibernation:

  • Click "^" on the bottom task bar to pop up the small window
  • Right-click the OpenVPN GUI icon
  • Click the configuration file name, and click Connect
  • Enter user name and password for the first time.

A message will appear if connected successfully.

Open file manager and enter \\10.7.0.1 to access the files.

Map folders to drive names for easy access.

Do not manually connect for other automatically connected configuration files.

If not connected successfully due to insufficient number of TAP-Windows adapters, search Windows as follows, add a new adapter and connect again:

If connection is still not successful, try opening Windows' command prompt and execute ipconfig /release and ipconfig /renew, and connect again.

Restart Windows as the last resort.

Map drive for quick access

Define a drive to be listed in the file manager directory to represent the server:

  • Open file manager and enter \\10.8.0.1 to access the vpn server.
  • Right-click the desired folder.
  • Click Map network drive.
  • Choose a drive name to represent the folder.
  • Click Finish.

 

Map the top shared folder to a drive number for easy access.

Set up QVPN to connect from internet

On computers, download "QVPN Device Client" from https://www.qnap.com/en/utilities/networking.

Set up as follows:

Set as follows:

  • The profile name can be decided by the user.
  • The Username should be the user name registered on the NAS. He will be able to access folders and files owned by him or shared with him.
  • The Pre-Shared Key should be the same as that set on the NAS (described later below).
  • The VPN Port should be the external port of the router connecting the NAS which can be the default port of "443" but should preferably be another port.
  • Note the selections of blue bullets.

Adjust "Connection" settings as follows, and select "Save" to connect:

Open Windows' File Explorer.

Click "Network" on the folder tree.

Enter "10.6.0.1" at the top to display the shared folders (if successful):

Map the folder to a drive number:

Install new hardware

Insert hard disks in the storage case.

Connect to a computer through router.

Connect to power supply and switch on the NAS.

Download Qfinder Pro from Essentials | Utilities | QNAP on computer and install.

Use Qfinder Pro to search for the NAS on the same network.

Update the firmware

Click the row showing the existence of the NAS to bring up the following to start smart installation:



Initialise

Enter the user name and password of the first user to become the administrator to replace the default "admin":







Create storage volume

Log in as the new administrator:




Click "+" against "No Volume" to create new volume:

Select "Change Type":

Select "Static volume" to use full disk size (backups are done elsewhere not in this NAS):

Unclick "Alert threshold" (optional).

Select "Set to Max".

 

A single volume without name is created.

If create thick volume (not used)

If "Thick volume" was selected, click "+" against "Create Storage Pool":

Storage Pool 1 is created (not in the case of static volume):

Create storage pool

Do not enable "Create SSD secure storage pool" if no SSD:

Click to select both hard disks:

Select "RAID 1" meaning two hard disks contain the same data and either disk can stand alone to function if another disk is faulty:

DataVol1 being created for static volume:

DataVol1 being created for thick volume:

Datavol 1 is created for thick volume:

Final result:

Disable snapshot backup

No snapshot volume created yet:

Snapshots are used to capture the current data and their changes. Since backup of the data is done on other media, no snapshot is to be set here.

If snapshots are enabled and huge volume of data is imported to the hard disks for the first time, the snapshots taken during the long time of import will use up the snapshot space and may cause system hang and downgrading of the RAID system.

Unclick all three in green to disable "Smart Snapshot Space Management" and "Snapshot Directory" since the full disks are to be used for live data:

Select "OK":

View to ensure correct selections:

Scroll down the page to view the rest.

Select "Apply" to confirm:

Set general settings

Change the default System port of "8080" to another port to reduce the chances of unauthorised access, change the external port of the router, apply proper port forwarding from the router port to the NAS port:

Set date and time format:

Set up network

Select "Network & File Services" as shown on the left above.

Select "Win/Mac/NFS/WebDAV":

Set Microsoft Networking as follows and select "Advanced Options" for connection by Windows computer on the same local area network::

Set as follows:

Set NFS Service as follows for mounting as a drive on linux computers::

Set up shared folder

Select "File Station" on desktop:

Select "Create folder":

Select "Shared Folder":

Define folder name and access privileges:

Folder is created:

Edit shared folder permissions:

Click open "Advanced Permissions" menu:

Select "NFS host access" (select "Microsoft Networking hot access" later):

Select "Access right" and add "*" to enable all hosts and IPs:

Select "read / write", and "Apply":

Check and apply Microsoft Networking host access as follows:

Enable as follows:

Set up QVPN service

Select "QVPN Service" on NAS desktop:

Enable QBelt service as follows:

Enable OpenVPN service as follows:

  • Change the default VPN client IP pool of "10.8.x.x" to "10.7.x.x" to avoid conflict with that used by OpenVPN on the server (not NAS).
  • Change the default Sever Port of "1191" to another port to reduce the chances of unauthorised access.
  • Change the external port of the router to the same.
  • Apply proper port forwarding from the router port to the NAS port:
  • Download configuration file (from the left box) for use.

 

Set up myQNAPcloud

myQNAPcloud is another way to access the NAS from the internet whereby an QNAP ID is given to link to the NAS through the internet.

QNAP ID can be used on computers and smartphones, and can connect to home computers without fixed external IP addresses.

However, myQNAPcloud requires access by administrators, and should not be used by non-administrators.

Select "myQNAPcloud" on the NAS:

Select 'Get Started":







Download and install "Let's Encrypt":



End of Page