As the title very well expresses it, this blog will deal with how to set up and use mutt with the gmail server as a relay in Ubuntu. In my case this became necessary after leaving UNSW and needing to use gmail as the main mail exchange, but while doing so, I always missed mutt. The feature that I needed most, was proper archiving. Gmail tends to put everything in one place and then put tags, and the user is shown a list of mails according to tags: inbox and send-mail for example, are just tags. But sometimes you want to actually put away a bunch of emails by category, and be able to access them in the future, and perhaps have some hierarchy on how these are accessed. Perhaps I am too used to the file system… And mutt allows you to save your messages away in files, facilitation backups. I also prefer the view of threads in mutt as opposed to conversations in gmail.
For comprehensive information about mutt, follow this link from Arch Linux or here. There, you can find directions for installing and running mutt along with the different options for configuration. The documentation for ubuntu can be found here. In one of the blogs I visited, somebody was suggesting Cyrus for handling email at a company level. Sounded interesting but have not investigated as there is no need at the moment. For the future…
I found a set of instructions from the official Ubuntu documentation site, to set up the following process: retrieve mail from gmail, store it in the local computer, view it using mutt and send mail back using the gmail server as the proxy. The instructions are here and are specifically for Ubuntu.
Following I will review what I did based on these instructions, adding my comments and questions.
Existing accounts
The obvious assumption is that there exist a gmail account where you receive your email and that you have a local computer running Ubuntu with a username, under which account the email will be downloaded. I’ll take the same pseudos used in the source document, but PLEASE throughout the process, substitute for your own:
Gmail Address: john.example@gmail.com Gmail Password: rover Computer Username: john
Editor
The best editor for mutt is vi or vim. It will allow the user to be more efficient (once to get used to it) without needing extra windows. This is very useful if you are reading your email across a slow connection.
Retrieving the mail
To download the mailbox from the gmail server into your local computer, you need to set up three things:
- Certificates to authenticate passwords and allow the transfer (SSL certificates) using POP
- To fetch your mail (fetchmail)
- To distribute the mail locally (procmail)
- SSL certificates and POP
- Gmail uses the POP3-over-SSL protocol for the secure transmission of username and password over the Internet.
The Post Office Protocol (POP) is an application-layer Internet standard protocol used by local e-mail clients to retrieve e-mail from a remote server over a TCP/IP connection. POP and IMAP (Internet Message Access Protocol) are the two most prevalent Internet standard protocols for e-mail retrieval. Virtually all modern e-mail clients and servers support both. The POP protocol has been developed through several versions, with version 3 (POP3) being the current standard.
Gmail uses both IMAP and POP3. IMAP allows the bidirectional communication between mail server and client. But given that mutt and gmail store messages fundamentally differently, the bidirectional changes don’t make any sense. IMAP is more efficient in retrieving mail: it assigns unique consecutive UID’s, so it only needs to load the messages with higher UID’s than the biggest message’s UID stored locally. POP3 on the other hand, assigns an arbitrary unique identifier using UIDL, and the number may be repeated for identical messages. To retrieve the new messages, POP3 must fetch the whole UIDL table. For long mailboxes this might be lengthy. POP3 can retrieve messages twice or miss some. Nevertheless, we will use POP.
(Will IMAP work anyway with mutt? The answer is YES. Reading some documentation on the Web, IMAP seems to be the default, so I don’t know why POP3 was preferred in the instructions I’m following.)
Secure Sockets Layer (SSL), are cryptographic protocols that provide communication security over the Internet.
http://en.wikipedia.org/wiki/Transport_Layer_SecurityTo use SSL, you will need to install both open SSL and a certificate pack:
sudo apt-get install openssl ca-certificates
- fetchmail: setting up the Mail Retrieval Agen (MRA)
-
As it names suggests, this program fetches mail from a POP (and other) capable server.
Install as follows:sudo apt-get install fetchmail
As with many programs, fetchmail needs a configuration file (.fetchmailrc) stored in the home directory. This file must include the SSL configuration. An initial file, after changing the pseudos, follows:
# Gmail account details poll pop.gmail.com with proto POP3 user "john.example@gmail.com" there with password "rover" is "john" here # Options for everybody mda "/usr/bin/procmail -d %T" options no keep ssl sslcertck sslcertpath /etc/ssl/certs
The commnand poll in the first line, can be used as many times as needed for number of servers to be queried.
Lines 1–5 state: connect to pop.gmail.com using the POP3 protocol, login using username “john.example@gmail.com” uisng password “rover” deliver to local user “john”.
Line 6 with keyword mda, invokes a MDA, or Message Delivery Agent. %T will the replace by the local delivery addresses for the MDA command. We will use as MDA procmail (instructions following later).
The keyword options in line 7 indicates that the following lines are all options to the command fetchmail:
- no keep deletes seen messages in the server. As far as I understand, this has no effect with gmail. There is however, a setting in gmail allowing you to do something with the messages accessed via POP3 (details later).
- ssl causes the connection to the mail server to be encrypted via SSL: connect to the server using the specified base protocol (POP3) over a connection secured by SSL.
- sslcertck causes fetchmail to strictly check the certificate sent by the server against a set of local trusted certificates specified by sslcertpath
- sslcertpath sets the directory fetchmail uses to look up local certificates. /etc/ssl/certs is the default directoy used by OpenSSL.
Since the username and password are stored in this file, the permissions need to change to make the file readable only by its owner. If this is not done, fetchmail will not even run:
chmod 600 ~/.fetchmailrc
To change or enable the forwarding protocol to POP3 in gmail, go to gmail settings, choose Forwarding and POP/IMAP. The option POP download has three points. One is "1.Status" to enable POP in three different ways as per your convenience, The second point is "2.When messages are accessed with POP" which allows you to choose what to do with the messages once there are downloaded to you local machine using POP. Point three is a link to instructions for set ups with other mailers.
- procmail: setting up the Mail Delivery Agent (MDA)
-
(From procmail man pages).This program is an autonomous mail processor. When invoked, it first sets some environment variables to default values, reads the mail message from stdin until an EOF, separates the body from the header, and then, if no command line arguments are present, it looks for a file named $HOME/.procmailrc. According to the processing recipes in this file, the mail message that just arrived gets distributed into the right folder (and more). If no configuration file is found, or processing of this file falls off the end, procmail will store the mail in the default system mailbox.
In other words, the program receives the mail and sorts it according to the information in .procmailrc file. These recipes can be quite complicated but are also very flexible.
To install:
sudo apt-get install procmail
Procmail needs a default file to store the mail (in case the message does not fit any of the recipes). This file is usually known as the mailbox or the inbox, and is stored in the $MAIL environment variable. This variable is not set in Ubuntu. Set it by adding these lines to your ~/.bashrc file:
# Sets the Mail Environment Variable
MAIL=/var/spool/mail/john && export MAIL
If you want to store different mail sources (if you have different accounts in gmail, yahoo, etc) you have to have the appropraite recipes for this purposes. (I need to work this out if I want different gmail accounts to be stored in different places, for example.) As mentioned before, these recipes are stored in the .procmailrc configuration file in your home directory. Note that is compulsory to have such file. A simple .procmailrc is given below:
# Environment variable assignments PATH=/bin:/usr/bin:/usr/local/bin VERBOSE=off # Turn on for verbose log MAILDIR=$HOME/Mail # Where procmail delivers recipes LOGFILE=$HOME/.procmaillog # Keep a log for troubleshooting. # Recipes :0: * ^TOmutt-user mutt
This includes a very simple sorting recipe: it intercepts everything addressed to “mutt-user” and directs it to $HOME/Mail/mutt. (This is the mutt-user mailing list which I would advise all new mutt users to join.) The rest of the mail is redirected to the default inbox. I like to keep a log per day so I change the logfile setting to the following:
LOGFILE=$MAILDIR/.procmaillog.`date +%F`
This is usefull when the volume of email is high or you need to look at specific days.
The configuration file assumes that $HOME/Mail exits, Create it now if you haven’t:
mkdir $HOME/Mail
Sending mail: setting up the Mail Sending Agent (MSA)
As of now, we can retrieve the mail, sort it as we wish in different mailboxes or files. Now we need to send mail from the local machine using the gmail server. We will use msmtp for this purpose.
msmtp is a SMTP (Simple Mail Transfer Protocol). Here is a full installation documentation with lots of details. In the page you are presented with the abridged version.
Simple Mail Transfer Protocol (SMTP) is an Internet standard for electronic mail (e-mail) transmission across Internet Protocol (IP) networks.
SMTP uses TCP port 25. The protocol for new submissions (MSA) is effectively the same as SMTP, but it uses port 587 instead.
http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol
A MSA is a mail submission agent that receives electronic mail messages from a mail user agent (MUA), in our case, mutt.
To install msmtp:
sudo apt-get install msmtp
The configuration file should include the following for a gmail server account:
# Set default values for all following accounts defaults tls on tls_starttls on tls_trust_file /etc/ssl/certs/ca-certificates.crt logfile ~/.msmtp.log # Only one account for now... gmail account default host smtp.gmail.com port 587 auth on from "john.example@gmail.com" user "john.example" password "rover"
The log file should exist, so:
touch ~/.msmtp.log
and after having created the configuration file, its permissions should be changed:
chmod 0600 ~/.msmtprc
The meaning of the configuration file is as follows:
- defaults: set configuration commads to all account definitions in this file.
- tls on: this command enables TLS (also known as SSL) encrypted connections to the SMTP server. The next two lines is required for this one to work.
- tls_starttls on: TLS encryption is activated using the STARTTLS SMTP command.
- tls_trust_file /etc/ssl/certs/ca-certificates.crt: this command activates strict server certificate verification. The filename must be the absolute path name of a file in PEM format containing one or more certificates of trusted Certification Authorities (CAs).
- auth on: this command enables SMTP authentication; msmtp chooses the best method available.
- logfile ~/.msmtp.log: when logging is enabled by choosing a log file, msmtp will append one line to the log file for each mail it tries to send via the account that this log file was chosen for.
- account default: information about SMTP servers is organized in accounts. Each account describes one SMTP server: host name, authentication settings, TLS settings, and so on. Each configuration file can define multiple accounts. This line starts a new account definition, this time is the default.
- host smtp.gmail.com: host is the SMTP server capable of sending email. The argument can be a name or an address and is mandatory for each account.
- port 587: specifies the port that the SMTP server listens on.
- from "john.example@gmail.com": sets the envelope-from address.
- user "john.example": sets the user name for SMTP authentication. An empty argument unsets the user name. Authentication must be activated with the auth command.
- password "rover": sets the password for SMTP authentication. As with user, authentication must be activated with the auth command.
The next lines describe the settings of one account:
Reading mail: Setting up the console Mail User Agent (MUA)
And finally, mutt.
To download:
sudo apt-get install mutt
Mutt’s configuration file is called .muttrc and as usual it must be located in the home directory or in the /etc directory for global definition. For more information in understanding this file, I encourage you to browse its man pages: man muttrc.
A simplified beginner’s version of the configuration file is given below:
#======================================================#
# Set up for fetchmail
set realname = "John The Big Brother"
set from = john.example@gmail.com
set use_from = yes
set envelope_from ="yes"
set sendmail="/usr/bin/msmtp"
# If not set in environment variables:
set spoolfile = /var/spool/mail/john
#======================================================#
# Folders
set folder="~/Mail" # Mailboxes in here
set record="+sent" # where to store sent messages
set postponed="+postponed" # where to store draft messages
set move=no # Don't move mail from the spool.
#======================================================#
# Watch these mailboxes for new mail:
mailboxes ! +Fetchmail +slrn +mutt
set sort_browser=alpha # Sort mailboxes by alpha(bet)
#======================================================#
# Order of headers and what to show
hdr_order Date: From: User-Agent: X-Mailer \
To: Cc: Reply-To: Subject:
ignore *
unignore Date: From: User-Agent: X-Mailer \
To: Cc: Reply-To: Subject:
#======================================================#
# which editor do you want to use? And how?
# vim of course!
set editor="vim -c 'set tw=70 et' '+/^$' "
set edit_headers # See the headers when editing
#======================================================#
# Aliases
set sort_alias=alias # sort aliases in alpha order by alias name
#======================================================#
# Odds and ends
set markers # mark wrapped lines of text in the pager with a +
set smart_wrap # Don't wrap mid-word
set pager_context=5 # Retain 5 lines of previous page when scrolling.
set status_on_top # Status bar on top.
push # Shows mutt version at startup
#======================================================#
Now you can view your mail. Well not quite. The mail has to be fetched first.
fetchmail -v
will bring all of your mail, actually in chuncks of 300 messages. If you want to delete all of the messages from the gmail server, be sure to change the setting in your gmail account before fetching the mail for the first time. After the mail is fetched, mutt will show you local mailbox.
Once in mutt, you can fetch mail by typing ! to open a shell prompt, and typing fetchmail -v. Alterantively, you can include the following macro in your .muttrc file and invoke fetchmail by typing I:
macro index,pager I ' fetchmail -v'
Et voilá!
Super Spread Sheet S³ 