Using an SSL certificate with OTRS

Reasons for using HTTPS with OTRS

… well really, there is NO reason NOT to.browserbar

  • When you’re NOT using HTTPS you’re exposing your password in plain text when logging in. Even if you’d be only using OTRS on your corporate network, can you trust everything and everyone on your network?
  • You’re having data about your customers in OTRS. You should make sure this data is not exposed or leaked; so you should be using HTTPS when accessing OTRS from your web browser.
  • Apart from the encryption; there is also the aspect of MITM-attacks on your web server. If you have a certificate on your web server you can be fairly sure the server you’re looking at, and the password for you’re typing your password into, is actually YOUR server and not some guy using the same hotel wifi poisoning DNS.
  • Even if you would not care about security, using SSL and HTTPS is a pre-requisite for using mod_spdy with Apache, which brings many of the upcoming HTTP/2.0 features to the Apache webserver and modern web browsers such as Google Chrome and Mozilla Firefox. This makes OTRS faster!

Self-signed versus purchased

The ‘easiest’  way is to slap a self-signed certificate on your web server. This will lead to the well known nasty warning signs on your website. This can be perfectly acceptable if your OTRS system is just used by you and your co-worker and no customers log in to it. And after the first login, even the MITM-prevention will work because your web browser would complain if the certificate would change.

You can also purchase a certificate at a vendor such as Verisign. This does money but because the certificate authority is trusted by your web browser, the nasty warnings are gone. And certificates are not so expensive as they used to be, really.

Wildcard certificates, which you can use on many subdomains such as support.example.com, sales.example.com and www.example.com are still pretty pricey at a hundred euros or more per year, and Extended Validation certificates are also still expensive, but a simple certificate for www.example.com and example.com would be not even 10 euros per year at providers as NameCheap (full disclosure: affiliate link. I know this is never going to make any money).

Extended Validation and Organisation Validation levels only differ in the amount of work you have to do to make sure you prove to actually be this company before you can get a certificate, and do not make the encryption any stronger or weaker.

Configuring apache

So you’ve purchased or generated your certificate for your web server, great! Now you’ll have to install it, that will be difficult, right? WRONG! It’s super easy. Let me show you how.

You’d need to activate SSL. On Debian or Ubuntu, it’s pre-installed with Apache and you’d just need to enable it, like this:

sudo a2enmod ssl
sudo a2ensite default-ssl

On CentOS or Red Hat, it would be:

sudo yum install -y mod_ssl

now in the SSL configuration file, /etc/apache2/sites-enabled/default-ssl on Ubuntu/Debian and /etc/httpd/conf.d/ssl.conf on RHEL/CentOS, you’d add this for your purchased certificate:

SSLCertificateFile    /etc/apache2/certs/mydomain.crt
SSLCertificateKeyFile /etc/apache2/certs/mydomain.key
SSLCertificateChainFile /etc/apache2/certs/intermediate-rapidssl.crt

now you simply restart your web server, on Debian/Ubuntu:

sudo service apache2 restart

or on RHEL/CentOS:

sudo service httpd restart

Now you can log in OTRS using the ‘https://’ prefix. It’s THAT easy!

It would still be great if you can forward existing links into http:// URLs that might already exist in notifications and such to https:// – but that’s easy. Just add this to the HTTP virtualhost configuration:

RewriteEngine On
RewriteCond %{HTTPS} off 
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Verification of your certificate

The people at Qualys SSL Labs have a nice service which allows you to verify if your certificate was set up properly, you can find it here.

Configuring OTRS

In OTRS the configuration change is also quite easy. Just navigate to Admin > SysConfig > Framework > Core and set ‘HttpType’ to https.

httpsThis will make the links in outgoing email notifications contain ‘https’ and it will also set the secure attribute on your login cookies.

Conclusion

With just a couple of minutes of effort you can put a self-signed certificate on your system. If you’d throw in some ten euros you can grab a commercial certificate which increases the user experience. It’s really a no-brainer, and it should be part of every proper OTRS installation! I hope this article helped you setting it up.

Using strong password hashing with OTRS

In the last years we saw several cases of websites getting hacked and having the passwords of their users leaked. This happened for instance at LinkedIn in 2012. The user passwords were not stored in the database in plain text but hashed with SHA-1 and this algorithm is pretty weak nowadays meaning if a hacker would be able to get the password hashes, it’s possible to expose the original passwords.

As a result of this, OTRS added support for hashing passwords for agents and customers using bcrypt starting OTRS 3.3.  Since it depends on an external module, it is not enabled by default! This means that, if you want to preventing the nasty event your systems would be compromised your users’ passwords are out there in the open, you’d better enable it manually. Luckily this is not hard.

Details of the hashing – skip if you want!

As all modern systems, OTRS uses two techniques called ‘salting’ and ‘hashing’ to store passwords. This means that OTRS does not store the users’ password in the database, so the passwords can’t be stolen directly. Instead, it stores the hashed value, and this is supposed to be a one way hash. This means, if your password would be ‘secret123’ (which would obviously be a terrible example of a password) OTRS takes this, along with a random value, the salt, and calculates the hash. It then stores the salt and the hash in the database. Salting is used to prevent a situation where two users with identical passwords would end up having the same hashes and thus preventing the use of pre-constructed rainbow tables. If the user tries to log in, he supplies his password in clear text. After that, the salt is read from the database and together with user input a hash is calculated and this hash is compared with the value in the database. If these are equal, access is allowed.

If the hashing operation is ‘expensive’ enough, i.e. it costs a relative lot of time to perform the hashing, crackers can not easily find out your passwords would they obtain a copy of your database.

If you don’t do anything by default, OTRS uses SHA-256 hashing which is not as bad as MD5, but it’s not industry best practice anymore either. Using special hardware you can calculate SHA-256 hashes pretty fast. bcrypt is really much stronger; especially because OTRS uses a better salting mechanism with bcrypt than what it uses with SHA-256. You don’t need to know any of these implementation details, just follow the simple steps below!

Installing the module

It might just be you already have the needed module installed. Log into your OTRS machine and run bin/otrs.CheckModules.pl

$ perl bin/otrs.CheckModules.pl 
  o Apache2::Reload..................ok (v0.12)
  o Archive::Tar.....................ok (v1.76)
  o Archive::Zip.....................ok (v1.30)
  o Crypt::Eksblowfish::Bcrypt.......ok (v0.008)
  ....

If the Crypt::Eksblowfish::Bcrypt module is installed, the output would look like above. If not, you’d need to install it. You can best do that from your package manager. On Red Hat or CentOS this is as simple as

sudo yum install "perl(Crypt::Eksblowfish::Bcrypt)"

although if you do not yet have the EPEL repository enabled, you should do that first!

sudo yum install epel-release

On Debian or Ubuntu, you would use

sudo apt-get install libcrypt-eksblowfish-perl

Now you can run the bin/otrs.CheckModules.pl script again to verify the installation.

Configuring OTRS

In OTRS, you should change the password hashing mechanism for both customers and agents. For customers, you can simply navigate to Admin > SysConfig > Frontend::Customer::Auth and change the CryptType setting to ‘bcrypt’.

customer-bcrypt

For agents, there is no such convenient configuration setting available and you should modify the Kernel/Config.pm file with a text editor and add this option:

$Self->{'AuthModule::DB::CryptType'} = 'bcrypt';

After this, the changes are effective immediately for newly saved passwords. Passwords that were already set are NOT automatically upgraded.

Verifying the change

If you’d want to make sure the new mechanism works, you can change your password and log out & in in OTRS, then check the log. You’ll see the hashing method used:
bcrypt-log

Since this configuration change does NOT update existing user passwords, it is best to make this change before you take OTRS in production.

Sorting mail using SpamAssassin in OTRS

0f3b3bf1Fighting spam is more difficult than ever and many people let other parties handle their spam filtering. This is just one of the reasons products like Office365 and Google Apps for Work have become so successful. After all, there is not much glamor in running your own mail server.

If you run your own mail server you’ll have to handle spam yourself. If you have OTRS on its own server and you’re receiving spam, this is very annoying as it might send out false notifications and wastes time.

Luckily, it is very easy to use SpamAssassin to filter out spam emails in order to make sure you only receive valid mails in OTRS. SpamAssassin is an Apache foundation open source software that is used very widely to combat spam.

Installing SpamAssassin on your system

You can simply install SpamAssassin from your linux package manager, on CentOS/RHEL based systems:

sudo yum install -y spamassassin

and on Debian- or Ubuntu-systems:

sudo apt-get install spamassassin

After you’ve installed the package you can test if it’s available by typing  /usr/bin/spamassassin –version in your console.

Updating rules

Of course spammers and spam-fighters are in a never-ending battle. New heuristics to determine spam messages are created often. On some systems the spamassassin rules can be out of date, run the sa-update command as root to update the rules. If you want you could run this as a cron job.

Setting up OTRS to use SpamAssassin

First, there are two options for handling spam. One is to delete incoming email marked as spam and one is to move it to a separate queue. The last option I would strongly recommend. If a message would falsely be marked as spam, it will be deleted from your system and you’d not know about it! It’s much better if the message is still available, at least for a small period of time, so you can restore it if needed.

So in order to make this happen, you can set up a special queue on the system where the message should be sorted into. If you make sure your agents only have read-permissions on this queue (and probably also move-into, but at least not read-write) they will not see the queue in the dashboard and in the queue view, they’ll not get notifications, but they are able to search through tickets in this queue.

In SysConfig, under Ticket > Core::PostMaster, there is a pre-defined filter called PostMaster::PreFilterModule###6-SpamAssassin. Just please make sure to fill in the queue you want under the Set: option, check the box, and save the SysConfig. Your setup is active now.

SpamAssSysConfig

Verifying your setup

If a spam message reaches your system, a nice short message is added to your system log and the ticket is created in the spam queue automatically!

spamfilter

How to install OTRS 4 on CentOS 7

Centos_full.svgIn this post I’m going to walk you through installing OTRS 4 on CentOS 7. The procedure will be very similar for Red Hat Enterprise Linux (RHEL) version 7 as this is binary compatible.

Please note that there are some differences between CentOS 6 and CentOS 7: it now ships with systemd and with firewalld so the instructions to install OTRS are pretty different.

Setting up your production server or migrating from one is something you don’t want to do every day. This means you better take a distribution that will receive security upgrades for a long time. This is why I would recommend CentOS version 7 over version 6 at this point in time.

Preparation: deactivation of SELinux

OTRS does not ship with a profile for SELinux. This means that you’ll have problems if you don’t turn it off. If you’re an advanced system administrator, you’d be able to create a profile for OTRS. This is beyond the scope of this post.

You can check the status of SELinux with the sestatus command:

[root@localhost ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Edit the file /etc/selinux/config and set SELINUX=permissive.  This will make sure after a reboot selinux will not be enabled.

Type setenforce Permissive to set the current SELinux status to ‘permissive’. I chose Permissive here, rather than disabled, because otherwise you might loose the security context on files and would you want to enable SELinux on some later point you’d need to re-label files which is difficult.

[root@localhost ~]# setenforce Permissive
[root@localhost ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Preparation: installation of a database

Of course you can use OTRS with a database that is on some central location in your setup. You can use OTRS with PostgreSQL or MySQL, or even with Oracle if you need to. In this example, I’m going to assume that you’ll use a database installed on the OTRS machine itself, which is the most common setup and recommended for all except very big installations.

The most widely used database for OTRS is MySQL. In CentOS 7, MySQL Server is no longer available; the fork MariaDB is available and you can use that as a drop-in replacement.

If you want to install MySQL instead of MariaDB, this is no problem; the MySQL project has provided a yum repository that you can use.

Otherwise, if you’d want to install MariaDB, just use these commands:

yum install -y mariadb-server
echo -e "[server]\nmax_allowed_packet=20M\nquery_cache_size=32M" > /etc/my.cnf.d/otrs.cnf
systemctl enable mariadb.service
systemctl start mariadb.service

The echo command is used to create a small configuration file called /etc/my.cnf.d/otrs.cnf which contains specific settings in order to make OTRS happy. The contents of this file is:

[server]
max_allowed_packet=20M
query_cache_size=32M

Get and install OTRS

Now you can get and install the OTRS software itself. You can find RPM installation files on the web server of OTRS. For the current version the install command is:

yum -y install http://ftp.otrs.org/pub/otrs/RPMS/rhel/7/otrs-4.0.2-01.noarch.rpm

Please note this will install loads of dependencies so it might take a brief while.

Install additional dependencies

Now you can install additional dependencies from EPEL, the enterprise quality package repository maintained by the Fedora project. Note that this step is kind of important as it also will bring you mod_perl which is really needed to have proper performance of the web server!

yum -y install epel-release
yum install -y mod_perl "perl(Crypt::Eksblowfish::Bcrypt)" "perl(JSON::XS)" "perl(GD::Text)" "perl(Encode::HanExtra)" "perl(GD::Graph)" "perl(Mail::IMAPClient)" "perl(PDF::API2)" "perl(Text::CSV_XS)" "perl(YAML::XS)"

Configure firewall and start Apache

Now you can start the Apache web server.  You should also add a rule to the firewall to allow access to the web server. CentOS 7 ships with firewalld, a new generation firewall that allows you to make these changes pretty easily.

You might want to remove the ‘welcome page’ of CentOS as it is kind of annoying.

rm /etc/httpd/conf.d/welcome.conf
systemctl enable httpd.service
systemctl start httpd.service
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --reload

At this point you can continue using the Web Installer as explained in the OTRS documentation. As database you should choose ‘MySQL’ , also if you’re using MariaDB, because they really are forks and in this regard compatible. The database administrative password is empty. Note that this is not a security risk per se as the database only listens on localhost, so you can only access it from the local machine.

Of course there are many more tasks you should perform before considering your OTRS installation ready, but this is a nice quick start into setting up OTRS on a very popular, long-supported server OS.

OTRS Survey Export

A while ago I noticed someone in the OTRS community wrote a very simple but nice and useful plugin that allows you to export survey results to CSV files. I fixed some small issues with it with a pull request, about nine months ago.

SurveyExportLast week someone contacted me because he upgraded his OTRS system to version 3.3 and the package would no longer work. I ported the module to the 3.3 framework now, and it’s available for download from here: https://download.huntingbears.nl/SurveyExport-0.1.1.opm

Of course I also contributed my changed back to the original author.

Making it work on OTRS 3.3 was relatively easy. The original version had a rather nasty hack to display the extra link in the Survey Zoom screen, where basically the whole template file and frontend module were duplicated in order to add the extra option. In the last version of the Survey module, the screen now has a regular MenuModule registration, as in other parts of OTRS, so now the menu option could be added by simply adding some XML to the configuration file. I love it when porting a feature to a new framework version exists mostly of deleting stuff!

Update: Upon request I now have a version for the OTRS 5 framework available here: https://download.huntingbears.nl/SurveyExport-0.3.1.opm

What Operating System should I choose for deploying OTRS?

Many times I heard the question what OS would be best to install or setup OTRS on. In this post I’ll give an answer.

Choose what fits you best

Typically, the answer is simple: it should be the operating system you (or your organization) is most comfortable with. If your organization is full of debian-lovers, please do install OTRS on Debian. If your organization is all about Microsoft Windows, by all means, go and use Windows.  Short historical note: the Windows Installer for OTRS used to be kind of cumbersome and it was really not easy to upgrade OTRS on Windows, but this is no longer the case since the 3.0 installer was introduced in summer 2013. Continue reading What Operating System should I choose for deploying OTRS?

No mod_perl in RHEL 7 and CentOS 7

I was reading up on the documentation for the newly released RHEL 7 beta the other night. The section Removed Packages showed that RHEL 7 no longer includes mod_perl as it is ‘Incompatible with HTTP 2.4‘ and mod_fcgid is recommended as its replacement.

200px-RedHat.svgWith ‘HTTP 2.4‘ the Red Hat folks actually mean the Apache Webserver 2.4, which package is called ‘httpd’ in the RHEL repositories.

While it is true that there still is no ‘official’ mod_perl release that has support for Apache httpd 2.4, there is a mod_perl branch with support for 2.4 and this has been used by the Fedora project in Fedora 19 and later. Since RHEL 7 would be based on Fedora 19 I find it kind if weird to read there will be no mod_perl in RHEL 7: there has not been a single release of Fedora without mod_perl!

Furthermore, mod_fcgid is by no means a drop-in replacement for mod_perl; mod_perl allows very deep integration with the Apache web server. If you’d just use mod_perl as a way to make your CGI scripts run faster, then you can use it as an alternative though.

Most ‘modern’  perl applications have moved off from mod_perl to plack, which is a more standardized way to integrate your application with your web server, which has lots of deployment options. So if you have moved your application to plack in the last years, you’d be good.

OTRS is typically deployed on mod_perl; and last summer (when I still worked at OTRS Group) we created a wrapper around our frontends which allows you to deploy on Plack. This is still marked as ‘experimental’, and it’s not documented, but it does work. I’ve been running in Plack mode on my ‘personal’ OTRS system since August last year. I’ll want to post a how-to on deployment soon.

I find it a bit disheartening to see that there has been no real activity on mod_perl and there is still no ‘official’  support for mod_perl on Apache 2.4; but also I can’t really understand why there is no mod_perl in RHEL7 as it is readily available in Fedora. Of course, I do consider mod_perl as ‘old’ technology, Plack is really the more modern choice. That said, it’s not always trivial to port your application to Plack. But if you haven’t done so already, you really should start now!

UPDATE: luckily, you can now install mod_perl on CentOS or RHEL 7 via EPEL.

EPEL is the high-quality RPM collection for RHEL and CentOS linux, maintened by the Fedora project. You can install mod_perl like this:

yum install -y epel-release
yum install -y mod_perl