New tar, paxheaders and installing from CPAN

I work most often with CentOS or Red Hat Enterprise Linux (RHEL) platforms, which are widely deployed in enterprise environments. Last year version 7 was released, and while it’s great it has some features like systemd that scares some sysadmins off. This means the version most widely used is version 6, which comes with perl 5.10.1.

Recently, I wanted to install SOAP::Lite using CPAN because I wanted the latest version – it’s pretty simple to install version 0.710 from repositories using

sudo yum install perl-SOAP-Lite

But anyway I wanted a new feature so I decided to install from CPAN.

I got this error message:

$ cpan SOAP::Lite
CPAN: Archive::Tar loaded ok (v1.58)
PaxHeader/SOAP-Lite-1.13
SOAP-Lite-1.13/
SOAP-Lite-1.13/PaxHeader/bin
....
CPAN: File::Temp loaded ok (v0.22)
CPAN: Time::HiRes loaded ok (v1.9721)
Package seems to come without Makefile.PL.
  (The test -f "/home/vagrant/.cpan/build/PHRED-wxPcgc/Makefile.PL" returned false.)
  Writing one on our own (setting NAME to SOAPLite)

  CPAN.pm: Going to build P/PH/PHRED/SOAP-Lite-1.13.tar.gz

Checking if your kit is complete...
Looks good
Warning: prerequisite Class::Inspector 0 not found.
Warning: prerequisite Crypt::SSLeay 0 not found.
Warning: prerequisite IO::SessionData 1.03 not found.
Warning: prerequisite IO::Socket::SSL 0 not found.
Warning: prerequisite Task::Weaken 0 not found.
Bareword found where operator expected at ./Makefile.PL line 1, near "17 gid"
    (Missing operator before gid?)
Number found where operator expected at ./Makefile.PL line 2, near "18"
    (Missing semicolon on previous line?)
....

That looks strange, right? Especially since version 1.12 of the SOAP::Lite module installs perfectly fine. Do you notice the message “Package seems to come without Makefile.PL”?

The problem here is that the archive is built with a ‘newer’ tar, using the ‘extended headers’ which are described in POSIX.1-2001. Yeah that’s right – these extended headers were standardized in 2001, have not found their way into RHEL6, but are available in newer tars on OS X and Linux now. The only problem is, they aren’t really backwards compatible.

The cpan on my CentOS machine expands the archive, finds all kind of headers it does not know what to do with, and creates separate directories called PaxHeaders.

It puts files in these directories with the extended headers; one is called Makefile.PL and cpan tries to execute ALL Makefile.PLs it finds, however the file containing the extended headers is not valid perl and that is the actual error you see above.

The directory structure created looks like below, you can inspect it by typing ‘look SOAP::Lite’ in your cpan client on RHEL6:

Makefile.PL  <-- this is actually created by cpan
PaxHeader
  |-- SOAP-Lite-1.13  <-- this contains extended headers
SOAP-Lite-1.13
  |-- bin
  |-- lib
  |-- Makefile.PL  <-- actual Makefile.PL
  | ...
  |-- PaxHeader  <-- all files below only contain extended headers
        |-- bin
        |-- lib
        |-- Makefile.PL # this is not valid perl!

How widespread is this issue?

Well of course if it’s ONLY SOAP::Lite that is affected by this, it would not be so bad. I wrote a little script and let it loose on my offline CPAN mirror, ran it on all dists from authors starting with ‘A’ and found hundreds of tarballs that had this issue. I did not bother running it on the complete CPAN but I guess you can say a pretty considerable portion of CPAN would have the ‘new’ tar archives.

What can you do as a CPAN author?

I asked the author of SOAP::Lite, Fred Moyer, if he did anything special or changed anything to his setup between the 1.12 and 1.13 release. He said he did not think anything particular should have changed. He just used his OS X laptop and build chain and out came the extended headers.

Tux wrote on perlmonks about this, he ran into the issue on his openSUSE laptop, and has a nice solution: add some tarflags to Makefile.PL like below

use ExtUtils::MakeMaker;
WriteMakefile (
    ABSTRACT     => "Comma-Separated Values manipulation routines",
    VERSION_FROM => "CSV_XS.pm",
    macro        => { TARFLAGS   => "--format=ustar -c -v -f",
                    },
    );

Workaround for cpan users

There are actually two possible workarounds for this:

 

Upgrading Archive::Tar

The cpan client extracts the tarballs using the module Archive::Tar. If you’d upgrade Archive::Tar using cpan, it will handle the archives properly and you can install SOAP::Lite properly. Of course this would only work as long as the author of Archive::Tar does not upload its module with extended headers.

Upgrading cpan

If you’d upgrade your cpan module, it will start using /usr/bin/tar instead of the perl module Archive::Tar for the extracting. /usr/bin/tar on RHEL 6 actually ignores the extended headers and will complain loudly about them, but it DOES work. Upgrading cpan also only will work as long as the new tarball itself is not created with extended headers!

Resolution upstream

Being a good citizen I thought it might be helpful to file this issue at Red Hat, it’s #1184194  – it was very swiftly handled by backporting a fix from Archive::Tar so it ignores PaxHeader files; that’s great! It’s in QA now for a little over three weeks. I’m not sure exactly about how this would progress, but hopefully it’ll be released and you can just yum upgrade and have this issue done with soon!

Using an SSL certificate with OTRS

Reasons for using HTTPS with OTRS

… well really, there is NO reason NOT to.browserbar

  • When you’re NOT using HTTPS you’re exposing your password in plain text when logging in. Even if you’d be only using OTRS on your corporate network, can you trust everything and everyone on your network?
  • You’re having data about your customers in OTRS. You should make sure this data is not exposed or leaked; so you should be using HTTPS when accessing OTRS from your web browser.
  • Apart from the encryption; there is also the aspect of MITM-attacks on your web server. If you have a certificate on your web server you can be fairly sure the server you’re looking at, and the password for you’re typing your password into, is actually YOUR server and not some guy using the same hotel wifi poisoning DNS.
  • Even if you would not care about security, using SSL and HTTPS is a pre-requisite for using mod_spdy with Apache, which brings many of the upcoming HTTP/2.0 features to the Apache webserver and modern web browsers such as Google Chrome and Mozilla Firefox. This makes OTRS faster!

Self-signed versus purchased

The ‘easiest’  way is to slap a self-signed certificate on your web server. This will lead to the well known nasty warning signs on your website. This can be perfectly acceptable if your OTRS system is just used by you and your co-worker and no customers log in to it. And after the first login, even the MITM-prevention will work because your web browser would complain if the certificate would change.

You can also purchase a certificate at a vendor such as Verisign. This does money but because the certificate authority is trusted by your web browser, the nasty warnings are gone. And certificates are not so expensive as they used to be, really.

Wildcard certificates, which you can use on many subdomains such as support.example.com, sales.example.com and www.example.com are still pretty pricey at a hundred euros or more per year, and Extended Validation certificates are also still expensive, but a simple certificate for www.example.com and example.com would be not even 10 euros per year at providers as NameCheap (full disclosure: affiliate link. I know this is never going to make any money).

Extended Validation and Organisation Validation levels only differ in the amount of work you have to do to make sure you prove to actually be this company before you can get a certificate, and do not make the encryption any stronger or weaker.

Configuring apache

So you’ve purchased or generated your certificate for your web server, great! Now you’ll have to install it, that will be difficult, right? WRONG! It’s super easy. Let me show you how.

You’d need to activate SSL. On Debian or Ubuntu, it’s pre-installed with Apache and you’d just need to enable it, like this:

sudo a2enmod ssl
sudo a2ensite default-ssl

On CentOS or Red Hat, it would be:

sudo yum install -y mod_ssl

now in the SSL configuration file, /etc/apache2/sites-enabled/default-ssl on Ubuntu/Debian and /etc/httpd/conf.d/ssl.conf on RHEL/CentOS, you’d add this for your purchased certificate:

SSLCertificateFile    /etc/apache2/certs/mydomain.crt
SSLCertificateKeyFile /etc/apache2/certs/mydomain.key
SSLCertificateChainFile /etc/apache2/certs/intermediate-rapidssl.crt

now you simply restart your web server, on Debian/Ubuntu:

sudo service apache2 restart

or on RHEL/CentOS:

sudo service httpd restart

Now you can log in OTRS using the ‘https://’ prefix. It’s THAT easy!

It would still be great if you can forward existing links into http:// URLs that might already exist in notifications and such to https:// – but that’s easy. Just add this to the HTTP virtualhost configuration:

RewriteEngine On
RewriteCond %{HTTPS} off 
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Verification of your certificate

The people at Qualys SSL Labs have a nice service which allows you to verify if your certificate was set up properly, you can find it here.

Configuring OTRS

In OTRS the configuration change is also quite easy. Just navigate to Admin > SysConfig > Framework > Core and set ‘HttpType’ to https.

httpsThis will make the links in outgoing email notifications contain ‘https’ and it will also set the secure attribute on your login cookies.

Conclusion

With just a couple of minutes of effort you can put a self-signed certificate on your system. If you’d throw in some ten euros you can grab a commercial certificate which increases the user experience. It’s really a no-brainer, and it should be part of every proper OTRS installation! I hope this article helped you setting it up.

How to install OTRS 4 on CentOS 7

Centos_full.svgIn this post I’m going to walk you through installing OTRS 4 on CentOS 7. The procedure will be very similar for Red Hat Enterprise Linux (RHEL) version 7 as this is binary compatible.

Please note that there are some differences between CentOS 6 and CentOS 7: it now ships with systemd and with firewalld so the instructions to install OTRS are pretty different.

Setting up your production server or migrating from one is something you don’t want to do every day. This means you better take a distribution that will receive security upgrades for a long time. This is why I would recommend CentOS version 7 over version 6 at this point in time.

Preparation: deactivation of SELinux

OTRS does not ship with a profile for SELinux. This means that you’ll have problems if you don’t turn it off. If you’re an advanced system administrator, you’d be able to create a profile for OTRS. This is beyond the scope of this post.

You can check the status of SELinux with the sestatus command:

[root@localhost ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Edit the file /etc/selinux/config and set SELINUX=permissive.  This will make sure after a reboot selinux will not be enabled.

Type setenforce Permissive to set the current SELinux status to ‘permissive’. I chose Permissive here, rather than disabled, because otherwise you might loose the security context on files and would you want to enable SELinux on some later point you’d need to re-label files which is difficult.

[root@localhost ~]# setenforce Permissive
[root@localhost ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Preparation: installation of a database

Of course you can use OTRS with a database that is on some central location in your setup. You can use OTRS with PostgreSQL or MySQL, or even with Oracle if you need to. In this example, I’m going to assume that you’ll use a database installed on the OTRS machine itself, which is the most common setup and recommended for all except very big installations.

The most widely used database for OTRS is MySQL. In CentOS 7, MySQL Server is no longer available; the fork MariaDB is available and you can use that as a drop-in replacement.

If you want to install MySQL instead of MariaDB, this is no problem; the MySQL project has provided a yum repository that you can use.

Otherwise, if you’d want to install MariaDB, just use these commands:

yum install -y mariadb-server
echo -e "[server]\nmax_allowed_packet=20M\nquery_cache_size=32M" > /etc/my.cnf.d/otrs.cnf
systemctl enable mariadb.service
systemctl start mariadb.service

The echo command is used to create a small configuration file called /etc/my.cnf.d/otrs.cnf which contains specific settings in order to make OTRS happy. The contents of this file is:

[server]
max_allowed_packet=20M
query_cache_size=32M

Get and install OTRS

Now you can get and install the OTRS software itself. You can find RPM installation files on the web server of OTRS. For the current version the install command is:

yum -y install http://ftp.otrs.org/pub/otrs/RPMS/rhel/7/otrs-4.0.2-01.noarch.rpm

Please note this will install loads of dependencies so it might take a brief while.

Install additional dependencies

Now you can install additional dependencies from EPEL, the enterprise quality package repository maintained by the Fedora project. Note that this step is kind of important as it also will bring you mod_perl which is really needed to have proper performance of the web server!

yum -y install epel-release
yum install -y mod_perl "perl(Crypt::Eksblowfish::Bcrypt)" "perl(JSON::XS)" "perl(GD::Text)" "perl(Encode::HanExtra)" "perl(GD::Graph)" "perl(Mail::IMAPClient)" "perl(PDF::API2)" "perl(Text::CSV_XS)" "perl(YAML::XS)"

Configure firewall and start Apache

Now you can start the Apache web server.  You should also add a rule to the firewall to allow access to the web server. CentOS 7 ships with firewalld, a new generation firewall that allows you to make these changes pretty easily.

You might want to remove the ‘welcome page’ of CentOS as it is kind of annoying.

rm /etc/httpd/conf.d/welcome.conf
systemctl enable httpd.service
systemctl start httpd.service
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --reload

At this point you can continue using the Web Installer as explained in the OTRS documentation. As database you should choose ‘MySQL’ , also if you’re using MariaDB, because they really are forks and in this regard compatible. The database administrative password is empty. Note that this is not a security risk per se as the database only listens on localhost, so you can only access it from the local machine.

Of course there are many more tasks you should perform before considering your OTRS installation ready, but this is a nice quick start into setting up OTRS on a very popular, long-supported server OS.

No mod_perl in RHEL 7 and CentOS 7

I was reading up on the documentation for the newly released RHEL 7 beta the other night. The section Removed Packages showed that RHEL 7 no longer includes mod_perl as it is ‘Incompatible with HTTP 2.4‘ and mod_fcgid is recommended as its replacement.

200px-RedHat.svgWith ‘HTTP 2.4‘ the Red Hat folks actually mean the Apache Webserver 2.4, which package is called ‘httpd’ in the RHEL repositories.

While it is true that there still is no ‘official’ mod_perl release that has support for Apache httpd 2.4, there is a mod_perl branch with support for 2.4 and this has been used by the Fedora project in Fedora 19 and later. Since RHEL 7 would be based on Fedora 19 I find it kind if weird to read there will be no mod_perl in RHEL 7: there has not been a single release of Fedora without mod_perl!

Furthermore, mod_fcgid is by no means a drop-in replacement for mod_perl; mod_perl allows very deep integration with the Apache web server. If you’d just use mod_perl as a way to make your CGI scripts run faster, then you can use it as an alternative though.

Most ‘modern’  perl applications have moved off from mod_perl to plack, which is a more standardized way to integrate your application with your web server, which has lots of deployment options. So if you have moved your application to plack in the last years, you’d be good.

OTRS is typically deployed on mod_perl; and last summer (when I still worked at OTRS Group) we created a wrapper around our frontends which allows you to deploy on Plack. This is still marked as ‘experimental’, and it’s not documented, but it does work. I’ve been running in Plack mode on my ‘personal’ OTRS system since August last year. I’ll want to post a how-to on deployment soon.

I find it a bit disheartening to see that there has been no real activity on mod_perl and there is still no ‘official’  support for mod_perl on Apache 2.4; but also I can’t really understand why there is no mod_perl in RHEL7 as it is readily available in Fedora. Of course, I do consider mod_perl as ‘old’ technology, Plack is really the more modern choice. That said, it’s not always trivial to port your application to Plack. But if you haven’t done so already, you really should start now!

UPDATE: luckily, you can now install mod_perl on CentOS or RHEL 7 via EPEL.

EPEL is the high-quality RPM collection for RHEL and CentOS linux, maintened by the Fedora project. You can install mod_perl like this:

yum install -y epel-release
yum install -y mod_perl

Vagrant base image for CentOS 6.5

Vagrant is a very nice system for automating and standardizing your development environments. The standard documentation is geared towards Ubuntu but at $work we standardize on CentOS and RHEL 6.

I created a base image for Vagrant based on CentOS 6.5 Minimal Install.Vagrant

Features:

  • Default vagrant ‘insecure’ key
  • root password is ‘vagrant
  • Added EPEL repository
  • Installed Virtualbox Guest Additions and dkms
  • Disabled iptables
  • Set selinux to ‘permissive’

How to start this image locally using Vagrant:

 $ vagrant box add centos6 http://vagrant.huntingbears.nl/vagrant-centos-6.5-x86_64.box
 $ vagrant init centos6
 $ vagrant up

Enjoy!