CentOS 7: Get loggerhead to work

I have been trying to get loggerhead to work on a CentOS 7 system. I do have a functional loggerhead on a CentOS 6 system and an older Mandriva Enterprise Server 5.2 system. It was trivial to get loggerhead to work on these other systems, but CentOS7 is proving to be difficult. I have been testing serve-branches on localhost using `lynx http://localhost:8080`.

CentOS 6 installation has the following (?relevant?) packages:

$ rpm -qa | egrep '(^bzr)|(simpletal)|(paste)|(loggerhead)|(flup)|(simplejson)'
python-paste-deploy-1.3.3-2.1.el6.noarch (CentOS)
bzrtools-2.1.0-1.el6.i686 (EPEL)
bzr-2.1.1-2.el6.i686 (CentOS)
python-paste-1.7.4-2.el6.noarch (CentOS)
loggerhead-1.18.1-1.el6.noarch (EPEL)
python-paste-script-1.7.3-5.el6_3.noarch (CentOS)
python-simpletal-4.2-4.el6.noarch (EPEL)
python-simplejson-2.0.9-3.1.el6.i686 (CentOS)

These were all provided by either CentOS or EPEL packages as indicated.

My testbed branch was created like this:

$ mkdir ~/bzr/project
$ cd ~/bzr/project
$ bzr checkout bzr+ssh://host/path/to/repo blah

Getting loggerhead to work was as simple as:

$ serve-branches ~/bzr/project/blah
$ lynx http://localhost:8080/

Note that this was done with SELinux in enforcing mode and I didn't have to do anything special to make it work.

CentOS 7 complicates things. Neither the CentOS nor EPEL distribute loggerhead or python-simpletal as packages. I went through various contortions trying to re-compile source rpms to provide missing resources, but never got anywhere, so I backed off and did python setup.py install from tarballs obtained from loggerhead and simpletal web sites.

The CentOS 7 (?relevant?) installed packages for comparison are:

$ rpm -qa | egrep '(^bzr)|(simpletal)|(paste)|(loggerhead)|(flup)|(simplejson)'
python-paste-1.7.5.1-9.20111221hg1498.el7.noarch (CentOS)
python-simplejson-3.3.3-1.el7.x86_64 (EPEL)
bzrtools-2.5-5.el7.x86_64 (EPEL)
bzr-doc-2.5.1-14.el7.x86_64 (CentOS)
bzr-2.5.1-14.el7.x86_64 (CentOS)
python-paste-deploy-1.5.0-10.el7.noarch (EPEL)

To avoid mucking with the distribution site-packages directory permissions, I set up my own user-accessible site-packages and confirmed the following python setup deployed the expected resources.

$ python setup.py install --prefix=/home/app
$ export PYTHONPATH=/home/app/lib/python2.7/site-packages

Since this server is hosting the Bazaar repo, I replicated the test with something like:

$ mkdir ~/bzr/project/blah
$ bzr checkout /path/to/repo blah
$ /home/app/bin/serve-branches ~/bzr/project/blah
$ lynx http://localhost:8080/

Lynx reports:

Alert!: HTTP/1.0 404 Not Found

I've had this same behavior in all the various configurations I've tried. Even though I can't see evidence that SELinux is getting in the way, I set SELinux to permissive to simplify the unknowns.

After a lot of struggling with serve-branches (I do not have experience debugging python applications), I looked at the start-loggerhead and stop-loggerhead scripts. They aren't particularly friendly scripts, but I did get them "working" in an environment something like the following (but I don't know for sure that all the steps are required at this point):

$ mkdir ~/bzr/logs
$ mkdir ~/bzr/cache
$ mkdir ~/bzr/loggerhead-files
$ cp /home/apps/bin/st*-loggerhead ~/bzr
$ cd ~/bzr

I created /etc/loggerhead.conf containing something like:

title = 'branches in loggerhead'
log.roll = 'weekly'

[bazaar]
    name = 'bazaar'

    cachepath = '/home/kbulgrien/bzr/loggerhead-files'

    [[bzr.dev]]

        folder = '/home/kbulgrien/bzr/project/blah'

        description = "blah development branch"

I don't know that the above is an appropriate loggerhead.conf setup, BUT, this DOES NOT result in a 404 error! serve-branches ALWAYS gave a 404 error. This does not work though, as the browser page contains only the text "loggerhead branches" in the upper right corner. There are no projects or other links shown. This seems like progress, but I don't know why serve branches doesn't give this minimal result, and, I don't know what path to give the browser to have any hope of getting branch data to show.

The nice thing about the start-loggerhead script is that it writes access.log, error.log, and debug.log files (serve-branches gave this info in the console). There are no errors. The debug log shows messages characteristic of those that appear in a working installation. The access log shows the accesses by lynx.

I posted to the Bazaar users mailing list yesterday with no hits so far and I don't really know what to do/try next in terms of effecting a repair or collecting more debug data that would help someone figure out what is going wrong.

On the CentOS 6 server, I see a /etc/loggerhead/bazaar.conf file, so I made one on the CentOS 7 server, but I doubt this is needed as it is not present on the MES 5.2 server. It does have a setting (http_sql_dir) that makes me wonder though...

While trying to build RPMs to install missing dependencies of loggerhead, I found that rebuilding one loggerhead RPM requires python-sqlite2 for the RPM to install. I don't see information on the loggerhead that lists this as a dependency otherwise, but suppose that if a back-end database isn't working, then perhaps that could explain some issues.

The CentOS 6 server has:

sqlite-3.6.20-1.el6.i686
python-sqlite2-2.3.5-2.el6.i686
qt-sqlite-4.6.2-28.el6_5.i686

The CentOS 7 server has:

sqlite-3.7.17-6.el7_1.1.x86_64
sqlite-devel-3.7.17-6.el7_1.1.x86_64
python-sqlite3dbm-0.1.4-6.el7.noarch

The MES 5.2 server has:

libsqlite0-2.8.17-8mdv2009.0
php-pear-MDB2_Driver_sqlite-1.5.0-0.0.b3.0.1mdvmes5.2
libsqlite3-devel-3.7.15.2-0.1mdvmes5.2
php-sqlite-5.3.22-0.2mdvmes5.2
libsqlite3_0-3.7.15.2-0.1mdvmes5.2

CentOS 7 has systemd. None of the other working systems have systemd, but I could try this all out on a Mageia system to prove that should not be an issue.

Any ideas how to proceed with getting loggerhead up on this system, or get some better data on what is broken?