blog.dt.org

My experimental search engine is now available in 10 languages…

About 18 months ago, I created an experimental search engine to play with some new ways to extract and distill information from the web and present it in a more topic-focused way. The site is called doryoku (which is Japanese — 努力 — meaning supreme effort). It has been doing well, and my experiments and enhancements continue. My latest addition involves automatic translation: the site is now available in 10 languages: English, Japanese, Chinese, Korean, French, German, Italian, Spanish, Russian, and Greek. The translation capabilities are courtesy of the Google Language API.

More new features to come. Stay tuned!

Tags: machine learning, search, translation

Adobe LiveCycle Express — Cloud Computing Meets Adobe LiveCycle ES

Adobe LiveCycle Express is a new software-as-a-service (SaaS) product offering from Adobe that takes a cloud computing approach to delivering Adobe’s LiveCycle ES enterprise software suite using Ruby on Rails, Adobe Flex, and Amazon’s EC2 and S3 services. LiveCycle Express grew from a research project I conceived while working in Adobe’s Advanced Tchnology Labs division.

The site was launched on December 15, 2008 at http://livecycle.express.adobe.com.

The joint press release from Adobe and Amazon announcing LiveCycle Express can be found here: http://finance.yahoo.com/news/Adobe-Announces-LiveCycle-bw-14025946.html.

Tags: Adobe, LiveCycle, Rails, Ruby, saas

Installing Lighttpd, Ruby on Rails, FastCGI, and MySQL on RedHat Enterprise Linux 5

Recently, I needed to configure a RedHat Enterprise Linux 5 box with lighttpd, FastCGI, Ruby on Rails, and MySQL. The box was subscribed to RHN, so I assumed a few simple commands like “sudo yum install lighttpd”, etc., would do the trick. Imagine my surprise to find that lighttpd, Ruby, gem, and FastCGI were all not available via RHN. I don’t know what the folks at RedHat are thinking by not including these packages, but it made me wish the box were running Debian Linux (which, for the record, has all of these packages available via aptitude).

Anyhow, switching to Debian was not an option, so I had to go back and do this the old fashioned way. The following is a quick walkthrough of the steps to install lighttpd, FastCGI, Ruby on Rails, and MySQL on RHEL 5. I am going to assume that you possess a reasonable level of Linux kung-fu if you are going to attempt this, so I won’t go too deeply into the individual details.

First, make sure your box is fully up to date, Linux-wise:

sudo yum update

Next, stop and deactivate Apache, which is running by default (I think) on RHEL 5:

sudo /etc/init.d/httpd stop
sudo chkconfig httpd off

Next, install necessary libraries and build lighttpd from the source code. I am adding some configuration options to ./configure to install files into the /usr/* directories, rather than the /usr/local/* directories as would be typical. This is a personal choice, so feel free to strip the options if you prefer to have your manually built applications in /usr/local…

sudo yum install pcre-devel zlib-devel bzip2-devel openssl-devel
wget 'http://www.lighttpd.net/download/lighttpd-1.4.20.tar.gz'
tar xvfz lighttpd-1.4.20.tar.gz
cd lighttpd-1.4.20
./configure --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-openssl
make
sudo make install

Next, create a non-root user and group that will run lighttpd. I typically use “www-data”, since I am a Debian guy, but feel free to use whatever you want. The important bit, of course, is that the user is something other than root for (hopefully) obvious reasons:

sudo /usr/sbin/adduser -s /sbin/nologin www-data
sudo /usr/sbin/addgroup www-data

Next, configure lighttpd and create necessary config and log files and directories. The contents of lighttpd.conf will depend on the location of your Rails application, among other things. I have included a simple Rails config below…but I assume if you are doing this you probably already have that…

sudo mkdir /etc/lighttpd

sudo echo > /etc/lighttpd/lighttpd.conf <<!

#
# Global config
#

server.port = 80
server.username = "www-data"
server.groupname = "www-data"
server.pid-file = "/var/run/lighttpd.pid"
server.errorlog = "/var/log/lighttpd/error.log"
server.indexfiles = ( "index.php", "index.html" )

accesslog.filename = "/var/log/lighttpd/access.log"

url.access-deny = ( "~", ".inc" )

server.modules = (
  "mod_rewrite",
  "mod_redirect",
  "mod_alias",
  "mod_access",
  "mod_auth",
  "mod_fastcgi",
  "mod_accesslog"
)

var.your.app = "/www/path-to-your-app"
server.document-root = var.your.app + "/public"
server.error-handler-404 = "/dispatch.fcgi"
url.rewrite = ( "^/$" => "index.html", "^([^.]+)$" => "$1.html" )
fastcgi.server = ( ".fcgi" =>
  ( "localhost" =>
    ( "min-procs" => 4,
      "max-procs" => 4,
      "socket" => "/tmp/your.app.fcgi.socket",
      "bin-path" => var.your.app + "/public/dispatch.fcgi",
     "bin-environment" => ( "RAILS_ENV" => "development" )
    )
  )
)
!

sudo chown -R www-data.www-data /etc/lighttpd

sudo echo "LIGHTTPD_CONF_PATH=/etc/lighttpd/lighttpd.conf" > /etc/sysconfig/lighttpd

sudo echo > /etc/init.d/lighttpd <<!

#!/bin/sh
#
# lighttpd Startup script for the lighttpd server
#
# chkconfig: - 85 15
# description: Lighttpd web server
#
# processname: lighttpd
# config: /etc/lighttpd/lighttpd.conf
# config: /etc/sysconfig/lighttpd
# pidfile: /var/run/lighttpd.pid

# Source function library
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/lighttpd ]; then
  . /etc/sysconfig/lighttpd
fi

if [ -z "$LIGHTTPD_CONF_PATH" ]; then
  LIGHTTPD_CONF_PATH="/etc/lighttpd/lighttpd.conf"
fi

prog="lighttpd"
lighttpd="/usr/sbin/lighttpd"
RETVAL=0

start() {
  echo -n $"Starting $prog: "
  daemon $lighttpd -f $LIGHTTPD_CONF_PATH
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
  return $RETVAL
}

stop() {
  echo -n $"Stopping $prog: "
  killproc $lighttpd
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
  return $RETVAL
}

reload() {
  echo -n $"Reloading $prog: "
  killproc $lighttpd -HUP
  RETVAL=$?
  echo
  return $RETVAL
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart)
    stop
    start
    ;;
  condrestart)
    if [ -f /var/lock/subsys/$prog ]; then
      stop
      start
    fi
    ;;
  reload)
    reload
    ;;
  status)
    status $lighttpd
    RETVAL=$?
    ;;
  *)
    echo $"Usage: $0 {start|stop|restart|condrestart|reload|status}"
    RETVAL=1
esac

exit $RETVAL
!

sudo chmod 755 /etc/init.d/lighttpd
sudo /sbin/chkconfig --add lighttpd
sudo /sbin/chkconfig lighttpd on

sudo mkdir -p /var/log/lighttpd
sudo chown www-data.www-data /var/log/lighttpd

Next, build and install Ruby from the source code. Again, I am adding configuration options to ./configure to install files into the /usr/* directories:

wget 'ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz'
tar xvfz ruby-1.8.7-p72.tar.gzcd ruby-1.8.7-p72./configure --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-openssl
make
sudo make install

Next, install MySQL. This is the one step that RHN handles, thankfully…

sudo yum install mysql mysql-server mysql-devel
chkconfig mysqld on ; /etc/init.d/mysqld start

You should probably take this opportunity to change your MySQL root password, for (hopefully) obvious reasons:

/usr/bin/mysqladmin -u root password 'new password'
/usr/bin/mysqladmin -u root -p -h 'hostname' password 'new password'

Next, install FastCGI for use by lighttpd to spawn Rails dispatchers (again, using options for the /usr/* directories…):

wget http://www.fastcgi.com/dist/fcgi.tar.gz
tar xvfz fcgi-2.4.0.tar.gz
cd fcgi-2.4.0
./configure --prefix=/usr
make
sudo make install

Next, install RubyGems, so that we can grab Rails:

wget 'http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz'
tar xvfz rubygems-1.3.1.tgz
cd rubygems-1.3.1
sudo ruby setup.rb

Next, grab Rails and install some other useful Gems:

sudo gem update --system
sudo gem update
sudo gem install rails --include-dependencies  # installs Rails 2.1.2
sudo gem install mysql -- --with-mysql-config=/usr/bin/mysql_config  # installs mysql-2.7
sudo gem install fcgi  # installs fcgi-0.8.7
sudo gem install packet   # installs packet 0.1.14

And, finally, if all has gone according to plan, you can bring up lighttpd and access your Rails application:

sudo /etc/init.d/lighttpd start

Hope this helps. Maybe some day soon this will be condensed to about five yum install commands via RHN…

Tags: configuration, lighttpd, linux, Rails, Ruby

Rails 2, Flex 3, and Form Authenticity Tokens

Recently, I was working with a Ruby on Rails application and I had the need to call a Rails controller method, with some parameters, from a remote Flex client. I would have thought that this would be a simple HTTP GET or POST to the Rails controller/method URL, using a Flex HTTPService object, with a tweak to the Rails method to render XML back to the client for parsing within Flex. However, Rails introduced the concept of form authenticity tokens in Rails 2.0, and these tokens are designed to block naive attempts to call Rails controller methods from outside of views rendered by Rails.

In simple terms, form authenticity tokens are one-time hashcodes that are generated as a hidden parameter for any form that is rendered by Rails. When the form is submitted, the hashcode is passed as a hidden parameter to the Rails controller, and Rails validates this hashcode to ensure that the form submission came from a view generated by Rails. This provides a measure of security against naive attempts to submit the form from other clients, since they will not have the proper hashcode needed to pass the Rails authenticity filter for the form submission. The specifics of the hashcode generation algorithm are covered elsewhere, but it suffices to say that they will resist uninspired hacking attempts, and it requires significant kung fu to bypass them without access to the Rails application.

In my case, I am not trying to hack the application — I just need to allow my Flex client to call my Rails methods. So I need to emulate the control flow of form generation in Rails, so that the view that kicks off my Flex client will contain a generated form authenticity token that can be passed to the Flex client as a startup parameter. There are 3 parts to this (or two if you want to condense parts 1 and 2):

• Store the generated form authenticity token for the Flex launch view in a javascript variable, so that it can be substituted intto the flashvars parameter of the Flex AC_FL_RunContent() javascript method. I chose to put this in the layout for the Flex launch view with:

<%= javascript_tag "const AUTH_TOKEN = #{form_authenticity_token.inspect};" if protect_against_forgery? %>

• Modify the call to AC_FL_RunContent() in the Flex launch view to include the form authenticity token. The line of code for this in the AC_FL_RunContent parameters list (if you are using my method of storing this is javascript as AUTH_TOKEN) is:

AC_FL_RunContent(
[...]
"flashvars","authenticityToken="+AUTH_TOKEN,
[...]
);

• I can now access the form authenticity token within Flex Actionscript code with a reference to:

Application.application.parameters.authenticityToken

Now that we have the form authenticity token in Flex, all that is left is to pass it as a parameter in the GET or POST to the Rails controller method. I found this last step to be surprisingly tricky. The Flex HTTPService object allows you to specify the parameters for a HTTP POST operation via an XML structure. Rails happily accepts and parses XML in POST operations, provided that the content type is set appropriately to application/xml. The tricky part is that the XML structure that is submitted by the Flex HTTPService object will be wrapped with a root XML tag of <request></request>, and all of the specified parameters will be contained within these tags. Rails will look for the form authenticity token as a root level tag named <authenticity_token>, and if it sees only a single root level tag of <request> (as sent by the HTTPService in Flex), then it fails the form authenticity test.

The workaround is to pass the form authenticity token as a URL parameter in the target URL of the HTTPService object, and to pass the other form variables within the standard request block of the HTTPService object, e.g.:

<mx:HTTPService id="httpService"
  url="http://mysite/method/?authenticity_token={Application.application.parameters.authenticityToken}"
  [...]>
  <mx:request>
    [...]
  </mx:request>
</mx:HTTPService>

The result of this is that Rails sees two parameters in the form submission: an XML document with a root tag of <request>, and the form authenticity token with its proper name of <authenticity_token>. The form parameters are accessible via the XML document, and the form authenticity token is automatically found and validated by the form authenticity filter in Rails.

That’s it. Hope this helps.

Tags: development, Flex, Rails, Ruby