The Freaky Sandbox

One thing that can always be useful is the ability to run "unsafe" code inside a sandbox, where it can do limited damage. For a project I am working on, I wanted to do this very thing. Users would be able to upload their own code for execution on the server.

Naturally, this leads to a whole whack of security issues. If you just run code unchecked, users could get access to files, threads, processes, all sorts of things.
Fortunately there is this gem called the Freaky Sandbox which lets you run Ruby code, and it disables access to many security sensitive areas.

Unfortunately the gem doesn't work anymore. If you check out the source and install it, all sorts of bad things happen when you actually try to use the gem.
So before diving in to fix this, I decided to see if JRuby had a similar gem. Fortunately, it does. Now when I first tried it with JRuby 1.1.something it also didn't work, but I made some tweaks to it and now it does. Also with JRuby 1.2 RC1 it works (and also with RC2).
UPDATE (Mar. 28/10): The gem in the repositories no longer works, however you can check out the code to the javasand gem and install it using these instructions.

So now you can do fancy things like this:

require 'sandbox'

s = Sandbox.safe

File.open("untrusted_code.rb") do |f|
  s.eval f.read
end

What it does is filters out things like Thread and Kernel.fork, and file manipulation, things like that. It also doesn't see any classes that you have defined unless you import them into the sandbox.

So how could this be useful? One thing I'm thinking of is to allow users to write custom scripts for a site. Of course most users wouldn't be able to take advantage of this feature, but if script writers made their scripts available then they could just grab it and install it.

You could try something like this:

sandbox = Sandbox.safe

sandbox.import RandomAPIClassThatIWrote

res = s.eval untrusted_code

# do stuff with res

Unfortunately importing modules doesn't seem to work, it gives a NullPointerException when you try. If I get some free time soon I may look into it and try to fix it, but for now I'll just have to live with importing classes.

Now the question is: is this perfectly safe? Well, no. The code you run in the sandbox can still have infinite loops in it which will halt your program. Unfortunately it is impossible to tell if arbitrary code will have infinite loops in it or not, so we're out of luck for fixing this problem. However what you can try is the following:

sb_thread = Thread.new do
  s.eval untrusted_code
end

sleep TIMEOUT

sb_thread.raise if sb_thread.alive? # use raise since kill doesn't seem to work

Normally this is unsafe, because of blocking operations like synchronous IO and things like that. However with the sandbox, you cannot use the require keyword. You also have no access to the File or Dir classes (they are not defined inside the sandbox). So correct me if I'm wrong but unless you go and import these classes into the sandbox there are no unsafe things that could happen by just killing the thread.

OORegress: Another Stats Package

A while back I complained about how statistics functionality in OpenOffice is sadly lacking. Then I discovered that JRuby can fairly easily tie into OpenOffice, so I started thinking I might be able to tie the two together.

I mucked around a bit and rolled up something that can help out. It is a stats program that ties into OpenOffice from the command line, allowing you to enter commands to do statistical things. For example, the following code will run a regression:

regress(Y = X)

It will evaluate the model:

Yi = α + β * X + ei

Note that you don't specify the intercept, it is implied. I'll be adding a feature eventually where you can force a zero intercept.
You can also do some fancier things:

regress(ln(Y) = X1^2 + D*X2)

Which will obviously regress

ln Yi = α + β1*X1i2 + Di*X2i + e1

Where X₁, X₂ and D are independent variables.

The interface follows some simple conventions. The columns of the spreadsheet contain the data, and the very first row contains the name of each column. In your regression equation you address the variables by those names. When you run the regression, the program will open up a new sheet in Calc with the regression output with a bunch of info about the coefficients, their significance, some properties of the variance of the regression, R², etc. I pretty much just copied the stuff that Excel prints out when you run a regression because this is what my stats classes want. However I make it easier here since you can use an actual regression formula instead of having to copy-paste columns and apply formulas in the spreadsheet itself.

I'm working on documenting how to use the program, and also working on some new functionality like lagged variables and having ΔY_i instead of just some function of Y_i. However the regression itself doesn't completely work at the moment, so new stuff will have to wait.

You can check out the code if you like from here: http://code.google.com/p/ooregress/. I should have a more functional version coming out soon.

Erb Comments in JRuby vs. Ruby

With my current job, we built up most of the software with JRuby. More recently I discovered that development is faster using Ruby and then deploying on JRuby.

However then we discovered some bugs that were in code that hadn't changed in a long time. For some reason, certain chunks of our view templates were just disappearing. Nobody could figure out why.

This is where Vim came unexpectedly to the rescue. You can write comments in erb by writing something like this:

<%# this is a comment %>

You can also go like this:

<% # this is a comment %>

However with the latter type of comment, it would always mess up the syntax highlighting in Vim. It wouldn't recognize that the %> was closing the erb comment tag, because it thought it was part of the comment. I had noticed this before but didn't really care that much, because it wasn't my code and I didn't really want to mess with it. But this time in order to find the bug I figured I should have correct syntax highlighting.
Anyway after fixing the comments, I couldn't find anything wrong with the file. I was still baffled. So I figured I'd just go and look at the page in the browser again, and boom! It worked!

My hypothesis is that Ruby and Vim probably use the same logic for parsing those erb comments, but JRuby uses something different. JRuby would behave exactly the same if we had <%# or <% #. Ruby on the other hand will think that anything after the # is part of a comment, and therefore will ignore the %> that is on the same line.

Adventures in Haskell

I must apologize, I've been uncharacteristically quiet as of late and haven't posted much stuff. And even when I do, it's usually to report about (J)Ruby things that I've discovered. For the most part it is because I rarely have time/motivation to finish off the many half-finished posts I have sitting around. That's mainly because any free time I get is either lost in the world of Oblivion or spent in my geeky pursuit of learning Haskell.

Yes, I'm attempting to learn Haskell. Some people come home, sit down and fire up the TV, or read a book, or even write blogs. I come home, pour myself a glass of wine, and crack open Real World Haskell. Usually this ends up with me getting another glass of wine.
It's quite a well written book. It's freely available online, and the online version allows people to comment on each paragraph so there's usually extra tidbits of information that you can get from other smart people explaining tricky things in a different way than the book.

Anyway to the language itself. I saw Haskell while I was at school, in a lab in CSC101 one exercise asked us to write insertion sort in Haskell, which is about 2 lines of code. However it still managed to baffle pretty much everyone in the class, myself included. Of course, after learning Scheme in later classes the functional programming aspect comes much more easily now, but Haskell is still quite weird.

This is what I've come out with so far:

1. Type inference is awesome. No need to write things like std::vector< std::pair >::const_iterator or junk like that (don't ask why I have that particular data structure, I made it up on the spot). The Haskell compiler usually looks at your code and says, "Aha! You are probably wanting this type!" Of course it may not always be right, or you want to restrict the types a bit better than what it gives you, but it gives you the option of not specifying things.
2. Haskell is lazy. This is different than any other language I have used. It does exist in other languages too, although less completely. This should be familiar:

   if (1 || infiniteLoop())

   In C this will work fine, because the infiniteLoop function is never actually called. However, this will result in an infinite loop:

   foo(1, infiniteLoop())

   , even if foo looks like this:

   foo(int i, int j){
     if (i || j)
       return i;
   }

   In Haskell, this is not the case. Expressions are only ever evaluated when they are needed. In the previous example, the value of j is never actually needed, but it is calculated anyway.
   This results in some very interesting results. The standard library of Haskell has a function called repeat, which takes one parameter and just repeats it as a list. Infinitely. You can even write a list like this: [1..], which is just [1, 2, 3, 4, 5, ...]. These aren't all that useful on their own, but combine it with a function like take which takes the first n values of a list:

   take 5 (repeat 'a') == ['a', 'a', 'a', 'a', 'a']
   take 5 [1..] == [1, 2, 3, 4, 5]

   This may be geeky, but I think that's pretty cool.
3. No side effects - Code is not allowed to have side effects, except is special circumstances. So all variables are immutable once they are set (in Haskell-speak: "bound"). This has some interesting results. Suppose you have this:

   f (g x) (h y)

   Where f, g and h are functions, and x and y are values. Since there are no side effects, it doesn't matter whether you execute g or h first inside of f. In fact, you can run them simultaneously. No side-effects means easy concurrency, no need for locks or the complications that go with it.

I could probably go on for a while about the nifty things I've discovered (like how you can make *-~!!! a valid operator), but I'd recommend checking it out for yourself. If you're up for it anyway, it is a weird language compared to, well, most other languages out there, but I think it is one of those ones worth learning just for the ideas it gives you.

JRuby on Rails and Development Efficiency

I've been working with JRuby for about six months now, and it has been pretty good. It has native thread support, and you have a number of different options available from the Java world.

As a deployment strategy, JRuby is pretty solid IMO. However from a development perspective it is a bit slower than MRI. The biggest one you notice is that JRuby takes a while for the JVM to warm up. This is fine if you're just running Mongrel or WEBrick or something, but when you have a bunch of small scripts or Rake tasks to run or something to play around with in irb, it is quite annoying to have to use JRuby and wait that extra few seconds for the JVM to load. Also for some reason it takes way longer for my test suite to run in JRuby than with MRI. Oh well, whatever.

Another problem is that many gems are native, and therefore not available to JRuby. At this point JRuby has enough of a following for popular gems to have a JRuby port somewhere, but the matter of finding it and getting it to work on all your developers' machines is a pain in the ass. Better to just do it once on the deployment machine(s) and be done with it. Some examples of gems that don't work in JRuby: rcov, RMagick, mysql, anything to do with datamapper. The memcache-client gem used to work, if you use version 1.5.0 it works fine but the latest one fails.
EDIT: There's been a bit of confusion by what I meant here. What I mean is that the gems in the repository do not work with JRuby, so going 'jgem install GEM' does not work, you have to find the port online. This isn't usually that difficult, but a bit more time-consuming than the standard way of doing things.

However I'd say JRuby is great for production for a few reasons. First off, it has access to native threads. I believe Ruby 1.9 uses native threads, but my Rails app currently does not work with the Ruby 1.9 available in the Ubuntu repositories and I'd rather not have to maintain a new Ruby install unless absolutely necessary.
JRuby also has access to a wider range of application servers. Mongrel works well with JRuby, and any other web server written in Ruby should work fine as well. JRuby can also be deployed as a WAR with any application server that uses WAR files. We're using Glassfish, but I think you can do it with Tomcat and others too.
Finally, JRuby has access to Java libraries. Say what you will about Java the language, there are a ton of Java libraries out there. For basic stuff, Ruby has pretty much everything it needs, but when you want to move outside of web development things get sparse quickly. Want to write an OpenOffice plugin? JRuby can do it by using OpenOffice's Java API. Want to use a NLP tool like GATE? The API is in Java. Where are things like this for Ruby?

Anyway, IMO ideal setup is:
development - Ruby, unless you're using some Java libraries like I mentioned above
production - JRuby
This may change as Ruby 1.9 gets better, but at the moment I'm liking the above setup.

Peach

A coworker referred me to this little Ruby gem called Peach, which is a parallel processing gem designed to speed up each/map/delete_if by dividing up the work among several threads - something always good.

I did some checks to see how much faster it is. Unfortunately I couldn't see a speed improvement, and for the basic map: i => 2i the peach versions were actually much slower, likely due to the overhead of splitting up the collection and merging the results.

A couple gotchas - make sure to set $peach_default_threads, or you'll be sorry when working with massive arrays. The default is to use one thread for each element in the array. This is fine for arrays with like 3 elements and the operation takes a long long time, but for arrays with 100 000+ elements, that's just insane - the overhead is not worth the gain.
The gem doesn't show much improvement on MRI, probably because MRI uses green threads. Also I can't test this, but it may not show too much improvement on a single-core processor, depending on what you're doing. So basically it is much better to use JRuby for this on a multi-core machine, since JRuby uses native threads and can actually take advantage of the hardware available.

Digging through the code a bit, I can see some points of potential optimization due to the natures of the operations. Right now the code splits up the array into a number of sub-arrays based on the number of threads to run, then executes the function, and finally merges the results. For Array#each, this can be done in place as Array#each just returns itself - no need to split the array and remerge it afterward.
Array#map on the other hand cannot be done in place. However since the size of the output is the same as the size of the input, the new array can be allocated before the threads begin and each thread works with indices. This saves a massive amount of time merging the arrays afterward, since Array#+ creates a new array and copies all the elements from the old arrays into the new one.

The gem is still in a young state, there are plenty of places for optimization (I think I will try my hand at this). It only works with Array, so most other enumerable types are not supported yet. Also there are only the three operations that are supported, no inject or anything yet. Finally, you must use your brain when doing parallel processing. Side effects = bad. They introduce all sorts of problems with race conditions. So try to avoid them when using peach.

All in all, this is an awesome idea and with some publicity, the open-source community will improve this gem big time.

UPDATE: I made my own version of pmap which allocates a new array and modifies that array directly, and it doesn't provide a huge boost in speed for larger collection sizes. For smaller sizes it is faster (by a lot) but later on not so much. I would gather this is because the overhead of Peach is a much smaller chunk of the processing time in the long run than the processing of each sub-array. So the tweak shaves off a bit of time, but not a lot. I will post my tweaks at a later date because I want to add some more functionality like inject.

Using MySQL with JRuby outside of ActiveRecord

UPDATE(Jun. 12/2012): Turns out things have changed a bit since this post was written, JDBC has been put directly into JRuby. You can check it out here.

I was playing around yesterday with more JRuby stuff and wandered my way into needing to access MySQL from JRuby, but outside of an ActiveRecord environment. Of course naively I originally tried to use the mysql gem for Ruby, but it failed since it seems that gem is native code only (one of the annoyances of JRuby). But I've been using JRuby with MySQL for nearly 6 months now and never had problems. So I dug into the activerecord-jdbcmysql-adapter gem and some related ones and discovered some things that were a bit annoying. It turns out that those things all use the java.sql stuff, instead of some Ruby-baked solution.

Now I haven't really ever worked with java.sql. In fact, I don't recall ever working with a DB outside of a dynamic language. And it looks painful. There's all sorts of getString() this and getRef() that, blah blah.

So I decided to roll up a nice and simple JRuby class for JDBC. Here's the whole code:

require 'java'
require 'rubygems'
require 'jdbc/mysql'
include_class "com.mysql.jdbc.Driver"

class JdbcMysql
  def initialize(host = nil, username = nil, password = nil, db = nil, port = nil)
    host ||= "localhost"
    port ||= 3306

    address = "jdbc:mysql://#{host}:#{port}/#{db}"
    @connection = java.sql.DriverManager.getConnection(address, username, password)
  end

  def query sql
    resultSet = @connection.createStatement.executeQuery sql

    meta = resultSet.getMetaData
    column_count = meta.getColumnCount

    rows = []

    while resultSet.next
      res = {}

      (1..column_count).each do |i|
        name = meta.getColumnName i
        case meta.getColumnType i
        when java.sql.Types::INTEGER
          res[name] = resultSet.getInt name
        else
          res[name] = resultSet.getString name
        end
      end

      rows << res
    end
    rows
  end
end

It may not be the most flexible of classes right now, but it should get the job done. And it is wide open for improvement.

For this to work you'll need the jdbc-mysql gem:

jgem install jdbc-mysql

While you'd think the jdbc-mysql gem would have something like the above, all it does is include the Java MySQL driver (aka com.mysql.jdbc.Driver).

So now if you want to use the class:

db = JdbcMysql.new("localhost", "me", "secret", "my_database")

res = db.query "SELECT * FROM my_table"

res.each do |row|
  puts row["value"]
end

Much easier!