Stats App Revisited

A while back I mentioned that I was working on an addon for OpenOffice called ooregress, which adds some more advanced statistics functionality to OpenOffice Calc.

Unfortunately when I wrote it, it was for OpenOffice 2.4. I think they changed the API for OpenOffice 3.0, so my code no longer works.

Rather than try to repair my code, I decided to port the thing to Javascript so that I could release it as an easily accessible website. The issue with the original version was that it was written in JRuby so you would need JRuby actually installed to run it, and it needed the development files for OpenOffice installed on the machine (not to mention OpenOffice).

This one is in a website, so all you need installed is a supported web browser. Since this is a personal project, IE is not a supported web browser :P I am currently supporting Firefox, Chrome and Safari - I can probably support Opera too, I'd just have to install it.

A temporary link is here, although I won't guarantee that this will always be active. I've got it hosted on my SheevaPlug, which I turn off sometimes. UPDATE: The site is now statsan.com, and I've called it StatSan for STATiStical ANalysis.

I'm planning on putting all the code as open source (GPL) so that anybody can host their own version of it should you want to, I'll put up the link once I get something set up - I've got a local git repo of it, it's just a matter of creating something on GitHub and pushing everything over.

Anyway, why am I writing this? Well, a few reasons. First, my stats classes at school have required various stats software, including Excel, EViews and Stata. I've also tried R. For the most part, these programs fall into one or more of the following categories: tedious, hard to use, expensive, not cross-platform. I want to make something that is more streamlined, easy to use, free, accessible and portable. You don't get much more accessible and portable than JavaScript, since the vast majority of people have something installed that supports it to some degree or another (however I guess this isn't 100% portable since I'm not supporting IE).
There is one downside: JavaScript is slow. I'm sure if you were running a regression on 20 variables with hundreds of thousands of observations, it would make your computer cry. This is an issue that I've been thinking about a bit and will hopefully come up with some solution for (if that native code plugin that Google is talking about ever comes out, that would be awesome!).

So if you have to do stats calculations, feel free to check out this app and let me know what you think. If you find any bugs or come up with any features you'd like, feel free to let me know.

Javascript Dataset Speedup

I've been working on a little project and I noticed it was loading very very slowly when working with large datasets (large being ~6000 numbers, so it is "large"). After doing some digging, I realized that it was because I was rendering my objects as Javascript objects:

<%= @workspace.as_js %>

Since the objects had some very large arrays in them, it took quite some time for the browser to parse that Javascript and load it in.

However, the weird part was then when I'd use Javascript code to process these arrays once they were loaded, it was nice and speedy.

The solution was simple: don't put the objects as Javascript. Parsing it is slow.
Initially I decided I would put the data in some raw form inside a <div> and then process the innerHTML. However that also turned out to affect the rendering time of the page (surprise surprise).

My final solution was to load the data from an AJAX action. This way the rendering time of the page is snappy, and there is not a lot of Javascript to parse.

The raw format is pretty simple. It is a set of name-value pairs separated by pipes (|), and each element is separated by colons - as I write this it might make more sense to use a URL formatting with & and =. The datasets themselves are arrays, so the name is preceded by a @ and is stored in the vars subfield of the object that I'm loading in. Here's the code:

<script type = "text/javascript">
  var workspace = {vars: {}};
  $(function(){
    $.get('data_set_url', {},
      function(response){
        $.each(response.split("|"), function(i, obj){
          if (obj.substring(0, 1) == "@"){
            // dataset
            obj = obj.substring(1).split(":");
            
            workspace.vars[obj[0]] = $.map(obj[1].split(","), function(obj2, i2){
              return parseFloat(obj2);
            });
          }else{
            obj = obj.split(":");
            workspace[obj[0]] = obj[1];
          }
        });
      }
    );
  });
</script>

Even with the AJAX, this turns out to be much faster than putting the data as Javascript (or JSON), or putting the raw data in the HTML.

Scalability Presentation Slides

Someone requested that I put up my slides for the scalability presentation I did the other day, they are available here.

Unfortunately they are only the slides, the presentation was not recorded. If anybody has questions about a certain page, feel free to ask here. I'd prefer if people ask via comments, so that others can see the answers if they have the same question.

Scalability Presentation

I'm giving a presentation at Bishop's University in Sherbrooke on Friday at noon if anybody is interested. It's called "Managing Millions: Tips and techniques for building high traffic web applications." It will talk about some of the problems that we faced back when I was working on a high traffic website like various database bottlenecks, issues faced when using cache, etc. plus the solutions we used on how to fix them.

Some of it will cover some techniques that we researched but didn't really employ, so I won't be going into as much detail on those. For example, doing asynchronous processing (aka offline processing) which in hindsight I don't think we didn't do as much as we should have and the stuff we did do wasn't very elegantly handled - cron jobs are decent but when you aren't sure how long the script will take you have to do some hacks to make sure that you don't have too many running at once, etc. There's better tools out there for this.

Dual Head and Sauerbraten

Normally when I play fullscreen games in Linux it is really annoying, since they tend to display in the middle of the displayable area: that is, half of the game on one monitor, and the other half of the game on the other monitor. It's slightly annoying, although I just disable one of my monitors when I go into a fullscreen game.

However I discovered today with Sauerbraten that it seems to map itself across the entire viewable area. So I end up having the game spread fully across both monitors. It's actually pretty sweet, and if you have dual head and Linux set up, I suggest you try it out.

Multiple Inheritance in Scala vs. C++

I've made an interesting discovery today on how Scala's type system is slightly more awesome than C++'s type system. It has to do with traits.

There's a C++ project I'm working on where I'm pulling out some shared functionality from a few classes and putting it into another class that will be used like a mixin (one reason to learn a language with mixins - you get better ideas when you go back to languages that don't have them). It's fairly simple: I have a base class Creature, and I have a number of classes which inherit from Creature to define some specific behaviour. Two examples are Giraffe and Tiger. These two subclasses have very different habits when it comes to how they find their food, however there's some basic functionality involved in the mechanics of eating and starvation and stuff that is common. So I put that basic functionality into a class called Hungry:

class Hungry{
 ...
public:
  void act();
};

In the act() method, it makes the creature slightly more hungry and if it is too hungry, it makes the creature die. Unfortunately, the Hungry class has no idea what die() is. So I could have Hungry inherit from Creature as well, however that will cause diamond problems when we start getting into many different traits for different creatures. The next step was to try putting die() as a pure virtual method in Hungry. Unfortunately that didn't work unless I explicitly defined die() in both Giraffe and Tiger, which would be annoying since die() is specified in Creature and works just fine. With that, the code ends up like this:

class Hungry{
  ...
public:
  ...
  void act() {
    if (--hunger <= 0)
      die();
  }

  virtual void die() = 0;
  ...
};

class Giraffe : public Creature, public Hungry{
  ...
  void die() { Creature::die(); }
  ...
};

This is really a minor detail, but slightly annoying. Shouldn't it be able to figure out that there is one concrete implementation of die() in Giraffe and Tiger?

Turns out that in Scala, it does figure this out:

class Creature{
  def die(){
    ...
  }
}

trait Hungry{
  def act() = {
    hunger -= 1
    if (hunger <= 0)
      die
  }

  def die  // abstract method

  var hunger : Int = 0
}

class Giraffe extends Creature with Hungry{
  override def act() = {
    super[Hungry].act
  }
}

This works perfectly. It is type-safe too, if you remove the "extends Creature" from Giraffe, it will give you an error saying that you haven't defined die.

I'm confused, why can't C++ do this? This doesn't really have anything to do with the fact that Scala is using a trait, because if you made Hungry into a class and allowed Scala to inherit from multiple classes, it wouldn't be any different. The fact of the matter is that Giraffe and Tiger do have a die() method, but the C++ compiler isn't recognizing this and saying that they are abstract classes.

Multiple Method Inheritance in Scala

I had a little issue today that likely crops up once in a while when using Scala's traits. I have a class with a particular method foo(), and a trait with a method of the same name. When I created a derived class that used this trait, I didn't know how to specify which version of foo() I wanted to call. Let's take a look at our picture:

class Blah{
  def foo(){
  }
}

trait Tleh{
  def foo(){
  }
}

class Dingle extends Blah with Tleh{
  override def foo(){
    super.foo  // this calls Tleh.foo
  }
}

As I mentioned in the comment there, using super.foo calls the trait's version of foo(). But what if you want to call Blah's version of foo()?

It's actually pretty simple, and a bit intuitive if you know a bit of Scala:

class Dingle extends Blah with Tleh{
  override def foo(){
    super.foo  // calls Tleh.foo
    super[Blah].foo  // calls Blah.foo
    super[Tleh].foo  // calls Tleh.foo
  }
}

This is pretty simple, but it took me a little while to figure it out since there doesn't seem to be any mention of this in the docs I found!