Javascript Dataset Speedup

I've been working on a little project and I noticed it was loading very very slowly when working with large datasets (large being ~6000 numbers, so it is "large"). After doing some digging, I realized that it was because I was rendering my objects as Javascript objects:

<%= @workspace.as_js %>

Since the objects had some very large arrays in them, it took quite some time for the browser to parse that Javascript and load it in.

However, the weird part was then when I'd use Javascript code to process these arrays once they were loaded, it was nice and speedy.

The solution was simple: don't put the objects as Javascript. Parsing it is slow.
Initially I decided I would put the data in some raw form inside a <div> and then process the innerHTML. However that also turned out to affect the rendering time of the page (surprise surprise).

My final solution was to load the data from an AJAX action. This way the rendering time of the page is snappy, and there is not a lot of Javascript to parse.

The raw format is pretty simple. It is a set of name-value pairs separated by pipes (|), and each element is separated by colons - as I write this it might make more sense to use a URL formatting with & and =. The datasets themselves are arrays, so the name is preceded by a @ and is stored in the vars subfield of the object that I'm loading in. Here's the code:

<script type = "text/javascript">
  var workspace = {vars: {}};
  $(function(){
    $.get('data_set_url', {},
      function(response){
        $.each(response.split("|"), function(i, obj){
          if (obj.substring(0, 1) == "@"){
            // dataset
            obj = obj.substring(1).split(":");
            
            workspace.vars[obj[0]] = $.map(obj[1].split(","), function(obj2, i2){
              return parseFloat(obj2);
            });
          }else{
            obj = obj.split(":");
            workspace[obj[0]] = obj[1];
          }
        });
      }
    );
  });
</script>

Even with the AJAX, this turns out to be much faster than putting the data as Javascript (or JSON), or putting the raw data in the HTML.

Scalability Presentation Slides

Someone requested that I put up my slides for the scalability presentation I did the other day, they are available here.

Unfortunately they are only the slides, the presentation was not recorded. If anybody has questions about a certain page, feel free to ask here. I'd prefer if people ask via comments, so that others can see the answers if they have the same question.

Scalability Presentation

I'm giving a presentation at Bishop's University in Sherbrooke on Friday at noon if anybody is interested. It's called "Managing Millions: Tips and techniques for building high traffic web applications." It will talk about some of the problems that we faced back when I was working on a high traffic website like various database bottlenecks, issues faced when using cache, etc. plus the solutions we used on how to fix them.

Some of it will cover some techniques that we researched but didn't really employ, so I won't be going into as much detail on those. For example, doing asynchronous processing (aka offline processing) which in hindsight I don't think we didn't do as much as we should have and the stuff we did do wasn't very elegantly handled - cron jobs are decent but when you aren't sure how long the script will take you have to do some hacks to make sure that you don't have too many running at once, etc. There's better tools out there for this.

Dual Head and Sauerbraten

Normally when I play fullscreen games in Linux it is really annoying, since they tend to display in the middle of the displayable area: that is, half of the game on one monitor, and the other half of the game on the other monitor. It's slightly annoying, although I just disable one of my monitors when I go into a fullscreen game.

However I discovered today with Sauerbraten that it seems to map itself across the entire viewable area. So I end up having the game spread fully across both monitors. It's actually pretty sweet, and if you have dual head and Linux set up, I suggest you try it out.

Multiple Inheritance in Scala vs. C++

I've made an interesting discovery today on how Scala's type system is slightly more awesome than C++'s type system. It has to do with traits.

There's a C++ project I'm working on where I'm pulling out some shared functionality from a few classes and putting it into another class that will be used like a mixin (one reason to learn a language with mixins - you get better ideas when you go back to languages that don't have them). It's fairly simple: I have a base class Creature, and I have a number of classes which inherit from Creature to define some specific behaviour. Two examples are Giraffe and Tiger. These two subclasses have very different habits when it comes to how they find their food, however there's some basic functionality involved in the mechanics of eating and starvation and stuff that is common. So I put that basic functionality into a class called Hungry:

class Hungry{
 ...
public:
  void act();
};

In the act() method, it makes the creature slightly more hungry and if it is too hungry, it makes the creature die. Unfortunately, the Hungry class has no idea what die() is. So I could have Hungry inherit from Creature as well, however that will cause diamond problems when we start getting into many different traits for different creatures. The next step was to try putting die() as a pure virtual method in Hungry. Unfortunately that didn't work unless I explicitly defined die() in both Giraffe and Tiger, which would be annoying since die() is specified in Creature and works just fine. With that, the code ends up like this:

class Hungry{
  ...
public:
  ...
  void act() {
    if (--hunger <= 0)
      die();
  }

  virtual void die() = 0;
  ...
};

class Giraffe : public Creature, public Hungry{
  ...
  void die() { Creature::die(); }
  ...
};

This is really a minor detail, but slightly annoying. Shouldn't it be able to figure out that there is one concrete implementation of die() in Giraffe and Tiger?

Turns out that in Scala, it does figure this out:

class Creature{
  def die(){
    ...
  }
}

trait Hungry{
  def act() = {
    hunger -= 1
    if (hunger <= 0)
      die
  }

  def die  // abstract method

  var hunger : Int = 0
}

class Giraffe extends Creature with Hungry{
  override def act() = {
    super[Hungry].act
  }
}

This works perfectly. It is type-safe too, if you remove the "extends Creature" from Giraffe, it will give you an error saying that you haven't defined die.

I'm confused, why can't C++ do this? This doesn't really have anything to do with the fact that Scala is using a trait, because if you made Hungry into a class and allowed Scala to inherit from multiple classes, it wouldn't be any different. The fact of the matter is that Giraffe and Tiger do have a die() method, but the C++ compiler isn't recognizing this and saying that they are abstract classes.

Multiple Method Inheritance in Scala

I had a little issue today that likely crops up once in a while when using Scala's traits. I have a class with a particular method foo(), and a trait with a method of the same name. When I created a derived class that used this trait, I didn't know how to specify which version of foo() I wanted to call. Let's take a look at our picture:

class Blah{
  def foo(){
  }
}

trait Tleh{
  def foo(){
  }
}

class Dingle extends Blah with Tleh{
  override def foo(){
    super.foo  // this calls Tleh.foo
  }
}

As I mentioned in the comment there, using super.foo calls the trait's version of foo(). But what if you want to call Blah's version of foo()?

It's actually pretty simple, and a bit intuitive if you know a bit of Scala:

class Dingle extends Blah with Tleh{
  override def foo(){
    super.foo  // calls Tleh.foo
    super[Blah].foo  // calls Blah.foo
    super[Tleh].foo  // calls Tleh.foo
  }
}

This is pretty simple, but it took me a little while to figure it out since there doesn't seem to be any mention of this in the docs I found!