Wages Over The Years

I recently read this article which makes a claim that real average wages (aka inflation-adjusted wages) have fallen over the years and supposedly "continue to fall." I looked at this and thought the idea was preposterous, so I rolled up my sleeves and did some real data analysis. Since I've been talking about statistics, I figured some of you might be interested in this knowledge.

tl;dr: Wages overall have actually increased, but not for university grads. University grads still make more than non-university grads. Income inequality has increased a lot, especially among university grads.

First off, my sources. I grabbed the data from the 1986 and 2006 Canadian Censuses (Censi?). Unfortunately I can't share this data with you since I am not legally allowed to distribute it, but if you have access to a university you should be able to dig it up somehow. This data is probably some of the best you can get, since it is much less likely to have selection biases compared to other surveys - people are legally required to fill this data out.
Second source was the Consumer Price Index (CPI) which can be used as a measure of inflation. That's how I adjust the raw figures in the census data for inflation. Statistics Canada is kind enough to list these figures here. All dollar values in this post will be inflation-adjusted to 2010 Canadian dollars.

Now I know that many readers here are not Canadian, however these results should be similar for countries with a similar economy to Canada like the US, the UK, or Australia. For those of you so inclined you can probably find the same data for your respective country to do the same analysis.

Let's get started. How would you go about figuring this stuff out? Well, you need the data. Once you get that, it's pretty straight-forward. I did my analysis using the following steps:
1) Filter the data. I want to look at people who are at least 15 years old, and have a regular old job. This excludes self-employment. This isn't a huge deal, just keep in mind that the averages here are for employed people.
2) Adjust wages for inflation. This is done by dividing by the CPI for the year of the data (1986 or 2006) and multiplying by the CPI for 2010.
3) Construct a confidence interval for the average wage. This lets us see if the wages between the two periods are actually statistically different. The formula for a 95% confidence interval in R-pseudocode is:

mean(wages) ± 1.96 * sd(wages) / sqrt(length(wages))

The 1.96 is the critical value of the normal distribution (sample averages follow a normal distribution) for a 95% confidence interval.

What are the results? Here's the R output:

[1] "Average wages for all employed individuals:"
[1] "Confidence for 1986: 30221.611097 to 30436.550551"
[1] "Confidence for 2006: 38479.736828 to 38790.160974"
[1] "Standard Deviation for 1986: 27382.937843"
[1] "Standard Deviation for 2006: 52381.845010"
[1] "Median Confidence for 1986: 25083.328237 to 25325.457532"
[1] "Median Confidence for 2006: 29704.694341 to 30054.387142"

What are we looking at here? Well, the average wage in 1986 was roughly $30k/year, where the average wage in 2006 was roughly $38.5k/year. Looks like wages in general are not falling.
The median is a better measure here though, since medians are a bit more robust to outliers (aka those few people who make hundreds of thousands of dollars a year). As we can see, the shift in the median is not quite as big as the shift in the mean, meaning that while the wages have gone up, they haven't gone up quite as much as the mean might indicate.
What does all this tell us? Well it is difficult to say for certain, but it would appear that in 2006 people are in fact making more money (as shown by the higher median), but there are also more people making giant salaries than before which will skew the average. Given the big jump in the standard deviation, we can see that there is an increase in income inequality.

Now, here's the real interesting part. I decided to run this again, but with one more twist: I filtered for people who have a bachelor's degree or higher. Let's see the results:

[1] "Average wages for bachelor's degree or higher:"
[1] "Confidence for 1986: 49543.651801 to 50383.047856"
[1] "Confidence for 2006: 59994.274103 to 61070.608811"
[1] "Standard Deviation for 1986: 37181.295254"
[1] "Standard Deviation for 2006: 82837.561885"
[1] "Median Confidence for 1986: 47938.828738 to 48884.408393"
[1] "Median Confidence for 2006: 46310.094270 to 47522.585319"

The average and the median wages here are much higher in both periods than for the entire group. This gives us a pretty good indication that university graduates make more money than non-university graduates.
However, it would appear that while the average university graduate makes far more in 2006 than they did in 1986, this result is misleading. When we use the median we can see that university salaries have actually gone down slightly over this time period. One conclusion that we can guess from this data here is that there are some university graduates in 2006 that are making huge salaries, while the bulk of the university grads aren't doing quite as well.

So there we have it, some statistical data. While the interpretations of the data and the methods for analysis here are up for debate, the numbers calculated are not. They're taken directly from census data, which is pretty darn good data (unfortunately it might not be quite as good for 2011, since the Conservatives have scrapped the long-form census and I'm not sure if this data will be on the short-form one).

Keep in mind that these are purely correlations, not causations. This data is not saying, "if you get a university degree then you will make more money." This data is saying, "people who currently have university degrees make more money on average."
It is also looking at the aggregate. I'm sure most of you can come up with examples of non-university graduates who are making good salaries, and of university graduates who are not making great salaries. These people are the exceptions, not the rule.

One thing that I could do in a later analysis is split the groups up into the different types of degrees. Both these censuses provide the level of education the people get (Bachelor's, Master's, etc.) and the discipline (sciences, engineering, arts, etc.). However this post is getting long enough, so that can wait for another day.

For the code, it is written using R and is available here as a gist on Github. Feel free to fiddle with it if you feel like it. You'll notice I generate some histograms of the data, however I found that they don't really reveal much info so I didn't include them here.

Oops!

In case you read the post earlier today, just ignore it. I got lazy and didn't do enough research beforehand, which was pointed out. My apologies!

Rug

Over the last few months I've been working with Concordia's CART CGD group, which is a group of computer art students who want to make video games. We did a few things called "game jams", which is where a group of people meet together and try and churn out a game in a few hours. We didn't actually come up with anything that great this last semester, but then again there was only two game jams (if you're interested in participating, they have them every Saturday starting January 15th, send me an email and I can give you more details).

We did the game jams with Lua and a game framework called Löve2D, which helps simplify game development.

I found after learning a bit of Lua that it is a lot like Javascript (although I have to say I still prefer Javascript). It isn't an object-oriented language in the same way that C++ or Ruby are, but it is still possible to create objects with methods and inheritance and so on. I found in the end that I need a little bit more structure when I'm programming than is offered by Lua - I actually also found the same thing with Javascript when starting to build some larger Javascript apps like Colonial.

However, I found that I really liked Löve2D. A lot of things were very simple and straight-forward. Before that the nicest gaming library I worked with was pygame which is a wrapper around the C version of SDL and a few extra bells and whistles like collision detection and sprite handling. There's a port to Ruby called RubyGame, but I find it suffers from the same problem as pygame: it's just a wrapper around SDL with a few bells and whistles.

I decided to go and roll out my own Ruby gaming library called Rug. It's mostly written in C++ for speed, but the API is much more Ruby-ish than Rubygame or Pygame with a few hints from Löve2D. At the moment it doesn't have a lot of features; it supports some rudimentary collision detection and animations. It doesn't stick completely to Löve2D's API, there were a few things in there that I found kinda clunky when it came to animations that I fixed up.

You can see an example of Pong here with a screenshot:

Or a very simple platformer here, with a screenshot:

The library is still very new, and there are a few things that I'm not happy with that will change in the near future (in other words if you choose to fiddle with this, don't be surprised if things change - but then again if you've had any experience with Rails you will be used to this sort of thing ;) ). Documentation still has a ways to go, I started documenting the features religiously but in the end I found that I changed the way I wanted the library to work often enough that documenting can wait until things are a little bit solid. For example the physics module for collision detection has undergone quite a few API changes, and after having to update the documentation several times I decided that I will wait until I am happy with how the module works before telling everyone else how to use it.

So anyway, feel free to play around with it and tell me what you think. Installing it is pretty easy in Linux, however in Windows it can be a bit annoying with all the Ruby headers and the various SDL libraries. I managed to get it to compile in Windows, so shortly I'll put up a zip with all the various DLLs that you need to run Rug.
I have no idea how easy/hard it will be to set up on a Mac, if it works anything like Linux you can probably just use MacPorts to install SDL and the Ruby development headers.

Your 30 Before 30

One thing that I see a lot of these days is all sorts of things like this blog or this book telling you to do your own thing, do your own projects, branch out on your own, take initiative, etc. Whenever I read these things, I can't help but find myself inspired by them: I enjoy doing my own projects, and I would love to be able to just work on them all day and maybe someday be able to live off them.

So here is my first try. I'm launching a project (it's too small for me to consider it a startup) called Your 30 Before 30 that might be a little fun to do. The idea behind it is to create a "30 before 30" list (similar to a bucket list), where you write down 30 things you want to do before you're 30. They don't have to be anything special, but they can be. Then at some point when you accomplish what you want to accomplish, you mark it off and tell people what you did.

I'm putting it here to invite any of my readers to check it out and "beta" test it. Let me know what you think. What's cool, what's not? Do you have any suggestions/constructive criticisms? Any feedback is welcome.

Second Thoughts on IronRuby

Unfortunately after using IronRuby at work a bit, I've found that there are still a few too many bugs with the thing for me to want to depend on it for production software. The bugs are already reported, however they are tagged as low priority and will not likely be fixed any time soon. Rather than spending lots of time fixing the bugs myself (I get paid to make profit, not to work on open-source pet projects), I'll just continue doing things the way that I have been!

Hopefully someone else will find IronRuby useful!

In case you're wondering, the bug that really killed it for me was the inability to use gems with RVM under Linux. The following command fails with issues installing RubyGems:

rvm install ironruby

No Rubygems drastically reduces the usefulness of Ruby, enough that I'm not really wanting to use it anymore.

Getting Your Files

I suppose I'm talking to the wrong group of people here, but I decided to write a quick post about how to access anybody's files without needing their username/password for their computer.

It's quite simple:
1) Put in an Ubuntu LiveCD/LiveUSB
2) Boot off it
3) Access their files by going to Places -> X GB/TB Volume

This doesn't work if they have any sort of encryption on their drives or a BIOS password, but otherwise you can do whatever you feel like! So if you have a laptop or a machine that is easily reachable by random people, then you should probably secure your files from something like this if there are things on there that you don't really want people getting to.

Embedding a Ruby REPL in a .NET App

I've finally gotten fed up with using VB.NET as a scripting language at work, so I decided to try dropping an IronRuby interpreter into the system. It's fairly simple to do, I'll describe here how to build a REPL within your app.

Step 1: Make a window for the REPL. Give it a box for your input and a box for your output. How you do all this is up to you, the interesting part is how to use Ruby within it.

Step 2: Initialize the scripting engine. You need a config file, I call it ironruby.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
 <configSections>
   <section name="microsoft.scripting" type="Microsoft.Scripting.Hosting.Configuration.Section, Microsoft.Scripting,
     Version=1.0.0.5000, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="false" />
 </configSections>

 <microsoft.scripting>
   <languages>
     <language names="IronRuby;Ruby;rb" extensions=".rb" displayName="IronRuby 1.0" type="IronRuby.Runtime.RubyContext, IronRuby,
       Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
   </languages>
 </microsoft.scripting>
</configuration>

You need to make sure you have the DLLs available that IronRuby comes with. Just put them in the same folder as your app.

Now the code to load in your Ruby stuff. Initialize a few objects:

var runtime = new ScriptRuntime(ScriptRuntimeSetup.ReadConfiguration("ironruby.config"));
var engine = runtime.GetEngine("Ruby");

// create a scope - you need to do this in order to preserve variables declared across executions
var globalScope = engine.CreateScope();

// I'm assuming you have a textbox or something named command, just adjust
// this to what you want if you have something else
var result = engine.CreateScriptSourceFromString("(" + command.Text + ").inspect").Execute(globalScope);

And, jackpot! The result variable is a string that is the result of whatever is in the command.Text variable.

This code could be a little bit more robust, so make sure to handle exceptions nicely. An exception will be thrown if say the syntax is invalid, or there are other problems.

The nice thing about this is that you can tinker with the classes within your own application. In the REPL if you put:

# replace MyApp.exe with the .exe or .dll file that your app uses
require "MyApp.exe"

You can then access your namespaces and classes within your own app, and fiddle with them while the app is running!