Letter Frequency in Text and Rails

The other day a coworker and I were discussing keyboard layouts and this led into whether or not the character frequency (how often various letters are used) in a Rails application would be the same as it would be in normal text (though obviously, the code will have many more and more varied special characters). I thought that it would be but, he didn't think that would be the case. Since I had a bit of free time this weekend, I thought I'd write a bit of code to figure it out. For the text, I used Project Gutenberg's copy of Moby Dick and for the code, I used a smallish Rails project of mine.

Here's the code and results ...



As you can see, the histograms are pretty similar. For this particular Rails project, I was using HAML, so I'm not sure if there'd be any differences if you used erb or not. Also, as I noted, this was a fairly small project.

The code runs against a single file that's passed in on the command line. So run it like this ...

ruby letter_freqency.rb moby10b.txt > moby_results.txt

for example. For a rails project, I cat'd all the files together and then ran the code like this ...

cat `find . -iname \*.rb -or -iname \*.haml` > rails_files.txt
ruby letter_freqency.rb rails_files.txt > rails_results.txt

Let me know if you try this on one of your projects and post the results in the comments.

Rails and Windows

In a word ... "Don't".

I often see posts, especially on LinkedIn, with titles like "How do I learn Rails" etc. My advice is always to start with Michael Hartl's Rails Tutorial. A related question that often comes up though is about using Windows to do this. There are often comments that you can use Windows for this, cross-platform is good, etc. The truth though is that you're not going to have a lot of success developing Rails on Windows and if you're looking to move into Rails development as a profession, it will not probably help. There are a few different reasons for this, but first let's take a look a why you should develop for Windows.

If you've decided that Windows is a great platform and that you're ready to do your part to bring it to the Rails community, then by all means, go for it. Very few in the Rails community do use Windows, but if someone was willing to do the work to make everything that's commonly used work together under Windows, then I think it would be welcomed with open arms.

If you're just trying to learn some Rails though, this is probably not a good approach. While you're developing, you're going to have issues come up and those that can help you are going to be few and far between. In fact, even if it's not a problem with Windows per se, the fact that it works on Mac/Linux (or even appears to work there), is going to make it hard to get others interested in the problem. You're almost certainly going to have to come up with a failing test case (you should anyway) and show that this is the case. The problems you are, in general, going to be different than you'd see on a either a Mac or Linux and so most Rails developers won't have seen them and won't be able to help you solve them. Also, a big part of Rails development is done with the command line and while you can get bash or some other Unixy shell running, it's going to be sub-optimal.

For someone who is looking to move into Rails development, this is probably even more true. You're going to have all the issues listed above and those problems and their solutions won't be relevant to the issues your co-workers are going to be having. Additionally, whatever tools (here I'm thinking of editors and editor plugins especially) get adopted internally are unlikely to work well or even at all on Windows. A big part of the Rails development experience is collaboration and you won't really be able to benefit from or contribute to this collaborative process.

So, I recommend either breaking down and buying a Mac (a pretty expensive idea if you're just looking to experiment) or dual booting using Ubuntu (a pretty common platform where you'll be able to get some help easily if you run into issues).

Good luck and I'd be glad to hear your thoughts in the comments and in this case I wouldn't mind all that much being proven wrong.

Interviewing Redux

In my last post I talked about interviewing and overrepresenting your knowledge. We had a similar situation with a different candidate and I'd like to talk about that a bit.

A typical interview question that I've used and had used on me goes something like "What blogs do you read regularly?". It's generally used as a conversation starter to see what you're interested in and how much you stay up on things. The candidate in question told us that he read "lots of blogs" but when pressed couldn't name a single one or even talk about a post he'd read recently. As with our other candidate, he could have probably redeemed himself but didn't.

You should be reading blogs. I'd suggest at least three or four with a mix of ruby/rails and general technology. thoughtbot is good along with Ruby News. and ruby source. Before any interview make sure that you've read one or two posts pretty in depth and be prepared to talk about it a bit.

If you have other suggestions for blogs to read or thoughts on interviewing (either side of the table), be sure to leave them in the comments.

Interviewing Classic Mistake

We were interviewing a for a position the other day, when the interviewee made the pretty classic mistake of overrepresenting his knowledge on a particular subject. In this particular case it was puppet/chef, although I've seen it happen a number of times over the years. His resume should have been a warning, he listed a large number of technologies that he knew ... actually far more than he could have known. You should keep your list down to a handful of things that you know really well. Anyway, we've been looking at puppet recently for building our machines and using them for development also and since this was on his resume, one of my coworkers asked him about it. "Oh yeah, I love puppet and chef." was the reply. "Great, where have you used them and what did you like about them? Also, where do you think each is strongest?". "Well ... I haven't really used them in production, more just played with them." Hmmmm... at this point he's not making a very good showing. If you come out that you "love" a technology, then you probably should have done more than "play" with it. "OK, then exactly how did you play with them? Did you run the tutorial?". "Yes, I ran the tutorial". By this time, we weren't even sure that he'd run the tutorial or just read a blog post or something. He probably could have redeemed himself after this (he didn't), but it would have been difficult.

In general in interviews, you don't want to be talking about things you don't know about. You really want to be talking about things that you do know about. If you don't know something, don't put it on your resume and certainly don't say much more than you've "read about it, it seems really interesting, and you'd just love to learn more about it here at XYZ Corp when you start."

Rails / Ajax / Autocomplete

As you all know if you're regular readers here, I started out in the Ruby web framework world with Ramaze and I'm definitely glad that I did. However, it doesn't seem like there's much of a job market for Ramaze programmers, so I've taken up using Rails, specifically Rails 3.1. There's a number of great books for Rails and the one I'd recommend is Ruby on Rails 3 Tutorial by Michael Hartl. In this book, he develops a Twitter like application in Rails starting with basic pages and moving on to more complex topics.

I've been working on a simple project for a greeting card application and I came up with a little something that might be interesting. The cards have a recipient and can have multiple signers. I added autocomplete for the recipient, but it was a bit harder for the comma separated list of signers. What I ended up doing was was simply grabbing the last piece of the list in the controller and passing that back to the view.

Here's the view code ...

 Create a Card 
<%= image_tag "card_images/#{@template.image_name}", :size => "200x200" %>
<%= form_tag ("create_from_image") do %>

    <%= label_tag("Add the recipient's email: ") %>

    <%= text_field_tag(:recipient_email, "#{@recipient_email}")%>


    <%= label_tag("Add A Greeting: ") %>

    <%= text_field_tag(:greeting, "#{@greeting}")%>


    <%= label_tag("Add the signers' emails (comma separated): ") %>

    <%= text_field_tag(:signers_email, "#{@signers}")%>


    <%= hidden_field_tag :template_id, @template.id %>

    <%= submit_tag("Preview the card!") %>
    <%= submit_tag("Send the card!") %>
<% end %>

There's nothing very interesting in here, but the main field we're interested in is the :signers_email text field.

Here's the javascript code for it ...

$(document).ready(function(){
    // Below is the name of the textfield that will be autocomplete
    $('#recipient_email, #signers_email').autocomplete({
        // This shows the min length of charcters that must be 
        // typed before the autocomplete looks for a match.
                    minLength: 2,
        // This is the source of the auocomplete suggestions. In this case a 
        // list of emails from the users controller, in JSON format.
                    source: '/users/index.json'
    })

});

What this says is that for both the recipient_email and signers_email fields, use autocomplete, send when we have two characters, and use the /users/index.json function to get the data we need.

Finally, we have the controller code ...

    def index
        @title = "All users"
        @suggestion = "Pick someone and send them a card!"
        if params[:term]
            search_term = params[:term].split(",").last.strip
            @users = User.find(:all, :conditions => ['email LIKE ?', "#{search_term}%"])
            @users_hash = []
            @users.each do |user|
                @users_hash << { "label" => user.email }
            end

        else
            @users = User.where(:active => true).paginate(:page => params[:page])
        end

        respond_to do |format|
            format.html

            # Here is where you can specify how to handle the request for "/people.json"
            format.json { render :json => @users_hash }
        end
    end

Here, we have a couple of things going on ... for the Ajax side, we'll get the params[:term] field so we know that we need to get the constrained list. First, we're going to grab the params[:term] value, which should look something like "abc@test.com, def". Here we'd like to look for emails that start with the "def" and ignore the "abc@test.com" piece. So ... we split on the comma, grab the final element in the array, and then remove any leftover spaces at the end or beginning of the string. Next, we'll create an array where each element is a hash of the form { "label" => "defghi@test.com" }. This is the form that the jquery autocomplete needs. Finally, we use the respond_to section to send this back as json.

As always, let me know if you have questions.

Majority Voting

I saw this one over at Programming Praxis. It's pretty simple with ruby, but there's one slightly interesting piece in there that might come in handy sometime. Here's the code ...

def majority(l)
    h = Hash.new(0)
    l.each { |v| h[v] += 1 }
    max = h.max {|a,b| a[1] <=> b[1]}
    max[1] >= l.length / 2.0 ? "#{max[0]} won with #{max[1]} out of #{l.length} total votes" : "No winner"
end

v1 = %w[A A A C C B B C C C B C C]
v2 = %w[A B C A B C A]

puts "v1 = #{majority(v1)}"
puts "v2 = #{majority(v2)}"

OK, so we create a Hash where the value in the key/value pair is initialized to 0 and we add one each time time we see a vote. Here's the cool part, we use the max function from Enumerable knowing that max itself uses each which will give us the key/value pair as a two element array. So, we set the function to compare the second element, the value, of the pair. The max that gets returned will also be a two element array and we use that to calculate whether there's a winner or not.

These types of problems are great for sharpening your programming skills in general and your ruby skills in particular. They're exactly the types of questions that end up on programming interview tests.

Let me know if you have questions or comments.

Centroids

Here's another one from my clustering code, slightly modified for explanatory purposes. This post was spurred by a conversation with a couple of friends when we were talking about programming, who said that programming languages mostly had the same things (i.e. everything old is new again). While I'm not completely in opposition to that, I think that some of the newer languages (read Ruby) do have a lot to offer when it comes to compactness.

Here the problem is to find the centroid of a number of points. Basically, we want the averages of all the first values of the points, the second values of the points, etc. Let's show an example ...

If we have [[x0, y0, z0], [x1, y1, z1], ... [xn, yn, zn]] as our points then the centroid would be the point given by [sum(x's) / n, sum(y's)/n, sum(z's) / n] (and here I'm too lazy to figure out how to create the capital Sigma for sum). Think about how you might solve this in your favorite programming language. In C/C++ anyway, you'd need to know in advance how many points and how many dimensions (the example above uses three, but it could be more or less). In Java, you don't need that, but I don't think you could do something like this ...

    def centroid(x)
        x[0].zip(*x[1..x.length-1]).map { |v| v.inject(:+) / v.length.to_f }
    end

OK, so what's going on here? First up we're going to take the first element of the array (above the [x0, y0, z0]) and zip it with the rest of the array with *x[1..x.length-1]. This should give us, once again from the example above [[x0, x1, ...xn], [y0, y1, ... yn], [z0, z1 ... zn]]. So basically, a new array with all the x's gathered together, the y's, etc. Now comes map. Here, we'll take each element, which is itself an array and then use inject to run through each of the items, say [x0, x1, ... xn] and sum them together. Finally, we'll divide by the length of the element to get an average. If it's hard to see this, break it into pieces and then run it through using irb. It should be a bit easier to see there.

So, what's the point here. Well, essentially you could this in any programming language or in fact any Turing machine, but the language itself can make it easier freeing you to do other things or harder.

Let me know if you have any questions or comments.