Unescaping UTF-8 Strings in Ruby 1.9

Today Radar and I encountered a little issue with String encodings in Ruby 1.9. In this project, some Merb UTF-8 params needed to be unescaped. We spent some trying to force strings into UTF-8 encoding but for some reason while the encoding was UTF–8, the actual contents of the strings were getting massacred.

Long story short:

# In Ruby 1.9

# Broken
puts URI::unescape # => "Baden-Württemberg"

# Working
puts CGI::unescape # => "Baden-Württemberg"

Another lesson we learnt on our adventures today is the following:

m1 = "Munich"
puts = m1.encoding  # let's suppose it is something other than UTF-8, such as ASCII-8Bit

m2 = "München"
puts = m2.encoding  # let's suppose we have a UTF-8 encoded string


# The following will raise an exception about mismatched encodings
m3 = m1 << m2

# The easiest way to get around this is to do the concatenation like so:
m3 = m3 = m1 << m2.force_encoding

Colemak

About a year ago I found out about the Colemak keyboard layout. This is what it looks like:

I was instantly intrigued and was drawn to a few key features:

• Easy to learn
• Caps lock is replaced with backspace (making backspace closer to fingers and getting rid of the useless caps lock)
• Obvious health benefits (just look how much your fingers need to move when using QWERTY here)
• It’s not QWERTY, which was designed without any consideration to ergonomic or efficiency aspects.

Long story short: it took me a few tries to commit to learning it (3 tries over year in fact). Eventually I went cold turkey for a week and haven’t looked back. Typing is more comfortable and faster already.

Getting Colemak on OS X is as simple as following the simple Installation instructions. The instructions seem out of date when it comes to re-mapping caps lock to backspace. It is as simple as downloading PCKeyboardHack, installing it, rebooting, and choosing the option in its preference pane to remap caps lock to delete (key code 51).

Snow Leopard

Colemak itself works fine under Snow Leopard, however the PCKeyboardHack utility does not work, because it is hardcoded to only load the kernel driver if it recognises the OS version. Luckily it is open source, and I was able to import the Mercurial project into a GitHub project, modify it, and build a version that works fine on Snow Leopard.

Dead Keys that Apple uses on the QWERTY layout

One thing I loved about the QWERTY layout on OS X was the sensible dead key and alternate character behaviours, allowing users to quickly and logically input characters such as: , é, ¡, ¿, », ‡, °, etc. The version of Colemak, for one reason or another, moved or simply didn’t have many of these shortcuts.

So, I created my own Colemak layout bundle that moves the dead key positions to preserve the associations with the character position changes from QWERTY to Colemak. This means, for example,  can be created by pressing ⌥⇧K on either keyboard layout (where K is the key that outputs K, not the key that physically says K on it).

My version of the Colemak bundle is available on my GitHub and a direct download to a zip file is available here. Simply put the .bundle file in your ~/Library/Keyboard Layouts directory. The bundle includes both the original keyboard layout and my re-created one so you can switch between them if you need both, for any reason.

RubyCocoa and Keychain access

There have been a few posts asking how to access and manipulate the Keychain from within RubyCocoa but no answers supplied. As someone who is just getting into RubyCocoa (and Cocoa all together for that matter), I thought I’d document once and for all the process I took to get it working.

I am running on Leopard and while 10.5 is supposed to ship with BridgeSupport for most frameworks, the Security framework must have been left out. After playing around with using other frameworks (such as the AddressBook framework) and seeing how they were used, I did a bit of digging in the Apple Developer Documentation (incredible resource for anyone starting out) and stumbled across this gem: Generate Framework Metadata

After reading this, I was accessing the keychain within 5 minutes. Here’s how:

1. Run the following commands (first one will take some time):

   gen_bridge_metadata -f Security -o Security.bridgesupport
   mkdir -p /Library/BridgeSupport
   mv Security.bridgesupport /Library/BridgeSupport

2. Open an IRB session to test it:

>> require 'osx/cocoa'
=> true
>> OSX.require_framework 'Security'
=> true
>> defined? OSX::SecKeychainAddGenericPassword()
=> "method"

Hoorah! Two steps! Well, one because the second one was just testing… Now go make password-able RubyCocoa application! I am not sure how to pull this off with deployable apps but I suggest your apps could ship with the file or run those commands on first-run or if the Security.bridgesupport file doesn’t exist. Note: I have not tested either of those scenarios, but I would guess the latter would be more reliable for cross-version development (i.e. Tiger + Leopard)

How to use these methods

# Require needed libraries

require 'osx/cocoa'
include OSX
require_framework 'Security'

# Set up some relevant variables

service = "Test Service"
account = "MyUsername"
password = "MyPassword"

# Add password to default keychain (first param)

SecKeychainAddGenericPassword(nil, service.length, service, account.length, account, password.length, password, nil)

# Finding a password in default keychain (first param)

status, *password = SecKeychainFindGenericPassword(nil, service.length, service, account.length, account)

# Password-related data
password_length = password.shift # => 10
password_data = password.shift # OSX::ObjcPtr object
password = password_data.bytestr(password_length)

# The last item is another OSX::ObjcPtr. I haven't figured 
# out how to cast or use this yet but will post when I've
# figured it out
keychain_item = password.shift

# Yay it works! You can also check the password exists
# in the Keychain Access utility. I've confirmed that
# this works
puts password # => "MyPassword"

[UPDATE 4/8/08]: Below is a fix for the REXML bug some people have

been getting (myself included)

There is good news for anyone that has been getting the following error:

There is a very simple fix. This is caused by a typo in the source of REXML. It’s such a massive bug I can’t believe it made it into the release of Ruby but i did some poking around and it looks like they renamed a method and didn’t change all references to it.

There are two ways to fix it—patching REXML; and patching gen_bridge_metadata.

Personally I prefer patching the generator for peace of mind and have submitted the patch at http://bridgesupport.macosforge.org/trac/ticket/1, but I’ll show both fixes.

Patching gen*bridge*metadata

1. Open /usr/bin/gen_bridge_metadata and find the method definition forgenerate_xml`
2. Replace 0 in the xml_document.write() with -1 (both occurrences)
3. Save!

Patching REXML

1. Open /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rexml/document.rb
2. Find the line that has if trans and change it to if transitive
3. Save!

Generating Sound from Sine Waves on the iPhone

A few months ago I saw this extremely addictive flash game/toy called ToneMatrix.

My immediate thought was: "This would make quite possibly the coolest iPhone application ever."

So my good friend, Anthony Mittaz, and I decided to have a crack at it but hit one immediate hurdle; however proficient an iPhone developer Anthony is, he hadn’t done anything to do with sound, and nor had I, in any language/framework, let alone on the iPhone. Let me tell you this: Apple’s documentation in this area is severely lacking. Tangent: well, almost all Apple documentation on specialised things is pretty sparse.

I think we spent about two weeks’ worth of nights reading documentation, source code (Cocoa, iPhone, Java, C, and others), and plain outright stabbing in the dark code-wise getting not much more than ugly pops and crackles. Whenever we thought we had a clean note working, we’d change the frequency to what should be another nice clean note, and would instead get ear-shattering screeches.

Nonetheless we persevered and were even able to come up with a nifty little piece of code that could take an array of frequencies and play them together in what could be made to sound like a pleasant chord.

Here is the relevant code:

Code

TonePlayer.h

#import <Foundation/Foundation.h>
#include <AudioUnit/AudioUnit.h>
#include <math.h>

#define kTonesAvailable 16
#define kOutputBus 0
#define kInputBus 1
#define kNumChannels 2

@interface TonePlayer : NSObject {
  AudioComponentInstance audioUnit;
  AudioStreamBasicDescription audioFormat;
  float phase[kTonesAvailable];
  float frequencies[kTonesAvailable];
}

-(OSStatus)start;
-(OSStatus)stop;
-(void)resetFrequencies;
-(void)cleanUp;
-(void)intialiseAudio;
@end

// Some C function headers to get arount compile errors while I refactor to be more readable
float phaseOffsetFromFrequency(float);
OSStatus RenderCallback(void*, AudioUnitRenderActionFlags*, const AudioTimeStamp*, UInt32, UInt32, AudioBufferList*);

TonePlayer.m

#import "TonePlayer.h"

typedef struct {
  @defs(TonePlayer);
} SinewaveDef;

@implementation TonePlayer

static float gSampleRate = 44100.;
static float notes[111]  = {
      8.1758, 8.6620, 9.1770, 9.7227, 10.3009, 10.9134, 11.5623, 12.2499, 
      12.9783, 13.7500, 14.5676, 15.4339, 16.3516, 17.3239, 18.3540, 19.4454,
      20.6017, 21.8268, 23.1247, 24.4997, 25.9565, 27.5000, 29.1352, 30.8677, 
      32.7032, 34.6478, 36.7081, 38.8909, 41.2034, 43.6535, 46.2493, 48.9994, 
      51.9131, 55.0000, 58.2705, 61.7354, 65.4064, 69.2957, 73.4162, 77.7817, 
      82.4069, 87.3071, 92.4986, 97.9989, 103.8262, 110.0000, 116.5409, 123.4708,
      130.8128, 138.5913, 146.8324, 155.5635, 164.8138, 174.6141, 184.9972, 
      195.9977, 207.6523, 220.0000, 233.0819, 246.9417, 261.6256, 277.1826, 
      293.6648, 311.1270, 329.6276, 349.2282, 369.9944, 391.9954, 415.3047, 
      440.0000, 466.1638, 493.8833, 523.2511, 554.3653, 587.3295, 622.2540, 
      659.2551, 698.4565, 739.9888, 783.9909, 830.6094, 880.0000, 932.3275, 
      987.7666, 1046.5023, 1108.7305, 1174.6591, 1244.5079, 1318.5102, 1396.9129, 
      1479.9777, 1567.9817, 1661.2188, 1760.0000, 1864.6550, 1975.5332, 2093.0045, 
      2217.4610, 2349.3181, 2489.0159, 2637.0205, 2793.8259, 2959.9554, 3135.9635, 
      3322.4376, 3520.0000, 3729.3101, 3951.0664, 4186.0090, 4434.9221, 4698.6363
  };

OSStatus    RenderCallback(void                          *inRefCon,
                           AudioUnitRenderActionFlags    *ioActionFlags,
                           const AudioTimeStamp          *inTimeStamp,
                           UInt32                        inBusNumber,
                           UInt32                        inNumberFrames,
                           AudioBufferList               *ioData)
{
  SinewaveDef* def = inRefCon;

  float *outL = ioData->mBuffers[0].mData;
  float *outR = ioData->mBuffers[1].mData;

  float wave, j;
  float freqs[kTonesAvailable];

  memset(freqs, 0.0f, sizeof(freqs));

  for (int i=0; i< inNumberFrames; i++){
    wave = 0.0f;
    j = 0.0f;

    for(int m = 0; m < kTonesAvailable; ++m) {
      if(def->frequencies[m] != 0) {
        if(freqs[m] == 0) freqs[m] = phaseOffsetFromFrequency(def->frequencies[m]);

        wave += 0.5 * sinf(def->phase[m]);
        def->phase[m] += freqs[m];

        ++j;
      }
    }

    wave /= j;

    *outL++ = wave;
    *outR++ = wave;
  }

  return noErr;
}

float phaseOffsetFromFrequency(float frequency) {
  return (float)(frequency * 2.0 * M_PI / gSampleRate);
}

-(void)resetFrequencies{
  memset(frequencies, 0.0f, sizeof(frequencies));
  memset(phase, 0.0f, sizeof(phase));
}

-(OSStatus)start{
  [self resetFrequencies];
  OSStatus status = AudioOutputUnitStart(audioUnit);

  // See http://www.phys.unsw.edu.au/jw/notes.html for note numbers
  frequencies[0] = notes[60]; // C
  frequencies[1] = notes[64]; // E
  frequencies[2] = notes[67]; // G

  sleep(1);

  frequencies[0] = notes[64];
  frequencies[1] = notes[67];
  frequencies[2] = notes[71];

  sleep(1);

  frequencies[0] = notes[74];
  frequencies[1] = notes[67];
  frequencies[2] = notes[70];


  return status;
}

-(OSStatus)stop{
  return AudioOutputUnitStop(audioUnit);
}

-(void)cleanUp{
  AudioUnitUninitialize(audioUnit);
}

-(AudioStreamBasicDescription)audioFormat{
  return audioFormat;
}

// Below code is a cut down version (for output only) of the code written by
// Micheal "Code Fighter" Tyson (punch on Mike)
// See http://michael.tyson.id.au/2008/11/04/using-remoteio-audio-unit/ for details
-(void)intialiseAudio{
  OSStatus status;

  // Describe audio component
  AudioComponentDescription desc;
  desc.componentType = kAudioUnitType_Output;
  desc.componentSubType = kAudioUnitSubType_RemoteIO;
  desc.componentFlags = 0;
  desc.componentFlagsMask = 0;
  desc.componentManufacturer = kAudioUnitManufacturer_Apple;

  // Get component
  AudioComponent inputComponent = AudioComponentFindNext(NULL, &desc);

  // Get audio units
  status = AudioComponentInstanceNew(inputComponent, &audioUnit);

  UInt32 flag = 1;
  // Enable IO for playback
  status = AudioUnitSetProperty(audioUnit, 
                  kAudioOutputUnitProperty_EnableIO, 
                  kAudioUnitScope_Output, 
                  kOutputBus,
                  &flag, 
                  sizeof(flag));

  // Describe format
        audioFormat.mSampleRate = gSampleRate;
  audioFormat.mFormatID = kAudioFormatLinearPCM;
  audioFormat.mFormatFlags = kAudioFormatFlagsNativeFloatPacked | kAudioFormatFlagIsNonInterleaved;
  audioFormat.mFramesPerPacket = 1;
  audioFormat.mBytesPerPacket = sizeof(Float32);
  audioFormat.mBytesPerFrame = sizeof(Float32);
  audioFormat.mChannelsPerFrame = kNumChannels;
  audioFormat.mBitsPerChannel = 32;

  //Apply format
  status = AudioUnitSetProperty(audioUnit, 
                  kAudioUnitProperty_StreamFormat, 
                  kAudioUnitScope_Input, 
                  kOutputBus, 
                  &audioFormat, 
                  sizeof(audioFormat));

  // Set up the playback  callback
  AURenderCallbackStruct callbackStruct;
  callbackStruct.inputProc = RenderCallback;
  //set the reference to "self" this becomes *inRefCon in the playback callback
  callbackStruct.inputProcRefCon = self;

  status = AudioUnitSetProperty(audioUnit, 
                  kAudioUnitProperty_SetRenderCallback, 
                  kAudioUnitScope_Global, 
                  kOutputBus,
                  &callbackStruct, 
                  sizeof(callbackStruct));

  // Initialise
  status = AudioUnitInitialize(audioUnit);
}

@end

I really wish that I could remember how most of it worked, or, more importantly, the multitude of projects whose code we had to look at to get it working. If you recognise any of the code or can help add references on how people can build up a nice list of references for people to use if are following a similar pursuit.

If you are wondering what ever happened to the application we were building, it seems we weren’t the only ones who recognised the potential of ToneMatrix as an iPhone application. During the 2-3 weeks of us developing the sound generation code, about 5-6 ToneMatrix clones were added to the AppStore. While most were crap, there was one that we felt left little to be improved upon and we therefore recommend everyone goes and downloads Melodica. It’s just as addictive as the Flash version, only portable!

Snow Leopard for the Ruby Developer

After the 2009 WWDC release, I decided to give Snow Leopard a try on my production machine. Crossing my fingers, expecting a plethora of broken gems and databases, I wiped my hard drive clean and installed the latest build.

Much to my surprise, most things worked flawlessly and exactly as it did on Leopard.

What Ships with Snow Leopard

$ gem --version
1.3.1

$ ruby --version
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0]

$ gem list

*** LOCAL GEMS ***

actionmailer (2.2.2, 1.3.6)
actionpack (2.2.2, 1.13.6)
actionwebservice (1.2.6)
activerecord (2.2.2, 1.15.6)
activeresource (2.2.2)
activesupport (2.2.2, 1.4.4)
acts_as_ferret (0.4.3)
capistrano (2.5.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
dnssd (0.6.0)
fastthread (1.0.1)
fcgi (0.8.7)
ferret (0.11.6)
gem_plugin (0.2.3)
highline (1.5.0)
hpricot (0.6.164)
libxml-ruby (1.1.2)
mongrel (1.1.5)
needle (1.3.0)
net-scp (1.0.1)
net-sftp (2.0.1, 1.1.1)
net-ssh (2.0.4, 1.1.4)
net-ssh-gateway (1.0.0)
rails (2.2.2, 1.2.6)
rake (0.8.3)
RedCloth (4.1.1)
ruby-openid (2.1.2)
ruby-yadis (0.3.4)
rubynode (0.1.5)
sqlite3-ruby (1.2.4)
termios (0.9.4)
xmpp4r (0.4)

Quite possibly these default gems and versions will change between now and the final release of Snow Leopard. For instance, this seed comes with Rails 2.2.2, even though we are already at 2.3.2. Snow Leopard doesn’t come out till September and by then we could have an even newer version of Rails that might be bundled.

What doesn’t work, and how to fix it.

Textmate - Everything works fine except for the following shortcuts: ⌘←, ⌘→, ⌘⇧←, and ⌘⇧→. These shortcuts are for moving the text insertion point to beginning and end of line, and selecting to beginning and end of line, respectively. The fix is easy, simply download and double click the TextMate macros attached to this ticket

Nokogiri - Nokogiri installs fine but gives error “native.bundle: mach-o, but wrong architecture” when used. Getting around this is as easy as running sudo gem install nokogiri -s http://tenderlovemaking.com/ to get the latest development release which includes a fix for this.

do_sqlite3 - I haven’t personally used this or tried to fix it but I saw report of it not working by @benlovell

Passenger PrefPane - Preference pane does not load. No known fix (it’s probably an easy fix but I haven’t looked enough into it to try).

Ghost - Manage hostnames effortlessly

Background

Ghost is a gem that provides a command and a Ruby API for managing hostnames locally, much as people do manually by editing /etc/hosts now.

Sometime last year I was having to add lots of Passenger virtual hosts on my development machine and felt that having to constantly edit the /etc/hosts file was a bit archaic for my liking and that there must be a nicer way of accomplishing the same task.

Having seen the Passenger Preference Pane added hostnames without requiring the user to do anything special, I had a peak into its source and discovered the dscl (man page) command that comes with Leopard.

For the hell of it I decided to write my Ghost gem using this command and use it to make a nice wrapper that has an easier syntax than dscl, and that is less effort than manually editing my hosts file. Happy with the result, a few other people requested support for other unix systems and Mitchell V Riley was kind enough to patch Ghost to fall back to processing and editing the /etc/hosts file when not on Leopard.

Usage

$ ghost add mydevsite.local
  [Adding] mydevsite.local -> 127.0.0.1

$ ghost add staging-server.local 67.207.136.164
  [Adding] staging-server.local -> 67.207.136.164

$ ghost list
Listing 2 host(s):
  mydevsite.local      -> 127.0.0.1
  staging-server.local -> 67.207.136.164

$ ghost delete mydevsite.local
  [Deleting] mydevsite.local

$ ghost list
Listing 1 host(s):
  staging-server.local -> 67.207.136.164

$ ghost modify staging-server.local 64.233.167.99
  [Modifying] staging-server.local -> 64.233.167.99

$ ghost list
Listing 1 host(s):
  staging-server.local -> 64.233.167.99

$ ghost empty
  [Emptying] Done.

$ ghost list
Listing 0 host(s):

Other

Since creating Ghost I’ve used the Ruby library to add an initializer to my Rails projects that add the domain locally if running under Passenger in development mode (some configuration is needed to let it do this as it needs sudo).

Also, I have heard reports that entries in the /etc/hosts file are ignored in Snow Leopard. I haven’t tried because I only ever use Ghost now, but I do know that Ghost still works well.

Caveats

It only works on Unix-based systems and the /etc/hosts path is hard-coded. Windows support is entirely possible if somebody wants to use it. It isn’t something I am likely to do as I haven’t booted Windows in at least a year. In fact, I don’t think I have even ever installed Ruby on Windows before…