Sunday, December 28, 2008

Windows and problem playing DVDs

The other day on a short road trip we tried playing a DVD on our laptop and got the following message:

"Windows Media Player cannot play this DVD because there is a problem with digital copy protection between your DVD drive, decoder, and video card. Try installing an updated driver for your video card."

As soon as I got an internet connection, I tried downloading a driver for my video card. After installing it, I assumed that the DVD would play (after all, the message told me it would!).

No luck. I then downloaded new firmware and drivers for my DVD drive. The message did say something about my DVD drive too ...

Again, no luck. Well, what else is there? Then I noticed the word 'decoder' in the message and decided to pursue that option.

I tried a few things, but the one that actually did the trick was to download the XP Codec Pack. After downloading and installing these codecs, I was finally able to play the DVD that we brought to occupy our time on our short road trip!

Thursday, November 27, 2008

Comparing XML Documents Semantically

I have been interested in comparing the contents of XML documents for some time now.

What I wanted to do is to perform a 'Diff' on 2 XML documents and determine whether or not they are the same document (regardless of whitespace or child element ordering).

In JAVA, I came across XmlUnit. This piece of software is excellent for determining whether or not 2 separate XML documents are equal.

In Perl, I came across XML-SemanticDiff. I thought it was great until I re-ordered the elements in one of my documents. Then this module wasn't so great anymore.

Since I really needed a piece of software equivalent to XmlUnit in Perl, I decided to create my own module and to call it XML-SemanticCompare. This new module really does perform a semantic diff on XML documents:
  • Child element re-ordering doesn't result in false negatives.
  • Whitespace is trimmed from text by default when comparing text and attribute values [can be turned off].
  • Attributes can be ignored [turned off by default].

Using the module is extremely straightforward:
  use XML::SemanticCompare;
my $x = XML::SemanticCompare->new;

# compare 2 different files
my $isSame = $x->compare($control_xml, $test_xml);
# are they the same
print "XML matches!\n"
if $isSame;
print "XML files are semantically different!\n"
unless $isSame;

# get the diffs
my $diffs_arrayref = $x->diff($control_xml, $test_xml);

# test xpath statement against XML
my $success = $x->test_xpath($xpath, $test_xml);
print "xpath success!\n" if $success;

The only downside to this piece of software is that it isn't very efficient (although, it isn't terribly inefficient either).

If you find yourself trying to compare XML documents DOM trees for equality and you are using Perl, please check out XML-SemanticCompare. If you can make the code more robust and efficient, please do!

Tuesday, November 25, 2008

A perl script to rename your files

For those of you looking for a cool way to rename your files, here is a little script that could work wonders for you.

Basically, this scripts takes in a PERL substitution expression and applies it to the filename of interest.

To get started, copy the script below into a file called ''. Once you have done that, then read the usage instructions by running the script with the '-h' option. Of course, you will need perl installed on your machine.

So why write this script? Because I got sick and tired of manually renaming files by hand. Some people have files called 'some_file.txt'. I prefer that file to be called 'SomeFile.txt'.

To accomplish this, all I have to do is run the script like so:

   perl -u "s/_/ /g" some_file.txt

followed by
   perl "s/ //g" "Some File.txt"

Note: The -u option causes the first letter of each word separated by a space to be capitalized.

Okay, maybe its quicker to manually do this for a single file, but if you are on Windows and have a whole folder full of files like that, then the process is:

   for %v in (*.txt) do perl -u "s/_/ /g" "%v" 
   for %v in (*.txt) do perl "s/ //g" "%v"

That is all there is to it! All of the files will be renamed with 'camel' text names.

Please make sure that before you attempt to batch rename files, that you have them backed up first!

Script start [hint: dont copy this line ;-)]
#!/usr/bin/perl -w

 use Getopt::Std;
 use vars qw/ $opt_h $opt_u /;

 # usage
        sub usage {
  print STDOUT <<'END_OF_USAGE';

  Usage: rename [-hu] sub_regex [files]

                sub_regex is any expression that you would like to
                apply to filename. You can use capturing or just 

                -h .... shows this message ;-)
                -u .... makes first letter of each word uppercase


         perl "s/_/ /g" rename_me.txt
                   This renames rename_me.txt to "rename me.txt"

         perl -u "s/_/ /g" rename_me.txt
                   This renames rename_me.txt to "Rename Me.txt"

                3. MS Windows example
         for %v in (*.mp3) do perl -u "s/_/ /g" "%v"
                   This renames all mp3 files in the current directory such that
                   every word begins with a capital letter and all underscores
                   are replaced with spaces.


 if ($opt_h) {


# get the substition regex
$op = shift or (usage() and exit(1));

# go through file names
chomp(@ARGV) unless @ARGV;
for (@ARGV) {
    $was = $_;
    eval $op;
    die $@ if $@;
    # cap first letter of each word
    my $newname = "";
    my @components = split(/ /, $_);
    foreach (@components) {
      my $x = $_;
      $x = ucfirst lc $x if $opt_u;
      if ($newname eq "") {
         $newname = "$x";
      } else { 
         $newname .= " $x";
    rename($was,$newname) unless $was eq $newname;

Script End [hint: dont copy this line ;-)]

Wednesday, August 27, 2008

Freebase & Google App Engine - Write Queries

If you would like to use Freebase with your application that is hosted with Google App Engine, then you may have run into some difficulties trying to get write queries to work!

Now that I have been able to get writes to work, I am going to write down what it is that I did so that other people don't run into the same problems that I did!

Basically, to get writes to work, I created a proxy between my app engine page and freebase. In other words, I had a form that allows you to modify an entity in freebase and upon submit, the mql write statement was created and sent to the app engine. The app engine then forwarded this request to Freebase and then waited for the response. The response was then forwarded back to the to the client page.

To get this to work, you need to first obtain the cookies that are set when you authenticate with Freebase. To do this, we have a simple python subroutine:

# domain is either or
# user is the username
# pass is the password
def login(domain, user, pass):
result = urlfetch.fetch(
+ domain
+ '/api/account/login?username='
+ user
+ '&password='
+ pass
+ '&rememberme=true',

return result.headers['set-cookie']

This subroutine basically calls the login service for Freebase and returns any cookies that their server says to save.

The next thing that you need to do is to forward your write mql to Freebase. This can be done in a similar manner:

# domain is either or
# user is the username performing the write
# pass is the password for the user performing the write
# mql is the write query wrapped in the query envelope
def write_to_freebase(domain, user, pass, mql):
form_fields = {
form_data = urllib.urlencode(form_fields)
cookies = login(domain, user, pass)
result = urlfetch.fetch(
url='http://' + domain+ '/api/service/mqlwrite',
'Content-Type': 'application/x-www-form-urlencoded',
'Cookie' : cookies
return result.content;

This method simply calls the mqlwrite service and forwards the JSON object to you.

The imports required are:

from google.appengine.api import urlfetch
import urllib

That is all that is required in order to get writes to work with google app engine!

Please let me know if I missed anything so that I can update this post!

Thanks and good luck!

Wednesday, June 18, 2008

My Solaris .profile

I wanted to post my .profile file online because I have had to re-create it many times and I thought that others might find it useful!

Some of the things that the profile does are:
  • colors the prompt
  • adds shortcuts for ll and l (ls -lF and ls -laF)
  • fixes the delete (among others) key so that delete actually deletes characters, rather than add tildes!
  • adds mouse support to pico and makes 'nano' refer to pico (our system doesn't have nano)
  • sets up CDPATH (adds the directories in this variable to the search path when using cd)

# set the term to xterm
export TERM=xterm

# Set up VISUAL, EDITOR for crontab editing

# Set up a prompt
PS1="\u@\h:\w# "
export PS1="\e[1;31m$(echo $PS1)\e[m"

# Export all the above

# set up CDPATH
export CDPATH

# set up some aliases
alias ll="ls -lF"
alias l="ls -laF"

alias nano="pico -w -m"
alias pico="/usr/local/bin/pico -w -m"

# fix some keys
case $TERM in
bind '"\e[1~": beginning-of-line'
bind '"\e[3~": delete-char'
bind '"\e[4~": end-of-line'
bind '"\177": backward-delete-char'
# eof

All that you need to do is save the above text in your home directory under the filename '.profile'.

I have tested this on Solaris, SuSe Linux and Ubuntu Linux.

Good luck!

Tuesday, June 17, 2008

Deleting shared memory segments

Recently, I found myself using IPC in a multi-threaded perl script that I wrote.

After running the script a few times I eventually ran out of memory segments and so the script would just die (turns out that the script wasn't robust enough ;-)

So after searching around, I came up with 2 commands that go through and clean, (read delete) all segments (and semaphores) on the computer. The commands need to be run from the command line.

On Solaris:

ipcs -s | awk ' $5 == "ekawas" {print $2, $5}' | awk '{ print $1}' | while read i; do ipcrm -s $i; done

ipcs -m | awk ' $5 == "ekawas" {print $2, $5}' | awk '{ print $1}' | while read i; do ipcrm -m $i; done

On Linux/Unix:

ipcs -s | awk ' $3 == "ekawas" {print $2, $3}' | awk '{ print $1}' | while read i; do ipcrm sem $i; done

ipcs -m | awk ' $3 == "ekawas" {print $2, $3}' | awk '{ print $1}' | while read i; do ipcrm -m $i; done

Of course, you will need to change "ekawas" to a more appropriate username. To find out the username that you should be using, run 'ipcs -s' or 'ipcs -m' and see who created the segments.

Please be careful, because all segments created by all users will be listed; so don't delete ones that you didn't create!

Friday, April 25, 2008

dos2unix On Every File in a Dir

I am by no means a Linux expert and I find myself forgetting obvious commands if I haven't used them for a while.

Today, I found myself needing to apply the app 'dos2unix' on a bunch of files recursively in a directory.

After fumbling for a few minutes, this is what I came up with:

    find . -type f -exec dos2unix '{}' \;

Now hopefully, I wont forget again!

Thursday, January 17, 2008

Enabling the Home/End/Delete Keys on a *nix Machine

Make the Home/End/Delete key function like you would assume that they would!

In your .profile file (create it if necessary in your home directory), if you add the following to the bottom (or top), you can enable the Home/End/Delete keys:

case $TERM in
bind '"\e[1~": beginning-of-line'
bind '"\e[3~": delete-char'
bind '"\e[4~": end-of-line'
bind '"\177": backward-delete-char'
This would enable the hot keys for your profile.

If you want to set them up system wide, then (as root) add the following to the end of your /etc/inputrc file:

"\e[1~": beginning-of-line
"\e[4~": end-of-line
"\e[3~": delete-char
"\177": backward-delete-char