Freebase & Google App Engine - Write Queries

If you would like to use Freebase with your application that is hosted with Google App Engine, then you may have run into some difficulties trying to get write queries to work!

Now that I have been able to get writes to work, I am going to write down what it is that I did so that other people don't run into the same problems that I did!

Basically, to get writes to work, I created a proxy between my app engine page and freebase. In other words, I had a form that allows you to modify an entity in freebase and upon submit, the mql write statement was created and sent to the app engine. The app engine then forwarded this request to Freebase and then waited for the response. The response was then forwarded back to the to the client page.

To get this to work, you need to first obtain the cookies that are set when you authenticate with Freebase. To do this, we have a simple python subroutine:

# domain is either sandbox.freebase.com or www.freebase.com
# user is the username
# pass is the password
def login(domain, user, pass):
    result = urlfetch.fetch(
             url='http://'
                  + domain
                  + '/api/account/login?username='
                  + user
                     + '&password='
                  + pass
                  + '&rememberme=true',
                 payload={},
              method=urlfetch.POST,
                 headers={
                 'X-Metaweb-Request':'x-metaweb-request'
                 }
       )
    return result.headers['set-cookie']

This subroutine basically calls the login service for Freebase and returns any cookies that their server says to save.

The next thing that you need to do is to forward your write mql to Freebase. This can be done in a similar manner:

# domain is either sandbox.freebase.com or www.freebase.com
# user is the username performing the write
# pass is the password for the user performing the write
# mql  is the write query wrapped in the query envelope
def write_to_freebase(domain, user, pass, mql):
    form_fields = {
        "queries":mql,
    }
    form_data = urllib.urlencode(form_fields)
    cookies = login(domain, user, pass)
    result = urlfetch.fetch(
                url='http://' + domain+ '/api/service/mqlwrite', 
                payload=form_data,
                method=urlfetch.POST,
                headers={
                    'Content-Type': 'application/x-www-form-urlencoded', 
                    'X-Metaweb-Request':'x-metaweb-request', 
                    'Cookie' : cookies
                }
             )
    return result.content;

This method simply calls the mqlwrite service and forwards the JSON object to you.

The imports required are:


from google.appengine.api import urlfetch
import urllib

That is all that is required in order to get writes to work with google app engine!

Please let me know if I missed anything so that I can update this post!

Thanks and good luck!

My Solaris .profile

I wanted to post my .profile file online because I have had to re-create it many times and I thought that others might find it useful!

Some of the things that the profile does are:

• colors the prompt
  
• adds shortcuts for ll and l (ls -lF and ls -laF)
• fixes the delete (among others) key so that delete actually deletes characters, rather than add tildes!
• adds mouse support to pico and makes 'nano' refer to pico (our system doesn't have nano)
  
• sets up CDPATH (adds the directories in this variable to the search path when using cd)

# set the term to xterm
export TERM=xterm

# Set up VISUAL, EDITOR for crontab editing
VISUAL=/usr/local/bin/pico
EDITOR=${VISUAL}

# Set up a prompt
PS1="\u@\h:\w# "
export PS1="\e[1;31m$(echo $PS1)\e[m"

# Export all the above
export VISUAL EDITOR PS1

# set up CDPATH
CDPATH=$HOME
export CDPATH

# set up some aliases
alias ll="ls -lF"
alias l="ls -laF"

alias nano="pico -w -m"
alias pico="/usr/local/bin/pico -w -m"


# fix some keys
case $TERM in
xterm*)
    bind '"\e[1~": beginning-of-line'
    bind '"\e[3~": delete-char'
    bind '"\e[4~": end-of-line'
    bind '"\177": backward-delete-char'
    ;;
esac
# eof

All that you need to do is save the above text in your home directory under the filename '.profile'.

I have tested this on Solaris, SuSe Linux and Ubuntu Linux.

Good luck!

Deleting shared memory segments

Recently, I found myself using IPC in a multi-threaded perl script that I wrote.

After running the script a few times I eventually ran out of memory segments and so the script would just die (turns out that the script wasn't robust enough ;-)

So after searching around, I came up with 2 commands that go through and clean, (read delete) all segments (and semaphores) on the computer. The commands need to be run from the command line.

On Solaris:

ipcs -s | awk ' $5 == "ekawas" {print $2, $5}' | awk '{ print $1}' | while read i; do ipcrm -s $i; done 

ipcs -m | awk ' $5 == "ekawas" {print $2, $5}' | awk '{ print $1}' | while read i; do ipcrm -m $i; done

On Linux/Unix:

ipcs -s | awk ' $3 == "ekawas" {print $2, $3}' | awk '{ print $1}' | while read i; do ipcrm sem $i; done

ipcs -m | awk ' $3 == "ekawas" {print $2, $3}' | awk '{ print $1}' | while read i; do ipcrm -m $i; done

Of course, you will need to change "ekawas" to a more appropriate username. To find out the username that you should be using, run 'ipcs -s' or 'ipcs -m' and see who created the segments.

Please be careful, because all segments created by all users will be listed; so don't delete ones that you didn't create!