Learning Go (and hitting C libraries with it, too)

Update: This is already underway in a real fashion. Check out go-python for more information. Looks like a solid project!

Being the type of guy that doesn’t like to learn anything at the surface level, I decided to look into how one would integrate an existing C library with a Go application. My curiosity was two fold. First, I hoped to learn a bit more about what lives under the Go-hood.  Secondly, being a new language, my assumption is that there are a lot of native libraries that haven’t yet been wrapped.

I spent a while trying to hunt down a simple, step-by-step guide that would lay out all of the steps needed to do this.  Python has excellent documentation when it comes to handing extending and embedding. I was hoping that I’d come across such a document written from a Go perspective, but I wasn’t able to find one.  Using examples and tool documentation, however, I was able to get some test code put together. Below you’ll find a rundown of what I did and how I did it.

My goal was to link in Python and execute Python code from within a Go executable. I wouldn’t recommend ever doing this for any reason, ever. Actually, you may want to just stop reading now.

Using the cgo Compiler

When building Go code that needs to interface with an existing C library, you’ll compile using the cgo compiler instead of the usual ${num}g.  So, the approach I took was to isolate all of that code into a specific library and then access those functions as I would any other package.

The “C” Package

Generally speaking, C provides a flat, non-hierarchical namespace. Protection is limited to source file scope and is dictated by the programmer’s use of the static keyword. Simply put, everything is visible everywhere unless declared static. This is why C API definitions usually start with an identifying prefix. For Python developers, consider the Py_ prefix. It provides a way to create independent namespaces (though, with no protection between them).
When interfacing with C libraries, Go wraps everything C under the “C” package.  For example, printf as defined in stdio.h becomes C.printf. This provides a nice demarcation between the Go world and the everything else world.

Including Headers

So, now we know what we’ll use to build the code and how we’ll interface with it.  The next step is to ensure Go knows where it can find the necessary headers.  To do this, we embed the information in comments immediately preceding the import “C” line.

// #include <Python.h>
import "C"

This ensures that the definitions in Python.h are visible to cgo. Everything found here will be made available under the C namespace and is accessible to code within this source module.  Now, as an aside, I noticed that C preprocessor macros were not expanded. So, because PyRun_SimpleString is simply a macro that calls PyRun_SimpleStringFlags, I had to use the latter.

The Makefile

We’ll be using the following makefile.  This comes from the example code under $GOROOT/misc/cgo/gmp.

# Copyright 2009 The Go Authors.  All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

include ${GOROOT}/src/Make.inc

TARG=pygo

# Can have plain GOFILES too, but this example doesn't.

CGOFILES=\
  pygo.go

CGO_LDFLAGS=-lpython2.6
CGO_CFLAGS = -I/usr/include/python2.6

# To add flags necessary for locating the library or its include files,
# set CGO_CFLAGS or CGO_LDFLAGS.  For example, to use an
# alternate installation of the library:
# CGO_CFLAGS=-I/home/rsc/gmp32/include
# CGO_LDFLAGS+=-L/home/rsc/gmp32/lib
# Note the += on the second line.

CLEANFILES+=pygo

include ${GOROOT}/src/Make.pkg

# Simple test programs
pygo:
  $(GC) pygo.go
  $(LD) -o $@ pygo.$O

There’s a not very much going on here. If you compare this file to the example, we’re simply changing a few internal variables.

• TARG is the name of the library we’ll be building.
• CGO_LDFLAGS & CGO_CFLAGS are generally the same as LDFLAGS & CFLAGS. They’ll be passed on during the compilation and linking phases.
• CLEANFILES is pretty self-explainatory.

Finally, we add the pygo rule at the bottom and remove the example rules for gmp.

The pygo Package

Now that we have our Makefile setup and ready to go, we need to put together our actual Go package. Below you’ll find the listing as I used it. Note that in a few places I purposefully used the long way as a learning tool.

package pygo

/* "C" isn't a real package. Rather, it's a virtual one created when building
 * via cgo. Cgo, as opposed go 6g, allows us to access C code/libraries from
 * within a Go application. Everything in the C-world is then jammed under the
 * 'C.' namespace.
 */

// #include <Python.h>
// #include <pythonrun.h>
import "C"

/* Some basic file IO helpers live here. */
import "io/ioutil"

/* Create a PythonCode object, which is a struct, containing
 * just one field. There's no reason this couldn't have
 * just been a type PythonCode string. When I started,
 * my intention was to be a bit more complicated and store
 * return values and interpreter state and all that, but
 * there's really no point.
 */
type PythonCode struct {
  code string
}

/* In this case, a pointer to a PythonCode object is the receiver,
 * our function is called ExecPy, and there is no return value.
 * we call our internally defined init/exec/term functions.
 *
 * Two things to note here:
 * 1. i/e/t are lower case, which means that they are the equivilent
 *    of a C top level 'static void F().' Not visible outside of
 *    the package (Encapsulation FTW!).
 *
 * 2. This is a METHOD because it has a defined receiver.
 */
func (p *PythonCode) ExecPy(result chan int) {
  initPython("PYTHON_EXEC")
  py_response := execPython(p.code)
  termPython()
  result <- py_response
}

/* This is our init method. There is no __init__ or __del__
 * equiv.  This builds a new object and returns a pointer to it.
 * Note that here, it's 100% okay that we're returning what
 * appears to be a pointer to a local variable. I know,
 * makes me feel a bit uncomfortable, but the docs clear
 * that up.
 */
func NewPythonCode(src *string) *PythonCode {
  contents, _ := ioutil.ReadFile(*src)
  return &PythonCode{code: string(contents)}
}

/* Here we tickle the Python C API. We set the program name,
 * initialize state, and ensure we init correctly.
 */
func initPython(interp_name string) {
  C.Py_SetProgramName(C.CString(interp_name))
  C.Py_Initialize()
  if C.Py_IsInitialized() == 0 {
    panic("Could not init interp.")
  }
}

/* This executes the code. This probably could also be a method on
 * our structure-based type above, but I wanted to highlight
 * the differences between a Go method and a Go function.
 */
func execPython(python_code string) int {
  return int(C.PyRun_SimpleStringFlags(C.CString(python_code), nil))

}

/* Fall Py_Finalize and free up interp. data structures. Again,
 * this probably would work better using an interpreter Go
 * object and then a method to pass in code to exec, but this
 * covers more ground
 */
func termPython() {
  C.Py_Finalize()
}

There’s quite a bit going on in there, so let’s step through it in a bit more detail.

1. The first few lines are straight forward. We name our package pygo, include the correct C headers, and import io/ioutil.
2. Next, we create a PythonCode type, which is struct. This object simply contains a string which will be actual Python code.  As the comment states, we could have done this simpler, but I think the struct approach is probably more representative of idiomatic Go (correct me, someone?).
3. Next, we define ExecPy. Since it has a capital ‘E’, it is exported from the package and available elsewhere.  This method does three things, all by calling other functions internal to the package.  Notice that it takes a channel of int objects as its sole parameter. We pass the return value of execPython back up the channel instead of simply returning it.  In the real world, a return value would be simpler, however, this illustrates the usage of channels.
4. After that, we have NewPythonCode. As Go doesn’t provide initializers/constructors, we simply do the build in an appropriately named factory function like this.  Here, you see the calls directly into the Python library. In this case, C.Py_SetProgramName, C.Py_Initialize, and C.Py_IsInitialized. In actuality, Go is simply calling Py_Initialize, Py_SetProgramName, and Py_IsInitialized. The other thing to note here is the use of C.CString(). This converts a Go string object into a C char *.
5. Next up is execPython. Here, we actually return the int value from C.PyRun_SimpleStringFlags. The int call/cast is necessary as a C int is not the same as a go int. Again, you’ll see the call to C.CString converting from string to char *.
6. Finally, we simply call Py_Finalize from within termPython.

Pretty simple if you ask me.  I’m becoming a fan of the “C.$identifer” approach as it allows code authors to avoid having to write stub modules in C. (SWIG Status, anyone?).

The Driver

The driver is pure Go. There’s no C code whatsoever. We simply import our new module and call it as we would any other Go library.

/* Everything goes into a package. */
package main

/* Could import individually, but this is cleaner. Sort of like wrapping Python
 * imports in order to group them on multiple lines.
 */
import ("pygo"
  "flag"
  "fmt"
  "stdio"
)

/* Flag Parsing. This is silly easy. */
var pythonSource = flag.String("pysource", "test.py", "Python Source File")

/* Main entry point. Just like C/C++ */
func main() {
  flag.Parse()

  /* Print a goofy little header so we can visualize the Go exec
   * vs. the Python exec.
   */
  fmt.Printf("Exec Python: %s\n", *pythonSource)
  fmt.Printf("-----------------------------\n\n")
  stdio.Stdout.Flush()

  /* Create a channel, which is like a queue. We'll block on
   * this for our Pythoning to finish.
   */
  ch := make(chan int)

  /* Create, set code, Exec Py. But.. do it in a different thread
   * of control! All we did was fire it after the keyword "go."
   * Sweet.
   */
  go pygo.NewPythonCode(pythonSource).ExecPy(ch)

  /* This says "fill python_result with whatever comes out
         * of the channel." This is a blocking call.
   */
  python_result := <-ch
  fmt.Printf("\n\n-----------------------------\n")

  /* Another way of printing... */
  fmt.Println("Channel Returned Code: ", python_result)
}

That’s all there is to it. Again, let’s walk through this source listing.

1. We name our package and import our dependencies. Note that our new package appears here and it is listed like any other package.
2. Next we setup a –pysource flag, so we can specify a Python file on the command line. This lets us run arbitrary Python code from our Go executable.
3. We print a header and flush standard output. This is to ensure Go finishes printing (buffered?) before Python starts running.
4. Next, we create a channel of integers. The channel is used to send the result of our Python execution back to our driver code. Note the use of the make function versus the new function here.
5. The next line passes the location of the Python source file to a new PythonCode object (that we created via NewPythonCode). See how this call is prefixed with the “go” keyword? This allows the Go runtime to run both the driver code as well as the Python system concurrently.
6. Finally, we wait on our channel and print the exit code.  If we didn’t wait, the driver would complete execution before our Python library ran. In short, much like terminating the main thread before a worker thread has a chance to complete.

The Test Code

The following snippet of Python code is what I used to test. The metaclassery is not needed, but I was curious as to whether it would cause any issues (I didn’t think it would!).

import sys

class GoofyMeta(type):
  def __new__(*args, **kw):
    print "In Go->Python Metaclass"
    return type.__new__(*args, **kw)

class HelloFromGo(object):
  __metaclass__ = GoofyMeta
  def __str__(self):
    return "Hi, you are running Go."

if __name__ == '__main__':
  h = HelloFromGo()
  print h

  print "See, not kidding: "
  print sys.version

  print "Here's my global namespace: "
  print dir()

So, as a control, let’s run the Python code from the command line first, to ensure it runs as we think it should.

mcjeff@martian:~/pygo$ python test.py
In Go->Python Metaclass
Hi, you are running Go.
See, not kidding:
2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3]
Here's my global namespace:
['GoofyMeta', 'HelloFromGo', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'h', 'sys']
mcjeff@martian:~/pygo$

Now that we’re certain everything works, we can move forward with compilation and testing.

Compilation and Running

For this example, I’m going to use the same little shell function I used in my first Go post. Everything else is new.  Let’s take a look at what’s required to build this extension.

mcjeff@martian:~/pygo$ export GOROOT=~/go
mcjeff@martian:~/pygo$ make
CGOPKGPATH= cgo -- -I/usr/include/python2.6 pygo.go
6g -o _go_.6  pygo.cgo1.go _cgo_gotypes.go
6c -FVw -I"/home/mcjeff/go/pkg/linux_amd64" _cgo_defun.c
gcc -m64 -g -fPIC -O2 -o _cgo_main.o -c -I/usr/include/python2.6  _cgo_main.c
gcc -m64 -g -fPIC -O2 -o pygo.cgo2.o -c -I/usr/include/python2.6  pygo.cgo2.c
gcc -m64 -g -fPIC -O2 -o _cgo_export.o -c -I/usr/include/python2.6  _cgo_export.c
gcc -m64 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o pygo.cgo2.o _cgo_export.o -lpython2.6
cgo -dynimport _cgo1_.o >__cgo_import.c && mv -f __cgo_import.c _cgo_import.c
6c -FVw _cgo_import.c
rm -f _obj/pygo.a
gopack grc _obj/pygo.a _go_.6  _cgo_defun.6 _cgo_import.6 pygo.cgo2.o _cgo_export.o
mcjeff@martian:~/pygo$ make install
cp _obj/pygo.a "/home/mcjeff/go/pkg/linux_amd64/pygo.a"
mcjeff@martian:~/pygo$

That’s it. The files included in our Makefile handle the setup and compilation for us. There are a series of intermediary files created that we really don’t have to concern ourselves with.  If you’re curious, they’re left around by the compiler. A simple ‘ls’ will display them.

Finally, we can run our Go binary and execute our small Python example.

mcjeff@martian:~/pygo$ source ~/.profile
mcjeff@martian:~/pygo$ gogogo main.go
Exec Python: test.py
-----------------------------

In Go->Python Metaclass
Hi, you are running Go.
See, not kidding:
2.6.5 (r265:79063, Apr 16 2010, 14:15:55)
[GCC 4.4.3]
Here's my global namespace:
['GoofyMeta', 'HelloFromGo', '__builtins__', '__doc__', '__name__', '__package__', 'h', 'sys']

-----------------------------
Channel Returned Code:  0
mcjeff@martian:~/pygo$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

Well, there you have it. A quick run down on how to access C libraries from a Go application.  I find it rather elegant, honestly. Now, as for the embedded Python?  I wouldn’t use this anywhere, ever, for any reason. I’ve no idea what’s going on with regards to internal thread states and structures. This is nice little example, but probably not such a hot idea!

• About

  

  Bio: I'm a Linuxy Engineer with a Pythonic Twist!

• Categories

  Select Category books development django Go linux open source politics pycon pylons python religion Uncategorized virtualization work xen zope

• Book Links

  • Django 1.1 Testing & Debugging
  • Grok 1.0 Web Development
  • Python 3 Object Oriented Programming
  • Python Testing: Beginner's Guide
  • Python Text Processing: Beginner's Guide
  • wxPython 2.8 Cookbook

• Blog Archives

  Select Month August 2012 July 2012 April 2012 March 2012 February 2011 December 2010 September 2010 August 2010 July 2010 June 2010 May 2010 March 2010 February 2010 January 2010 December 2009 November 2009 August 2009 May 2009 February 2009 January 2009 December 2008 November 2008 October 2008 August 2008 July 2008 June 2008 May 2008 April 2008

• Twitter Feed

  • No public Twitter messages.