The morning begins with Guido’s keynote, answering questions posted via Twitter: #python.

More to come…

Importing Addresses

I started this project with an Excel spreadsheet someone had sent me, and of course immediately saved it to CSV so I could read in from Python.  I’ve changed the names and addresses below to protect the innocent, indemnify the guilty, and remove any traces linking me to the incident.  Instead, I’ll just list my favorite places to find sweet tea along with delicious chicken and biscuits.

name,address,city,state,zip,phone,county
Apex - Williams Street,1581 East Williams Street,Apex,NC,27539,919-362-6796,Wake
Apex - Laura Village Drive,1209 Laura Village Drive,Apex,NC,27502,919-362-1416,Wake
Durham - Garrett Road,4600 Garrett Road,Durham,NC,27701,919-489-5942,Durham
Durham - Guess Road,2801 Guess Road,Durham,NC,27705,919-477-2362,Durham
Durham - Hillsboro Road,3558 Hillsboro Road,Durham,NC,27705,919-383-6797,Durham
Durham - Miami Boulevard,1712 Miami Boulevard,Durham,NC,27703,919-596-4330,Durham
Durham - South Miami Boulevard,5425 South Miami Boulevard,Durham,NC,27703,919-941-5620,Durham
Durham - Roxboro Road,4521 Roxboro Road,Durham,NC,27702,919-471-0581,
Fuquay Varina,1400 N. Main Street,Fuquay Varina,NC,27526,919-557-0749,Wake
Garner - Jones Sausage Road,3920 Jones Sausage Road,Garner,NC,27529,919-662-1621,Wake
Garner - NC 42 East,5497 NC 42 East,Garner,NC,27529,919-773-9116,Wake
...

Using the csv module in the Python standard library, it’s a piece of fried chicken cake to read in the CSV and return a dictionary for each row.  In my situation, I had a spreadsheet with some missing values, so I filtered the rows to only include the ones with complete addresses.

import csv

def read_addresses(filename):
    """Retrieve addresses from the given CSV filename."""
    required_fields = set(['name', 'address', 'city', 'zip'])
    reader = csv.DictReader(file(filename, 'rU'))
    for row in reader:
        if not all(row.get(f, '').strip() for f in required_fields):
            continue
        yield row

And then a little code to just use the first argument to the script as the filename.

import sys

if __name__ == '__main__':
    for address in read_addresses(sys.argv[1]):
        print address

Geocoding

Now, we need to use the Google Maps API to geocode our addresses and return the latitude and longitude in CSV format.

import urllib
import urlparse

def geocode(address):
    """Geocode the given address, updating the standardized address, latitude,
    and longitude."""
    qs = dict(q=address['address_string'], key=GMAPS_API_KEY, sensor='true',
              output='csv')
    qs = urllib.urlencode(qs)
    url = urlparse.urlunsplit(('http', 'maps.google.com', '/maps/geo', qs, ''))
    f = urllib.urlopen(url)
    result = list(csv.DictReader(f, ('status', 'accurary', 'latitude', 'longitude')))[0]
    if int(result['status']) != 200:
        raise RuntimeError, 'could not geocode address %s (%s)' % \
                            (address, result['status'])
    address['latitude'] = result['latitude']
    address['longitude'] = result['longitude']

To iterate over the addresses and format them for geocoding, we use the following code:

for address in read_addresses(sys.argv[1]):
    address['address_string'] = \
            '%(address)s, %(city)s, %(state)s %(zip)s' % address
    geocode(address)

Geocoding to Make Google Happy

The geocoding works, well almost.  It seems to start out fine, then ends up returning quite a few errors as it goes along.  When we look at the status code returned, it’s a 620 – G_GEO_TOO_MANY_QUERIES.  So we just add a sleep to the geocode function and it makes it through all the addresses, albeit a little more slowly now.

import time

def geocode(address):
    ...
    time.sleep(1.0)

Generating KML

Now than we have addresses with their corresponding coordinates, we can start to create a KML document with our placemarks.

import xml.dom.minidom

def create_document(title, description=''):
    """Create the overall KML document."""
    doc = xml.dom.minidom.Document()
    kml = doc.createElement('kml')
    kml.setAttribute('xmlns', 'http://www.opengis.net/kml/2.2')
    doc.appendChild(kml)
    document = doc.createElement('Document')
    kml.appendChild(document)
    docName = doc.createElement('name')
    document.appendChild(docName)
    docName_text = doc.createTextNode(title)
    docName.appendChild(docName_text)
    docDesc = doc.createElement('description')
    document.appendChild(docDesc)
    docDesc_text = doc.createTextNode(description)
    docDesc.appendChild(docDesc_text)
    return doc

And to use this method to create a basic document, we call it with our title and description parameters. The second line gets the Document element of the overall KML doc; this is the element to which we’ll append other child elements.

    kml_doc = create_document('RDU Bojangle\'s',
                              'Sweet Tea, Chicken and Biscuits')
    document = kml_doc.documentElement.getElementsByTagName('Document')[0]

Now we need a way to create a placemark using the address information we’ve obtained from geocoding. We put the phone number in the description, if there is one available.

def create_placemark(address):
    """Generate the KML Placemark for a given address."""
    doc = xml.dom.minidom.Document()
    pm = doc.createElement("Placemark")
    doc.appendChild(pm)
    name = doc.createElement("name")
    pm.appendChild(name)
    name_text = doc.createTextNode('%(name)s' % address)
    name.appendChild(name_text)
    desc = doc.createElement("description")
    pm.appendChild(desc)
    desc_text = doc.createTextNode(address.get('phone', ''))
    desc.appendChild(desc_text)
    pt = doc.createElement("Point")
    pm.appendChild(pt)
    coords = doc.createElement("coordinates")
    pt.appendChild(coords)
    coords_text = doc.createTextNode('%(longitude)s,%(latitude)s,0' % address)
    coords.appendChild(coords_text)
    return doc

To use this method to build our KML document from the list of addresses, we add the following code:

        placemark = create_placemark(address)
        document.appendChild(placemark.documentElement)

And finally, we need a way to print the whole document to stdout:

print kml_doc.toprettyxml(indent="  ", encoding='UTF-8')

Generating KML to Make Google Happy

If we upload this file to a web server, and point to its URL from Google Maps, we have a problem. It doesn’t work. Apparently Google doesn’t like the default pretty formatting of our KML, in particular the nodes containing only text, because of the extra newlines inserted before and after the text. We can either turn off pretty printing, which I’d rather not do since it makes the output hard to read, or apply this little monkey patch to fix the problem:

class Element(xml.dom.minidom.Element):

    def writexml(self, writer, indent="", addindent="", newl=""):
        # indent = current indentation
        # addindent = indentation to add to higher levels
        # newl = newline string
        writer.write(indent+"<" + self.tagName)

        attrs = self._get_attributes()
        a_names = attrs.keys()
        a_names.sort()

        for a_name in a_names:
            writer.write(" %s=\"" % a_name)
            xml.dom.minidom._write_data(writer, attrs[a_name].value)
            writer.write("\"")
        if self.childNodes:
            newl2 = newl
            if len(self.childNodes) == 1 and \
                self.childNodes[0].nodeType == xml.dom.minidom.Node.TEXT_NODE:
                indent, addindent, newl = "", "", ""
            writer.write(">%s"%(newl))
            for node in self.childNodes:
                node.writexml(writer,indent+addindent,addindent,newl)
            writer.write("%s%s" % (indent,self.tagName,newl2))
        else:
            writer.write("/>%s"%(newl))

# Monkey patch Element class to use our subclass instead.
xml.dom.minidom.Element = Element

Changing Placemark Styles

KML files also support the notion of styles, which allow you to customize the appearance of your placemarks.  For this example, I’ll just use different colored icons provided by Google to correspond to the county in which the address is located (based on the “county” column in my CSV file). The code below is used to generate the style elements for the KML document:

def create_style(style_id, icon_href):
    """Create a new style for different placemark icons."""
    doc = xml.dom.minidom.Document()
    style = doc.createElement('Style')
    style.setAttribute('id', style_id)
    doc.appendChild(style)
    icon_style = doc.createElement('IconStyle')
    style.appendChild(icon_style)
    icon = doc.createElement('Icon')
    icon_style.appendChild(icon)
    href = doc.createElement('href')
    icon.appendChild(href)
    href_text = doc.createTextNode(icon_href)
    href.appendChild(href_text)
    return doc

In our main function, we use this code to create three different styles:

    style_doc = create_style('Wake', \
        'http://maps.google.com/mapfiles/kml/paddle/red-blank.png')
    document.appendChild(style_doc.documentElement)
    style_doc = create_style('Durham', \
        'http://maps.google.com/mapfiles/kml/paddle/blu-blank.png')
    document.appendChild(style_doc.documentElement)
    style_doc = create_style('Orange', \
        'http://maps.google.com/mapfiles/kml/paddle/wht-blank.png')
    document.appendChild(style_doc.documentElement)

Now, we add this code to our create_placemark method in order to associate the placemark with a style based on the “county” field:

    if address.get('county', ''):
        style_url = doc.createElement("styleUrl")
        pm.appendChild(style_url)
        style_url_text = doc.createTextNode('#%(county)s' % address)
        style_url.appendChild(style_url_text)

And now there is a different colored icon associated with the addresses, based on county.

Putting It All Together

I’ve attached the final script combining all of the snippets above (minus my Google Maps API key).

makemaps.py

To view the custom placemarks in your browser, you need to make the KML file available on the web, and enter its URL into the search field in Google Maps.  You can also generate a URL with a query string such as the following:

http://maps.google.com/?q=<kml-url>

Or view the results of my sample data:

http://maps.google.com/?q=http://www.ninemoreminutes.com/bojomap.kml

And Then?

It would be relatively simple to include this code in a web app to read addresses from a database and generate a KML file on the fly. For use in any Python web framework you choose.

And Then?

No more “and then”.

So this update script is on an embedded Linux device and can be completely wiped away if the user does a factory reset or firmware update from the manufacturer.  If they do that, my enhancements to the device and my upgrade script are gone, making the device useless for the particular application.  Fortunately, the manufacturer gives me a way to store a one-line setting in an area of flash that doesn’t get erased.  And even better, the default firmware will read and evaluate this one line setting in the shell on startup.  AHA!

The aforementioned Python script also takes the install script, packs it into one line, escapes everything again, and creates a bootstrap script, whose sole purpose is to write the one setting that gets read on startup, that runs the install script, which creates the update script, which periodically checks for updates to the whole thing.

And now my head hurts.  But it works!