Building a (Simple) Mashup in 24 Hours
January 21, 2008
By: Andrew Mattie
Author's Website: http://www.diversesolutions.com
When Zillow released their neighborhood data last Wednesday night, I was floored. My company had talked about trying to acquire similar data off and on over the past two years, but it was always too cost prohibitive to do so. As I thought about the magnitude of what they had done that night, I realized that it was practically required for me as a coding geek to see what I could do with the data.
I knew next-to-nothing about GIS before I able to download the data on Thursday night and had thus never heard of ArcGIS, ESRI, or a shapefile. I do know a good bit about Google Maps though, and like any self-respecting software developer, I figured I could just do a bit of research and cobble together a quick demo showing what the data is all about. 24 hours later, I had a functioning demo; check it out here. Following is a simplified rundown of what I did to build it.
(Step 1) I did quite a bit of research to find out what the heck a shapefile even is. I found out that the ArcGIS software costs over $1000 for a license, and since that was way to much money for me to spend, I had to find an alternative. I found this page that details a number of ways one can convert a shapefile to Google Earth’s KML format, but I couldn’t use those KML files because of the way I wanted to highlight the area on Google Maps with a filled-in polygon. I considered writing my own shapefile parser based on their technical specs, but it would have taken far more time than I was willing to invest in this little project. I also tried The Google Map Creator, which takes a shapefile and cuts up hundreds or thousands of Google map tiles based on the file and then gives you all the code necessary to use it on your own site. It wasn’t what I was looking for though.
Ultimately, I found a little program called shp2text that I was able to run against the California shapefile zip I had downloaded. The data it gave me back was in tab-delimited spreadsheet format, and I was then able to easily move that data into our database. As an aside, the same site I found shp2text on also suggested Christine GIS as a free alternative to ArcGIS; I didn’t try it out though, so if you want to try it, do so at your own risk.
(Step 2) I had to do some research to figure out how to convert the raw data I had into a format Google Maps could take in. While I could have just fed it all of the latitude / longitude pairs in order, some neighborhoods had more than a thousand different pairs of points to define the boundary. I was worried that if I fed it all of those points, it would be slow to initially load and then slow to map out in the browser. To combat this, Google created an encoding algorithm to condense all of the data into a shortened form based off a somewhat standardized encoding format. Basically, they made something that acts like a zip file program for all of those latitude / longitude pairs.
I started coding my own implementation of the encoder, but coding it and debugging it was taking more time than I thought it would. I looked around on the web for something that I could just plug right into my program, and I found a page called Encoding for Performance. The code was written by some guys in Austraila and worked well for me in my initial tests (with modifications), so I used it. I ran their code against all of the latitude / longitude pairs that I had previously imported into our database. When it was finished running, I had a single entry in our database for every neighborhood in California that Zillow had made outlines for. Each of those entries in turn had the encoded latitude / longitude pairs stored next to them in the database.
(Step 3) Now that I had all of the data I needed in the form I needed it in, the rest was cake. I was able to throw together a quick page using a Google Map and a tree outline “control” that the brilliant guys over at Yahoo! had made for exactly this kind of purpose. I used my favorite JavaScript library to make AJAX calls whenever one of the neighborhoods was clicked. When the server receives one of those AJAX requests, it does a lookup in the database for the neighborhood that was clicked. The server then takes that information and sends off a request to Zillow’s GetDemographics API to see if they know anything more about that area. If they do have information, I add all of the charts they have for that area to the AJAX response and then send it back to the browser. The browser then takes that information, draws a polygon around the area, and displays all of Zillow’s charts for that area in a little info window on the map. Easy!
Anyway, I hope this just goes to show that you (agents / brokers / software developers / everyone else) don’t need to write off something just because you don’t know anything about it. I knew next to nothing about the details of what Zillow was freely offering, but I knew that I wanted to learn more. In the same way, I hope and expect that a least a few of Geek Estate’s readers will want to run off now and create some mashups of the data in the way I just did. You (usually) don’t need any money nowadays, but you do need time, patience, and a willingness to learn.
- Stumble it!
- Categories: API's, Mapping
Enjoy this post? Subscribe to the Geek Estate Blog feed or get updates via e-mail
Comments
View Comments
Kurtis Holsapple on January 21, 2008 1:26 pm
As a fellow coder, I must applaud you. This is a very nice demo with lots of potential. Well done! The world is a better place with free APIs and open source standards. Glad you could work around that $1000 license.
Drew Meyers on January 21, 2008 1:46 pm
Love it — well done Andrew!
Brian Wilson on January 21, 2008 2:29 pm
Andrew, it is amazing what you were able to do in 24 hours including all of the research of these open source tools. Wow.
A Mashup with Zillow Neighborhood Shapefiles and the Zillow API - Zillow® Blog - Real Estate News and Analysis on January 21, 2008 3:03 pm
[...] be do-able and sure enough, just 5 days later, Andrew Mattie (a developer at Diverse Solutions) has created a mashup using both the boundary files and the GetDemographics API call. The mashup has a pretty slick [...]
DarrinClement - neighborhood boundaries on January 22, 2008 1:23 pm
Wow – very impressive for just 24 hrs!
Marc Grayson on January 23, 2008 12:28 am
24 hours uninterrupted…nice…get some sleep
Such neighborhood data would be very applicable to city websites looking to provide more consumer-centric data to prospective and current residents. http://www.GlendaleAZ.com (host city where this year’s 2008 Super Bowl) could use more GIS, neighborhood centric data.
DarrinClement on January 23, 2008 5:49 am
What did you think of the topological quality or accuracy of the data?
Andrew Mattie on January 23, 2008 10:21 am
Heh, I never said 24 hours uninterrupted! I started working with the data on Thursday night, went to work on Friday, and had everything pretty much wrapped up by the end of Friday night. Ergo, that’s 24 hours to me.
Like I said, I’m certainly not a GIS guy, so I don’t know that I’m the one to ask regarding the quality or accuracy of the data. All I know is that they effectively provided coordinates for polygon outlines, and I was able to draw them on a map.
On a subjective level, it seems very accurate for the neighborhoods around where I live (Irvine, CA in Orange County). I’m not sure how one would be able to measure the data objectively though. As far as topographical data, I don’t believe that the shapefiles they provided represent anything in 3D space (only 2D).
Jennifer Kyrnin on January 24, 2008 3:45 pm
Very cool. This is something that a lot of people would be interested in – especially as mashups tend to be built by people with tons of time and “geekspertise”. It’s nice to see articles explaining how it can be done more easily.
Great job!
Giles@DevelopingAtlanta.com on February 3, 2008 10:28 am
Man… Great Work!!!
I am amazed at the ability people have to pick something up like this and run with it to a finished product!!
Giles
Now Available: Google Street View via API | GeekEstate Blog on March 27, 2008 12:07 pm
[...] a good idea and you post it below, I’ll might be able to whip up another quick mashup (see Building a Simple Mashup in 24 Hours) for [...]
Zillow’s API Expands the Reach of Real Estate Data on September 29, 2008 10:34 pm
[...] walkability ranking web site Walk Score, which uses this data in conjunction with heat maps, and Andrew Mattie’s blog post on how to use the neighborhood data to build a mashup in 24 [...]
Jack Smith on January 3, 2009 11:00 am
Good work Andrew. I followed your guide to essentially replicate what you did so I can tailor it to suit my needs. However, I couldn’t figure out how you got the hierarical data for state>country>city>neighborhood and mapping the neighborhood ID in the Zillow shapefile to this hierarchy. Can you talk about that? Thanks.
Andrew Mattie on January 5, 2009 12:11 pm
@Jack: I didn’t actually structure the data hierarchically. Instead, I just created a quick SQL query that loaded all of the data from a single table in our database. I then processed that resulting data in a few lines of application layer code and turned it into a string representing a JavaScript object that I wrote to the page when the page is first loaded. Once that JSON object is fed to Yahoo’s treeview control, it’s visible as the hierarchy that you see in the demo above.
wedphp on April 1, 2009 2:43 am
Great work Andrew!
Vipin on May 21, 2009 11:38 am
Gr8 work Andrew !
I tried to follow the same steps.. but when I tried the shp2txt, it gave me the error which said “unable to open shape file”. Any idea why this is hapening?
Thx.
Nowfal Akash on September 5, 2009 4:44 pm
This is great!! I'd love to tinker around with this project, could you make the database and scripts available for download?
hudsonbar on September 22, 2009 2:24 am
I am to submit a report on this niche your post has been very very helpfull
Regards