Building a (Simple) Mashup in 24 Hours
When Zillow released their neighborhood data last Wednesday night, I was floored. My company had talked about trying to acquire similar data off and on over the past two years, but it was always too cost prohibitive to do so. As I thought about the magnitude of what they had done that night, I realized that it was practically required for me as a coding geek to see what I could do with the data.
I knew next-to-nothing about GIS before I able to download the data on Thursday night and had thus never heard of ArcGIS, ESRI, or a shapefile. I do know a good bit about Google Maps though, and like any self-respecting software developer, I figured I could just do a bit of research and cobble together a quick demo showing what the data is all about. 24 hours later, I had a functioning demo; check it out here. Following is a simplified rundown of what I did to build it.
(Step 1) I did quite a bit of research to find out what the heck a shapefile even is. I found out that the ArcGIS software costs over $1000 for a license, and since that was way to much money for me to spend, I had to find an alternative. I found this page that details a number of ways one can convert a shapefile to Google Earth’s KML format, but I couldn’t use those KML files because of the way I wanted to highlight the area on Google Maps with a filled-in polygon. I considered writing my own shapefile parser based on their technical specs, but it would have taken far more time than I was willing to invest in this little project. I also tried The Google Map Creator, which takes a shapefile and cuts up hundreds or thousands of Google map tiles based on the file and then gives you all the code necessary to use it on your own site. It wasn’t what I was looking for though.
Ultimately, I found a little program called shp2text that I was able to run against the California shapefile zip I had downloaded. The data it gave me back was in tab-delimited spreadsheet format, and I was then able to easily move that data into our database. As an aside, the same site I found shp2text on also suggested Christine GIS as a free alternative to ArcGIS; I didn’t try it out though, so if you want to try it, do so at your own risk.
(Step 2) I had to do some research to figure out how to convert the raw data I had into a format Google Maps could take in. While I could have just fed it all of the latitude / longitude pairs in order, some neighborhoods had more than a thousand different pairs of points to define the boundary. I was worried that if I fed it all of those points, it would be slow to initially load and then slow to map out in the browser. To combat this, Google created an encoding algorithm to condense all of the data into a shortened form based off a somewhat standardized encoding format. Basically, they made something that acts like a zip file program for all of those latitude / longitude pairs.
I started coding my own implementation of the encoder, but coding it and debugging it was taking more time than I thought it would. I looked around on the web for something that I could just plug right into my program, and I found a page called Encoding for Performance. The code was written by some guys in Austraila and worked well for me in my initial tests (with modifications), so I used it. I ran their code against all of the latitude / longitude pairs that I had previously imported into our database. When it was finished running, I had a single entry in our database for every neighborhood in California that Zillow had made outlines for. Each of those entries in turn had the encoded latitude / longitude pairs stored next to them in the database.
(Step 3) Now that I had all of the data I needed in the form I needed it in, the rest was cake. I was able to throw together a quick page using a Google Map and a tree outline “control” that the brilliant guys over at Yahoo! had made for exactly this kind of purpose. I used my favorite JavaScript library to make AJAX calls whenever one of the neighborhoods was clicked. When the server receives one of those AJAX requests, it does a lookup in the database for the neighborhood that was clicked. The server then takes that information and sends off a request to Zillow’s GetDemographics API to see if they know anything more about that area. If they do have information, I add all of the charts they have for that area to the AJAX response and then send it back to the browser. The browser then takes that information, draws a polygon around the area, and displays all of Zillow’s charts for that area in a little info window on the map. Easy!
Anyway, I hope this just goes to show that you (agents / brokers / software developers / everyone else) don’t need to write off something just because you don’t know anything about it. I knew next to nothing about the details of what Zillow was freely offering, but I knew that I wanted to learn more. In the same way, I hope and expect that a least a few of Geek Estate’s readers will want to run off now and create some mashups of the data in the way I just did. You (usually) don’t need any money nowadays, but you do need time, patience, and a willingness to learn.
Kurtis Holsapple
Posted at 13:26h, 21 JanuaryAs a fellow coder, I must applaud you. This is a very nice demo with lots of potential. Well done! The world is a better place with free APIs and open source standards. Glad you could work around that $1000 license.
Drew Meyers
Posted at 13:46h, 21 JanuaryLove it — well done Andrew!
Brian Wilson
Posted at 14:29h, 21 JanuaryAndrew, it is amazing what you were able to do in 24 hours including all of the research of these open source tools. Wow.
A Mashup with Zillow Neighborhood Shapefiles and the Zillow API - Zillow® Blog - Real Estate News and Analysis
Posted at 15:03h, 21 January[…] be do-able and sure enough, just 5 days later, Andrew Mattie (a developer at Diverse Solutions) has created a mashup using both the boundary files and the GetDemographics API call. The mashup has a pretty slick […]
DarrinClement - neighborhood boundaries
Posted at 13:23h, 22 JanuaryWow – very impressive for just 24 hrs!
Marc Grayson
Posted at 00:28h, 23 January24 hours uninterrupted…nice…get some sleep 🙂 Such neighborhood data would be very applicable to city websites looking to provide more consumer-centric data to prospective and current residents. http://www.GlendaleAZ.com (host city where this year’s 2008 Super Bowl) could use more GIS, neighborhood centric data.
DarrinClement
Posted at 05:49h, 23 JanuaryWhat did you think of the topological quality or accuracy of the data?
Andrew Mattie
Posted at 10:21h, 23 JanuaryHeh, I never said 24 hours uninterrupted! I started working with the data on Thursday night, went to work on Friday, and had everything pretty much wrapped up by the end of Friday night. Ergo, that’s 24 hours to me.
Like I said, I’m certainly not a GIS guy, so I don’t know that I’m the one to ask regarding the quality or accuracy of the data. All I know is that they effectively provided coordinates for polygon outlines, and I was able to draw them on a map. 🙂 On a subjective level, it seems very accurate for the neighborhoods around where I live (Irvine, CA in Orange County). I’m not sure how one would be able to measure the data objectively though. As far as topographical data, I don’t believe that the shapefiles they provided represent anything in 3D space (only 2D).
Jennifer Kyrnin
Posted at 15:45h, 24 JanuaryVery cool. This is something that a lot of people would be interested in – especially as mashups tend to be built by people with tons of time and “geekspertise”. It’s nice to see articles explaining how it can be done more easily. 🙂 Great job!
[email protected]
Posted at 10:28h, 03 FebruaryMan… Great Work!!!
I am amazed at the ability people have to pick something up like this and run with it to a finished product!!
Giles
Now Available: Google Street View via API | GeekEstate Blog
Posted at 12:07h, 27 March[…] a good idea and you post it below, I’ll might be able to whip up another quick mashup (see Building a Simple Mashup in 24 Hours) for […]
Zillow’s API Expands the Reach of Real Estate Data
Posted at 22:34h, 29 September[…] walkability ranking web site Walk Score, which uses this data in conjunction with heat maps, and Andrew Mattie’s blog post on how to use the neighborhood data to build a mashup in 24 […]
Jack Smith
Posted at 11:00h, 03 JanuaryGood work Andrew. I followed your guide to essentially replicate what you did so I can tailor it to suit my needs. However, I couldn’t figure out how you got the hierarical data for state>country>city>neighborhood and mapping the neighborhood ID in the Zillow shapefile to this hierarchy. Can you talk about that? Thanks.
Andrew Mattie
Posted at 12:11h, 05 January@Jack: I didn’t actually structure the data hierarchically. Instead, I just created a quick SQL query that loaded all of the data from a single table in our database. I then processed that resulting data in a few lines of application layer code and turned it into a string representing a JavaScript object that I wrote to the page when the page is first loaded. Once that JSON object is fed to Yahoo’s treeview control, it’s visible as the hierarchy that you see in the demo above.
wedphp
Posted at 02:43h, 01 AprilGreat work Andrew!
Vipin
Posted at 11:38h, 21 MayGr8 work Andrew !
I tried to follow the same steps.. but when I tried the shp2txt, it gave me the error which said “unable to open shape file”. Any idea why this is hapening?
Thx.
Nowfal Akash
Posted at 16:44h, 05 SeptemberThis is great!! I'd love to tinker around with this project, could you make the database and scripts available for download? 🙂
hudsonbar
Posted at 02:24h, 22 SeptemberI am to submit a report on this niche your post has been very very helpfull
Regards
Canadian prescriptions and percocet.
Posted at 18:17h, 31 DecemberPercocet….
Percocet….
Buy adipex.
Posted at 02:33h, 01 JanuaryAdipex….
Adipex p phentermine ecureme com. Pillinc buy phentermine adipex meridia online. Adipex. Adipex results. Adipex generic form phentermine lowest prices….
Free animal sex.
Posted at 19:44h, 01 JanuaryFree animal sex….
Free animal sex. Free animal sex videos….
Buy percocet online.
Posted at 07:37h, 02 JanuaryPercocet online….
Percocet 5. Percocet. Percocet addiction. Krons disease percocet. Percocet abuse….
Mens sexy underwear.
Posted at 03:51h, 03 JanuaryBritney spears no underwear….
Men underwear. Hairy men in underwear. Custom underwear. Child underwear models. Gay underwear. Crotchless underwear. Men in underwear….
Zoo sex.
Posted at 15:14h, 05 JanuaryZoo sex….
Zoo sex….
Buy targeted traffic.
Posted at 15:42h, 05 JanuaryBuy guaranteed signups….
Buy and sell. Buy web site traffic. Buy guaranteed sign ups. Buy guaranteed signups. Buy car. Buy phentermine….
Oxycodone.
Posted at 02:35h, 06 JanuaryOxycodone….
Oxycodone….
Xanax.
Posted at 12:08h, 06 JanuaryXanax….
Xanax….
Percocet.
Posted at 17:52h, 06 JanuaryPercocet addiction….
Percocet….
Cialis.
Posted at 09:43h, 07 JanuaryCialis….
Cialis….
Diflucan.
Posted at 17:51h, 07 JanuaryDiflucan….
Diflucan….
Viagra.
Posted at 18:48h, 08 JanuaryViagra….
Viagra….
Provigil.
Posted at 07:50h, 10 JanuaryProvigil….
Provigil….
Cymbalta.
Posted at 21:12h, 10 JanuaryCymbalta….
Cymbalta….
Viagra.
Posted at 05:19h, 11 JanuaryViagra reviews….
Buy viagra online. Cheap viagra. Viagra patent. Buy generic viagra. Viagra….
Viagra uk.
Posted at 11:18h, 12 JanuaryViagra side effects….
Viagra….
Vicodin.
Posted at 15:58h, 12 JanuaryVicodin….
Vicodin….
Viagra.
Posted at 03:23h, 13 JanuaryViagra uterine thickness….
Cheap viagra….
Viagra.
Posted at 03:59h, 13 JanuaryBuy viagra….
Viagra soft tabs. Viagra no prescription. Female viagra. Natural viagra. Viagra substitute….
Percocet.
Posted at 19:42h, 13 JanuaryPercocet….
Percocet….
Viagra.
Posted at 02:58h, 14 JanuaryFemale viagra….
Viagra….
Bulk alprazolam.
Posted at 06:42h, 14 JanuaryBuy alprazolam….
Buy alprazolam without a prescription. Buy alprazolam. Buy at discount alprazolam free consult. Xanax alprazolam no prescription. Order alprazolam. What is alprazolam rss feed….
Ambien cr.
Posted at 15:21h, 14 JanuaryAmbien online….
Ambien side effects. Ambien. Ambien overdose….
Benefits of valtrex.
Posted at 18:54h, 15 JanuaryValtrex australia….
Valtrex. Valtrex side effects….
Paxil side effects.
Posted at 03:00h, 16 JanuaryPaxil….
Paxil. Paxil lawyers southern california….
Sam
Posted at 20:17h, 25 MaySadly, almost none of these links work. But very cool.