While the development of this plugin was funded by Zillow, all the different components in the plugin are now completely independent from each other and can be used as individual modules if desired. Version 3 is a complete rewrite of the plugin that brings with it a ton of features that had been requested on the Local Market Explorer feedback page as well as many bug fixes. Following are the big things that have changed.

• Performance has been significantly increased by making all of the external data requests in parallel instead of in series. This means that the modules that have slower APIs won’t slow down the page as much as they did in the past.
• The modules can now be used on individual pages and posts instead of only on the Local Market Explorer virtual pages. This adds a great deal of flexibility for embedding this valuable data into your pages. Here’s an example of the market stats module for Los Angeles loaded into a random page. This functionality is implemented via WordPress shortcodes and includes a handy toolbar icon in the page / post editor for inserting these shortcodes.
• The HTML markup and CSS styling has been significantly pruned so that skinning is easier and so that the default styles work better with many more themes.
• It’s now easier to use a custom page template for all LME pages.
• A real-time IDX real estate module has been added to all city and zip pages to display the newest listings in those areas. This only works if you also have the dsIDXpress WordPress IDX plugin (disclaimer: I work for Diverse Solutions, the company that sells dsIDXpress).
• The admin UI has been significantly enhanced to make it easier to use. A help section has also been added in the admin.
• A neighborhood module has been added on city pages that links to all the neighborhoods within that city. These neighborhood names are provided by Zillow’s GetRegionChildren API.
• Support for canonical link tags has been added so that search engines will better index the true URL and won’t see duplicate content within your domain.
• This version works way better with some of the more obscure WordPress installs as well as with WordPress 3.x.
• Support for XML sitemaps has been added via the Google XML Sitemaps plugin.
• Lots of bugs have been fixed relating to data not displaying when it should (ex. homes for sale in certain areas) and blank data displaying when it shouldn’t (ex. Zillow Home Value Index showing $0 when Zillow doesn’t have that data).
• More charts and data have been added to a few of the modules.

The Local Market Explorer v3 beta can be downloaded as a zip file from here. After you’ve downloaded it, simply install it in your blog by uploading it through the plugin installer. If you have v2 installed, you’ll probably need to delete it first. Deleting v2 won’t delete your settings or area descriptions though; installing v3 will then use these settings / descriptions. Additionally, you should be able to downgrade to v2 if you really want to for some reason.

While this release is considered to be an open beta and hasn’t encountered any major issues during testing on a number of other blogs, you may want to consider back up your site before installing this to make sure nothing happens with your descriptions and settings if you already have v2 installed. You may want to consider WordPress Backup or WP-DB-Backup; I’ve personally never tried either. You’ll probably be safe without doing this, but it just never really hurts to double-check your backup system.

Anyway, I’m sure this will have bugs, so please discuss in the comments. If you have feature requests from here on out or encounter other bugs after v3 is released, please use the feedback page. The final release will be out soon!

The BIG change in v2 is that the plugin now supports loading data on both neighborhood AND zip code level. To see an example, take a look at the Belltown neighborhood in Seattle, WA. The module was previously limited to just being able to display data on a city level. You can add your own descriptions (including HTML now) for each city, neighborhood, and zip code that you set up. You can still also link to cities, neighborhoods, and zips that you haven’t set up / added descriptions for.

Another great new feature is the addition of more market data in the “Market Statistics” module. Data such as the median list price, median sale price, median list price per sq ft, and the number of homes for sale has been added. More data will likely be available in the very near future.

Other changes include usability improvements in the admin area, the ability to turn the Flickr panel off independently of turning off the area descriptions, and the ability to turn off the “Schools” and “Market Statistics” modules.

Anyway, please give it a whirl. You can download the zip file here. If you don’t have Local Market Explorer installed at all, you should be able to simply download the plugin and install it via the plugin upload area in your admin. If you already have Local Market Explorer installed, you’ll need to follow the steps below to upgrade for now (since it’s not officially released yet):

1. Deactivate the Local Market Explorer plugin.
2. Go to the “Recently Active” plugins link.
3. Click the “Delete” link on the plugin to delete all of the files for the plugin. You won’t lose your settings as they’re stored in the database, not in the files.
4. Install the plugin using the plugin “Upload” link.
5. Activate the plugin.

That’s all! It’d be really great to get feedback before this thing is officially released, so if you have anything to say, bugs you found, features you’d like to see, etc, feel free to leave it in the comments.

Over on the Diverse Solutions blog, Mike Price posted an interview with Russ Bergeron, CEO of SoCal MLS, about what CARETS is specifically trying to accomplish in spite of all of the other proposals going around. It’s an interesting read for sure, especially considering that many other MLS’s around the country are keenly watching to see what happens in California so that can learn how to handle the complexities of creating their own partnerships. Some of the excepts from the interview are posted below.

Mike: Many MLS’s have tried similar data share program in California what makes CARETS different and successful thus far?
Russ: CARETS is not a true datashare – it is more than that. Others have done datashare (as have we), but that is usually between two or more disparate systems with no common data definitions. The result is that data is not common from system to system once the datashare has been done. With CARETS, we have normalized the database so that each and every MLS adheres to the same definitions and rules regarding content and display. The result for the user is that it looks like they are members of one big MLS. The other huge difference with CARETS is that the end user does not have to learn any new software or systems. They continue to use whatever software they’re comfortable with – in our case Tempo, eNeighborhoods, rDesk, MLS Office, ListingBook, etc. – with no obvious changes other than the fact they now have access to many, many more listings.

Mike: CARETS has moved quickly. How did you and the others get this done at such a rapid pace?
Russ: This is kind of like becoming a star overnight. We have actually been hard at work on this for 2 years and still have much to do but we have definitely proven that this concept is the best way to go. What made it so successful is that because of the history of the 5 MLS involved we did not have to overcome the politics, paranoia, personalities and protectionism that you are normally faced with in a project like this – we had addressed all of that years ago.

Mike: CAR is working on a statewide MLS. Do you see them or have they contacted you to adopt what you have done?
Russ: A statewide MLS is not necessarily a bad thing. I think if you polled the practitioners they would agree that a statewide MLS, or at least a standard statewide database would be a very good thing. [...] I think there is consensus that the common database is the best approach. [...]

Read more to hear the rest of the interview, including thoughts on how the idea got started, the benefits to the MLS’s involved in such a partnership, and what he thinks of a national MLS.

I’m actually surprised there aren’t more listings up there, but maybe it’s just that only a handful of people know about it. If you have a bunch of photos from a listing — or any other collection of photos — I encourage you to visit the site and give it a shot. You may want to read “Synthing to sell: using Photosynth to help sell a house“and “Synthing tips: How to synth a room” before you get started.

Once you’ve got one you’re proud of, show it off by linking to it in a comment below. It seems to me that this would be a pretty nifty way to make a pitch to potential clients to try to pick up their listing.

(Rule III.13) Address fields are included in the IDX data set for the sole purpose of facilitating mapping functionality. The addresses of property must not be viewable (and may not be printable) on or from any IDX website.

How unfortunate, and really, what’s the point? Is the point to protect the house from the squatters who use their laptop with it’s high-speed Internet connection to search for their next place to squat? Maybe the rule is in place to prevent the next door neighbors from performing an address search on Estately to find out that your house is in foreclosure. Actually, I bet they have the rule in place just because RMLS thinks it would be funny to frustrate buyers who just want to live on a particular street.

Okay, okay, I know. The rule is in place so that buyers can’t just search for properties on a website and then go for a drive to take a look at the properties themselves. Basically, easily displayable information is intentionally withheld from potential buyers in order to facilitate a business model that prizes information asymmetry and requires potential buyers like myself to essentially waste someone’s time to find out the information required to see a property with my own eyes. I know that there’s probably a lot of agents out there who want to egg my car for saying that while other agents would defend me, but hey, it’s just the way a lot of people in my generation (Generation Y) feel.

The best (worst?) part about the rule is that it only applies to “any IDX website”. Does that mean it’s okay to show the address in our RSS feeds? How about our property update emails? What about the printable brochures that don’t have to be part of the IDX website? If we’re allowed to show addresses in any of those venues as an IDX vendor (as the rule reads), then what’s the point of making the site harder to use if the information is available if the user just looks hard enough? We’ve loaded in IDX data from over 40-something of the largest markets in the US now, and RMLS is the only MLS that we’ve encountered who’s had that rule

Anyway, enough of picking on RMLS. Let’s take a jab at some other rules that we’ve encountered in our travels.

• NorthStar MLS (coincidentally enough, they also refer to themselves as RMLS):

  “Once during each viewing session of the BRS’s web site, the BRS’s web site must require the visiting consumer to assent to the terms of the “End-User License Agreement for consumers accessing public MLS and broker web sites, including Broker Reciprocity sites” (the “EULA”), promulgated by the Service. The BRS’s web site must obtain the consumer’s assent by means of the consumer’s affirmative act before the consumer views any portion of the Broker Reciprocity Database.”

  Are they serious? Yes, deadly serious. Here’s their EULA (Word doc) that all visitors are expected to read in its entirety and consent to before searching. No other MLS requires this.

• A handful of MLSs require us to put their logo on every single property. Most MLSs require us to display some sort of broker reciprocity logo on listings that aren’t owned by the agent or broker who is using the IDX, and I understand that, but what’s the point of putting the MLS’s logo on every single listing? We’re trying to encourage people to remember and contact the agent or broker, not remember what the MLS’s logo looks like.
• Realcomp (Detroit) requires us to display the school district on every single listing and in all of the search results. I understand that it’s nice to know, but it’s very odd to me that they consider it to be so important for people to know as to be codified into their IDX rules. No other MLS requires this.
• Three MLSs mandate that we don’t show the number of days that the property has been on the market. Redfin and others have worked around this by calling the number “Days on Redfin” instead of “Days on Market”. Again though, if there are workarounds, what’s the point?
• MetroList in northern California required us to remove all logos from our IDX solution. It’s not a big deal at all — it’s just weird is all. None of the other MLSs mentioned it.

There are bunch of other aggravating rules we’ve encountered, but you probably get the point by now. I really love what I do, and it’s actually quite enjoyable to search each new MLS that we load into our system, but sometime’s it feels like we hit the negative lottery (to quote from The Death of Common Sense) each and every time we load in a new IDX feed. Speaking of The Death of Common Sense, I think there’s a couple of MLS execs, lawyers, and compliance-checking employees out there who probably need to read that book as well.

Because I’m employed by an IDX vendor, most of the ideas I’m coming up with as I think this through have to do with enhancing the way consumers search for properties. However, there’s plenty of other great ideas out there on how this could be used for real estate professionals. I encourage you to check out the demo on the blog post or in their street view demo gallery.

What other ideas are out there? Have you thought of something really great that you’d love to see this new functionality used for? If you do have a good idea and you post it below, I’ll might be able to whip up another quick mashup (see Building a Simple Mashup in 24 Hours) for y’all.

We have a complete set of APIs for your app to talk directly to the contacts DB on the iPhone, and an entire database API with SQLite. … Core Location – we’ve taken that and patched it into an API so you can create location-aware applications. [emphasis added]

Imagine the possibilities this brings! Zillow can now make an app that shows information about all of the properties within a half-mile radius of your current location. IDX developers can create apps that show users properties for sale or open houses as they drive around a neighborhood. MLSs vendors can create locatation-aware comp programs to help agents impress sellers and pick up contracts. The list goes on. Yay!

I knew next-to-nothing about GIS before I able to download the data on Thursday night and had thus never heard of ArcGIS, ESRI, or a shapefile. I do know a good bit about Google Maps though, and like any self-respecting software developer, I figured I could just do a bit of research and cobble together a quick demo showing what the data is all about. 24 hours later, I had a functioning demo; check it out here. Following is a simplified rundown of what I did to build it.

(Step 1) I did quite a bit of research to find out what the heck a shapefile even is. I found out that the ArcGIS software costs over $1000 for a license, and since that was way to much money for me to spend, I had to find an alternative. I found this page that details a number of ways one can convert a shapefile to Google Earth’s KML format, but I couldn’t use those KML files because of the way I wanted to highlight the area on Google Maps with a filled-in polygon. I considered writing my own shapefile parser based on their technical specs, but it would have taken far more time than I was willing to invest in this little project. I also tried The Google Map Creator, which takes a shapefile and cuts up hundreds or thousands of Google map tiles based on the file and then gives you all the code necessary to use it on your own site. It wasn’t what I was looking for though.

Ultimately, I found a little program called shp2text that I was able to run against the California shapefile zip I had downloaded. The data it gave me back was in tab-delimited spreadsheet format, and I was then able to easily move that data into our database. As an aside, the same site I found shp2text on also suggested Christine GIS as a free alternative to ArcGIS; I didn’t try it out though, so if you want to try it, do so at your own risk.

(Step 2) I had to do some research to figure out how to convert the raw data I had into a format Google Maps could take in. While I could have just fed it all of the latitude / longitude pairs in order, some neighborhoods had more than a thousand different pairs of points to define the boundary. I was worried that if I fed it all of those points, it would be slow to initially load and then slow to map out in the browser. To combat this, Google created an encoding algorithm to condense all of the data into a shortened form based off a somewhat standardized encoding format. Basically, they made something that acts like a zip file program for all of those latitude / longitude pairs.

I started coding my own implementation of the encoder, but coding it and debugging it was taking more time than I thought it would. I looked around on the web for something that I could just plug right into my program, and I found a page called Encoding for Performance. The code was written by some guys in Austraila and worked well for me in my initial tests (with modifications), so I used it. I ran their code against all of the latitude / longitude pairs that I had previously imported into our database. When it was finished running, I had a single entry in our database for every neighborhood in California that Zillow had made outlines for. Each of those entries in turn had the encoded latitude / longitude pairs stored next to them in the database.

(Step 3) Now that I had all of the data I needed in the form I needed it in, the rest was cake. I was able to throw together a quick page using a Google Map and a tree outline “control” that the brilliant guys over at Yahoo! had made for exactly this kind of purpose. I used my favorite JavaScript library to make AJAX calls whenever one of the neighborhoods was clicked. When the server receives one of those AJAX requests, it does a lookup in the database for the neighborhood that was clicked. The server then takes that information and sends off a request to Zillow’s GetDemographics API to see if they know anything more about that area. If they do have information, I add all of the charts they have for that area to the AJAX response and then send it back to the browser. The browser then takes that information, draws a polygon around the area, and displays all of Zillow’s charts for that area in a little info window on the map. Easy!

Anyway, I hope this just goes to show that you (agents / brokers / software developers / everyone else) don’t need to write off something just because you don’t know anything about it. I knew next to nothing about the details of what Zillow was freely offering, but I knew that I wanted to learn more. In the same way, I hope and expect that a least a few of Geek Estate’s readers will want to run off now and create some mashups of the data in the way I just did. You (usually) don’t need any money nowadays, but you do need time, patience, and a willingness to learn.

What exactly is IDX and what does it mean?
At the most basic level, IDX, or Internet Data Exchange, is a branded property search website. Although some MLS’s only allow brokers to get an IDX website, most of them allow both brokers and agents to get an IDX website. Additionally, some MLS’s will actually give out free links to a (usually) stripped down version of an IDX website that the MLS either built themselves or contracted out to a vendor to build. Once the agent or broker has an IDX website, they usually either frame it in or link to it from their own website so that their website visitors can interact with it and search for properties by themselves.

Can I use an IDX website to look up or search for active listings on a national level?
Not right now. While it is possible to build an IDX website aggregates listing data on a national level to allow someone to search for properties across the nation, the only way I’m aware of doing it would be to create a company that would have some mondo bucks behind it. First, they’d have to be legally licensed as a broker in all 50 states. Next, they’d have to pay membership dues to approximately 1800+ MLS’s and a plethora of other associations through the country. They would also have to make sure that they, and their IDX vendor, maintained compliance with all of the different rules and regulations for each one of those MLS’s. Finally, they’d have to have an army of agents to be able to respond all of the requests for real estate services by people all across the county. Depending on who you ask, those restrictions that prevent an IDX website from functioning on a national level are either a good thing or a bad thing.

Are all IDX feeds the same?
Not at all. The raw data that comprises the IDX feeds to power the IDX websites are provided to vendors in almost any and every way you can imagine, from FTP downloads to, literally, UPS shipments of hard drives. Additionally, each MLS or association puts their own spin on the data, making it really difficult to maintain a good IDX system that can accommodate any MLS across the country. Each MLS also requires vendors to change the way their IDX solution displays the data to follow their own rules. Basically, there’s just not a single point of contact for anyone or a single API that allows those who need the data to query or download it at will.

I’ve heard a lot about RETS lately. Is RETS different from IDX?
RETS is just one of the numerous ways that IDX vendors and other direct consumers of raw real estate can get information from the organizations who collect that information. The whole idea of RETS is that operational costs will be lower to the data providers and tech vendors will be able to quickly hook their systems up to those providers without having to spend lots of time to accommodate each new system. The goal is definitely a noble one, but from my own perspective of actually working with the different RETS systems, it really hasn’t saved me that much time due to the numerous differences among each provider’s data.

Can I get listings from the IDX for my RSS reader?
The ability to get new / updated listings and property updates via RSS, or Really Simple Syndication (YouTube vid), is a feature that the individual IDX vendors would have to implement. I know that our IDX has full support for RSS feeds, and I have definitely seen other companies that do this (although, unfortunately, I can’t recall them right now). I’d bet that quite a few vendors who don’t currently support RSS are actually working on this feature right now.

My MLS or association only gives out a VOW feed. Is IDX different from VOW?
I recently discussed IDX vs VOW, or Virtual Office Website, at length with my coworkers, and from our perspective, we’re convinced that the definition between IDX and VOW is, at best, fuzzy. A VOW data feed is supposed to be a raw data feed that can be only be displayed on a VOW website, and, by definition, VOW websites require visitors to register as a user in the system to search for and view listings. In reality though, many data providers actually give vendors what they call an IDX feed that is really a VOW feed; vendors are then supposed to strip out the VOW-only fields to turn that data feed into something that can be displayed on an IDX website. It’s a really confusion world when it comes to IDX vs VOW, so if you want to know more, I suggest you check out MetalSoftware’s writeup on IDX vs VOW.

What do I need to do to get an IDX feed?
First, you should probably contact your association or MLS. Again, if you’re an agent, you might not be able to get an IDX website. Once you contact your organization, ask them what paperwork you need to fill out to be able to have your own IDX website. You might also want to ask them if they provide a free, pre-branded IDX link you can use, although those links usually pretty crippled with less features and without the visitor login / registration system you’d find elsewhere. After you have a basic idea what hoops you’ll have to jump through to get an IDX website, you’ll want to pick a good IDX vendor to go with. I’ve listed a few of the ones I know of below.

• Advanced Access
• Birdview
• Diverse Solutions (my employer)
  
• IDXCentral
• IDXSolutions.com
• iHomefinder
• Real Estate Webmasters
• MetalSoftware
• Smart Realty Solutions
• WebCubic
• Wolfnet
• Xoompad

Anyway, I hope this post addresses some of the more basic questions that people have about IDX. I love to talk with people about the gooey technical details of what I do, so if you want to know more about anything on any level, leave a comment and I’ll do my best to answer anything you can throw at me.

For the purposes of sharing my background and fully disclosing my employer, I’ll should tell you a little bit about what I’ve been working on lately. For a bit more than a year now, I have been working on the client-facing portion of our very own mapping MLS IDX solution, dsSearchAgent. Additionally, I’m responsible for working with a myriad of MLS’s across the country to load their data into our system. I’m a coder at heart, and while I’m confident that I can translate the tech stuff I work on into something everyone who reads this blog can understand, I’m always excited to get down into the über-geek details of anything and everything pertaining to what I work on.

Anyway, while we do employ a sales staff just like any other business, I’m hoping that my position as a software engineer, my experience, and my expertise will allow me to contribute a unique perspective to this blog without sounding like I’m trying to pitch any sort of products or services. Also, since the sales department is usually the known mortal enemy of any engineering department in any company, I think you’ll be especially safe from me.