Geospatial Queries with Google App Engine

Introduction

While developing an Android app that uses Google Cloud Messaging and integrates with +Google App Engine, I ran into a common App Engine issue. Geospatial queries using High Replication Data Store.  In my application I connect users via their location, when a user performs an action in the app, users in their selected location receive a message via GCM.

DataStore vs Rational Databases

Coming for the rational database world of Oracle and SQL server, I attempted to execute a query to locate devices inside the users selected area.  I used a bounding box to determine what devices are within a given area.  Pleased with my approach, I constructed my filter and updated to App Engine for execution.

query.setFilter( CompositeFilterOperator.and(

   new FilterPredicate(COFFEERUN_ACTIVE_PROPERTY, Query.FilterOperator.EQUAL, 1),

   CompositeFilterOperator.and( 

         new FilterPredicate(COFFEERUN_LAT_PROPERTY,        

                             Query.FilterOperator.GREATER_THAN, 
                             boundbox[0].getLatitudeInDegrees()),

         new FilterPredicate(COFFEERUN_LAT_PROPERTY, 
                                Query.FilterOperator.LESS_THAN, 
                             boundbox[1].getLatitudeInDegrees())
         ),                       

   CompositeFilterOperator.and( 

         new FilterPredicate(COFFEERUN_LON_PROPERTY,        

                             Query.FilterOperator.GREATER_THAN, 
                             boundbox[0].getLongitudeInDegrees()),

         new FilterPredicate(COFFEERUN_LON_PROPERTY, 
                                Query.FilterOperator.LESS_THAN, 
                             boundbox[1].getLongitudeInDegrees())

   )));                    

BOOM!!! Exception (IllegalArgumentException: Only one inequality filter per query is supported)


What! this can't be right, this type of query executes all the time in Oracle and SQL, lets check the documentation.

Google Developers - App Engine - Datastore Queries

A query can have no more than one not-equal filter, and a query that has one cannot have any other inequality filters.

Research & Development

Ok, that's fine, time to do some research, GeoHashing seems to be the answer and the GeoModel project has an App Engine implementation. However, it's a lot of work to integrate and I still have to read my matches to determine if they are within my search area.  

There must be something in native App Engine to handle this type of query, after all Google does it all the time.  I find out there is a native GeoPt datatype in App Engine, I will try using that first to see if it works. 

I update the code to store GeoPt values for device locations and change my query to use the new GeoPt value, I upload my code to App Engine and give it a try.

GeoPt geoLocG = new GeoPt((float)boundbox[0].getLatitudeInDegrees(),              
                          (float)boundbox[0].getLongitudeInDegrees());

GeoPt geoLocL = new GeoPt((float)boundbox[1].getLatitudeInDegrees(), 
                          (float)boundbox[1].getLongitudeInDegrees());

     

query.setFilter( CompositeFilterOperator.and(

   new FilterPredicate(COFFEERUN_ACTIVE_PROPERTY, Query.FilterOperator.EQUAL, 1),

   CompositeFilterOperator.and( 

         new FilterPredicate(COFFEERUN_GEO_PROPERTY,        

                                Query.FilterOperator.GREATER_THAN, geoLocG),

         new FilterPredicate(COFFEERUN_GEO_PROPERTY, 
                                Query.FilterOperator.LESS_THAN, geoLocL)                            
   )));                    

     

             

Iterable<Entity> entities =   
                    datastore.prepare(query).asIterable(DEFAULT_FETCH_OPTIONS);


Yes it worked, it didn't raise an exception and I get some results in my rather large area query. I move on and go back to working on the Android app.

Then during testing I find that I'm not receive some GCM messages and other devices are getting messages that are outside the visual search area on the map. I then start adding integration with Google Places API and custom POI data from various US cities.  I set my testing environment location for my devices to San Francisco, then it becomes very clear.  I'm getting back entries that are outside my query area.

I take each matching entries geographical location and plot it using +Google Maps , I add my bounding box and quickly see there is an issue.  I guess the lesson learned here is, just because it executes doesn't mean it worked.

Conclusion

So the question is "When searching the datastore by location, you can not use an inequality filter for latitude/longitude. Many older articles suggest using geomodel, however with the newer GeoPt datatype, does datastore correctly handle greater than/least than on GeoPt's or does geomodel still have to be used for location queries?"

Thankfully I recently joined +Google+ , and was able to ask that very question to the +Google Cloud Platform Developers . If you want to view the post directly, click here.

They suggested I use "App Engine Search API (Experimental)", which comes with the following note

Experimental!
Search is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Search. We will inform the community when this feature is no longer experimental.

I changed all my code to use the Search API and it works awesome.  I double checked all matches using   +Google Maps and it was perfect.

So if your looking to perform location based searches, have a look at +Google App Engine Search API.

Here is a link to the Java Search API reference doc "Search API - Performing Location Based Searches"

Let me know if you have any issues or need more info, you can find me on +Google+  at  +Tim O'Brien


Thanks for reading.

+Tim O'Brien
+Kyght
www.kyght.com
tobrien@kyght.com

Note: The above icon is the property of Google and is only here to provide readers with

a visual recognition of the Google product.