Geocoding with the Quick ‘n Dirty Geocoder

The geocoder described in this post is free software, and can be used and distributed however you like.

It installs in a servlet container like Tomcat, and accesses the geocoder.us web service to translate the names and addresses you supply (in a text file) into geographic coordinates, which it then writes back to your PC and/or a database.

Download the free geocoder

For details on how to install and use this software, frequently asked questions, configuration, etc., refer to the wiki article. The short version is just drop geo.war into Tomcat’s webapps directory, start it, then follow the instructions at http://localhost:8080/geo/code. This post describes some of the Java code used to implement the communication between the Q&D geocoder and the geocoder.us web service.

Most lines of code in this application are used in IO on your own PC, rather than the web service. The GeoServlet receives requests, and uses the GeoImporter to read in and parse files from the text file on your PC. It then uses the GeoTransport class to communicate with the geocoder.us service, and finally writes the results back to a file and/or database with the GeoExporter class.

[Read more]

Geocoding with Geocoder.us and Google Maps

Geocoding is the process of assigning geographic identifiers to map features — a specific example is assigning a latitude and longitude to a given street address. A common technique uses address interpolation. Using this method, if we know a street address and the endpoints of that street, we can interpolate the approximate location of the specific address.

The address information comes from the TIGER/Line files, which are extracts of selected geographic and cartographic information from the US Census Bureau’s TIGER (Topologically Integrated Geographic Encoding and Referencing) database.

So the task of a geocoder is to parse an address for street numbers, names, cities, states, and zip codes, and then interpolate the coordinates of that address by finding its endpoints in the dataset. I recently used two geocoders, Google Maps and Geocoder.us, and thought I’d share the results of my work along with free software that you can use to geocode your own addresses.

[Read more]

Job Scheduling with Quartz

I’ve recently become enamored with the Quartz job scheduling framework from OpenSymphony. While Quartz can be used in many types of applications, I ran across it looking for a solution to fire scheduled events in web applications. I wanted a Cron replacement that allowed me to instantiate and run any Java class in my application.

Before Quartz, I relied primarily on Caucho Resin’s runat tag to fire events. By adding a runat tag to the application’s web.xml file, you can call a servlet extending GenericServlet. Using Timers or application server-specific solutions is another option, but for the current project, I really wanted to start out with Tomcat, and I wanted the option to easily migrate to an application server if the need arose.

Quartz just works, and integrating it with an existing project was a snap. I approach most new tools and frameworks cautiously. The feature rich, easy to configure, and mature trifecta doesn’t come along as often as one would like, and I’ve been continually impressed with how carefully executed Quartz is.

There are three pieces that make a Quartz application: jobs, triggers, and the scheduler.

• Jobs: contain the logic or processing that you want to perform.
• Triggers: determine when jobs are run.
• Scheduler: the conductor that coordinates jobs and triggers.

Since I’ve started using Quartz, I’ve offloaded much of the on-demand processing into jobs of all sorts. Some jobs are used to perform routine maintenance, in a Ronco set it and forget it fashion, e.g. backups. Some jobs are used to generate pseudo-dynamic content, by running frequently in the background and generating static pages. This takes load off of the server, and conveniently aids in search engine spidering. Some jobs are traditional batch processing. For example, in the current project, a lot of geocoding is performed. Batching these requests, and running them every hour is much more efficient than firing them off on demand.

Adding to the flexibility of Quartz is the ability to schedule jobs both programmatically and declaratively, the ability to use listeners, the ability to persist jobs in a JobStore, and very versatile triggers.

Quartz is most definitely worth a look-see. The latest stable release is 1.5.2, and 1.6.0 alpha was recently released. Both can be download via OpenSymphony.

Yahoo! Go for TV

I’m a big TiVo fan, aside from its cost and single tuner (TiVo just released a dual-tuner version). I’ve been saying for two years now that I would get our home completely run via a tablet PC, which would control all audio, video, lighting, and security. I’ve since become a bit more realistic and have decided to put off that project for the next home.

I’m still sold on a Media Center PC (I like the HP Pavilion MC with warm swappable drives) as a hub with perhaps four TV tuners, and I’ve been working on getting all of my CD’s, DVD’s, and VHS tapes stored on a few external drives to make the transition much faster.

Part of the reason I’ve been putting this off is because it’s a relatively complex endeavor to integrate all of these disparate functions into one tablet. For those who want to focus just on the home automation, or just the audio, or just the DVR functionality, services like the recently launched Yahoo! Go for TV are a boon. As is common, the comments about the Go service via TechCrunch are almost as informative as the review.

bpwidgets.com Site Completed

The enthusiast site for bluepulse widgets has finally launched. This is a personal project that I’ve been working on for the past few weeks.

The intent of the site is to give developers a forum for showcasing their widgets, help foster the development community the folks at Bluepulse have created, and help bluepulse users find and use the best widgets the bluepulse platform has to offer.

Not sure what bluepulse is? Check out some previous posts (1 | 2), or head to bluepulse.com for details. And of course, head to www.bpwidgets.com to check out the web site.

Tags: mobile application, mobile development, web 2.0, web 2.0 sucks, bluepulse, widgets, Java, web application

• bpwidgets-21.png
• bpwidgets-41.png
• bpwidgets-51.png
• bpwidgets-31.png
• bpwidgets-61.png

IE Madness

This post has no point. None whatsoever.

So I’m putting the finishing touches on BP-Widgets, an enthusiast site for Bluepulse widgets. I decided that because I skimped on the site search (which I almost always implement with Lucene), I’d take the short path and just use MySQL’s built-in searching. That’s caused a little guilt, so I added a type-ahead feature, but instead of the usual Ajax variety, I’ve used an in-page array updated hourly from the master widget list. Sometimes I get weird / excessive / OCD about things like this, where I avoid querying the database at all costs. I’m still stuck in the 20th century, I suppose.

The way the type ahead works is that hourly, a JavaScript array is constructed that looks something like this: var arrOptions = ["item1", "item2", "item3"];. Easy enough. Script is done, tested in Firefox, and no problems. I’m ready to shut down for the day, and enjoy the beautiful weather we have here in Raleigh. Do a quick sanity check in IE, and what the heck?! Every character I type in the textbox that has the event handler attached is doubled. If I enter “a,” the textbox says “aa.” If I enter “qwerty,” the box says “qQwWeErRtTyY.” Yes, mixed case and all.

I am not an IE basher. I love and appreciate IE, and I love developing for it in my full-time job (my company has standardized with IE 6.0 as the only supported browser. Rich, eh? But outside of my day job, I’m frequently reminded how much effort is required to support both IE and Firefox, especially when you consider the disabled JavaScript monkey wrench. Because these are the three largest segments of the browser market. Sure, you do your best to make sure Opera, Safari, and Konqueror are supported, but I generally don’t worry about flawless execution on those last three, only acceptable execution.

Okay, on to the point. Wait, there is no point. One hour goes by, no joy. Two hours, no joy. In hour number three, after stripping out and adding back scripts and handlers, running everything through a debugger, and all of the usual tasks, I notice — as I’ve decided to just remove the script period — a final trailing comma in my array. It reads var arrOptions = ["item1", "item2", "item3",];. Remove the comma, and all is well.

Before you scoff and say “of course, look at the data,” bear in mind the symptoms. I never even dreamed of looking here because the symptoms were just so odd. JavaScript is not my specialty. I’m more of an enterprise and mobile Java developer, so the symptoms presented baffle me. I spent an extra hour Googling them, but I can’t seem to find any similar references. Does anyone have any idea why they would present this way?

So you see, no point. All’s well that ends well? Bahh! I lost an afternoon!

Tags: curmudgeonly, JavaScript, IE, Firefox

Tips for Writing Bluepulse Widgets

One of the neat things about the Bluepulse mobile platform is that it makes it very easy to write a mobile application (a widget). In Bluepulse’s case, you write a traditional web application, using whatever flavor server side code you like, and ensure the output adheres to the XHTML subset specified in the Bluepulse SDK.

Because of the ease with which widgets are created, I expect that we’ll see widgets that are created by those who have no previous exposure to mobile development. And because widgets are generally tightly focused, we’re also likely to see widgets created by newcomers to the web application development field.

So I thought some tips were in order. If you’re already familiar with web applications, mobile development, and XHTML, most of these will seem obvious, but hopefully you’ll find one or two points that will be of value. If you have some tips of your own, please add them to the comments, and together we can build a handy document that can be shared with the good folks at Bluepulse, and on the forthcoming bluepulse widgets site (bpwidgets.com).

General

Find an Idea

If you’re looking for ideas on widgets to create, check out the Bluepulse forums and the Bluepulse blog. Some of the widgets described in the blog might spur some creative thinking, and several people have posted widget wish lists in the forum.

Check the Pipeline

You might also want to see if anyone else is working on your idea before you begin your work. The forums and the blog are a start, and Luke Watson at Bluepulse is a good resource. Luke is involved in Sales & Marketing, and also developer relations, so he might be able to tell you what’s in the pipeline.

RTFM

The development kit is short, and it will answer most of your questions, so you might as well read it.

Enthusiast vs. Commercial

There are two categories of Bluepulse developers: enthusiast and commercial. Enthusiasts develop free widgets and commercial developers create fee-based widgets. I’ve attempted several times, but I have yet to figure out how one becomes a commercial developer.

XHTML Format

Syntax Rules

Your web application’s output must be properly formatted XHTML. The primary rules to keep in mind for proper formatting are:

• All tags must be in lowercase
• You must include a doctype
• All tags must be closed (even single barreled tags like <br/>)
• All tags must be properly nested
• All attributes must be in quotes

Watch your entities

Probably the most common mistake is the misuse (or non-use) of entities, and the most commonly misused entity is the ampersand (&).

In XHTML, the ampersand marks the beginning of an entity. An entity is comprised of an opening (&) and closing (;) delimiter, with the entity code in the center. So any ampersand is treated as the beginning of an entity. If the closing delimiter, the semicolon, cannot be found, your document is not properly formed.

This means that a typical query string like: ?&name=widget&id=7 is invalid. Instead, it should be written as ?&name=widget&id=7.

For those of us who use Dreamweaver for UI development, older versions of DW had a helpful and altogether frustrating setting that attempted to rewrite the entities you carefully wrote. Been here before? Go to Edit | Preferences | Code Rewriting and check both encoding options. Incidentally, DW8 has corrected this and has terrific support for CSS. CSS support is alone worth the price of admission.

Validation

If your widget runs fine while testing it in your browser, but you receive “Unsupported Page Type” errors when running it through an emulator, your XHTML is likely formatted improperly. There are a number of validators out there that will help you make short work of finding the problem. View the source via your browser, and paste it into a schema validator like the one found at http://schneegans.de/sv/. It should point you right to the offending part of your document.

Double List View

At a glance, the double list view appears to be a powerful way to display summary text to a user. On the top line, place your title, and on the second line, place a longer text summary.

Unfortunately, the majority of the devices I tested on did not support the double list view, which limits the utility of it. If something is important, make sure it’s on the top line. If it’s not important, you can place it on the second line, in which case most of the text will be truncated for phones that don’t support the view. My preference is to stick with the regular list view, and support all handsets consistently, according to the bluepulse spirit.

Use Templates

I’m not a big fan of developing UI’s. I’m not that good at it, either. One of the things I like most about developing widgets is that I never have to worry about the UI. I have several templates created: list view template, text form template, and text template, and I reuse them for every page in every widget that I create. I’d recommend that you do the same.

More information on XHTML:
http://www.w3schools.com/xhtml/

Emulators and Testing

Developer Suite

I might be in the minority, but I have always had difficulty installing and integrating emulators with my IDE of choice. Nokia’s Developer Suite installs smoothly and offers 40, 60, and 80 Series emulators, along with the ability to plug-in other Nokia emulators.

International Character Support

In two widgets that I created, Ebay OnTheGo and Amazon OnTheGo, international characters and currency symbols (e.g. yen, pound) resulted in “Unsupported Page Type” errors via Bluepulse during testing.

The quickest way around this (from a performance perspective) is to use regular expressions to filter out the offending characters. I have a couple utility methods that I use for this purpose. One returns a Boolean, and one returns a clean string.

For example, calling checkRegExp(shaun@bpwidgets.com, “[^a-zA-Z0-9]”) returns true, and calling cleanRegExp(shaun@bpwidgets.com, “[^a-zA-Z0-9]”) returns “shaunbpwidgets.com.”

public static boolean checkRegExp(String value, String regexp) {
Pattern pattern = Pattern.compile(regexp);
Matcher matcher = pattern.matcher(value);
if(!matcher.find()) {
return false;
}
return true;
}

public static String checkRegExp(String value, String regexp) {
Pattern pattern = Pattern.compile(regexp);
Matcher matcher = pattern.matcher(value);
if(matcher.find()) {
return matcher.replaceAll(“”);
}
return value;
}

Peter from Bluepulse has an additional handy tip regarding character support:

“If you make sure all special characters are UTF-8 encoded, you can include pound signs, vowels with funny lines/dots and more. Some phones, particularly Nokias, allow the user to type these characters into a Text Form View, so if you use that view, you should be prepared to deal with them.”

Test on Multiple Emulators

Bluepulse takes much of the concern out of this. If it works on one emulator, it should work on them all. This is in stark contrast to the field you play on when developing a typical mobile application! Even though the widget will likely work across all emulators, you might be surprised to see how it looks across different emulators. As a sanity check, see what your widget looks like in several different emulators. You’ll likely find yourself rearranging menu items, editing screen titles, and reworking text here and there.

Test on a Real Device

Similar to testing on multiple emulators, you should test your widget on one or more real devices. Emulators will primarily point you to UI differences between phones, but testing on a real phone might open your eyes to some performance implications of running your widget. It’s also nice to see how your widget will look on the phones with which you are most comfortable – your own.

Traditional Mobile Development Tips

Constrained Resources

Always be aware that users will be using your widget on devices with limited display and data transfer capabilities, and data input is only slightly more amusing than shaving with a cheese grater. Most of the tips in this section follow this constrained resource theme.

Titles and Text

The amount of text that will appear in certain views is limited. For example, in the text-form view, long strings are truncated. Short, terse instructions are better than incomplete instructions.

Menu item title display is generally left to the implementation, though I’m not sure if bluepulse has done any work around standardizing. In traditional mobile development, both short and long labels can be provided, and the implementation decides which one to use. To be on the safe side, try to keep your labels one word long, and front-load them.

Similarly, your screen titles might be truncated, so put the most important characters at the beginning. This has improved dramatically over the years, but not so long ago, it was a real art to determine how to intuitively name your functions or screens with as few characters as possible.

Less important, but still advisable, is to make the first characters of your labels unique from other labels. This will enable users to easily differentiate among the commands available to them. Also keep in mind the potential items added to the menu by the bluepulse client, and ensure that there are no naming collisions.

Use the title of a list page to tell users what to do with that list.

Provide Reasonable Defaults

• Rather than requiring text entry, allow users to select from a list, if at all possible.
• When using lists, offer the most popular choices near the top.
• If your list only has one item, what’s the point of having a list? Just drop users to the action page.
• If you are forced to use text boxes for data entry, provide a reasonable default value.
• Whatever value a user types into that box, save, and present to them as the default the next time around.
• Persist data in a database between visits, rather than asking the user to enter it again on the next visit.

Provide Reasonable Navigation

On any page within your widget, anticipate where your users might want to go next. Provide direct navigation via the menu, rather than requiring the user to back out of an interaction. At the same time, make your widget back friendly, for those who love to mash on those back buttons.

Conclusion

That’s about all I can think of off of the top of my head. As you can see, Bluepulse takes most of the headache out of widget development, but there are still some considerations we all should all keep in mind.

Tags: Bluepulse, widgets, cell phone, mobile application, mobile development, mobile 2.0, bpwidgets

Eureka: Interact with JavaScript in a Remote HTML Page

I’ve been on a bit of an Ajax kick lately (no groans, please), and one of the books I’ve read is Wrox Professional Ajax. I was browsing the lead author’s blog the other day, and ran across Eureka. It’s a pretty slick tool. I Dugg it a few days back, and judging from some of the comments, people are confused about what it can do for them.

It’s an interpreter that essentially enables you to load a remote page. Once loaded, you can interrogate and modify properties (works in IE only), which is pretty slick, although I’ll admit that I haven’t really figured out a good use for the tool.

Available via NCZOnline

Tags: Eureka, Web 2.0, JavaScript

« Previous Page