My previous post got me wondering what Apple’s EULA says about location data collected by and on the iPhone. Like most people I breezed past the 1.1.3 EULA agreement without studying it, and then found I had to dig around online to find the iPhone License Agreement, which I finally did on Apple’s site.

Section 4(b) is the relevant piece:

A couple of basic points here. First, Apple asserts that they may collect and share with its partners location data from the phone. They secondly assert that the location data is collected in a way so as not to “personally identify you.” I am not sure I agree with that statement based on my previous post, but let’s give them the benefit of the doubt for now.

Second, they go on to say that the user can “withdraw their consent at any time” by “turning off the location-based feature.” How? Where is that button in the preferences?

Third, they make the statement that “turning off or not using these features will not impact the functionality of your iPhone.” Uh, yeah it will (assuming it were possible). I won’t have location-based services or maps, which is a real reduction in the functionality of the phone.

Let’s take a look at the newly published iPod Touch January ’08 Software Upgrade License Agreement, which we would expect should address the newly-added location features specifically.

Once again, the same section 4(b) contains the same text as the iPhone Software License Agreement. So, they consider the location services in the iPhone and iPod Touch to be legally equivalent, which is interesting to note.

Seems to me Apple’s in an interesting place with this location data business. The location data that is collected is packaged in HTTPS, so we can’t inspect it. It is theoretically possible that it contains nothing that can identify a given user “personally,” but what does that mean?

Does it contain an iPhone serial number, or an IMEI number, or a phone number? How is this data stored? While that data, by itself, may not personally identify you, could it be correlated with data that does? Is that possibility covered by the SLA as written?

I will assume that Apple and its lawyers have thought this through. However, there are some interesting issues raised here. Now that the iPhone/iPod Touch has location support, we should expect Apple, and possibly third party developers, to leverage that location data in interesting ways.

The most interesting ways involve tying identity to location, so if anything is going to happen down this path, then the SLA, as written, is not going to suffice.

In the meantime, you can bet that somebody is going to consider whipping up a class action suit because there is no clearly marked way to turn off the location-based services, and because “turning off LBS” does affect functionality — your phone doesn’t know where you’re at!

And keep an eye on that SLA for future versions — you can bet that the wording on the location data is going to evolve.

When Apple announced yesterday that the iPhone would now be “location-aware” with the release of their 1.1.3 software, I was curious how they had done it.

I’ve been working with location information quite closely (see Twittervision, for example) for the last year or so and have had some conversations with different companies about how Apple might geo-enable the iPhone.

There are three options available:

• GPS
• Cell Tower ID
• Wifi Access Points

GPS is not an option at present. E911 laws in the US have required carriers to provide location information for some time, but that could be via GPS or from cell tower triangulation data. TruePosition makes this their entire business, and is the primary location information provider for AT&T and T-Mobile in the US. A pretty cozy gig, eh? They do this by way of tracking cell tower information within the network, from what I understand.

GPS may be an option later if Apple adds an AGPS (assisted GPS) chipset to the iPhone or supports external Bluetooth GPS units, but external bluetooth will never be a true mass market phenomenon, and AGPS is at least going to have to wait for the next iPhone refresh, probably not til next year.

Cell Tower ID is another option. Carriers know where their cell towers are (we hope), and by comparing the signal strength and the intersection of multiple cell tower antenna distribution patterns, you can make a pretty fair guess about where the user is. It’s not always spot-on accurate, but it’s pretty close.

Wifi AP’s are the third option. There are millions of Wifi AP radios running around the world at this point, and for the most part, they tend not to move around that much. They do, however, come and go from time to time. However, there are a lot of them, and with a modest investment in driving around populated areas, one could build up a pretty accurate database of what APs are where. Then they could sell that database to people who want to know where their Wifi client radios are.

This is exactly what my friends up at Skyhook Wireless have done. You can try out their Loki service for your laptop (Firefox/IE plugin). Suddenly, if you have Wifi, you also have a pseudo-GPS capability.

Judging by the fact that Skyhook invited me to stop in and see them today at MacWorld (which I would have loved to do, but am sadly unable due to my being at home in Maryland this week), it seems Skyhook got the contract to provide some location data to Apple. Apparently, the iPhone uses both Cell Tower ID and Wifi (Skyhook) data for location, while the iPod Touch uses Skyhook exclusively. Good Job, guys!

This explains why when I went to see Skyhook in June and said that a company like Apple might be very interested in their technology, there was a definitive “no comment.” This happens a lot; companies like to protect what might be a very early-stage negotiation, or even an intention, a lot of the time. But in this case it looks like Skyhook bagged what might be their killer deal.

Yesterday, I succumbed to the hype and “Revirginized” my iPhone (we had been engaged in some unsavory hacking) so I could safely install the new 1.1.3 software update that Steve said would be available. The revirginizing and upgrade went as clean as could be, and now my phone is running 1.1.3.

I thought I might “inspect” what the phone is doing when you do a location lookup. I have a bunch of resources on my home network, including a multipurpose Linux server, so I thought if I could pass the iPhone’s traffic through the Linux box, some tools like ngrep and tcpdump might reveal what exactly happens when the iPhone tries to position itself.

Well, turns out I was mostly right. In typical Apple fashion, though, they’re keepin’ it real with HTTPS, revealing nothing very interesting about how the location information works.

The iPhone is 192.168.1.199 and my proxy is 192.168.1.10.

Here’s what I saw:

T 192.168.1.199:49311 -> 192.168.1.10:2525 [AP]CONNECT iphone-maps.apple.com:443 HTTP/1.0.Host: iphone-maps.apple.com.User-Agent: Apple iPhone v1.1.3 Maps v1.0.0.4A93.

T 192.168.1.10:2525 -> 192.168.1.199:49311 [AP][..HTTPS DATA...]

T 192.168.1.199:49311 -> 192.168.1.10:2525 [AP][..HTTPS DATA...]

So, alas, nothing to see here, really… move along. However, we do now know that Apple is grabbing data from the phone via HTTPS, processing it network-side, and rendering a response to the phone about its position. We do not, for example, see a variety of calls to Skyhook, Google, or elsewhere, which is not inconceivable without verifying it.

After the HTTPS call, we see this unencrypted call:

T 192.168.1.199:49313 -> 192.168.1.10:2525 [AP]POST http://iphone-wu.apple.com/glm/mmap HTTP/1.1.Accept: */*.Accept-Language: en.Accept-Encoding: gzip, deflate.Cookie: s_vi=[CS]v1|46B904DB00003607-A290B210000599B[CE]; s_nr=1199572400032.User-Agent: Apple iPhone v1.1.3 Maps v1.0.0.4A93.Content-Type: application/x-www-form-urlencoded.Content-Length: 145.Connection: keep-alive.Proxy-Connection: keep-alive.Host: iphone-wu.apple.com.

...

T 192.168.1.199:49313 -> 192.168.1.10:2525 [AP]..*..m..DN..en_US..com.apple.iphone.1.0.0.4A93......@.......?...&_...>....&`...>.......&]...>....&^...>....&\...>....&_...>....&[...>....&`...>.

T 192.168.1.10:2525 -> 192.168.1.199:49313 [AP]HTTP/1.1 200 OK.Date: Wed, 16 Jan 2008 12:38:31 GMT.Server: GFE/1.3.Content-Type: application/binary.Content-Length: 113.Cache-control: private.Connection: close.

Not sure what this all is, but it looks like it has my iPhone serial number in there. It’s so nice that Apple wants to know so much about my phone, its serial number, its position. Why, if DHS ever has any doubts about me, perhaps they could simply just ask Apple? Maybe they know where I am.

What is Apple’s position (pun intended) on customer privacy, now that they seem to be in the location data business?

Other firms like Boost Mobile’s Loopt service have gone to great (ridiculous) lengths to inform their customers about location data privacy and to protect collected data. So as to avoid potential problems, Loopt does not even save a location track for its users, but instead stores only the current location of the user. (This was the case when I spoke with them in May 2007.) They figure this makes them less of a honeypot for DHS types, and keeps their customers happy.

I have never believed that consumers are as paranoid about location data as the press (and the most paranoid among us) would have us believe. Most people are willing to generate, share, and publish some limited amount of location data if it provides some value to them in return and they can control the data sufficiently.

What seems like a simple software update for the iPhone is actually the consent of millions (4M+ according to Steve) of users to potentially publishing their location information. And not just for the iPhone, but for the iPod Touch as well.

Now the question is what a theoretical 1.2.0 software release might hold: location of your iChat buddies? Location-enabled Twitter clients (using the Twittervision API)? Your friends conveniently plotted on the Google Maps client? All of this is now theoretically possible with the iPhone and iPod Touch now, and Apple holds the keys.

It will be very interesting to see how the iPhone SDK (Software Development Kit) works next month. If Apple opens up this location service to third party developers, we can expect to see some very interesting applications emerge this year.

The fact that the location service is not down to meter-accuracy is irrelevant (it put me, alternately, within a few feet of my house and across the river at the Annapolis Mall — I suspect because it was alternating between an accurate Wifi position and a more general cell tower position). To make social location services work, all we really need to know is generally where someone is (nearby) and that they are really there (device has reported location).

There are plenty of apps where approximate location is sufficient (stores nearby, friends nearby, homes for sale nearby, etc). Only for driving-direction or aviation applications do you need meter-accuracy. A later update to the iPhone hardware with an AGPS chipset will solve that problem, but even without that, this opens up an amazing array of possibilities.

Mostly, great credit should go to Apple for pushing out a technology so seamlessly, so effortlessly, that so many others have found so problematic and full of legal and perceived landmines. This is a big deal. Skyhook, Loopt, uLocate, Nokia, Navizon, and dozens of others have been grasping for this holy grail for some time, and they’ve been told variously that it’s “impossible to get the data,” or that “consumers won’t go for it”, or that “no one would fund it.”

Apple did it via iTunes with a software update. Agree? Kudos, Steve.

As we enter into 2008 and a new US presidential election cycle, I am once again struck that the available candidates are all compromises; compromises between our ideals and the political and economic realities that rule the world today.

As we look back on the arc of recent history, we can point to some truly great individuals who managed to crystallize the mood of their era and, like a flame, transform it into something greater. From the French Revolution and Rousseau’s social contract, Napoleon cast modern France and set the stage for many nations around the world.

Unlike Napoleon, who offered equality and efficiency but a near totalitarian government, Jefferson, Washington, Adams, and Franklin wanted a government that could somehow withstand the forces of upheaval that had plagued so many before. They wanted to build a nation which could outlast the lifespan of just one Great Man, for they correctly supposed that Great Men were hard to find.

The three governmental branches and the deference of power to local governments are major strengths of the American Constitution and, really, have enabled the nation to hobble along between its handful of Great Men. In a sense the American system does its best to transform Ordinary Men into Capable Men. Sometimes it has transformed Capable Men into Great Men.

Lincoln was faced with the task of reconciling our ideals (all men are created equal, we the people, etc) with the economics of the day. Slavery was an economic reality that some felt was unavoidable, insurmountable, and even sanctioned by scripture. However, Lincoln realized that a divided nation was vulnerable. Lincoln had the unenviable task of explaining this to the nation, which he did so eloquently and forcefully.

Lincoln was called to tell the nation that which it did not want to hear. Washington and Jefferson did much of the same. Even Napoleon had to make a case that efficient government was better than a nonexistent but ideal government. In each of these cases, intelligent and thoughtful men have been called to reach beyond what’s politically and economically expedient to inspire, coerce, cajole and charm their constituencies into doing what is in the long-term best interest. It has seldom been easy.

In the distant future, when we look back on the last 100 years, I believe that we will see is a nation which has not learned to reconcile its politics with its economics, and which has hobbled on on the shoulders of but a few Great Men. Arguably the greatest transformative leaders of the 20th Century were Churchill, Franklin Roosevelt, Hitler, and Stalin. Roosevelt and Churchill were called to greatness by being locked in conflict with Hitler and Stalin.

While it is very tempting to include other American presidents like Wilson (father of the UN), Kennedy (a leader of the civil rights movement and an inspiration in the light of the Cold War), and Reagan (a truly great communicator who deserves some credit for the fall of the USSR), the simple fact of the matter is that these people were reacting to their circumstances. In a sense, Kennedy and Reagan were cleaning up Stalin’s aftermath, and Wilson unknowingly laid the groundwork for HItler’s ascent by presiding over Versailles.

But the Great Men have reacted to their circumstances, as well. Somehow, though, the Great Men are called to do more than simply react, more than simply show up, more than simply hold to their convictions. The actions of Great Men are outsized, just-right, artful and inimitable.

They have led us forward at times when we did not want to be led; showed the way at times when economics and politics offered more expedient paths; cast a visionary spell on the populace at a time when we needed to believe. Great Men have the power to call ordinary people to become extraordinary.

Today, our politics is inextricably tied to our economics. The founders expected that; the state system is designed to give regional economic interests fair say in the process and to insure that no one state dominates our national political debate. Today, however, this seems quaint.

The economic debates that matter today are not about whether to build a canal in New York or in Michigan; our economic debates are about the regulation of global corporations, human rights in China, global free trade, and immigration. Somehow it doesn’t seem the founding fathers anticipated the rise of the multinational corporation, and specifically its potential to influence American and global politics.

Today, the measure of a candidate’s popularity or viability is their ability to raise money. The candidates with the most money are generally those who are most agreeable to American Corporations. The candidates who are the most agreeable to American Corporations are also those who are most agreeable to global, multinational corporations. Therefore, we have a system where global multinational corporations get the first and most influential vote.

In a world of uneven regulations and where global corporations have only one mandate — to make money for their shareholders — we face a situation where our candidates are hostage to companies, and indirectly of governments other than our own. If Exxon/Mobil supports a candidate because they believe they will take a laissez-faire attitude towards environmental regulations in China, and if the Chinese government prefers to operate without environmental regulations because it believes it will generate more revenue, we indirectly have candidates who are locked into a variety of unsavory positions around the globe.

This is not news. It is also not realistic to suggest that we decouple the American democracy from capitalism; it is reasonable to have our system of government be influenced by and aligned with our engine of economic progress. We can temper the flaws in capitalism with sensible regulation and policy. We do a fair job of it.

The question remains, however: where are the Great Men, or Women, of 2008?

We have various people who think it’s their turn, or who have been anointed by global capitalism as acceptable, but to be frank, it is hard not to wonder if Ronald Reagan was right: that the best minds are not in government, that if they were they would be hired away by business.

That maxim may be true, but the converse probably isn’t. If it was, the best possible candidates would be hedge fund managers. However, Bloomberg may still run.

It seems we have lost a sense of what it means to be a public servant. With the scrutiny, the media, and the schedules that come with presidential politics, you really do have to be crazy to run. Is it any wonder, then, that we ended up with the oddball mix that we have?

It is doubtful that a Roosevelt or a Lincoln would fare too well in today’s races. Why is that? Maybe Roger Ailes could tweak Lincoln up a bit. (Lose the top hat, ditch the beard, a layer of foundation and a $400 haircut and you still flub Hannity & Colmes?)

Perhaps we have some Great Men and Women in our midst here in 2008. Certainly Obama is using the soaring rhetoric of a great leader, however it’s tough to know if he has the mettle to withstand the global political/economic machine once in office. Ron Paul certainly has no problem saying things that are unpopular. Hillary is trying hard but has a hard time inspiring, not unlike the rest of the candidates.

I am left feeling, as I think we all are, that we still haven’t found that one great leader who can crystallize and transform the challenges of our time. But if history is any solace, we know that a great leader comes but a few times a century. Maybe it’s just not our turn.

With the recent release of Rails 2.0, many of us are reviewing our approaches to common problems. Many new features have been added to Rails, and some old tricks are either no longer necessary or no longer work.

I am developing a project with Rails 2.0 and am getting close to putting it into production. A recurring issue for today’s web developers is that of asset packaging, or the combination of multiple site assets into a single file. Specifically, we’re talking about Javascript and CSS.

A given “Web 2.0” (a term I wish had recently been found dead in a cramped apartment in Brooklyn) site might have a half dozen Javascript or CSS files to deliver to a user, and web browsers are not all that efficient at retrieving them. Each one requires a separate TCP connection to the server, and many browsers are only capable of getting two of these files concurrently. This means delays for your users.

In Rails 2.0 (and previously in Edge Rails), it’s possible to combine multiple Javascript and CSS files using the javascript_include_tag and stylesheet_link_tag functions in your html.erb files; simply add :cache => true to the parameters like this:

<%= javascript_include_tag 'prototype', 'effects', :cache => true %><%= stylesheet_link_tag 'main', 'shop', 'form', :cache => true %>

With :cache => true and when running in your production environment, Rails will automatically combine your Javascript and CSS assets into single files (all.js and all.css, respectively) and significantly reduce your site’s load time.

However, this really only solves part of the problem. A common technique used to further improve site performance is to compress Javascript and CSS by removing unnecessary whitespace and comments. I am not sure why this wasn’t included as part of Rails’ built-in caching features, but it seemed to me it should be easy to add.

Turns out I was mostly right. Google “javascript minimization” (or minification) and you’ll see it’s a pretty hot topic. The Asset Packager plugin from Scott Becker does this, as well as CSS compression, but is targeted at Rails 1.x and doesn’t really make sense in the face of Rails 2.0.

So I set out to solve this problem in an elegant way for Rails 2.0. Asset Packager uses a Ruby script called jsmin.rb by Uladzislau Latynski which is based on jsmin.c by Douglas Crockford. The thing is, jsmin.rb is not a class or library, but rather a standalone executable that operates on stdin and stdout. Asset Pacakger actually forks a ruby shell process to do its Javascript minimization, and this seemed like folly if it could be done internal to Rails.

Accordingly, I modified jsmin.rb to operate as a singleton class and with a class method you could pass Javascript data to. Then it was simply a matter of monkey patching this function into ActionView::Helpers::AssetTagHelper, home of javascript_include_tag and stylesheet_link_tag.

I also wanted to add in CSS compression, which turned out to be easy. The javascript_include_tag and stylesheet_link_tag functions both use the same underlying functions to package their assets, so it was a simple case of replacing them with equivalents that do compression appropriately, based on whether we are dealing with CSS or JS.

config/initializers/javascript_minimization.rb:

module ActionView  module Helpers    module AssetTagHelper      require 'jsminlib'

      def compress_css(source)        source.gsub!(/\s+/, " ")           # collapse space        source.gsub!(/\/\*(.*?)\*\/ /, "") # remove comments        source.gsub!(/\} /, "}\n")         # add line breaks        source.gsub!(/\n$/, "")            # remove last break        source.gsub!(/ \{ /, " {")         # trim inside brackets        source.gsub!(/; \}/, "}")          # trim inside brackets      end

      def get_file_contents(filename)        contents = File.read(filename)        if filename =~ /\.js$/          JSMin.minimize(contents)        elsif filename =~ /\.css$/          compress_css(contents)        end      end

      def join_asset_file_contents(paths)        paths.collect { |path|          get_file_contents(File.join(ASSETS_DIR, path.split("?").first)) }.join("\n\n")      end

    end  endend

By simply modifying join_asset_file_contents to use our new function get_file_contents instead of File.read, we quickly get to the heart of the matter. CSS files get compress_css run on them, while Javascript files get JSMin.minimize run on them. Your :cache => true Javascript and CSS assets will now be gloriously combined and compressed!

Note that the above monkey patch requires jsminlib.rb, which you can download here. It is just a modified version of the original jsmin.rb, and you will want to put it into your Rails lib directory.

A good next step would be to further enhance get_file_contents to do Javascript obfuscation, which allows for the replacement of variable names and thus even further compression; it also tends to make Javascript code nearly incomprehensible and thus harder to steal, which may be desirable for some developers. I haven’t found any native Ruby ways to do this yet, but it seems to me that this would be a good place for a C extension (or similar), and that this should all be put into a tiny and lightweight plugin.

I’m always amazed at how easy it is to bend Rails (and Ruby) to one’s will, and in this case it’s really quite elegant and straightforward. I’d love to hear your ideas about how to take this idea forward, potentially even including it in Rails itself.

Download the files here:

• lib/jsminlib.rb
• config/initializers/javascript_minimization.rb