The Kind of Press Corrupt Governments Love

While we still have a functional press, journalists have a duty to bring the truth to the public. When evidence leads us to wonder if government officials committed serious crimes, and much of the public desires the truth, there’s just no excuse for the press to look the other way.

Glenn Greenwald criticized NBC News political director Chuck Todd for joining the choir of pundits making excuses for avoiding the investigation of potential war crimes. When Todd offered Greenwald an interview, only under much pressure would he give lip service to the notion that an investigation should be done, but it’s obvious he feels no obligation to the public in pushing for the truth. He’s afraid a trial would be “cable catnip”, or a politicized media circus.

What message does that send if we have this political trial, and how do you know this won’t turn into a political trial? In fact, we know it’s going to turn into a political trial. I’ll take that back – we don’t know whether it’s going to turn into a political trial.

Government cover-ups are acceptable if they keep his news day orderly. He has other worries as well:

If you have this trial, and there is, inevitably, some appeals and some, where we have a back-and-forth, where there is some sort of, where it becomes a legal debate about whether so-and-so can go on trial, or not go on trial, what was allowed – they were, they thought that they were following the law, that they, you know, what message does that end up sending? Does that end up harming us down the road?

The message sent by such an event would be a very good one to send: criminals will be punished and organizations that permit or encourage them will be made in the least very uncomfortable.

Personal opinions don’t excuse Todd and other “journalists” from their responsibilities to the public. The free passes they give today lay the groundwork for future corruption and its quiet pardoning. Already Obama has had some shady and questionably legal actions (disregard of contract law, the embrace of state secrets privileges); more conservatives should realize the necessity of and demand a strong press persistently shining a light in the corners of the White House and Congress.

On Immigration

Radley Balko provides some good evidence toward debunking the myth that immigrant communities bring violent crime, but while these communities are safe, a report on identity theft makes a convincing case that there are serious costs unfairly imposed on the citizens whose identities are stolen to employ those communities (beyond the more distributed costs of social services).

If our society is to be permissive about the use of false documentation, what does one say to a person finding themselves on the hook for loans they didn’t borrow, wanted for crimes they didn’t commit, liable for taxes on income they didn’t receive, or being denied safety net benefits because of someone else’s actions? Also our desire to allow good-natured illegal immigrants to live quietly on false identities means criminals can as well.

The report claims that organizations like the IRS, SSA and credit bureaus knowingly maintain policies that ease ID theft, and certainly the private sector enjoys handling the money of (and preying on vulnerabilities of) the falsely documented, so it seems there are some nasty incentives at play.

I’m still very much torn on many issues, but the situation doesn’t seem sustainable and has an ugly effect on some Americans’ view of Hispanics. Certainly criminalizing a huge percentage of the population and economy isn’t a reasonable solution, and the idea of mass deportation is ludicrous (and would likely be an even bigger civil liberties disaster than we already have), but there must be practical and humane means to provide decentives to future border crossings and visa over-stays.

There’s a sane middle between the libertarian ideal of free borders and the ugly rhetoric coming out of the cultural warriors.

Prisoners of Endless Wars

Most reasonable people can agree that Gitmo detainees not proven to be enemy combatants at all (e.g. persons pulled off the street on whom we’ve never had anything more than suspicion) should be freed. The tougher question is, what about those obviously working for the enemy, but who are acquitted of committing war crimes.

Mark Kleiman points out that it would still be lawful to detain (not imprison) these individuals as PoWs.

Imagine that the Russians had captured Waffen SS Gruppenfuhrer Klaus Heinrich Schmidt in the summer of 1941 and put him on trial for, let’s say, ordering the massacre of civilians. And imagine that he was acquitted, because he was able to show that the massacre was actually ordered by another Gruppenfuhrer named Heinrich Klaus Schmidt.

Now what? Should the innocent Gruppenfuhrer Schmidt be sent back through the lines so he can resume fighting? I don’t think so. He goes to a PoW camp, to be held until the war is over. As a PoW, he has certain rights (he can’t be pressed for information other than name, rank, and serial number, or be forced to work) but the right to go back to fighting is not among them.

He points out we’ve been at war with the Taliban since 2001 and al-Qaeda with us since 1998 or so, so I think the natural question is just how long can we reasonably detain a PoW for? of course the legal answer is:

… as long as the conflict lasts, even if that turns out to be forever.

This being the case, I think we should consider changing our laws to better satisfy our desires for civil liberties and human rights in the new age of endless “wars”. Some random ideas:

  • PoWs should be given the conditions that must be met for the war to be considered “over” and these should be public along with the evidence we have on them.
  • Human interaction must be allowed and PoWs should retain their human dignity.
  • Detainments should grow more comfortable over time and the public should be kept aware of those conditions.
  • Simple soldiers/workers should age out.

PDF readers: Help us read.

PDF articles are notorious usability disasters, with the worst being multi-column documents that require you to constantly scroll in opposite directions as you move through the columns. PDF readers should let us draw a simple path through the document (maybe zoomed out) to outline the flow of text through the article (better, it could try to guess this for us). Once the path is set, the PDF reader could either:

A) tie the scrollwheel linearly to the text flow, so that simply scrolling down shifted the document to the next column to read. Maybe grey-out the columns not actively being read.

or B) rearrange the columns to make the document single-column.

Reading PDFs shouldn’t be so painful.

I should mention regular multi-column web pages have the same issues on mobile browsers. There’s a good case for using media queries to switch small viewport devices to see single-column layouts, and generally keeping all long articles single-column.

Minify 2.1.3 released

This is how I spend my time off? I squashed a few more bugs and got Minify 2.1.3 finally out the door. It fixes a few HTTP bugs, including working around a Safari/webkit bug that was preventing efficient cache usage. There’s also code to make it far easier to diagnose users’ URI rewriting problems when they come up (caused a lot of past headaches).

I updated mrclay.org to use it, so you can check out Minify’s HTTP handling using Mark Nottingham’s new Resource Expert Droid:

RED is a robot that checks HTTP resources to see how they’ll behave, pointing out common problems and suggesting improvements. Although it is not a HTTP conformance tester, it can find a number of HTTP-related issues.

Miško Hevery Programming Talks

Miško Hevery gave several presentations at Google last year that are worth checking out, I think even if you’re familiar with Dependency Injections and unit testing. They cover the ways that global state can sneak into applications, how undeclared dependencies make classes harder to test and reuse, and how DI in general eases a lot of pain. He’s just a good teacher and the examples are clear and made me want to attack a lot of old code. Hig blog also covers a lot of the same topics.

Thoughts on the Cost Conundrum

This New Yorker article on health care proposes that we shouldn’t focus on who writes the checks, but rather that hospitals have a well-coordinated team in place keeping costs down.

Imagine that, instead of paying a contractor to pull a team together and keep them on track, you paid an electrician for every outlet he recommends, a plumber for every faucet, and a carpenter for every cabinet. Would you be surprised if you got a house with a thousand outlets, faucets, and cabinets, at three times the cost you expected…

Even if facilities had identical rates for procedures, the cost of a given inpatient stay is impossible for anyone to predict. Neither the incoming patient nor the insurer are in a great position to judge what care given would be unnecessary or inefficient, yet hospitals in McAllen, Texas (and likely all over) appear to deliver a lot of care that might qualify as such. So what can we do about it?

In other news, there’s probably never been as much public support for single-payer care as there is now, but like many of Obama’s promises, it’s off the table. Again, a single-payer switch might only shave minor administrative costs, but it should also give us more control over providers to encourage more to operate like the Mayo Clinic.

I still find the free market health care dream intriguing. If we could just remove all government regulation and insurance, the market would just work its magic and care would be affordable (and for enough people to be considered a success). It would be a grand experiment* to just say, “OK, you get ten years to make it work.” (*Does any country have a freer market health care system I should look at?)

The reality:

  1. Voters will never ditch Medicare, so the experiment will never happen.
  2. Democrats, like Republicans, will be driven by existing industry dollars, so single-payer will go nowhere (unless unemployment got much, much worse).
  3. We’ll inevitably raise taxes to cover Medicare (no politician facing re-election will let it (or SS for that matter) go insolvent on ideology).

So we’ll continue to have a system that few are happy with, and we’ll pay too much for it.

You Probably Don’t Need ETag

(updated 3/4 to include the “Serving from clusters” case)

As I see more server scripts implementing conditional GET (a good thing), I also see the tendency to use a hash of the content for the ETag header value. While this doesn’t break anything, this often needlessly reduces performance of the system.

ETag is often misunderstood to function as a cache key. I.e., if two URLs give the same ETag, the browser could use the same cache entry for both. This is not the case. ETag is a cache key for a given URL. Think of the cache key as (URL + ETag). Both must match for the client to be able to create conditional GET requests.

What follows is that, if you have a unique URL and can send a Last-Modified header (e.g. based on mtime), you don’t need ETag at all. The older HTTP/1.0 Last-Modified/If-Modified-Since mechanism works just fine for implementing conditional GETs and will save you a bit of bandwidth. Opening and hashing content to create or validate an ETag is just a waste of resources and bandwidth.

When you actually need ETag

There are only a few situations where Last-Modified won’t suffice.

Multiple versions of a single URL

Let’s say a page outputs different content for logged in users, and you want to allow conditional GETs for each version. In this case, ETag needs to change with auth status, and, in fact, you should assume different users might share a browser, so you’d want to embed something user-specific in the ETag as well. E.g., ETag = mtime + userId.

In the case above, make sure to mark private pages with “private” in the Cache-Control header, so any user-specific content will not be kept in shared proxy caches.

No modification time available

If there’s no way to get (or guess) a Last-Modified time, you’ll have use ETag if you want to allow conditional GETs at all. You can generate it by hashing the content (or using any function that changes when the content changes).

Serving from clusters

If you serve files from multiple servers, it’s possible that file timestamps could differ, causing Last-Modified dates sent out to shift and needless 200 responses when a client hits a different server. Basically, if you can’t trust your mtime to stay synched (I don’t know how often this is an issue), it may be better to place a hash of the content in an ETag.

In any case using ETag, when handling a conditional GET request (which may contain multiple ETag values in the If-None-Match header), it’s not sufficient to return the 304 status code; you must include the particular ETag for the content you want used. Most software I’ve seen at least gets this right.

I got this wrong, too.

While writing this article I realized my own PHP conditional GET class used in Minify, has no way to disable unnecessary ETags (when the last modified time is known).

Safari Cache-Control:must-revalidate bug

Update Apr 8 2009: Apparently this bug existed in previous Safari versions (at least back to 3.1), i.e. including “must-revalidate” in Cache-Control means Expire and max-age will both be ignored by Safari. Here’s the Apple bug, if you happen to be an employee. I created an Apple account but still couldn’t find it.


Short version: Safari 4 beta incorrectly interprets the Cache-Control “must-revalidate” directive and re-requests the file each time despite freshness info sent via “max-age”.

Long version

When a server sends Cache-Control with “must-revalidate”, this tells clients/caches that, after it expires, the cached resource should not be used without first checking with the server (sending a conditional GET). From the spec (my emphasis):

When the must-revalidate directive is present in a response received by a cache, that cache MUST NOT use the entry after it becomes stale to respond to a subsequent request without first revalidating it with the origin server. [HTTP/1.1]

In other words, while the cache is fresh, there’s no need to revalidate.

E.g. I serve this Javascript file with the header “Cache-Control: public, max-age=1800, must-revalidate”. This tells clients: “consider this fresh for 30 minutes, then check back before using it again”. Unless you force a refresh, the browser won’t re-request the file for 30 minutes, after which it will send a conditional GET request. If the file hasn’t changed, the servers returns a “304 Not Modified” and the same Cache-Control header. This tells the browser to keep using the cache for another half hour and check back again. (Yes, half an hour is too short, I’ll change this later.)

Well, Safari 4 beta re-requests the file each time it needs it, seemingly ignoring the “max-age” directive. At least it’s sending a conditional GET, but it’s still a waste of bandwidth and time.