Some of you may not be familiar with the work of Soj. Soj writes for our sister site, European Tribune. She also writes, more extensively, at her own blog, Flogging the Simian.

Soj and I have been mutual admirers ever since we first laid eyes on each other’s writing. But, personally, I don’t feel my writing, or my brain, can be fairly compared to Soj’s. Quite simply, she is operating on another level.

Soj lives in Romania, but she used to work for some Federal agency that is responsible for keeping Americans safe. I’d feel a lot better if she still was working to keep us safe. (Though, I guess she still is, because) she is currently working to expose the greedheads and warmongers that have so corrupted our world.

Below the fold, she explains how she is able to distill so much information down to a useful size, and make it comprehensible to us mere mortals.

It’s a classic worthy of a Koufax Award. And the fact that she says nice things about me, only makes it more special. Wink.

My esteemed colleague, the gentleman from Pennsylvania, has written an important article about yesterday’s NYT piece that “revealed” that the U.S. military knew about 4 of the 9/11 hijackers nearly a year before the attack.

The topic has been well-discussed both on my colleague’s website as well as on others, so there’s no need to rehash it here. Nor have I much to add on the discussion for its quite well covered on its merits.

Instead what I want to pay attention to is that identifying Atta (and the other 3 hijackers) came from an Army project called the “Information Dominance Center” which is largely the data mining of open-source documents.

“Data mining” is a term that refers to sorting through data to identify patterns and establish relationships. The term “open source” means publically available documents – the opposite of “secret” or “classified” information or gathered “intelligence”.

Indeed, as I mentioned in my PDB just this Monday, the CIA is now looking to spend $100 million dollars on creating a center to data mine open source documents.

What makes this all so ironic and somewhat funny, in a tragic sense, is that Flogging the Simian has been doing this for more than a year, on a heck of a lot smaller budget!

I’ve seen comments about FTS articles both on this blog as well as on other websites, and there seems to be some minor consensus that FTS is produced by series of supercomputers hooked up in series with access to intercepted intel or uses programs such as STARLITE. While we consider that a compliment in the sense that it’s flattering, actually FTS operates on a much smaller scale.

I can also tell you that 99% of what you see on FTS comes from open-source documents, nearly all of them from the web. And I firmly believe that it was possible for the Army’s Information Dominance Center (IDC) to figure out that Atta was involved in nefarious activities in 2000 purely through open-source documents, which could then be checked against known intel and classified information, to identify him as a high-risk threat to the United States.

Two years ago I was on the “other side” of the equation – I was down in the bunkers with the computers and the gear and the antennas and I can tell you that the government of the United States suffers from myopia, often unable to see what is directly in front of it. The various intelligence agencies all spend enormous amounts of money (and effort) gathering data and assembling it and analyzing it and seeking to recognize patterns. But what they fail to do is corroborate this with secondary sources, particularly “open source” ones. And this is exactly how the CIA and others fail to predict such momentous events like the “fall” of the Soviet Union.

Even if we take just the publically-accessible internet, and in particular just Google, the problem is not the information but that there is so much of it. Google says it tracks 8,168,684,336 pages. Let’s assume that there’s some critical information about Al-Qaeda members out there. With 8 billion pages to search through, who can find the needle (critical information) in the haystack (8 bilion pages)? And how does one go about doing it?

The way we do it here at FTS is completely different than the Army’s method – which is to put powerful computers in synch to churn through the data. That does result in some valuable data but the “secret” that FTS takes advantage of is that while the internet is run by computers, it is used by people. And what understanding what people do, where people post, and what people say does not require the use of powerful computers.

Sorting through data to identify patterns and establish relationships when it comes to people is something that is a combination of a science and an art. And because of who I am and my own personal history, FTS has an advantage in this department. To begin with, I’ve been sorting through data to identify patterns and establish relationships via the internet since at least 1995 when this was my job. It was for a civilian agency and I was doing it for (mostly) civilian purposes. Ten years ago isn’t long when you’re discussing world history, but in terms of the internet, it’s a heck of a long time ago.

Later I also was trained by different organizations on how to do other things and all of this comes together with my personality and my ability to understand people to make what FTS is – a pacifist version of the Army’s Information Dominance Center.

Not that I’m the only one doing this. Some good people who are now known as E Pluribus Media did an excellent job at both identifying an inexperienced and unqualified man who was acting as a journalist, gaining unprecedented rapid access to the White House press briefings and who later turned out to be a prostitute with a criminal record. The work they did is absolutely outstanding and is an excellent proof of the power of data mining open source documents.

My focus is not on the domestic affairs of the United States (I’ll leave that to E. Pluribus) but the larger scope of geopolitical activity around the world, especially, but not solely, as it relates to the United States.

This is one of the powers of blogs which keep me writing here as opposed to becoming a commercial journalist or seeking another career in a different domain. The truth is that blogs do not just give the ordinary person a “voice”, which in itself is a tremendously good thing, but they also give people “power” by allowing them access to the billions of pages of information that govern our world and then allow them to report back on what they’ve found.

If you’ve been a regular reader of FTS for a while, you’ve seen how stories first picked up here later become mainstream stories later or else manifest into reality. For instance, you can be sure that before the end of 2005, the nation of Azerbaijan will formally announce that it will be hosting a U.S. base (or forward operating area) on its soil. That’s nearly a given, yet good luck in trying to find that in any commercial news source.

I wrote a few days ago how the internet (and blogging specifically) is today’s samizdat, the people’s free press. But today I’m here to tell you that the internet (blogging) is the people’s own intelligence agency, the “PIA” if you will.

I tend to focus on the macroscopic or global scale but it works just as well on the local scale, as organizations such as Indymedia prove day after day. Or take one of my favorite organizations, the NarcoNews, whose use of open journalism about local, domestic issues led to the proof that a Cuban terrorist had American help in fleeing to U.S. soil. And of course it is the blogs and the independent media which raised awareness amongst enough people that the administration was forced to arrest Carriles.

Years and years ago, before the internet was much more than a gleam in a few DARPA eyes, Amnesty International achieved amazing things by investigating cases of political prisoners. AI would then raise awareness about their plight and its members and supporters would write letters and make phone calls to the repressive, dictatorial governments holding these people prisoner solely because of their ideas. And amazingly, many times that pressure was enough to free these people. No guns or bribes or hostages-for-weapons necessary.

FTS is my effort to do something very similar – to raise awareness on key issues so that the people, the ordinary average person reading this blog and these articles can know what’s going on. And when the people know what’s going on, they can decide what they want to happen. Do you want the United States to have a base in Azerbaijan? Do you want the U.S. to start producing landmines again? Do you want the U.S. to spend money beaming a television channel into Venezuela and to fund groups who oppose the democratically elected president? Maybe you do and maybe you don’t, but you can’t make a decision until you know about these issues.

FTS is just one cog in the chain of independent media and independent intelligence- gathering organizations (which Al Giordano calls “Authentic Journalists”). FTS doesn’t seek to understand or report on the whole picture but to fill in the gaps that Indymedia and NarcoNews and E Pluribus Unum and Booman and the others miss. I shouldn’t even say “miss” because that implies failure. This is a collaborate effort, and unlike the commercial media, the more voices, the better.

In the same way that quantum physics revolutionized classical physics, the internet and independent media (blogging et al) has revolutionized both the classical media (reporting to sell advertising) and classical intelligence (information to suit a government) and has put the power in the hands of the people, perhaps for the first time in history.

There are plenty of bad things occurring in this modern world of ours but it’s worth taking a moment to realize the historical significance of where are – you, the reader of this blog, have access to more information than any king or empreror in history ever had. You have more access to more information than any citizen in any open society/democracy has ever had. You have more access to uncensored information about what’s going on around the world than the CIA, FBI, NRO, DIA and NSA combined.

Take a moment to appreciate the awesomeness of it… and then remember our work here is never done.