From Markos:
Our host used to lease space at a wholesaler data center. The host bought its own building, and is in the process of physically moving all their servers to the new facility. The process was supposed to take 2 hours, in the dead of night, and we were supposed to have a message on the site explaining the situation.
But, as is often the case with technology, things didn’t go according to plan. I’m not sure what the problem is, and since I know they’re scrambling trying to make things right I don’t want to call and further delay them demanding an explanation. There will be time for that later. But bottom line, moving 10 servers (in addition to all their other clients) from one physical location to another is not easy. Lots of things could go wrong, and apparently many of them did.
Does anyone else find it interesting that while Kos was doing this, we also had problems with other non Kos related sites? MLW was so slow last night I gave up…Booman was not letting me on either. It took forever. Other sites also were freakin out. I understand Markos was moving servers but this had nothing to do with the other site problems. I wonder if Kos was aware of the other site problems? Thanks for the message SusanG.
dKos is the 800lb gorilla of the left wing-blogosphere. When it goes down, its normal traffic goes elsewhere, straining everyone else’s servers.
on MLW My Left Wing. Diary by someone who’s an admin in email distribution.
Now for the fascinating stuff!
http://www.myleftwing.com/showDiary.do?diaryId=4358
yeah..i just got on…but sooo slow…it’s like it was frozen in time over there.
Give the server a break, will you?
interesting..it is like wandering through a ghost town. 10:28pm last night…frozen. Wild.
Might be a modern dkos record for amount of time a diary has spent on the recent diary list…much less the top of the list.
My bad. I should probably just wait until next Tuesday to log back on like the other few thousand logging into the live site.
I got a half page so I backed out. The last FP post is from 10:30ish Pacific last nite.
I’ll give the servers a rest & go back later. I have a lot of housework waiting.
I thought it was me, like maybe they didn’t like me anymore. (I know that this is still debatable, but please be nice to me. I’m sleepy.)
I wish them good luck. My mantra, as much as I love computers, is “computers suck.” Everything always takes more time than you think.
Yesterday, there was a bru-haha on a Voting Fraud Diary with armando threating to kick us all off, so I discontinued my feeds and came here. Nice to see I’m not the only one that can’t get in…;0)
Is he on meds or something? He’s new Armando one day, and prick Armando the next! I don’t get it…
one does not have to be a prick to, one day be a prick and the next be a non prick. Meds actually might help the prick problem not creat it.
Can sometimes take awhile for them to kick in!
tumescent, detumescent, tumescent, detumescent . . .
nothing new . . . nothing worth seeing . . . move along now . . .
T-up, T-down, T-up, T-down…
legit fraudsters are welcome here at the Trib. I really wanted to read that diary. The comments were in the 300 last night when I looked. I was going to open it and then Boom…the site went down.
That was my diary – I didn’t expect to create such a maelstorm of controversy!
All the diary was was a call to freep this article:
Lyn Davis Lear: Paging Frank Rich! GAO confirms – 2004 Election Was Stolen
After some quotes, I made my own comments, which included the phrase “if there was fraud”, which, in my opinion was an admission that the article may be misleading. However, I was threatened with banning, and also received my first troll ratings ever by “yetanotherliberal”
An interesting night to say the least.
I had also put up a poll, and when I last looked at it, 66% said the diary was useful, 1 person said it smeared the integrity of DailyKos, and 30% checked that it “unfortunately shows some of the problems that Kossacks have working together”.
It will be interesting to see if I truly have been banned…
Thanks for the link.
As for banning fraud diaries at dKos…no comment.
How about putting that diary up here? I’m sure you’ll find it well recieved.
Your diary is back up now. I went over and reco’d it ;o)
I knew that New Armando shyt wasn’t gonna last.
I don’t think Armando was threatening to ban anyone. I think he just warned that “certain types of diaries” are, hmmmm, discouraged on the site. He mentionned that people have been banned over these type of diaries but didn’t say anything about doing it himself.
He did say it alittle more forcefully…it felt like my brother was threating to tell my mother—I was being a very bad girl LOL.
It seems El Caudillo has just posted a diary entry on that subject.
In other words, go ahead and speak your mind, but if you do, your work will be deleted and you will be banned.
Maybe the Brain Trust over there should start referring to themselves as the Central Committee, or the Politburo. God knows that as time goes on, they seem more and more interested in promoting a strict party line, and the language in which they express themselves has become progressively more Orwellian; “policy set down by Markos,” “Markos made the decision that these irresponsible claims . . . were so deleterious to the cause . . . that they would be prohibited,” “Markos reiterated his policy that diaries of this nature were prohibited,” “[t]hese issues and the policy enacted by Markos were thoroughly debated.” Just throw in a few exhortations like “comrades!” and you’ve got a CPUSA communique circa 1936 on dealing with criticisms of the Popular Front policy.
In other words, go ahead and speak your mind, but if you do, your work will be deleted and you will be banned.
that is not what was said at all. there is not a ban on theoretical musings. if a diary presents a hypothesis and reasonably lays out how the author got there, it won’t be banned. i diaried my opinion that bush tried to invoke the insurrection act (i.e., impose martial law) in post-katrina new orleans. i didn’t get banned. another prime example would be susan g’s diary on “jeff gannon” that got that whole ball o’ wax rolling.
it was very clear from her diary that she was just following bread crumbs and inviting us to go along. she didn’t claim any aspect of her musings was proven and she didn’t present anything as fact. contrast that with the “well, bush & blair stood to benefit from the london bombings and so must be behind them” diaries. those are the unwelcome ones.
hate kos and armando and dailykos all you care to, but please play straight with the facts involved. but really, el caudillo? armando is a fascist now?
i love the orange. i love the green. they are two of my favorite colors; two of my favorite blogs. i am really disturbed and saddened by this bickering vibe. is it productive; does it matter? can’t we all just get along?
As is evidenced by the sneering, abusive comments he posts every time anyone disagrees with him. He also goes about the comment threads with a bodyguard of sycophants who rate up his comments, downrate those who criticize him, and address his foes in the profane/obscene terms he is no longer allowed to emply. El Caudillo is awfully mild compared to the extraordinarily vulgar abuse he’s directed at other people (something he was ordered to stop doing, which has the oddly ironic effect of making his snarling responses seem even more childish). People have had their diaries removed and been banned, and those who have protested this have been banned as well. Hell, people have been banned for diving good ratings to verboten-style comments.
Something very ugly has been going on over there during the past year, and a lot of people who made the site worth going to are no longer there. If you can stay blind to that, good luck to you. Just make sure you avoid whatever topics are deemed off-limits, lest it happen to you.
I just read my email and apparently there is a new round of Armando bashing going around the site. I don’t have time to figure it all out. I am about to disconnect my computer and I won’t be able to read the response to this or my email until Tuesday.
But it should be clear by now that I really don’t comments like ‘Armando is an autocratic thug’ on this site, or comparisons made of Armando to fascism.
Everyone should know that by now. So, why do people continue to make vicious personal attacks against him, when I have expressed my opinion on this repeatedly? Please cease and desist. See you Tuesday.
Can be found here.
Don’t bother banning me, you can just tell me to get lost and I’ll stay away. I’ve deleted the two diaries I’ve posted here, and while I’d like to have the option to come back here and post comments, I don’t plan on doing it much, so if you’d rather I didn’t, I’ll just vanish. I’m in the process of deleting all my diaries at dKos, and am determined to do nothing more than lurk there as it’s still a useful repository of information.
I have no problem with never discussing dKos or anybody who posts there if that’s the way you want it. Just spell it out very clearly so people know what’s what. Otherwise, all your rhetoric about providing a haven for people who been banned there for discussing certain proscribed topics rings very hollow.
Hope the move went well. All the best.
Funny. The same thought briefly ran through my mind, silly as it is: I’ve been BANNED!
And have you ever had to explain to someone that you don’t know how long it will take because you’ve never done this exact thing? I always say “i hate computers” which is of course not really true, but they can really suck.
My feeling is this–everything is connected.
And…they hate us. I don’t mean that Kos hates us. I mean, THEY hate us.
Time for tinfoil hats, perhaps. But…THEY HATE US.
They haven’t found a way to jam us…yet. That we know of…
This is a good reason to have the other sites abounding in loose affiliation with Daily Kos. I think there’d be more of a problem taking everyone down … and there’s always quick ways to throw up a blogspot or peer-to-peer so we don’t get dispersed.
to post an alert ahead of the move, since he knew it was going to be down for at least two hours.
SusanG see my comment below. Someone needs to organize this. We need a backup plan if, for whatever reason, the Liberal blogs are taken out or down. It could happen a couple of days or weeks before an important election or vote and we would not be able to mobilize. We need a plan B meet up place.
Yes, you’re right, Chamonix. I do expect that, at some most important moments in the days before elections or when incredible news is moving across, the “800 lb gorilla” will go down for a couple of hours. You should expect it, and have workaround plans ready for it.
It’s also a good way for someone’s once-a-day-permitted newly posted diary to get lost because by the time the site is available to all a few hours later, it’s scrolled off the front page and won’t get recommends.
Timing is everything.
I get around these issues by just making booman my main haunt 🙂 I didn’t even know KOS went down!
I hear you, but I HAD to just let that out.
People are waking up to Bush’s crimes; folks are turning to Kos and to Boo, Atrios, and the rest and getting the real poop and scoop…
The LeftBlogosphere is like Radio Free Europe blasting to the oppressed.
european servers rather than eurotrib using U.S.
Susan-Thanks so much for this update. I was wondering. Of course, I always think the worst.
The closer we come to the truth . . .oh well. Y’all know what I’m thinking.
But thanks again, Susan.
…oooh-oooo-oooo-ooooo…
Grab tin foil hat (hey! I look just like Tom Terrific…wait! Make that Manfred the Wonder Dog!)!
Wait patiently. Hope I don’t miss Al Rogers…
As sluggish as a server.
We liberal bloggers need a backup plan if for some reason we are not able to get on the blogs or they are down for a long period of time. We should all have a meet up place if there is a natural disaster or Terror attack, why not have a plan B for our blog friends. If they are able to get us progressive/liberal blogs and take us out..we have no way to ever mobilize. Just a thought.
Don’t be like New Orleans. Have a plan to link with each other in case more BS occurs.
There is the DU panic room.
I wonder if they’d let tombstoned radicals in.
In fact, I’ve been wondering about that a lot lately: once the liberals “take back the country” and get everything set up, what’s going to happen to the “far left” (say everyone and anyone to the left of Michael Moore).
What’s going to happen to the Ward Churchills, the Mary Dalys, the Starkravinglunaticradicals of the world (most of whom, interestingly enough, are “pull yourself up from the bootstraps” types who are clearly “infiltrators” to the world of academia and privilege) once the liberals take back the country?
I mean are the liberals who so love to participate in character assassination of radicals like us going to hire us to clean their latrines with our phds after they’ve rendered them meaningless in their efforts to assassinate our character and destroy our professional reputations and stuff? Maybe they’ve got some cyberpadded cells or something they’re going to put us in?
Just wondering.
My question is: would this be a matter of virtual or actual meet-up (which would be regional, obviously) ..?
It’s true, there really doesn’t seem to be a system in place to maintain the connections particular to the medium if the medium’s disabled.
Funny, but it almost seems as if there’s a place for an actual printed newsletter in here somewhere ..
The blogosphere has more impact on the political process and consensus in the last two years than citizen activists and marches have had in the last ten.
You need the instantaneous transmission of information to many thousands of people to have impact. You know the press won’t listen to your in-person meetups.
Information is everything.
Absolutely. So we’re looking at a way to assure this possibility if there’s the type of wide-reaching communication ‘shut down’ we’re invisioning.
Of course, we each may be invisioning something different in terms of coordinated internet disabling & the source of the problem. All I have, personally, to offer is the sense that the system as it exists is terribly vulnerable, considering our medium’s increasing importance to the overall political process.
i agree, and i think it ties neatly into the need for localized blogs. get in contact with your local lefty community, as they’re closer to where you are. as for maintaining the larger community in the event of a crackdown, smaller cliques are more manageable than the community as a whole. unless they take the whole internet down, we’ll probably resurface here and there.
So what do we do and who plans it? I think we need to get on this right away. It could happen before we know it. If we had to rely on local/National/Cable news for our information we would be fucked. Please wu ming consider posting a diary here, MLW and Dkos as to our “Plan B”, and how to mobilize our troops if we are taken out on the net. I am sure Rove has a plan somewhere to have this happen. If it wasn’t for the Progressive/liberal blogs none of us would know who Paul Hackett is and Ohio-2 would have been a local race with probably no National coverage. So many other things would never have been exposed or known to the public. We need a plan now. Thanks.
We do have two means:
bootrib_socialcmte@yahoogroups.com
I am so glad when I get an e-mail from any of you about any site problems — with technical stuff or with someone’s post. It’s a huge help to us.
In this case, several sites had problems today but there were two separate issues:
Our site suffers from this more than other AdvLib sites (such as Crooks & Liars or Juan Cole) because on this site, the ads load first, then the content.
I’ve written to our programmer to see if we can change that around so that, even if the ads are not loading properly, the content will load first.
also seemed to be sluggish early this morning — it affected both BooTrib and MLW. That’s not the first time I’ve noticed Sitemeter take a long time to load…
They really really hate us!
Sorry, Mattes: it’s driving me nuts.
It’s Feuchtwanger (h missing).
“We must assert control over and neutralize the Internets.”
Testing, Testing 1,2,3…er 6,6,6
There was a post the other day about internet addiction —
I am sara seattle and I am addicted to the internet.
Last night when I could not get in to my evening fix of dailykos – I started getting a little panicky – after multiple tries a more and more agitation – I made myself go to bed. Slept fitfully.
Wearing my tinfoilhat and before coffee this morning (horrors for anyone in Seattle but shows the addiction) I rushed to the computer to sign-in and get on to Kos — oh no horrors still no Kos
After much searching and bloodpressure soaring – finally find SusanG’s lifesaving message – still wearing my tinfoilhat however – and just a little calmer – but now really concerned, because like someone said downstream – how easy would it be for someone to shut us all up before an election.
So now I am not only addicted – but also more paranoid (but as they say just because you are paranoid does not mean that they are not after you)
to the other sites. At least, that’s what I do. It’s like switching to whiskey when the vodka runs out.
Imho, it may be more like switching to whiskey when the beer runs out ..
EVERYONE TOGETHER: “HI, SARA!”
(Want some coffee and a donut?)
Do you have a sponsor yet?
(I’m not volunteering … I haven’t quit yet.)
No – not need a sponsor yet – I think
there are guilty pleasures in life – surfing with my lefty buddies is something that – addicted as I am – I am not willing to give up
One day at a time is all we can do, right?
…and keep coming back,
…oh, wait, that won’t help the internet addiction.
Thank you everyone for your support –
I decided that I just had to occupy myself with something else — went on Amazon.com (another terrific Seattle company) and finished most of my Christmas shopping.
So now after spending a gezillian dollars – went back to Kos — and it is baaaaaack!!
My husband is relieved – he was not aware how much money we are saving by me just surfing Kos and Booman
amazon bad! they give money to the (R) party. please don’t shop there!
Aside from my theory that it’s the Dept. of Defense retaliating for Steven D’s story ….
our issue here is the Advertising Liberally advertising.
It’s also slowing down other sites.
I’ve taken them down very temporarily … I just tried to put them up again, and the same problem occurs — where you just see the froggie icon and nothing else.
I’ll keep trying periodically. So, if all of a sudden you can’t get the home page to load, don’t worry. It’s just me and my evil experiments.
The slow-loading everywhere and the ad thing probably IS the virus slowing the internet down.
Daily Kos being down because of its server screw-ups means it’s missing the sluggishness. Wonder how it will respond once it’s up again.
Better yours than ‘theirs’, susanhu 😉
Looks like the server is back online to some degree. Instead of not connecting it now goes to the apache server installation page.
It has now changed to an internal error/configuration error page that does give Kos’s email address indicating that at least something from the old site has been moved over. The good news is this is a sign of progress. The bad news is it is a sign that progress will be very slow. This is an indication that the problems are more severe than machines being moved and reconnected and slightly reconfigured (new IP addresses, etc.). It indicates that at least one server was probably damaged and that Kos is being moved over to a new box. Or that moving to all new boxes was part
of the transition. Moving to a new box is likely
to create all sorts of problems. Different OS and software versions and configurations.
At this point, Kos should be able to ssh into the server and look around.
I get nothing. Still answering pings, though.
Front page partially restored. It looks like diaries and comments (at least mine) are intact. Images, such as the main banner, were not restored being loaded but now are. Loading is very slow so only one server may back online or their are communication issues.
Some images still missing. Also, beware that new content posted might possibly be lost or not propagate to other servers properly if stuff is still being restored from backups.
Thanks for the status report and running the site in the first place.
Since kos has multiple servers, anyway, it might be a good idea to move some of the servers to introduce geographic diversity. This could also provide some
protection against denial of service or censorship attacks. At least one server could be located outside the US. This could also reduce backbone traffic and in some configurations you might be able to negotiate lower rates for bandwidth as a result.
I am not sure how well scoop handles geographically diverse servers or what logistical issues you might have.
Also, Markos, you probably should contact the hosting company using the least disruptive means possible.
Such as contacting a non-technical contact who isn’t directly involved in fixing things.
For all you know, the move may have actually been completed in the expected 2 hours; they may not know it is broken.
The impending outage should have been diaried and front paged. Among other things, knocking down Kos is a possible trick that could have been used to reduce attention to a friday surprise, so people are likely to be a bit concerned not to mention having delirium tremens.
giving directives.
do you have any idea what your great ideas would cost in time and money to investigate and to implement, OR
like the hands of god on the ceiling of the sistine chapel, you wave your fingers and speak and the world jumps to figure out the details?
Yes, I do have a very good idea what kinds of costs may be involved, but it is very dependent on the specifics of how kos is currently set up. I have designed geographically distributed systems that were much larger than Kos.
In some cases, the costs could actually be less due to lower backbone usage and
ISPs willing to donate or subsidize a portion of Kos’s bandwidth but not all of it, etc. Much of the time, physical access to the servers by the person running the site is not an issue. Kos may not be in the same city as the servers as it is, in which case it probably doesn’t make much difference whether some of the servers are even further away. The most significant cost could be just the time negotiating with multiple ISPs. But it depends a lot on how efficient scoop is at minimizing server to server configuration and dealing with the intermittent nature of long distance connections.
The last sentence should have been “server to server _communication_”.
I am quite capable of designing and implementing blogger software that would probablly be significantly be more efficient and scalable than scoop. It would require fewer servers to handle the same load. It would not care if the servers were in the same data center. It would even do the right thing if the servers were temporarily isolated from each other; comments could still be posted to any server but would not propagate to others
until the connection came back up. And it would handle disk failures. But it would be an awful lot of work.
I looked over the scoop administartors documentation, though there are some major gaps in that documentation.
My comments are based on my impressions from the docs
and the possible implementations that would be consistent with what is said in the docs and not based on an understanding of the scoop code.
It appears that scoop probably has scalability problems. Scoop depends on the mysql database and doesn’t appear to have mirroring independent of mysql.
I don’t have a lot of confidence in mysql’s ability to reliably mirror data between two servers in the same data center let alone in different ones. It appears that if most dailykos users stopped being anonymous lurkers, dailykos would
slow to a crawl. I have my doubts about scoops ability to scale up if the number of dailykos users
were to increase dramaticaly. You can probably move various datasets to seperate database servers but if one server wasn’t big enough to handle a given dataset, it might be hard to scale up even if you can afford more servers. And it looks like server to server bandwith requirements might be pretty high.
So, the use of scoop could cause problems for geographic diversity. Better designed systems, though, make geographic diversity pretty easy to put into practice. My original post showed an awareness that diversity might or might not be easy to implement
depending on the current implementation of dailykos and scoop. rmdSeaBos’s trollish and adhominem post, however, did not show the slightest awareness of how easy this can be to do if you have not
made choices (due to limited selection, knowledge, judgement, or external pressures) that limit your future options. If he had, the criticism would have been directed at scoop and not at me.
But then, he said elsewhere he had worked at Microsoft, so his experience with well designed software may have been limited. But good developers, if they are given
the resources to do so, understand the distributed nature of the internet and develop internet applications that are highly scalable.
I think Marcus has secretly changed his name, snuck out of town and has set up a new secret blog at an undisclosed underground location. His new blog is called SodK
BooMan exhorted us not to “go there” re the orange place.
In the spirit of ALL THAT IS BOOMAN — who, when I just called him, was moving a dresser (with one hand, he claimed) — let us not go there.
That is all.
Should I not have posted this here? Is Booman boycotting Daily Kos?
Or is he saying not to “go there” in terms of starting Big Orange bashing?
I apologize if I’m missing something and caused a problem. I may have missed the last few chapters in the lefty blogosphere smash-a-thon.
NO! I’m so glad you posted this because it was really a puzzle … my comment was just re the bashing.
Thanks, SusanG! You were a huge help to me because we were also having a site problems, and I was trying to figure out why. That DKos was down, and I didn’t know why, was worrying me. You solved that concern.
Susan..it is really amazing that when Dkos goes down it affects alot of the other Progressive Blogs. I guess it might be a traffic problem as the Kossacks look to other liberal blogs. I still think we need a back up if we are targeted and have no place to go to reach each other, especially during a time that it is urgent to get in touch. Pre-election, pre-voting on SCOTUS, stuff like that. I would not put anything past BushCo and Rove as to knocking us out. We cannot count on the Cable/MSM for anything. Maybe only KO on MSNBC
No flaming….some of us thought we had been banned…that’s all.
and doing email server (YECK YECK YECK) support then Sql Server support (kind of DBA stuff for a lot of sites)
whenever stuff would go haywire, which would happen a lot cuz we used the corporate net to try new stuff before they shipped it
the BIGGEST waste of time would be dealing with all the assholes:
WHAT IS WRONG !!
– uh, I am trying to figure it out, but I am talking to you on the phone.
HOW ARE YOU GOING TO FIX IT !!!???
– uh, I don’t know what is wrong yet, so I don’t know what to fix.
WHEN ARE YOU GOING TO FIX IT !!??
– when I get off the phone with you and I can start figuring out what is wrong.
Finally –
ALL fix it support groups SHOULD follow the rules for communications (wish the assholes at airports did it.)
+++++++++++++
scenario 1 – we are pretty sure what the problem is AND pretty sure of how long to fix it.
+++++++++++++
Estimated Time of Resolution (ETR):
14:00
+++++++++++++
scenario 2 – don’t know what the problem(s) is / are, therefore don’t know how long to resolution.
+++++++++++++
Investigating / Troubleshooting.
No Known Estimated Time of Resolution (ETR):
Next Update:
14:00
opps,
I forgot the little detail that when your web site is down it is kind of tough to put up a message that the web site is down …
I got in BIG trouble cuz when the email servers went down there would be people who would send email to tell their customers that there were problems with email
I suggested that people who were that dumb shouldn’t be employed, and I got in trouble.
oh well – I couldn’t count the times I or we did something that was supposed to take 15 to 30 minutes, and … 6 or 16 or 26 hours later …
I love those two posts. Back starting in 1993, I worked for Internet start-ups, including ISPs … oh, the problems we had back then. It was just horrible.
We had a weird UNIX set-up at one ISP, and it’d take hours to get someone connected sometimes. Then they’d call back and say they couldn’t connect. Turns out they’d typed in their own phone number instead our the ISP’s phone number … stuff like that. We had some good laughs along the way.
the email group had a lot of networking / unix geeks – holy shit.
their favorite statement was “what is your question, I can’t help you if I don’t know what your question is…”
they thought it was intelligent to know all the bizzare-o commands and switches and gotchas …, therefore those who didn’t know all the bizzare-o crap where stupid.
was it nature or nurture so that they were incapable of being helpful? I know the unix geek culture is to be obtuse, BUT
no wonder the rest of any company or organization hates the IT people !!
btw, I did have a lot of laughs, there were a lot of cool things happening, and the best / only 2 things I got ( I missed the yacht, champaigne and caviar) were that I learned I do hate the client side of computers, and the only thing I like about computers is database / SQL stuff.
I’d say “knowledge management” but I don’t want to sound like the kool aid drinking kahki wearers 😉
rmm.
I was able to get the Diary page downloaded just now, and then navigate to several of the most recently posted diaries. But the front page wouldn’t load.
For example these would load.
I got the apache notice page on my firefox browser. AOL browser was able to bring up kos but with no updates later than Friday. Didn’t try non-aol MIE.
gretel
Where’s the mojo on this thread?
This is Bootrib. Ya gotta show some lovin ;o)
Gotta pocketfull of 4’s and I’m heading upthread ;o)
Helloooo o o o o o o o
</crickets>
dkos must be back up ;o)
dKos is alive and well and well, as supersoling said, where’s the love?
Something is not making sense. As I said in earlier diary, I am on a public website, using IE (believe me, I wish fx was installed here). I dunno…