Current mood:

Needy-like

Virtual Snapshot: Portrait of a Cyber Generation

The day the fanwank died.

I like to think that I have a “retro” sensibility. As I type this, the sounds of Simon and Garfunkel’s not so silence is playing from my turntable. Classic Chucks will always be my footwear of choice. Old time radio suspense thrillers are my idea of a good time.

Despite these things, I’m abhorrently addicted to the internet as much, of not more so, than the next kid on the block. I have numerous email accounts for different purposes, all of which I check daily and a few that I check no less than every couple of hours. At the forefront of this techno love is my dependency on Live Journal. I need to post my random thoughts when they come to me, I must know that all my friends are up to, and I can’t be behind on the latest bit of fan wank that’s occurring within my beloved fandom.

Right now, Live Journal’s suffering from a massive power outage and loading any page housed on the LJ servers yields the following message:

“Our data center (Internap) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We're currently dealing with bringing our 100 servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.”

So, what’s the first thing that I do? I create an email meme to find out what everyone’s doing in lieu of posting “Which stereotypical television buddy are you?” Quizilla results.

I’m also checking CNN.com, MSNBC.com, and Msn.com for any reports on, what I consider to be a fairly substantial event in the lives of a decent armload of internet users. This may sound a bit unreasonable at first glance. So what? My online journal’s down. Who cares? But those who frequent LJ.com at the consistency that so many do, are intensely immersed in a sense of community that has grown out of blog based websites. Imagine attempting a visit to your favorite twenty-four hour diner or coffee house, only to find the doors locked and no one inside. [LJ user hecatehatesthat brought up a much more valid allusion. “It's not like your favorite coffeehouse being suddenly closed. It’s like EVERY Starbucks is shut down, and all the bookstores and malls and city blocks that have Starbucks' in them being shut down.” It’s true. The magnitude of this incident is not just akin to the local Mom and Pop hang-out. It’s equivalent to chain store capitalism and our dependency therein.]

The internet has become a place of camaraderie and fellowship where the like minded from every corner of the globe can link up and, ideally, not feel so alone. Unlike chat rooms or instant messaging, the blog format takes us back a couple of decades and utilizes a standard bulletin board system that gives the users their own personal space to post their thoughts and allows others to come along and voice their feelings at any given time. It’s easy, it’s laid back, and it’s a gateway to obsession.

Selections from Googlism.com

livejournal is brought to you by the letter j
livejournal is not just an online journal; it's an interactive community
livejournal is an online journal site with a number of extra features
livejournal is a service that essentially lets me easily post quick updates and/or rants
livejournal is just me posting about what i ate when i feel like it
livejournal is in an unusual situation for a piece of software or a web service
livejournal is an alcoholic
livejournal is totally sweet on or something
livejournal is awesome
livejournal is extremely easy to use
livejournal is updated more than anything
livejournal is at hand
livejournal is a site that allows people to make a journal for themselves online
livejournal is x
livejournal is a free/"voluntary subscription" web log
livejournal is just easier to use
livejournal is open source and there seems to be a lot of voluntary effort going at least into developing client's for multiple platforms
livejournal is a complete hassle
livejournal is so well liked
livejournal is becoming more and more of a blog lately and it's describing less and less of my life
livejournal is a good idea
livejournal is the popular table at school; everyone on it is hanging around
livejournal is perfect for you if you just want to get a feel of blogging
livejournal is more my style
livejournal is kinda annoying
livejournal is?
livejournal is a pretty interesting phenomenom
livejournal is considered a good place for emo lovers
livejournal is something different today through much innovation
livejournal is overloaded and they're getting more servers even as we speak
livejournal is for my bitchy asides and nasty commentary
livejournal is not that great
livejournal is down

Yes, there are other journal websites. Yes, they’re just as functional. Yes, a lot of us have accounts with several of these sites. But they’re not LiveJournal. Again, let’s go back to your favorite hangout. Is it the same to just kick back somewhere else? Does the coffee taste as good? Is the music just a tad too loud? Does the staff know you well enough to just let you sit where you want and be your regular wacky self? Of course not.

It’s never the same somewhere else. For some of us, anyway.

My email meme has yielded results reporting that some people are, in fact, seeking refuge in alternate journaling sites. Others are simply trying to distract themselves and finding it difficult, as their internet start up page is their LJ. Currently, I’m chatting, real time on AIM and IRC. But what about my need to post my random thoughts about today’s lunch or the fact that my hands feel unusually dry? Where’s my outlet for all the mayhem in my brain? How am I expected to function if I can’t, at the push of a button, tell a hundred people the stupid chicken joke I heard from my brother?

And let’s keep in mind, that currently, this essay is housed on JournalFen.net, which is, proudly, NOT LiveJournal. It’s an account that I maintain, strictly to keep tabs on what might be the most entertaining community on the web, fandom_wank. It’s that place back behind the bleachers, where all the cool kids hang out, slug back shots of Jack straight out of the bottle, and make out with each other. It’s damn fun. But it’s not home. [Since my initial post of this essay, JF.net has also gone down. It has now been moved to my blogger.com account. If this one crashes next, I’m leaving to stock canned goods and chocolate. Oh wait... JF's back up. Whew.]

I miss you LiveJournal. Wherever you are.

ETA:

Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.

Update #2, 10:11 pm: So far so good. Things are checking out, but we're being paranoid. A few annoying issues, but nothing that's not fixable. We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn't a total freak coincidence. C'est la vie.

Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.

Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.

Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.