|
Post by Cyrus on Jan 15, 2005 23:32:03 GMT
It's making me very twitchy.... I need my fix Me too. Wanting to post too!
|
|
|
Post by Cyrus on Jan 16, 2005 1:29:07 GMT
LIVEJOURNAL IS BACK ONLINE!And my journal looks to be just fine. edit1: But it's incredibly slow at the moment. edit2: Update #6: We're up again, but only partially. Some database clusters are still reconstructing/syncing. See status.livejournal.com.
|
|
|
Post by PokerKitten on Jan 16, 2005 15:07:28 GMT
It's kinda sad and scary how everyone was so freaked by the disruption. How did we cope before it was invented? Lol. So glad it's back
|
|
|
Post by girlmacbeth on Jan 16, 2005 17:31:21 GMT
Just got blogged it is mylifeandjames.blogspot.com i'm also thinkig of getting my lj back if I do I'll post addy please visit and dumd ? but and mods if not apporpriate please delet please someone help understand the friend function on lj please pm me with info please it's for lj i'm meaning. thanks again.
|
|
|
Post by PokerKitten on Jan 16, 2005 17:44:57 GMT
Yeah, come on to LJ and get addicted, too! ;D You can "friend" anyone whose journal you might enjoy reading on a regular basis and/or who you already know via forums or whatever and want to see what they have to say for themselves on LJ. It means you can access their journals from your own LJ without having to type in their individual urls each time you fancy reading. To "friend" someone you canstart off at their LJ, click on their User Info, and then on that page you'll see a little head and shoulders icon (next to the pencil etc). Just click on that. BTW, the url to your blog doesn't work for me
|
|
|
Post by girlmacbeth on Jan 16, 2005 17:56:02 GMT
will fix problem on blog i'll get right addy sorry and thanks for the help.
|
|
|
Post by PokerKitten on Jan 16, 2005 18:30:07 GMT
Maybe you just didn't make it public, pet.
|
|
|
Post by Cyrus on Jan 17, 2005 20:55:09 GMT
Yes, and look at what the withdrawl made me do... I went and messed with my journal layout again.
|
|
|
Post by Cyrus on Jan 18, 2005 0:48:05 GMT
Attention paid LJ users:bradfitz, posting in paidmembers: A full explanation to all LiveJournal users of Internap's massive power loss last Friday is still forthcoming (we haven't even been given the explanation from Intenap yet), but in the meantime, we wanted to compensate our paid users for the downtime.
Go here to collect two weeks of additional paid time:
www.livejournal.com/misc/claim-2005-01.bml
If you have any questions or problems, file a support request, because comments here might be missed.Cool. All you've gotta do is click on the link, then click the button that says you want it, and *bang* it shows you your new expiration date! ;D
|
|
|
Post by Cyrus on Jan 21, 2005 8:40:08 GMT
Reason for LJ's power loss: bradfitz, posting in lj_dev: The post you've all been waiting for! Why we lost all our power, and why it took us so long to come back up afterwards....
(Warning: it's late and I'm tired/rambly, so this post might be incoherent... go ahead and ask questions...)
Why we lost all our power... Another customer in the facility accidentally pressed the EPO button, then depressed it, replaced the protective case, and left the building. Intenap all thought it was their UPS systems failing, but then logged into them, saw EPO shutdown notifications, and couldn't find any EPO cases open or pressed, so probably freaked out for a bit thinking there was a short in the walls that triggered the EPO, only to get a confession a day or so later.
EPO, by the way, stands for Emergency Power Off and it's a national fire/electrical requirement for firefighters to be able to press these big red buttons near all exits that turn off all power in the entire data center. This is the second time this has happened to us in the years I've been there. The first time the button was unlabeled and unprotected and some dude thought it opened the door. This time we have no clue why it was pressed... maybe that dude tripped and fell onto it... mystery.
Internap will be putting alarms and tamper-proof indicators on the plastic cages surrounding the EPO buttons now, though, so at least if this happens again in the future they'll know why.
Anyway, moving on....
Why it took us so long to come back up... Ton of reasons:
Faulty mobos/NICs: We have 9 machines with faulty motherboards with embedded NICs that don't do auto-negotiation properly. They only work with certain switches, so they reboot fine, but then their gigabit network comes up at 100 half duplex or something that doesn't work. To get them back up they need somebody at the NOC to plug them into a compatible switch, let them autonego, then switch them to their real switch. Setting the speed/duplex settings on both the host and/or switch themselves doesn't work.... most annoying. We're getting working dual-port gigabit PCI NICs for those machines, rather than replacing their motherboards.
Database start-up: All but a couple of our machines came back up when the power was restored, less than an hour later, but on our databases we intentionally don't have the database start back up on boot. In normal circumstances, if a single machine dies, it died for a reason and we want to investigate it. Normally that doesn't matter either, because we have 2+ of everything. But when every single database restarts, that leaves us with no alive databases, and we have to manually start them all.
Data validation: We could've just blindly started all the databases and trusted they worked, but we didn't trust them. We ran innodb tablespace checksum checks on everything, and also did backups of a lot of the databases before we even tried to bring them back up. (the act of bringing them back up modifies the tablespace, and we didn't want them messing themselves up worse, so a pre-backup was for paranoia...)
MyISAM vs InnoDB: When you lose power to a MySQL db w/ MyISAM tables, the indexes are generally messed and you need to rebuild. Fortunately almost all our databases are purely InnoDB nowadays, so that wasn't a huge problem. Unfortunately, though, the global DB (which is required even to get the site up in partial mode where some users are available and others aren't) is still like 5% myisam and we just hadn't got around to converting those few remaining tables to innodb yet. So every machine in the global cluster required index rebuilds and data checks. That was annoying. The Chef user cluster was also MyISAM (our last MyISAM user cluster), so rather than trust Chef, we restored from an old Chef backup and replayed binlogs to catch it up. That took some time.
Disk cache issues: We have battery-backed RAID cards with write-back caches. That means the RAID card immediately acknowledges writes and tells the OS (and thus DB) that they're done immediately, before they're on disk. This speeds up the DB. But if you lose power, those writes would normally be lost, which is why we have battery-backups on all the cards, and we even monitor the battery health w/ our automated checks. But unknown to us, the raid cards didn't disable the write caching on the drives themselves.... which is frickin' useless! If the controller is already lying to the OS (but doing it safely!) why should the disks behind the controller also lie, but unsafely, for minimal benefit? Our bad there. We should've had that right. So a couple machines were just gibberish afterwards and had to be restored from backup and had their binlogs replayed to catch the backups up to present.
Binlog syncing: We weren't using the option to sync binlogs to disk, so we lost a small number of transactions right before the power loss in the case of clusters that we had to restore from backup. Regrettably, we won't be able to get those posts or comments back.
Slaves tuned for speed: A lot of slave servers (mostly in the global cluster, since the user clusters are almost all master-master now) were tuned for speed, with unsafe filesystem/sync options that favored speed over reliability. Which is normally okay, since you'd lose one machine, not all of them. But we lost all of them, so restoring them all from good slaves was time-consuming.
Things we're doing to avoid this crap in the future... We're:
-- getting working NICs in those 9 machines
-- all our DBs have redundant power supplies. we'll be plugging one side into Internap's, and the other side into our own UPS, which itself is plugged into Internap's other power grid. that way if EPO is pressed, we'll have 1-4 minutes to do a clean shutdown. (but if we do the rest of the stuff right, this step isn't even required, including having UPSes... in theory... but the UPSes would be comforting)
-- disable disk caching behind all our RAIDs. (bleh... wanna kick ourselves for this, but also the raid vendors for even defaulting and/or allowing it in a BBU write-back setup) but also testing all existing and new hardware to make sure data makes it to disk and pulling power in the middle of write-heavy operations, then verifying the resulting disk image later with the expected result.
-- finish our MyISAM to InnoDB migration, so we don't have to deal with MyISAM index rebuilds
-- enabling binlog sync options
-- stop tuning slaves for speed. this used to matter, but we don't really do the slave thing as much as we used to, so the gain isn't worth it.
-- user level backup harness. we already have a tool to backup a single user's to a GDBM file, incrementally. (so if we run it a day later on the same file, it only updates the changes) so we plan to wrap a major harness around that tool and be backup up all users, all the time. this means that in the event of a future major corruption/outage, we'll be able to restore user-at-a-time, and even to a different database cluster than the one they came from. this also means we can prioritize recovery based on account status, popularity, attempted activity, etc. (and yes, we'll continue doing system-level backups as well, but it's good to have data in different formats just to be paranoid...)
-- also, we already bought a bunch more disk space that we installed today, so we have more room to do backups for bizarre non-typical reasons, and don't need to compress/shuffle so much stuff to make room sometimes.
|
|
|
Post by Pixie on Mar 10, 2005 13:28:14 GMT
In addition to my normal LJ, I also now have a blog to inform my readers (if I have any!) about updates to my fanfic site. If you're interested, it's at slayergirl-fanfic.tripod.com/blog
|
|
Lily
Billy Johnson
Transylvanian Concubine
Posts: 170
|
Post by Lily on Jun 30, 2005 12:29:01 GMT
I got bored and made myself a blog at msn spaces- the addy is www.spaces.msn.com/bluejean82It's not much yet- just a few mindless rambles about my probably-not-very-interesting life and one naff picture, but hopefully once I'm online at home that will change!
|
|
|
Post by Cyrus on Aug 16, 2005 1:38:57 GMT
Not sure who on here has a paid LJ account, but they've got a promo going to promote their new "automatic payment" system. paidmembersTo coincide with the launch of automatic payments announced in the latest news post, we are providing an exclusive offer for existing paid account holders (as of 8/14/05). Get three months of bonus paid time when you switch your account to automatic payments and select the 12-month billing cycle. This offer ends October 15, 2005. Read the fine print.
By the way, our auto-magical new system will take into account your remaining paid time and prorate your charges, so if you have six months of remaining paid time and you sign up for automatic payments on a 12-month billing cycle, then you’ll have a total of 15 months of paid time and only be charged $9.98 (half of $19.95).
|
|
|
Post by PokerKitten on Aug 16, 2005 9:13:17 GMT
And don't forget, paid accounts can also buy extra avi space for a teeny tiny fee... and they have just doubled out allowance, woot! 100 avis! ;D
|
|
|
Post by PokerKitten on Oct 7, 2005 0:33:41 GMT
"My LJ" now available! Paid and Permanent accounts now have access to the beta version of an upcoming feature called My LJ (http://www.livejournal.com/portal/). We will open this up to free users once we are done monitoring server performance.
My LJ lets you personalize the way you interact with your LiveJournal. We will be adding more modules over time, but the beta list includes:
Quick Update Account Status LiveJournal Announcements Friends' Birthdays Frank Friends Account Management New Users Recent Comments Send a Text Message Popular With Your Friends
Please check it out and leave your feedback in comments to this post. Your feedback is much appreciated and will help us further refine this feature. Enjoy!
|
|
|
Post by Cyrus on Oct 7, 2005 0:37:39 GMT
Yeah I played with it a tiny bit when it was still in test. But since it was test, I could only play with my account on the test server and not use it for real.
|
|
|
Post by Cyrus on Dec 16, 2005 17:07:32 GMT
For LJ users:
Happy Holidays from LJ 2005-12-15
'Tis the season for giving to those who are important to you, so LiveJournal has some gifts for you. We know you experienced a few glitches during the data center move, but now you can reap the benefits.
More For You! To: Free Account Holders From: LiveJournal We're doubling your userpics! You can now upload up to 6 userpics.
To: Paid Account Holders From: LiveJournal We've increased your storage space from 100MB to 1GB! That's ten times the storage space you had before. Use it to upload your holiday photos to ScrapBook.
To: Permanent Account Holders From: LiveJournal We've increased your storage space to 10GB!
These gifts are permanent -- they're not just for the holiday! We hope you enjoy these increased levels!
Give More! If this puts you in a giving mood, we've got great news for you.
Starting today, when you buy someone a gift of paid time, you'll automatically give them the userpic add-on for free! That means when you buy a gift of 12 months paid time, the recipient also gets upgraded to 100 userpics for those 12 months. If you buy them a 2 month paid account, they get 100 userpics for 2 months, and so on.
The Give More offer only runs for a limited time -- from today until Wednesday December 21st at 12noon PST (-0800 GMT). So head to the Gift Shop and Give More!
|
|
|
Post by PokerKitten on May 25, 2007 23:28:39 GMT
The LJ problems this week - they say Thursday but I know people who've had problems most of the week, and one or two still having them.
Since approximately 3am PDT Thursday morning, we've been handling a particularly large amount of traffic. We believe it's the result of a distributed denial of service (DDOS) and we've taken a number of measures to mitigate the impact in order to bring LiveJournal back to full functionality.
During this time, we received some reports from users who were unable to post entries or comments, or in some rare cases, being unable to access LiveJournal entirely. We regret that this had happened -- we try to minimize the amount of legitimate traffic affected -- but it was an unfortunate consequence of making sure that the site was stable for the majority of users.
At this time, all services are back to normal. We continue to monitor LiveJournal's performance and will address any problems as soon as they arise. Apologies for the inconvenience caused by this problem, and thank you for your patience as we worked to resolve it.
|
|
|
Post by Ditto on Jun 2, 2007 21:22:00 GMT
You get tons of icons at GJ...I think we should all emigrate!
|
|
|
Post by PokerKitten on Jun 2, 2007 22:43:51 GMT
GJ is deader than a dodo, and I already have masses of user pics.
|
|