AT the last it shop i worked at they had a generator that ha enough diesel to power it for 48 hour straight. That's the entire building, parking deck arms, security system and computers. They could order a truck and have it on site to refuel the generator in 10 minutes.
The last two I worked for had very similar setups.
I've seen a broadcast storm take out the core switches (several pairs of 6513s) in a 4 acre data center and shut the place down for several hours while everything was put back online. The broadcast storm took out the primary switches, systems rolled to the alternates and they failed immediately after. That happened in a tier 4 data center owned by a major corporation, and I'm sure Turbine doesn't run/can't afford a tier 4 data center.
Sometimes, no matter how hard you plan, unless you have an alternate data center somewhere, bad stuff is going to happen. It doesn't necessarily have to be a "SPOF." Sometimes, you get a cascading failure that rolls through the data center impacting things in turn and you're stuck until you can resolve the root cause.
Still, these occurrences are rare. I can think of a huge number of things that can shut a place down, the earlier was just an example, but other things off the top of my head, corrupt DNS from a provider, fire suppression system activation in the data center. Some idiot pushing the EPO, all sorts of weirdness can happen.
I'm a lotro player who's account got hacked early Saturday morning and I know of several other players that got hacked Friday night as well. The hackers were smart enough to wait until after the account services department had closed to start. After talking to a few friends who were online when this all went down, including one friend who got booted offline when they logged into his account, I realized the same thing happened to me. They logged into the turbine account and changed the password and the email address on the account. Then logged into each toon on the account. My friends that noticed our buddy logging in and out of each toon and running between the vault and mailbox reported it and got the account banned to save whatever was left on his account, but apparently what they have been doing is logging into an account and selling everything you have and then mailing off all the money. None of us have been able to get back on our accounts because they have been banned until we call on Monday 12 noon EST with our billing CC etc. My guess is the problem isn't localized to a few people I know because it has been a growing problem since free to play started. They were stupid enough to make the forum login the same as your account when free to play went live. There is no lock out on failed login attempts on the site so all a hacker has to do is grab your account name and start running a password script to generate the password. My password was complex including both capital and lower case letters and numbers. Now I'm wishing I had included a random character or two, but too late now.
I'm a lotro player who's account got hacked early Saturday morning and I know of several other players that got hacked Friday night as well. The hackers were smart enough to wait until after the account services department had closed to start. After talking to a few friends who were online when this all went down, including one friend who got booted offline when they logged into his account, I realized the same thing happened to me. They logged into the turbine account and changed the password and the email address on the account. Then logged into each toon on the account. My friends that noticed our buddy logging in and out of each toon and running between the vault and mailbox reported it and got the account banned to save whatever was left on his account, but apparently what they have been doing is logging into an account and selling everything you have and then mailing off all the money. None of us have been able to get back on our accounts because they have been banned until we call on Monday 12 noon EST with our billing CC etc. My guess is the problem isn't localized to a few people I know because it has been a growing problem since free to play started. They were stupid enough to make the forum login the same as your account when free to play went live. There is no lock out on failed login attempts on the site so all a hacker has to do is grab your account name and start running a password script to generate the password. My password was complex including both capital and lower case letters and numbers. Now I'm wishing I had included a random character or two, but too late now.
That's what you get from a PoS company that only supports up to 16-character passwords. Also sad they (and others) haven't taken a queue from Blizzard...HELLO! AUTHENTICATORS!
Our greatest glory is not in never falling but in rising everytime we fall.
And like Blizzard is never having problems with hacked accounts.....right.
I have seen posts by people who claimed they are being hacked even with authenticators tho i have no idea how now that blizz has the history system in the acounts and the callback feature that calls your number to varify if you changed your data recently. The history part is there to see what you're previous info was so they can call the right number i suppose.
In LOTROS case, no news is good news. It actually works so well I dont feel the need to go on the forums and gripe about it or stay on the lookout for updates. Its a pretty solid MMO.
Comments
The last two I worked for had very similar setups.
I've seen a broadcast storm take out the core switches (several pairs of 6513s) in a 4 acre data center and shut the place down for several hours while everything was put back online. The broadcast storm took out the primary switches, systems rolled to the alternates and they failed immediately after. That happened in a tier 4 data center owned by a major corporation, and I'm sure Turbine doesn't run/can't afford a tier 4 data center.
Sometimes, no matter how hard you plan, unless you have an alternate data center somewhere, bad stuff is going to happen. It doesn't necessarily have to be a "SPOF." Sometimes, you get a cascading failure that rolls through the data center impacting things in turn and you're stuck until you can resolve the root cause.
Still, these occurrences are rare. I can think of a huge number of things that can shut a place down, the earlier was just an example, but other things off the top of my head, corrupt DNS from a provider, fire suppression system activation in the data center. Some idiot pushing the EPO, all sorts of weirdness can happen.
I'm a lotro player who's account got hacked early Saturday morning and I know of several other players that got hacked Friday night as well. The hackers were smart enough to wait until after the account services department had closed to start. After talking to a few friends who were online when this all went down, including one friend who got booted offline when they logged into his account, I realized the same thing happened to me. They logged into the turbine account and changed the password and the email address on the account. Then logged into each toon on the account. My friends that noticed our buddy logging in and out of each toon and running between the vault and mailbox reported it and got the account banned to save whatever was left on his account, but apparently what they have been doing is logging into an account and selling everything you have and then mailing off all the money. None of us have been able to get back on our accounts because they have been banned until we call on Monday 12 noon EST with our billing CC etc. My guess is the problem isn't localized to a few people I know because it has been a growing problem since free to play started. They were stupid enough to make the forum login the same as your account when free to play went live. There is no lock out on failed login attempts on the site so all a hacker has to do is grab your account name and start running a password script to generate the password. My password was complex including both capital and lower case letters and numbers. Now I'm wishing I had included a random character or two, but too late now.
Website is back up.
Let's party like it is 1863!
That's what you get from a PoS company that only supports up to 16-character passwords. Also sad they (and others) haven't taken a queue from Blizzard...HELLO! AUTHENTICATORS!
Our greatest glory is not in never falling but in rising everytime we fall.
And like Blizzard is never having problems with hacked accounts.....right.
Let's party like it is 1863!
I have seen posts by people who claimed they are being hacked even with authenticators tho i have no idea how now that blizz has the history system in the acounts and the callback feature that calls your number to varify if you changed your data recently. The history part is there to see what you're previous info was so they can call the right number i suppose.
In LOTROS case, no news is good news. It actually works so well I dont feel the need to go on the forums and gripe about it or stay on the lookout for updates. Its a pretty solid MMO.