PDA

View Full Version : ??


Crash
01-24-2008, 09:05 PM
Where'd we go?

Kipper
01-24-2008, 09:16 PM
we've been having tons of fun Steve, check this out from this morning:


just shoot me. just got off the phone with easycgi for a third time since they are all but ignoring the ticketing system and giving me 48 hour turn around times, only to tell me the same thing they told me the day before. is 9am too early to start drinking?
so here's what's gone down:
1. wes calls me on the way home from a job interview (long story, laid off last week), they shut us down for too much resource usage, i need a drink, a tall drink
2. i reviewed the logs on the server and found that it was was due to a denial of service attack and update the support ticket with all of the information needed to block the domain name that had instigated the attack (the dos attacks were coming from a domain, not ip's)
3. I called tech support after they ignored my response to their help ticket, telling them all of the same things over they phone. at that time they said they would block the ip addresses for me and turn on the site.
4. last night, when i saw that the site was STILL down, i logged into the control panel and saw that they now wanted the IP of the domain
5. I lost any grip i still had on sanity. it takes one frickin' second to ping a domain name to get an IP?!?! where's the Makers Mark?!
6. I logged in and posted the IP instead of the domain name of the server to the control panel. anything else i can do for you guys, how about i wash your car or mow your lawn while you figure out your firewall system?
7. this morning i login and see that, whadday know, the site is STILL down, so i logged into the control panel again and see that now they don't want to block the IP with their super (sh!tty) firewall, they want me to write a script to do it manually.
8. I am now boarderline insane/unibomber
9. I write a quick script to block given IP's blockip.php and load it on the server. I also found a cool load management tool for vbulletin, but can't install it until the site is up.
10. Call my ol' pal Mike at tech support again and let him know what's up. he tells me there is a guy working on this right now, as we speak and promises to call me here at the house if he runs into anymore issues instead of sending me more cryptic messages by donkey and to let me know when the site is back up.
11. It's now 10:10am, no word yet, fingers crossed...... gotta go get ready for another interview.


...never heard a word from those guys all day, tried calling all evening got the recorder. talked to wes, so he started trying to and finally got through to them and they turned on the site. we're installing all kinds of crazy scripts now, so let us know if you see anything funky.

60wag
01-24-2008, 09:17 PM
I'm just glad to be back.

A big thanks to Wes, Kipper, Matt and anyone else working to kept the site up.

Chris
01-24-2008, 09:18 PM
Good to see everything back, thanks to all involved!

Groucho
01-24-2008, 09:21 PM
Nice job, Kipper, Wes Matt and the like.:thumb:

timmbuck2
01-24-2008, 09:25 PM
thanks to all!! how soon until we find a new web host? ;)

T

Red_Chili
01-24-2008, 09:35 PM
Wow, I never realized how addicted I am to this site...

corsair23
01-24-2008, 10:06 PM
Wow, I never realized how addicted I am to this site...

x2 :thumb:

Thanks to all for your hard work is getting us back online...Boy it was a VERY tough few days so I can only imagine how bad it was for those trying to get everything put back in order. :cheers:

Convert
01-24-2008, 10:08 PM
Thanks for all the hard work guys :thumb: Sure glad the site is back :beer::thumb:

Romer
01-24-2008, 10:11 PM
Excellent work Kip and Wes. Some beverages in your honor are due.

bskey
01-24-2008, 10:40 PM
Wow..I was concerned that all the FJ's in the world had transformed into pirates after lying dormant for years and attacked their owners, leaving our fearless leaders trapped and without a ride.



......glad to hear it was just a host issue. Thanks for fixing it!!

Romer
01-24-2008, 10:56 PM
A better story would have been we all went to the Lone Star Jamborie. Not what happened but would have been a better story.

wesintl
01-24-2008, 10:59 PM
I bought a jeep

Uncle Ben
01-24-2008, 11:01 PM
I find it interesting how fast the guests/lurkers popped back on....guess they were without their RS fix too! ;)

Mendocino
01-24-2008, 11:18 PM
I'm glad to have the forum back. Its a great :beer:resource.

Cheeseman
01-25-2008, 12:00 AM
When you have the best in the business. Things happen. Great job guys.
________
****ube (http://www.****tube.com/)

Shark Bait
01-25-2008, 12:03 AM
Wow, I never realized how addicted I am to this site...

Yeah. No kidding. I had to over to MUD once in a while for a fix. :eek:

Hulk
01-25-2008, 12:24 AM
I was checking this site about every 2 hours for the last 3 days. Man, am I glad it's back online. Methadone just don't cut it.

Kipper and Wes, the :beer: is on us!

nuclearlemon
01-25-2008, 12:30 AM
congrats to our resident it guys!!! :cheers:

RicardoJM
01-25-2008, 07:46 AM
Glad to see everything is back. Good job to those that made it happen.

subzali
01-25-2008, 08:18 AM
You guys are awesome!

Jacket
01-25-2008, 09:04 AM
we've been having tons of fun Steve, check this out from this morning:
Originally Posted by Kipper
just shoot me. just got off the phone with easycgi for a third time since they are all but ignoring the ticketing system and giving me 48 hour turn around times, only to tell me the same thing they told me the day before. is 9am too early to start drinking?
so here's what's gone down:
1. wes calls me on the way home from a job interview (long story, laid off last week), they shut us down for too much resource usage, i need a drink, a tall drink
2. i reviewed the logs on the server and found that it was was due to a denial of service attack and update the support ticket with all of the information needed to block the domain name that had instigated the attack (the dos attacks were coming from a domain, not ip's)
3. I called tech support after they ignored my response to their help ticket, telling them all of the same things over they phone. at that time they said they would block the ip addresses for me and turn on the site.
4. last night, when i saw that the site was STILL down, i logged into the control panel and saw that they now wanted the IP of the domain
5. I lost any grip i still had on sanity. it takes one frickin' second to ping a domain name to get an IP?!?! where's the Makers Mark?!
6. I logged in and posted the IP instead of the domain name of the server to the control panel. anything else i can do for you guys, how about i wash your car or mow your lawn while you figure out your firewall system?
7. this morning i login and see that, whadday know, the site is STILL down, so i logged into the control panel again and see that now they don't want to block the IP with their super (sh!tty) firewall, they want me to write a script to do it manually.
8. I am now boarderline insane/unibomber
9. I write a quick script to block given IP's blockip.php and load it on the server. I also found a cool load management tool for vbulletin, but can't install it until the site is up.
10. Call my ol' pal Mike at tech support again and let him know what's up. he tells me there is a guy working on this right now, as we speak and promises to call me here at the house if he runs into anymore issues instead of sending me more cryptic messages by donkey and to let me know when the site is back up.
11. It's now 10:10am, no word yet, fingers crossed...... gotta go get ready for another interview.

...never heard a word from those guys all day, tried calling all evening got the recorder. talked to wes, so he started trying to and finally got through to them and they turned on the site. we're installing all kinds of crazy scripts now, so let us know if you see anything funky.

Man - that's a load of crap. Doesn't matter if your in IT support or selling car parts - There's just no excuse for bad customer service (especially when you are paying for it)!

Thanks to all that worked hard to get the site back on line.

Crash
01-25-2008, 09:34 AM
Kipper,
Okay, I've reread your detailed explanation and it kinda makes sense, but what was really going on? Was there malicious intent, the attack, by a person or persons against our the club website or was it just sporadic computer weirdness? I understand the part about our "host" being most incompetent but would like to know more about the root cause of the forum being down.
Thanks, Kipper. Now we know who the "man behind the curtain" is. Best of luck to you on your job search. Good things happen to good people. Thanks to Wes too!

Red_Chili
01-25-2008, 09:59 AM
It was malicious.

RockRunner
01-25-2008, 10:21 AM
Sounds like our "Tech Support.............Oh I mean Strike Team"

We owe all you guys a Big Tall Drink, thanks a bunch.

I did realize something these last couple of days...........I AM ADDICTED TO RS FORUM!!!!!!! There I said it and I feel great!

Kipper
01-25-2008, 10:27 AM
yeah steve, bill's right it was malicious. there are people out there that really have a good time trying to take down every vBulletin based bulletin board by using an automated process to flood the servers so much that the server won't respond. it never came to that because our host noticed it was happening and turned the site off. there is really nothing we can do about these things, it needs to be managed at a much higher level, but we now have some very strong means of prevention in place that should keep this from happening again, thanks to Wes. Thanks Wes!

you're welcome guys! i was so bummed it was down too. i'd get off the phone all stressed out from talking to the tech support guys and without thinking, try to go browse the forum to take my mind off things. i guess we're all addicts.

bh4rnnr
01-25-2008, 10:37 AM
Thanks for the hard work guys!!:beer::beer:

Rezarf
01-25-2008, 10:42 AM
I missed everyone! :crybaby:

Good to be back! :D Thanks for all the work on your guys end for getting this up and running agin! :thumb:

Red_Chili
01-25-2008, 12:06 PM
I bought a jeep
I bought a puppy.
I win.

Besides, he will almost certainly last longer... :lmao:

wesintl
01-25-2008, 12:18 PM
Sorry I had to pull in kipper. We (kipper and I) got notified by email of the issue and being at work it wasn't something I could tend to right away. To make matters worse I had company scheduled for dinner tues night. On the way home I called Kipper to see if he could help out since it would be 11pm before I'd have the time to look at our issue. Kipper was gracious enough to take the lead and figure out our issue and report back to our host even with all the things he has going on in life. We did have to do some maintenance to satisfy them. I apologize, we should have really only been down a day and half at most. The other day and half was them dropping the ball on customer service to get us back up.

Romer
01-25-2008, 12:25 PM
You have nothing to apologize for. You guys volunteer your time away from your families and other things you were going to do and it is much appreciated. No one expects you to drop your personal stuff and make this your number one priority.

You guys did awesome work and your volunteer efforts in this key position are much appreciated.

corsair23
01-25-2008, 01:46 PM
You have nothing to apologize for. You guys volunteer your time away from your families and other things you were going to do and it is much appreciated. No one expects you to drop your personal stuff and make this your number one priority.

You guys did awesome work and your volunteer efforts in this key position are much appreciated.

X a Brazillion :thumb:

What amazes me was that through all of it Kipper was able to compose one of the best updates I've ever seen! Nearly snorted my Diet Coke out my nose while reading it :lmao: - I've updated my sig line to pay Kipper a tribute :D

Hulk
01-25-2008, 02:12 PM
Wes, everything you and Kipper did was much appreciated. I think it was clear from the start that Easy CGI was dragging their feet. I still find it odd that they offer shared hosting w/o firewall protection -- how many people who buy hosting would know how to protect against threats?

So, what kind of scripts did you guys put in place? How do they block DoS attacks?

Rogue Leader
01-25-2008, 04:31 PM
Thanks a Lot

Now I'm gonna regain all the weight i lost while the site was down:D

thanks kipper and wes, i dont know what i would do with out this amazing resource. if i was older i would gladly buy you a beer:beer:

Kipper
01-30-2008, 03:31 PM
luvin' the sig corsair23! :thumb: glad you got a kick out of it. the whole situation at that point was comical to me.

leiniesred
01-31-2008, 09:59 AM
Right on, Kipper and Wes.

I work at a company that only does business in the USA. I've chosen to block a ton of international IP ranges at our firewalls. This has significantly reduced our spam mail influx that the spam filters have to deal with and reduced the number of security events I record every week.

Just whacking a few giant /8 non-USA networks made a big difference, but I still add more networks every week. (From whomever has decided I'm a target for their unwanted traffic). The masks keep getting smaller and smaller, so I must be making progress overall.

There are some international addresses that I can't block (like IBM's big ol' mail server overseas.) so it takes some care to keep from interfering with some domestic traffic.

If you think it would help, I can generate a list of the networks I'm blocking traffic from.

Chris
01-31-2008, 11:04 AM
Wes, everything you and Kipper did was much appreciated. I think it was clear from the start that Easy CGI was dragging their feet. I still find it odd that they offer shared hosting w/o firewall protection -- how many people who buy hosting would know how to protect against threats?

So, what kind of scripts did you guys put in place? How do they block DoS attacks?

I'm surprised EasyCGI didn't step in too, curious about what they had to say about this.