Huge edit/rewrite (no replies yet): That was too long and poorly written.
The Shenanigans I Inherited
Okay, I've got a load Balancer set up using two servers I don't know an awful lot about because I'm new here and an entire dev team walked out almost all at the same time months before I got here and our third part IT department has actually started deploying ninja smoke bombs to escape when I ask them about it. This setup is hosting all the development and staging versions of a bunch of asp.net sites we maintain for a client. Anybody who knew anything about what the hell happened here has left the building. My manager is new too.
The most obvious thing wrong with these servers is that the secondary server can't access the internet. It's showing that it last updated 6 months ago.
Before I discovered that, I wasn't even getting feedback in the IIS Admin tool and the icons for the servers in the arm suggested they shouldn't actually be serving anything. But sure enough if you tried to add a file on just one server, you'd get something 404ish. If on both, you could load it. I managed to fix the feedback/logging problem by getting the webfarm services up and running. I have no idea how anything was working without there were two and they had both crashed.
At one point, our third party IT department had some sort of robocopy process running that copied files from the primary server to the secondary server when they were edited. I do not believe it has been working since I got here and I have nuked the one process I couldn't be 100% certain about just to be sure.
I have also for reasons that will become apparent shortly, killed all application and platform provisioning in the web farm settings. We are literally copying file changes to both servers in order to see them on our dev site and copying a file to either server has never lead to that change being duplicated in the other server in the months I've been here. There is no version control in effect that I'm aware of and I've looked for it (yes, awful). As far as I can tell, platform and application provisioning have never worked properly if they've worked at all and turning them on/off seems to make no difference. I've left them off.
Here's where it gets freaky.
Files seem to revert occasionally but only when you attempt to make changes. It seems like there's a window of about 30 minutes where there's no problems and then bam, if you were manually moving a set of file changes, several all in a row, but not the changes you made just before those, will have their changes clobbered on the primary server (which can access the internet). I've fallen in the habit of copying to the secondary server first because its files never seem to revert and it seems more likely reverts will happen when copying from primary to secondary and secondary changes to do not always stick when you do it the "wrong" way.
I also noticed at one point that even if I removed the secondary server from the web farm, stopping the app on that server or not having like-named files on both would cause errors. Seeing that led me to the conclusion that it was time to shut the silly thing down.
After a little trial and error I shut the farm down completely and deleted via GUI the whole setup. I stopped running the apps on the secondary by logging into it and killing them there. Everything seemed more than fine just running off of the primary. In fact, performance improved dramatically and I observed that I could make changes on one server and two clients accessing the same files would load them just fine (the load balancer seems uncomplicated in this regard and always splits the difference if two clients try to access the same unsynced file). I'm just a client-side-specialized-web/now-generalist developer with just enough knowledge about this stuff to RTFM and get things up and running. It felt like a glorious victory.
For about 30 minutes.
Then, just as I started bragging about it, everything went to Hell and every single one of our dev and staging apps broke and started throwing the same error I recall seeing when I tried to just pull the secondary out of the farm and shut it down (more than just a 404 IIRC). I hastily rebuilt a web farm similar to the old one and got it not-quite-working-properly as before the next morning.
Assume:
- I can't just nuke it from orbit (don't know what else is on there that might be important just yet - but I supposed we could just back it all up). This ultimately is probably what we'll have to do barring somebody who knows exactly what this batshit server situation is likely all about.
- I'm not going to get the internet connection up and running on the secondary server (I spent days on that - I'm starting to suspect a hardware issue and I have no access to the hardware).
- I have no IT resources that aren't some combination of too busy, incompetent or possibly afraid to help me.
- I do not want to get the web farm working unless there's some duh a-ha! way to do so that will fix everything. I want it gone. I just want one server and one simple config for all of our dev and staging sites, ideally for longer than 30 minutes.
Hardware/OS/Software
- Something Xeon and probably old because 2 gigs of RAM on each server
- Win Server 2008 R2, also on both
- IIS 7
- I interact with the servers from a Mac using MS's Mac RDP app and I copy mostly ASPX files I've made edits to manually (with Sublime Text 2) via an SMB connection
The only theory I have so far is that some aspect of internet access or a component that the secondary hasn't been able to update is causing some sort of disconnect that results in current configuration being ignored by half the setup leading to some spectacularly weird behavior when both halves start guessing what they're supposed rather than simply erroring out at the appropriate time. I'm not 100% sure but I think sometimes the files have been reverting to versions that were older than the state they were in before I started making changes (or possibly there was already a mismatch between the two servers).
Or it's a GAC thing which would be exciting because I've heard nothing is as stupefying as a GAC problem.
So how do I shut this load balancer from Hell down in such a way that I can just run these unnecessarily load balanced and ASP.NET static web sites that will never see more than 100k visitors in a year off of one simple server config without this mystery entity slapping everything I try to do down so I can start deploying with git as opposed to hand-copying files to both servers in the right order only.
