62

Let's say it again, we all make mistakes, and I have just made one.

A brief history: I was doing some stuff on a VPS (Debian) I'm renting, when I noticed some strange behaviour. Using the netstat command I saw an non-authorized connection through SSH. I didn't know what to do, so I decided to close his connection using iptables:

iptables -A INPUT -p tcp --dport ssh -s IP -j DROP

But I am tired, and I wrote

iptables -A INPUT -p tcp --dport ssh -j DROP

and I kicked myself (and everyone else) out...

How do I fix this?

Improve this question
6
  • possible duplicate of Is it possible to SSH access a server with a misconfigured subnet?
    – kasperd
    Apr 22, 2015 at 15:15
  • 1
    no, because he's blocking all ssh packets at the machine. on a misconfigured subnet, you'd just tcpdump for that mac address, get the subnet it's arp-ing on then configure a virtual nic to the same subnet and talk to it.
    – jfalcon aka Don Fanning
    Apr 22, 2015 at 19:52
  • 51
    Well, you certainly removed access for the unauthorized individual, at least.
    – user
    Apr 22, 2015 at 22:28
  • 2
    For next time, perhaps a dead man's switch would be handy.
    – Wayne Conrad
    Apr 23, 2015 at 22:48
  • 2
    Who is your host? A lot of VM hosts offer a serial console that lets you SSH in cases where you are unable to from a remote location.
    – Jon
    Apr 26, 2015 at 18:59

5 Answers 5

Reset to default
60

There are several alternatives:

  • See if they have IPMI / "KVM" / console access to the server which lets you control it as if you had a physical keyboard plugged into it.
  • If they don't offer that, see if you can boot the VM to a recovery linux CD (some providers offer this) and then correct the firewall rules that way and then boot it like normal.

  • If you don't have console access, before you boot to recovery or attach the volume to another VM (as in the Amazon case, credit user3550767's answer), you can try Ankh2054's answer of rebooting first if you haven't saved the rules (likely the case since you kicked yourself out before you had a chance to save). Use the control panel or ask someone to power cycle it using a non-graceful reset / poweroff (aka hard reboot or hard shutdown) in case the init script saves the rules automatically when gracefully rebooting (credit @jfalcon, @joshudson).

    Weigh the drawbacks of this (such as data being written during reboot may be lost and filesystem check may be required on boot so longer boot up time, though that delay may be less than booting to recovery).

Improve this answer
2
  • 1
    depending on the vps server, there might not be a remote access that he can reach. systems like vmware,kvm,xen,etc.. have consoles but are not setup for public consumption. closes that does allow this form of public control is openstack.
    – jfalcon aka Don Fanning
    Apr 22, 2015 at 19:49
  • 4
    that said, if it's in amazon/google space, he could snapshot the drive and mount it from another virtual machine to do the fix.
    – jfalcon aka Don Fanning
    Apr 22, 2015 at 19:50
47

If you have not yet saved the IPtables rule, you can reboot server on VPS (if available) and the rule should disappear.

Improve this answer
5
  • unless the init script saves the iptables during shutdown.
    – jfalcon aka Don Fanning
    Apr 22, 2015 at 19:46
  • 3
    @jfalcon: That's why you can ask them to pull the virtual plug.
    – joshudson
    Apr 22, 2015 at 20:49
  • 22
    @jfalcon That's also why it's a bad idea to auto-save on shutdown... make the save a conscious decision by the sysadmin, not something blindly executed by the system.
    – user
    Apr 22, 2015 at 22:27
  • @joshudson: You have to tell the reboot monkey exactly what to do and force power off. Not just initiate a shutdown.
    – jfalcon aka Don Fanning
    Apr 23, 2015 at 4:05
  • @MichaelKjörling: This philosophy is wrong as well. This is a VPS so likely he is sandboxing his applications and he has no control over who hits the power switch. And you have no idea how the VPS hosting configured their builds. There is nothing wrong with saving on shutdown. His mistake was his mistake. The smart move in the first place would have been to put SSH on a non-standard port or use fail2ban instead of reacting with a panicked reaction.
    – jfalcon aka Don Fanning
    Apr 23, 2015 at 4:17
30

This is what human-staffed help lines are for. Call the service provider, and have one of their operators remove the rule for you.

Improve this answer
3

A general way to fix a broken instance is to shut it down and attach the root volume to a working instance. You can then mount the volume there and view logs or edit configuration files. You can then detach the volume and start it up in its own instance.

Improve this answer
7
  • 2
    True for AWS VPSes; not generally true.
    – MadHatter
    Apr 23, 2015 at 10:37
  • @MadHatter The exact details may vary, but if a VPS provider doesn't offer any method to boot with a known working image from where you can mount and fix your root file system, then I would expect them to go out of business once customers learned about that limitation.
    – kasperd
    Apr 23, 2015 at 11:12
  • Usually, they do it by offering console access, so you can boot in single-user mode, or from rescue media (usually abstracted as an ISO). I submit that that is in fact much more normal than AWS's odd way of handling the issue, where a second VPS is (temporarily) a requirement.
    – MadHatter
    Apr 23, 2015 at 11:16
  • @MadHatter Which of the two approaches seems odd may very well depend on what you are the most familiar with. And either approach gets the job done. Which is the most common I can't really comment on, since so far I have only needed such access on a single VPS provider.
    – kasperd
    Apr 23, 2015 at 20:11
  • @kasperd: No, which you use depends on which a given VPS provider supports; it's not a matter of choice. If you're using AWS, you mount the volume on another VPS; if you're using a more normal VPS provider, you boot single-user or from rescue media. I don't make this point to be a dick (well, not entirely) but because it's important to appreciate that one has to find out what one's provider's supported method(s) are, then use them. Trying to use the AWS method with, say, Hetzner simply won't work, because one Hetzner vserver can't see another's FS (afaik).
    – MadHatter
    Apr 23, 2015 at 20:29
3

Formal answer: go to VPS management panel, get local access (virtual KVM) somehow or call them.

Explanation of steps/rules to prevent falling for it again:

  1. There are ip, routing and firewall rule changes that may go bad and block your access.
  2. and it applies to dedicated network device configuration too, not to VPSes only

So unless you are 100% sure that you can recover.. I recommend to always make a way to reset network config to its previous state.. like, open background session with either screen, nohup, or tmux, even cron may work for this, and add iptables -F or other desired means to reset anything to the previous state.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.