Running Proxmox behind a single IP address 45


I ran into the battle of running all of my VMs and the host node under a single public IP address. Luckily, the host is just pure Debian, and ships with iptables.

What needs to be done is essentially to run all the VMs on a private internal network. Outbound internet access is done via NAT. Inbound access is via port forwarding.

Network configuration

Here’s how it’s done:

Create a virtual interface that serves as the gateway for your VMs:

My public interface (the one with the public IP assigned) is vmbr0. I will then create an alias interface called vmbr0:0 and give it a private IP address in /etc/network/interfaces. Note that this is needed for KVM and OpenVZ bridged interfaces; venet interfaces automagically work.

Create an iptables rule to allow outbound traffic:

There are a few ways to specify this, but the most straightforward is:

In one of your VMs, set the interface IP to something in 192.168.4.2-254, and set the default gateway to 192.168.4.1, with the subnet mask of 255.255.255.0. Feel free to adjust this as you see fit. Test pinging your public IP address, and perhaps even an external address (like 4.2.2.2). If this works, you’re on the right track.

At this point, you have internet access from your VMs, but how do you get to them? For your OpenVZ containers, sure, you could SSH into the host node and ‘vzctl enter’ into a CTID, but that’s probably not what you want. We will need to set iptables rules to dictate which ports point to which servers.

Assuming you want VM 100 to have SSH on port 10022, and let RDP of VM 101 ‘live’ on port 10189, we can do the following:

You can add as many of these as you’d like.

Once you have your configuration set up as you please, we will need to make it persistent. If you reboot at this point, all of your iptables rules will be cleared. To prevent this, we simply do:

This step saves the rules to an iptables-readable file. In order to apply them upon boot, you have several options. One of the easier ones is to modify /etc/network/interfaces as such (notice the third line):

At this point, you now have a functioning inbound/outbound setup on your own private LAN.

Assigning public ports to containers

With multiple containers potentially running the same types of services, you can’t easily just map pu.bl.ic.ip:80 -> 192.168.4.100:80 and 192.168.4.101:80.  Ports will collide, and you have to figure out the best way to work around that.  The section below details how to perform host-header switching/proxying for websites, but for other services, there aren’t such elaborate solutions.  SIP, for example, runs on port 5060.  If you have two SIP servers (perhaps one for testing, one production), you’ll have to map things.

A port-numbering algorithm I came up with is:

(CTID mod 100) x 100 + original port number + 1000

For example, with container 105 that needs SIP:

For FTP, port 22 on container 105:

Your weights and offsets might need tweaking for your particular purposes; this is just what works for me.

Supporting multiple websites

Now, what if you want to install multiple websites across multiple containers?  One easy way to do this is to do port forwarding so that, e.g., domain.com:1180 goes to container 101, domain.com:1280 goes to container 102, etc., but that’s ugly.  We can instead setup a proxy that takes ALL requests on port 80 and routes them to their appropriate destinations.  Let’s get started.

In this example, we’re going to have a dedicated container for nginx.  I also have a dedicated container for a MySQL instance that’s shared for all of my sites.  This allows the website containers to be very lightweight.

First, create a container using the OS of your choice, and enter it.  I recommend using one of the minimal templates provided by openvz.org.  View this post for information on how to install templates and create containers.

Here, we’ll be using the Ubuntu 14.04 template.  Once you’re in, you’re now ready to install nginx.

You’ll now have a default site, which you’ll probably want to change.  This site will be served for any requests NOT matching a site name of anything else nginx serves (e.g., if a request for hello.ameir.net comes in, but nginx only knows to serve www.ameir.net, the default site would show up).  Either change the default site, or delete it so that another site (the first config file nginx loads, in alphabetical order), is the default.  You can prefix the config filename with something like 000- to ensure it’s the default.  Alternatively, you can specify it in the config file, like  listen 80 default_server; .

Now, for each site you want to proxy for, you’ll need a config file, as follows:

Once you’ve created all of the config files, as shown above, simply restart nginx with  service nginx restart .

Now, assuming your nginx container is container 101 with IP address 192.168.4.101,  we can allow worldwide access as such:

Now, once you point DNS, you should be good to go.  If you’d like to test this prior, you can update your hosts file, or simply use curl to see if things are looking as expected:

I hope that helps!


Leave a comment

Your email address will not be published. Required fields are marked *

45 thoughts on “Running Proxmox behind a single IP address

  • Matt

    Thank you very very much!
    I have been searching for long time to figure out how to setup my VMs under single public IP via port-forwarding, but didn’t come up with anything usefull at all.
    I wish you could find some time to write more of your experience in managing network for host and guests in proxmox, because I find it the most complicated part of proxmox.

  • BLiN

    Thank you very much! 8 hours I tried to get this work, and than in 5 minute all works)

  • Stephan Wagner

    Using Proxmox, it is not a question if it works or not, but rather, if the configuration is an option in the web-interface or not. Most you can’t configure in the web-interface, you can make it run on the console with standard Linux tools. Great work.

  • Ric Moore

    I’m with Matt! I would like to see more step by step for handling html container output through the same IP address. I’m still wrapping my brains around virtual networking. Plus hardware recommendations for same, like router vs switch with what capabilities. THANKS!! Ric

  • Ameir Abdeldayem Post author

    Thanks for the feedback, guys. To clarify, are you interested in reading about how to host multiple websites in different containers/VMs, all sharing a single public IP? If so, that’s actually something I’ll be dealing with soon, and can do a writeup on it.

  • somendra saini

    hey!
    thank you very much for this helpful info.currently i am looking for help.i have a proxmox server with public IP and i have few virtual machines with Elastix setup in it.all of the elastix server is configured by private IP.all these VM is in bridge network mode in proxmox.how can i get the access of every elastix from outside the world with these private IP?
    thanks in advance.

  • Ameir Abdeldayem Post author

    Hello, what protocol do you need to access Elastix on? Is it the web interface or for something like SIP calling? I’ll be publishing a post soon to describe an easy way to host multiple sites on multiple containers/VMs over one IP. The latter case (e.g., SIP) can be done by port forwarding, and possibly by an L7 load balancer that understands the protocol.

  • somendra saini

    Hey Ameir!
    thanks for your quick reply.well,I need to access Elastix for SIP calling.all these Elastix will be used for ‘hosted PBX services’.so it will be better if you can help me out with how to do this!
    please try to help me with complete configuration as soon as possible.
    waiting for your response.and thanking you again for your kind help!
    Thanks!

  • Ameir Abdeldayem Post author

    Hello, you can certainly make use of iptables here to perform port forwarding on your different instances. For example, you could so something like:
    iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6160 -j DNAT --to-destination 192.168.4.101:5060
    iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6260 -j DNAT --to-destination 192.168.4.102:5060
    iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6360 -j DNAT --to-destination 192.168.4.103:5060

  • Ameir Abdeldayem Post author

    You can use the multiport feature of iptables for this, but it might not do exactly what you’re looking for.
    iptables -t nat -A PREROUTING -i vmbr0 -p tcp --match multiport --dports 7080,7022,7053 -j DNAT --to-destination 192.168.4.101
    iptables -t nat -A PREROUTING -i vmbr0 -p tcp --match multiport --dports 8000:8080 -j DNAT --to-destination 192.168.4.101

    IIRC, this would pass the ports as-is to the destination, so, e.g., port 7080 -> 192.168.4.101:7080. I’m honestly not sure if there’s a straightforward way of doing port mappings.

  • Sergio

    Thank you very much Ameir,
    I’ve tried to follow these steps, I’ve found them pretty straightforward, but my guest can’t reach public ips (via ping) rather than host public ip (for instance ping to 4.2.2.2 fails, but from host it’s ok).

    Host can reach guest (via ping). It seems that iptables rules are not forwarding data.

    Any suggestions?

    HOST MACHINE
    =================

    IPTABLES RULES
    ——————————
    iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o br0 -j MASQUERADE

    /etc/network/interfaces
    ——————————–
    auto eth2
    iface eth2 inet manual

    auto br0
    iface br0 inet static
    address A.B.C.12
    network A.B.C.0
    netmask 255.255.255.0
    broadcast A.B.C.255
    gateway A.B.C.1
    bridge_ports eth2
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp on
    # dns-* options are implemented by the resolvconf package, if inst$
    dns-nameservers A.B.C.3 A.B.C.10
    dns-search upc.es

    auto br0:0
    iface br0:0 inet static
    address 192.168.2.1
    netmask 255.255.255.0
    network 192.168.2.0
    broadcast 192.168.2.255

    GUEST MACHINE
    ===============
    /etc/network/interfaces
    ———————————–
    auto eth0
    iface eth inet static
    address 192.168.2.100
    netmask 255.255.255.0
    gateway 192.168.2.1

  • Dario Rugani

    You forgot one very importan thing in your post.

    You have to define the source when your forward the port with the iptable rule, otherwise all traffic on this port, outgoing or incomming, will be redirected to the specified port.

    For example: You are working on a windows vm via remotedesktop and want to forward port 80 on your apache webserver.
    After you use this rule:
    iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 192.168.4.100:80

    all traffic will be directed to the address 192.168.4.100 port 80

    So whenn you try to open google in your VM you will land on your own apache webserver because all traffic on the host will be redirected to the adress that is defined in the rule.

    To prevent this you have to use the source parameter.
    So all internal traffic will be ingored and goes to the desired address.
    Like this : ! -s 192.168.4.0/24

    The full rule would be like this:
    iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp ! -s 192.168.4.0/24 –dport 80 -j DNAT –to-destination 192.168.4.100:80

    I hope you can follow what i am trying to say.
    I would be very happy if you cann edit it in your blogpost

    Greetz from Germany
    Dario Rugani

    Ps. Please excuse my bad englisch =)

  • arif

    Hi Ameir

    I’ve tried this tutorial, but I still have a little problem, my VM (Debian) can not do apt-get update. How to fix it?

    Thx

  • Ameir Abdeldayem Post author

    Hi Arif,

    Are you working with OpenVZ or KVM? Are you able to ping the internal IP of your host node (e.g. 192.168.4.1)? What happens when you try to traceroute 8.8.8.8?

    Thanks,
    Ameir

  • Mike Waters

    Try to install pfsense on proxmox as a gateway on my server. Got most everything to work i think just if i set the pfsense ip to .1 it doesnt work. I have to set the pfsense ip to .2 for it to work. What am i doing wrong? The idea is to have one nic plugged into my internet connection, and the other into my hub for my LAN. PFSENSE running on the proxmox box and acting as a gateway/firewall ….

  • Ameir Abdeldayem Post author

    Are you certain that you’re not using that IP address elsewhere? Note that the IP addresses have to be unique throughout your network for this to work. It sounds like the host or another system is hanging onto your .1 IP. If pfSense is on .2, are you able to ping .1? Are there any ARP table entries for it (`arp -a)?

  • Ameir Abdeldayem Post author

    Could you provide some more details on what you’re trying to accomplish? Are you trying to support multiple *inbound* SMTP servers? If you only need one active SMTP server, you can easily just port forward. I don’t think nginx supports SMTP server switching per-domain like it can with HTTP and DNS Host: headers, but I could be wrong.

  • vinc

    it is really interessting!
    “Supporting multiple websites”
    but there are a couple of things i did not understand.
    maybe a picture showing the ISP Router, your Firewall/NAT, and container would help
    have a nice day
    vinc

  • vinc

    Hello
    i got a error
    ====================
    iptables v1.4.21: can’t initialize iptables table `nat’: Table does not exist (do you need to insmod?)
    Perhaps iptables or your kernel needs to be upgraded.

  • Daniel

    Is this HowTo suitable for Proxmox V3.3 to or can you update it ?

    Some Commands like:

    iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 192.168.4.101:80

    will be dropped

    btw thanks for your great work in this HowTo !

  • Ameir Abdeldayem Post author

    Hi Vinc,

    Are you running these commands on the host node or within a container? The iptables rules should be on the host node itself, as that is what will act as the router in this case. If you’re getting this error on the host node, take a look at http://stackoverflow.com/questions/21983554/iptables-v1-4-14-cant-initialize-iptables-table-nat-table-does-not-exist-d (although a proper Proxmox install shouldn’t have this issue).

    Thanks,
    Ameir

  • Ameir Abdeldayem Post author

    I’m currently using this on Proxmox 3.3 without issue. Could you explain what you mean by “will be dropped”? I’d be happy to update this if there have been significant recent changes in Proxmox that would alter this approach.

    Thanks!

  • Martin Cleaver

    Hey Ameir,

    Have you thought about how to generate the proxy CT’s definition, such that it automatically knows to route port 80 to each CT?

    So:
    – a CNAME record on a domain .dev.companyname.com -> public IP address.
    – port 80 of router for public IP address -> nginx proxy ct
    – nginx proxy automatically configures itself:

    configuration scans the proxmox CTs, looking for “Name: ” host attribute, and builds the default config files itself. CTs could also provide something in /etc/ to add to the rules, for example to expose a non-standard port.

    Thanks very much.

    Best,
    Martin.

  • Ameir Abdeldayem Post author

    Hi Daniel,

    After digging around more, I realized what you meant. Unfortunately, the firewall in Proxmox 3.3 is still quite limited, and does not have support for the nat table. It looks like you’ll have to maintain the iptables rules manually. I spent a bit of time looking for workarounds, but there doesn’t seem to be anything clean and straightforward out there.

    Thanks,
    Ameir

  • Ameir Abdeldayem Post author

    Hi Martin,

    There are certainly a few ways to achieve this. The simplest solution I can think of at the moment would require you to set “variables” in the container description (“Notes” in the web UI), like in
    http://cl.ly/image/0P0U0u230Q0T/Image%202015-01-11%20at%206.08.54%20PM.png .

    When these “variables” are saved, they are URL encoded in the backend. From the CLI, you can do
    # pvectl config 102 | grep description
    description: HTTP_PORT=80%0AHTTPS_ENABLED=true%0AHTTPS_CERT_PATH=/etc/ssl/cert.crt%0AHTTPS_KEY_PATH=/etc/ssl/key.pem%0A

    to see the encoded string.

    Now, with Chef (or a CM tool of your choice), you can — on an interval — check each of your VMs/containers for the description property, urldecode the string after “description: “, and import those as variables (e.g. using eval()). If nginx is in a container, then you have full access to the filesystem from the host node, and can update the config with Chef. You can also then reload nginx using vzctl exec.

    In my example, the HTTPS paths are relative to the container filesystem, so this would only work with OpenVZ. If HTTPS is not a concern, or if you’re using a shared certificate, then you should be fine regardless.

    In my setup, I have a DNS entry like:
    *.dev.companyname.com IN A pu.bl.ic.ip
    which works great.

    That’s just one way to go about this. confd or consul-template may be of interest as well, but possibly overkill here.

    Thanks,
    Ameir

  • TeoWood

    Hi Ameir
    Thanks for your guide.
    What if my destination web server container is running Apache. Do i need any special configuration to achieve the nginx “catch all http requests” you describe?

  • Ivan

    Martin, could you please suggest any good reads to understand the principles of vm networking setup (bridged in particular) better?

  • Naveed Ur Rehman

    Hi,

    I need your help in setting up public ip addresses in KVM based VM in proxmox. I have googled a lot, but could not find the solution. Please provide me help in setting up public ip addreses in kvm machines.

    Thanks

  • Vedat

    i can ping and ssh to public ip’s but port 80 seems to be blocked (cant wget or telnet to port 80 – conn refused) even though i didnt turn pve firewall on

  • Gregory Wilk

    Sir. I am pulling my hair out trying to make RDP work on my VM. I would appreciate it very much if we could get in touch via email to set up an appointment where you could dial in and show me exactly what I am missing. I am a paying customer. Thank You.

  • Elgs

    When doing port forwarding, if I had multiple public IP bound to vmbr0, can I use -d x.x.x.x instead of -i vmbr0 in order to restrict to one public IP?

  • alfiechan

    this port-numbering algorithm is not uniqueness.
    for example:
    mod(105,100)*100+5060+1000=mod(205,100)*100+5060+1000

  • Ameir Abdeldayem Post author

    Sorry for the delayed response. This algorithm needs tweaking based on your expectations; it’s not “one size fits all.” In my case, I don’t expect 205 containers, so the algorithm suits me just fine. If you do expect more containers, you could perhaps to ctid % 1000 or the like instead.

  • Eric

    Hey!

    Thanks for the tutorial. I am getting a 404 not found when visiting the nginx website

    Not sure where to begin to debug