I ran into the battle of running all of my VMs and the host node under a single public IP address. Luckily, the host is just pure Debian, and ships with iptables.
What needs to be done is essentially to run all the VMs on a private internal network. Outbound internet access is done via NAT. Inbound access is via port forwarding.
Network configuration
Here’s how it’s done:
Create a virtual interface that serves as the gateway for your VMs:
My public interface (the one with the public IP assigned) is vmbr0. I will then create an alias interface called vmbr0:0 and give it a private IP address in /etc/network/interfaces. Note that this is needed for KVM and OpenVZ bridged interfaces; venet interfaces automagically work.
|
1 2 3 4 5 6 |
auto vmbr0:0 iface vmbr0:0 inet static address 192.168.4.1 netmask 255.255.255.0 network 192.168.4.0 broadcast 192.168.4.255 |
Create an iptables rule to allow outbound traffic:
There are a few ways to specify this, but the most straightforward is:
|
1 |
iptables -t nat -A POSTROUTING -s 192.168.4.0/24 -o vmbr0 -j MASQUERADE |
In one of your VMs, set the interface IP to something in 192.168.4.2-254, and set the default gateway to 192.168.4.1, with the subnet mask of 255.255.255.0. Feel free to adjust this as you see fit. Test pinging your public IP address, and perhaps even an external address (like 4.2.2.2). If this works, you’re on the right track.
At this point, you have internet access from your VMs, but how do you get to them? For your OpenVZ containers, sure, you could SSH into the host node and ‘vzctl enter’ into a CTID, but that’s probably not what you want. We will need to set iptables rules to dictate which ports point to which servers.
Assuming you want VM 100 to have SSH on port 10022, and let RDP of VM 101 ‘live’ on port 10189, we can do the following:
|
1 2 |
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 10022 -j DNAT --to-destination 192.168.4.100:22 iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 10189 -j DNAT --to-destination 192.168.4.101:3389 |
You can add as many of these as you’d like.
Once you have your configuration set up as you please, we will need to make it persistent. If you reboot at this point, all of your iptables rules will be cleared. To prevent this, we simply do:
|
1 |
iptables-save > /etc/iptables.rules |
This step saves the rules to an iptables-readable file. In order to apply them upon boot, you have several options. One of the easier ones is to modify /etc/network/interfaces as such (notice the third line):
|
1 2 3 4 5 6 |
auto vmbr0 iface vmbr0 inet static pre-up iptables-restore < /etc/iptables.rules address pu.bl.ic.ip netmask 255.255.255.0 ... |
At this point, you now have a functioning inbound/outbound setup on your own private LAN.
Assigning public ports to containers
With multiple containers potentially running the same types of services, you can’t easily just map pu.bl.ic.ip:80 -> 192.168.4.100:80 and 192.168.4.101:80. Ports will collide, and you have to figure out the best way to work around that. The section below details how to perform host-header switching/proxying for websites, but for other services, there aren’t such elaborate solutions. SIP, for example, runs on port 5060. If you have two SIP servers (perhaps one for testing, one production), you’ll have to map things.
A port-numbering algorithm I came up with is:
(CTID mod 100) x 100 + original port number + 1000
For example, with container 105 that needs SIP:
|
1 |
(105 mod 100) x 100 + 5060 + 1000 = 500 + 5060 + 1000 = 6560 |
For FTP, port 22 on container 105:
|
1 |
(105 mod 100) x 100 + 22 + 1000 = 500 + 22 + 1000 = 1522 |
Your weights and offsets might need tweaking for your particular purposes; this is just what works for me.
Supporting multiple websites
Now, what if you want to install multiple websites across multiple containers? One easy way to do this is to do port forwarding so that, e.g., domain.com:1180 goes to container 101, domain.com:1280 goes to container 102, etc., but that’s ugly. We can instead setup a proxy that takes ALL requests on port 80 and routes them to their appropriate destinations. Let’s get started.
In this example, we’re going to have a dedicated container for nginx. I also have a dedicated container for a MySQL instance that’s shared for all of my sites. This allows the website containers to be very lightweight.
First, create a container using the OS of your choice, and enter it. I recommend using one of the minimal templates provided by openvz.org. View this post for information on how to install templates and create containers.
Here, we’ll be using the Ubuntu 14.04 template. Once you’re in, you’re now ready to install nginx.
|
1 2 3 |
echo "deb http://nginx.org/packages/ubuntu/ `lsb_release -cs` nginx" | tee /etc/apt/sources.list.d/nginx.list curl -L http://nginx.org/keys/nginx_signing.key | apt-key add - apt-get update && apt-get install nginx |
You’ll now have a default site, which you’ll probably want to change. This site will be served for any requests NOT matching a site name of anything else nginx serves (e.g., if a request for hello.ameir.net comes in, but nginx only knows to serve www.ameir.net, the default site would show up). Either change the default site, or delete it so that another site (the first config file nginx loads, in alphabetical order), is the default. You can prefix the config filename with something like 000- to ensure it’s the default. Alternatively, you can specify it in the config file, like listen 80 default_server; .
Now, for each site you want to proxy for, you’ll need a config file, as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
server { listen 80; server_name ameir.net *.ameir.net; location / { proxy_pass http://192.168.4.104; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 150; proxy_send_timeout 100; proxy_read_timeout 100; proxy_buffers 4 32k; client_max_body_size 8m; client_body_buffer_size 128k; } } |
Once you’ve created all of the config files, as shown above, simply restart nginx with service nginx restart .
Now, assuming your nginx container is container 101 with IP address 192.168.4.101, we can allow worldwide access as such:
|
1 2 |
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.4.101:80 iptables-save > /etc/iptables.rules |
Now, once you point DNS, you should be good to go. If you’d like to test this prior, you can update your hosts file, or simply use curl to see if things are looking as expected:
|
1 |
curl -i -L -H "Host: ameir.net" pu.bl.ic.ip |
I hope that helps!
Thank you very very much!
I have been searching for long time to figure out how to setup my VMs under single public IP via port-forwarding, but didn’t come up with anything usefull at all.
I wish you could find some time to write more of your experience in managing network for host and guests in proxmox, because I find it the most complicated part of proxmox.
You need to use the nat table, add “-t nat” to the beginning of those rules (right after the iptables command).
Thank you very much! 8 hours I tried to get this work, and than in 5 minute all works)
Thank you very much for this article, really saved my day!
Using Proxmox, it is not a question if it works or not, but rather, if the configuration is an option in the web-interface or not. Most you can’t configure in the web-interface, you can make it run on the console with standard Linux tools. Great work.
I’m with Matt! I would like to see more step by step for handling html container output through the same IP address. I’m still wrapping my brains around virtual networking. Plus hardware recommendations for same, like router vs switch with what capabilities. THANKS!! Ric
Thanks for the feedback, guys. To clarify, are you interested in reading about how to host multiple websites in different containers/VMs, all sharing a single public IP? If so, that’s actually something I’ll be dealing with soon, and can do a writeup on it.
hey!
thank you very much for this helpful info.currently i am looking for help.i have a proxmox server with public IP and i have few virtual machines with Elastix setup in it.all of the elastix server is configured by private IP.all these VM is in bridge network mode in proxmox.how can i get the access of every elastix from outside the world with these private IP?
thanks in advance.
Hello, what protocol do you need to access Elastix on? Is it the web interface or for something like SIP calling? I’ll be publishing a post soon to describe an easy way to host multiple sites on multiple containers/VMs over one IP. The latter case (e.g., SIP) can be done by port forwarding, and possibly by an L7 load balancer that understands the protocol.
Hey Ameir!
thanks for your quick reply.well,I need to access Elastix for SIP calling.all these Elastix will be used for ‘hosted PBX services’.so it will be better if you can help me out with how to do this!
please try to help me with complete configuration as soon as possible.
waiting for your response.and thanking you again for your kind help!
Thanks!
How to add a port range?
Hello, you can certainly make use of iptables here to perform port forwarding on your different instances. For example, you could so something like:
iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6160 -j DNAT --to-destination 192.168.4.101:5060iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6260 -j DNAT --to-destination 192.168.4.102:5060
iptables -t nat -A PREROUTING -i vmbr0 -p udp -m udp --dport 6360 -j DNAT --to-destination 192.168.4.103:5060
You can use the multiport feature of iptables for this, but it might not do exactly what you’re looking for.
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --match multiport --dports 7080,7022,7053 -j DNAT --to-destination 192.168.4.101iptables -t nat -A PREROUTING -i vmbr0 -p tcp --match multiport --dports 8000:8080 -j DNAT --to-destination 192.168.4.101
IIRC, this would pass the ports as-is to the destination, so, e.g., port 7080 -> 192.168.4.101:7080. I’m honestly not sure if there’s a straightforward way of doing port mappings.
Thank you very much Ameir,
I’ve tried to follow these steps, I’ve found them pretty straightforward, but my guest can’t reach public ips (via ping) rather than host public ip (for instance ping to 4.2.2.2 fails, but from host it’s ok).
Host can reach guest (via ping). It seems that iptables rules are not forwarding data.
Any suggestions?
HOST MACHINE
=================
IPTABLES RULES
——————————
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o br0 -j MASQUERADE
/etc/network/interfaces
——————————–
auto eth2
iface eth2 inet manual
auto br0
iface br0 inet static
address A.B.C.12
network A.B.C.0
netmask 255.255.255.0
broadcast A.B.C.255
gateway A.B.C.1
bridge_ports eth2
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp on
# dns-* options are implemented by the resolvconf package, if inst$
dns-nameservers A.B.C.3 A.B.C.10
dns-search upc.es
auto br0:0
iface br0:0 inet static
address 192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
GUEST MACHINE
===============
/etc/network/interfaces
———————————–
auto eth0
iface eth inet static
address 192.168.2.100
netmask 255.255.255.0
gateway 192.168.2.1
Hi Sergio,
Could you try setting
net.ipv4.ip_forward=1in /etc/sysctl.conf (and apply with sysctl -p)?You forgot one very importan thing in your post.
You have to define the source when your forward the port with the iptable rule, otherwise all traffic on this port, outgoing or incomming, will be redirected to the specified port.
For example: You are working on a windows vm via remotedesktop and want to forward port 80 on your apache webserver.
After you use this rule:
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 192.168.4.100:80
all traffic will be directed to the address 192.168.4.100 port 80
So whenn you try to open google in your VM you will land on your own apache webserver because all traffic on the host will be redirected to the adress that is defined in the rule.
To prevent this you have to use the source parameter.
So all internal traffic will be ingored and goes to the desired address.
Like this : ! -s 192.168.4.0/24
The full rule would be like this:
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp ! -s 192.168.4.0/24 –dport 80 -j DNAT –to-destination 192.168.4.100:80
I hope you can follow what i am trying to say.
I would be very happy if you cann edit it in your blogpost
Greetz from Germany
Dario Rugani
Ps. Please excuse my bad englisch =)
I wasn’t able to replicate this issue (un?)fortunately. Thanks for reporting it, regardless, as it may help someone in need.
Great, work perfectly.. thanks pals
Hi Ameir
I’ve tried this tutorial, but I still have a little problem, my VM (Debian) can not do apt-get update. How to fix it?
Thx
Hi Arif,
Are you working with OpenVZ or KVM? Are you able to ping the internal IP of your host node (e.g. 192.168.4.1)? What happens when you try to traceroute 8.8.8.8?
Thanks,
Ameir
Try to install pfsense on proxmox as a gateway on my server. Got most everything to work i think just if i set the pfsense ip to .1 it doesnt work. I have to set the pfsense ip to .2 for it to work. What am i doing wrong? The idea is to have one nic plugged into my internet connection, and the other into my hub for my LAN. PFSENSE running on the proxmox box and acting as a gateway/firewall ….
Are you certain that you’re not using that IP address elsewhere? Note that the IP addresses have to be unique throughout your network for this to work. It sounds like the host or another system is hanging onto your .1 IP. If pfSense is on .2, are you able to ping .1? Are there any ARP table entries for it (`arp -a)?
I’ts GREAT blogpost.
I have a qustion: what about proxy for mail services.It does not work.
Ngins by default does not compile support for email FORWARDING, but additionally includes.
Here’s a good example how:
http://xmodulo.com/compile-install-nginx-web-server.html
Bum, what’s next?
How to change this example, here from thih blogpost, in order to work email forwarding?
Could you provide some more details on what you’re trying to accomplish? Are you trying to support multiple *inbound* SMTP servers? If you only need one active SMTP server, you can easily just port forward. I don’t think nginx supports SMTP server switching per-domain like it can with HTTP and DNS Host: headers, but I could be wrong.
it is really interessting!
“Supporting multiple websites”
but there are a couple of things i did not understand.
maybe a picture showing the ISP Router, your Firewall/NAT, and container would help
have a nice day
vinc
Hello
i got a error
====================
iptables v1.4.21: can’t initialize iptables table `nat’: Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
have a look
http://www.wombat.ch/index.php?option=com_content&view=article&id=9475:proxmox-kvm-openbsd-pfsense-nat&catid=66:perfect-server&Itemid=108
Is this HowTo suitable for Proxmox V3.3 to or can you update it ?
Some Commands like:
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 192.168.4.101:80
will be dropped
btw thanks for your great work in this HowTo !
Hi Vinc,
Are you running these commands on the host node or within a container? The iptables rules should be on the host node itself, as that is what will act as the router in this case. If you’re getting this error on the host node, take a look at http://stackoverflow.com/questions/21983554/iptables-v1-4-14-cant-initialize-iptables-table-nat-table-does-not-exist-d (although a proper Proxmox install shouldn’t have this issue).
Thanks,
Ameir
I’m currently using this on Proxmox 3.3 without issue. Could you explain what you mean by “will be dropped”? I’d be happy to update this if there have been significant recent changes in Proxmox that would alter this approach.
Thanks!
Hey Ameir,
Have you thought about how to generate the proxy CT’s definition, such that it automatically knows to route port 80 to each CT?
So:
– a CNAME record on a domain .dev.companyname.com -> public IP address.
– port 80 of router for public IP address -> nginx proxy ct
– nginx proxy automatically configures itself:
configuration scans the proxmox CTs, looking for “Name: ” host attribute, and builds the default config files itself. CTs could also provide something in /etc/ to add to the rules, for example to expose a non-standard port.
Thanks very much.
Best,
Martin.
Hi Daniel,
After digging around more, I realized what you meant. Unfortunately, the firewall in Proxmox 3.3 is still quite limited, and does not have support for the nat table. It looks like you’ll have to maintain the iptables rules manually. I spent a bit of time looking for workarounds, but there doesn’t seem to be anything clean and straightforward out there.
Thanks,
Ameir
Hi Martin,
There are certainly a few ways to achieve this. The simplest solution I can think of at the moment would require you to set “variables” in the container description (“Notes” in the web UI), like in
http://cl.ly/image/0P0U0u230Q0T/Image%202015-01-11%20at%206.08.54%20PM.png .
When these “variables” are saved, they are URL encoded in the backend. From the CLI, you can do
# pvectl config 102 | grep descriptiondescription: HTTP_PORT=80%0AHTTPS_ENABLED=true%0AHTTPS_CERT_PATH=/etc/ssl/cert.crt%0AHTTPS_KEY_PATH=/etc/ssl/key.pem%0A
to see the encoded string.
Now, with Chef (or a CM tool of your choice), you can — on an interval — check each of your VMs/containers for the description property, urldecode the string after “description: “, and import those as variables (e.g. using
eval()). If nginx is in a container, then you have full access to the filesystem from the host node, and can update the config with Chef. You can also then reload nginx usingvzctl exec.In my example, the HTTPS paths are relative to the container filesystem, so this would only work with OpenVZ. If HTTPS is not a concern, or if you’re using a shared certificate, then you should be fine regardless.
In my setup, I have a DNS entry like:
*.dev.companyname.com IN A pu.bl.ic.ipwhich works great.
That’s just one way to go about this. confd or consul-template may be of interest as well, but possibly overkill here.
Thanks,
Ameir
Hi Ameir
Thanks for your guide.
What if my destination web server container is running Apache. Do i need any special configuration to achieve the nginx “catch all http requests” you describe?
Martin, could you please suggest any good reads to understand the principles of vm networking setup (bridged in particular) better?
Hi,
I need your help in setting up public ip addresses in KVM based VM in proxmox. I have googled a lot, but could not find the solution. Please provide me help in setting up public ip addreses in kvm machines.
Thanks
You will need to define a static route on the proxmox box so it knows where to route the traffic. For example if the VM is on vmbr1, you would run
i can ping and ssh to public ip’s but port 80 seems to be blocked (cant wget or telnet to port 80 – conn refused) even though i didnt turn pve firewall on
Pls ignore more post. Dario Rugani already gave the solution just took a while to understand 🙂
Sir. I am pulling my hair out trying to make RDP work on my VM. I would appreciate it very much if we could get in touch via email to set up an appointment where you could dial in and show me exactly what I am missing. I am a paying customer. Thank You.
When doing port forwarding, if I had multiple public IP bound to
vmbr0, can I use-d x.x.x.xinstead of-i vmbr0in order to restrict to one public IP?this port-numbering algorithm is not uniqueness.
for example:
mod(105,100)*100+5060+1000=mod(205,100)*100+5060+1000
Sorry for the delayed response. I’m pretty sure that would work, although I haven’t tested it myself. If you have any updates on it, please let me know!
Sorry for the delayed response. This algorithm needs tweaking based on your expectations; it’s not “one size fits all.” In my case, I don’t expect 205 containers, so the algorithm suits me just fine. If you do expect more containers, you could perhaps to
ctid % 1000or the like instead.Hey!
Thanks for the tutorial. I am getting a 404 not found when visiting the nginx website
Not sure where to begin to debug
hi, thank you for the tutorial, i haved follow this. but i get an error, I can not access the website under reverse proxy from other vm.
any suggestion?