iptables + multiple gateways not firewalling

[Werner Schram]

Hi Ian,

Ian Coetzee wrote:
> Hi all
>
> I have a small problem here (I may have just overlooking something here)
>
> I have a ubuntu 9.10 server here. On it I do web proxing through one
> provider, and mail through another.
>
> I had it set up with one NIC and it worked (in my testing
> environment). However I am unable get get it to work, after I inserted
> another NIC to route the mail out through.
>
> I *think* I narrowed it down to the fact that ip tables is not
> firewalling eth1(which is the new NIC I inserted).
>
> I can ping both my NIC's subnets that is connected to them.
>
> Any help will be greatly appreciated.
>
> My config is as such (public IPs is changed due to lurking eyes :) )
>
> eth0 (internal) -> 192.168.1.3/24 gateway 192.168.1.15
> eth1 (external) -> 111.111.111.52/29 gateway 111.111.111.49
>
> iptables rules:
>  cat /etc/iptables.up.rules
> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
> *mangle
> :FORWARD ACCEPT [0:0]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> :PREROUTING ACCEPT [0:0]
> :POSTROUTING ACCEPT [0:0]
> #mark outgoing tcp/25 packets for routing
> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
> COMMIT
> # Completed on Sat Dec 12 11:02:07 2009
> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
> *nat
> :OUTPUT ACCEPT [0:0]
> :PREROUTING ACCEPT [0:0]
> :POSTROUTING ACCEPT [0:0]
> #Set source address on outgoing packet eth1
> -A POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52
> COMMIT
> # Completed on Sat Dec 12 11:02:07 2009
> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
> *filter
> :FORWARD ACCEPT [0:0]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> #log incomming and outging traffic on eth1
> -A OUTPUT -o eth1 -j LOG
> -A INPUT -i eth1 -j LOG
> COMMIT
> # Completed on Sat Dec 12 11:02:07 2009
>
> ip ru sh
> 0:      from all lookup local
> 32765:  from all fwmark 0x10 lookup mail
> 32766:  from all lookup main
> 32767:  from all lookup default
>
> ip ro sh table mail
> default via 111.111.111.49 dev eth1
>   
Ok, thats looks pretty complicated. Anyway what I understood (I did some 
guessing about where your second ISP connection would be): You have 
three networks:
- your local network (network 192.168.1.0/24 with NAT gateway 
192.168.1.3 (your server) to ISP1 and with NAT gateway 192.168.1.15 to ISP2)
- ISP1 for smtp (network 111.111.111.32/29 with gateway 111.111.111.49)
- ISP2 for the rest (network unknown with gateway unknown)

And this is what you want:
- All traffic from the local network is routed to your server (default 
gateway for hosts is 192.168.1.3).
- Your server (192.168.1.3 and 111.111.111.52) has two routing tables, 
all traffic is routed by the default table except for port 25 traffic 
(which is marked with 0x10) which is routed by the 'mail' routing table.
- Iptables on your server applies NAT on your mail traffic, and the mail 
routing table sends traffic to ISP1 (via 111.111.111.49).
- All other traffic is routed by the default routing table to 
192.168.1.15 which will presumably perform NAT by itself and send it to 
ISP2 (via a unknown gateway).

Is this correct?

I don't understand this rule:
> #mark outgoing tcp/25 packets for routing
> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
>   
You are marking all outgoing traffic with source port 25, but smtp 
traffic has destination port 25. Shouldn't this rule read:
-A OUTPUT -p tcp -m tcp --dport 25 -j MARK --set-mark 0x10
or even better:
-A OUTPUT -p tcp -m tcp -i eth0 --dport 25 -j MARK --set-mark 0x10

If the above description is correct, then I think that the "-A 
POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52" rule is breaking 
stuff. It is NATing *all* traffic (where it should only NAT port 25 
traffic). Possibly 192.168.1.15 will only NAT traffic from the 
192.168.1.0/24 subnet, but because of the iptables rule, traffic will 
come from 111.111.111.52, which is thus ignored. This is remedied by 
changing the rule to:
-A POSTROUTING -o eth1 -j SNAT --dport 25 --to-source 111.111.111.52

I hope I understood your setup correctly, and this message is of any 
help ;)

BTW, for debugging these kind of problems, tcpdump and/or wireshark can 
be of great help.

Werner