iptables + multiple gateways not firewalling

[Ian Coetzee]

On Sat, Dec 12, 2009 at 1:36 PM, Werner Schram <wrschram at gmail.com> wrote:
> Hi Ian,
>
> Ian Coetzee wrote:
>> Hi all
>>
>> I have a small problem here (I may have just overlooking something here)
>>
>> I have a ubuntu 9.10 server here. On it I do web proxing through one
>> provider, and mail through another.
>>
>> I had it set up with one NIC and it worked (in my testing
>> environment). However I am unable get get it to work, after I inserted
>> another NIC to route the mail out through.
>>
>> I *think* I narrowed it down to the fact that ip tables is not
>> firewalling eth1(which is the new NIC I inserted).
>>
>> I can ping both my NIC's subnets that is connected to them.
>>
>> Any help will be greatly appreciated.
>>
>> My config is as such (public IPs is changed due to lurking eyes :) )
>>
>> eth0 (internal) -> 192.168.1.3/24 gateway 192.168.1.15
>> eth1 (external) -> 111.111.111.52/29 gateway 111.111.111.49
>>
>> iptables rules:
>>  cat /etc/iptables.up.rules
>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>> *mangle
>> :FORWARD ACCEPT [0:0]
>> :INPUT ACCEPT [0:0]
>> :OUTPUT ACCEPT [0:0]
>> :PREROUTING ACCEPT [0:0]
>> :POSTROUTING ACCEPT [0:0]
>> #mark outgoing tcp/25 packets for routing
>> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
>> COMMIT
>> # Completed on Sat Dec 12 11:02:07 2009
>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>> *nat
>> :OUTPUT ACCEPT [0:0]
>> :PREROUTING ACCEPT [0:0]
>> :POSTROUTING ACCEPT [0:0]
>> #Set source address on outgoing packet eth1
>> -A POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52
>> COMMIT
>> # Completed on Sat Dec 12 11:02:07 2009
>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>> *filter
>> :FORWARD ACCEPT [0:0]
>> :INPUT ACCEPT [0:0]
>> :OUTPUT ACCEPT [0:0]
>> #log incomming and outging traffic on eth1
>> -A OUTPUT -o eth1 -j LOG
>> -A INPUT -i eth1 -j LOG
>> COMMIT
>> # Completed on Sat Dec 12 11:02:07 2009
>>
>> ip ru sh
>> 0:      from all lookup local
>> 32765:  from all fwmark 0x10 lookup mail
>> 32766:  from all lookup main
>> 32767:  from all lookup default
>>
>> ip ro sh table mail
>> default via 111.111.111.49 dev eth1
>>
> Ok, thats looks pretty complicated. Anyway what I understood (I did some
> guessing about where your second ISP connection would be): You have
> three networks:
> - your local network (network 192.168.1.0/24 with NAT gateway
> 192.168.1.3 (your server) to ISP1 and with NAT gateway 192.168.1.15 to ISP2)
> - ISP1 for smtp (network 111.111.111.32/29 with gateway 111.111.111.49)
> - ISP2 for the rest (network unknown with gateway unknown)
>
> And this is what you want:
> - All traffic from the local network is routed to your server (default
> gateway for hosts is 192.168.1.3).
> - Your server (192.168.1.3 and 111.111.111.52) has two routing tables,
> all traffic is routed by the default table except for port 25 traffic
> (which is marked with 0x10) which is routed by the 'mail' routing table.
> - Iptables on your server applies NAT on your mail traffic, and the mail
> routing table sends traffic to ISP1 (via 111.111.111.49).
> - All other traffic is routed by the default routing table to
> 192.168.1.15 which will presumably perform NAT by itself and send it to
> ISP2 (via a unknown gateway).
>
> Is this correct?

Yes, you have this correct.

>
> I don't understand this rule:
>> #mark outgoing tcp/25 packets for routing
>> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
>>

You are correct yes, however it is also our main mail gateway, ie it
revcieves mail as well, that is what the -sport is for.

I just noticed where there may be a flaw, I will change and get back
to you (after I ate lunch :) )

> You are marking all outgoing traffic with source port 25, but smtp
> traffic has destination port 25. Shouldn't this rule read:
> -A OUTPUT -p tcp -m tcp --dport 25 -j MARK --set-mark 0x10
> or even better:
> -A OUTPUT -p tcp -m tcp -i eth0 --dport 25 -j MARK --set-mark 0x10
>
> If the above description is correct, then I think that the "-A
> POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52" rule is breaking

It is necessary so that the other end know what my IP is for back
routing, else it sends all packet as if from 192.168.1.3.

> stuff. It is NATing *all* traffic (where it should only NAT port 25
> traffic). Possibly 192.168.1.15 will only NAT traffic from the
> 192.168.1.0/24 subnet, but because of the iptables rule, traffic will
> come from 111.111.111.52, which is thus ignored. This is remedied by
> changing the rule to:
> -A POSTROUTING -o eth1 -j SNAT --dport 25 --to-source 111.111.111.52
>
> I hope I understood your setup correctly, and this message is of any
> help ;)
>
> BTW, for debugging these kind of problems, tcpdump and/or wireshark can
> be of great help.
>
> Werner
>
> --
> ubuntu-users mailing list
> ubuntu-users at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>