iptables + multiple gateways not firewalling

[Ian Coetzee]

On Sat, Dec 12, 2009 at 1:49 PM, Ian Coetzee <ubuntu at iancoetzee.za.net> wrote:
> On Sat, Dec 12, 2009 at 1:36 PM, Werner Schram <wrschram at gmail.com> wrote:
>> Hi Ian,
>>
>> Ian Coetzee wrote:
>>> Hi all
>>>
>>> I have a small problem here (I may have just overlooking something here)
>>>
>>> I have a ubuntu 9.10 server here. On it I do web proxing through one
>>> provider, and mail through another.
>>>
>>> I had it set up with one NIC and it worked (in my testing
>>> environment). However I am unable get get it to work, after I inserted
>>> another NIC to route the mail out through.
>>>
>>> I *think* I narrowed it down to the fact that ip tables is not
>>> firewalling eth1(which is the new NIC I inserted).
>>>
>>> I can ping both my NIC's subnets that is connected to them.
>>>
>>> Any help will be greatly appreciated.
>>>
>>> My config is as such (public IPs is changed due to lurking eyes :) )
>>>
>>> eth0 (internal) -> 192.168.1.3/24 gateway 192.168.1.15
>>> eth1 (external) -> 111.111.111.52/29 gateway 111.111.111.49
>>>
>>> iptables rules:
>>>  cat /etc/iptables.up.rules
>>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>>> *mangle
>>> :FORWARD ACCEPT [0:0]
>>> :INPUT ACCEPT [0:0]
>>> :OUTPUT ACCEPT [0:0]
>>> :PREROUTING ACCEPT [0:0]
>>> :POSTROUTING ACCEPT [0:0]
>>> #mark outgoing tcp/25 packets for routing
>>> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
>>> COMMIT
>>> # Completed on Sat Dec 12 11:02:07 2009
>>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>>> *nat
>>> :OUTPUT ACCEPT [0:0]
>>> :PREROUTING ACCEPT [0:0]
>>> :POSTROUTING ACCEPT [0:0]
>>> #Set source address on outgoing packet eth1
>>> -A POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52
>>> COMMIT
>>> # Completed on Sat Dec 12 11:02:07 2009
>>> # Generated by iptables-save v1.4.4 on Sat Dec 12 11:02:07 2009
>>> *filter
>>> :FORWARD ACCEPT [0:0]
>>> :INPUT ACCEPT [0:0]
>>> :OUTPUT ACCEPT [0:0]
>>> #log incomming and outging traffic on eth1
>>> -A OUTPUT -o eth1 -j LOG
>>> -A INPUT -i eth1 -j LOG
>>> COMMIT
>>> # Completed on Sat Dec 12 11:02:07 2009
>>>
>>> ip ru sh
>>> 0:      from all lookup local
>>> 32765:  from all fwmark 0x10 lookup mail
>>> 32766:  from all lookup main
>>> 32767:  from all lookup default
>>>
>>> ip ro sh table mail
>>> default via 111.111.111.49 dev eth1
>>>
>> Ok, thats looks pretty complicated. Anyway what I understood (I did some
>> guessing about where your second ISP connection would be): You have
>> three networks:
>> - your local network (network 192.168.1.0/24 with NAT gateway
>> 192.168.1.3 (your server) to ISP1 and with NAT gateway 192.168.1.15 to ISP2)
>> - ISP1 for smtp (network 111.111.111.32/29 with gateway 111.111.111.49)
>> - ISP2 for the rest (network unknown with gateway unknown)
>>
>> And this is what you want:
>> - All traffic from the local network is routed to your server (default
>> gateway for hosts is 192.168.1.3).
>> - Your server (192.168.1.3 and 111.111.111.52) has two routing tables,
>> all traffic is routed by the default table except for port 25 traffic
>> (which is marked with 0x10) which is routed by the 'mail' routing table.
>> - Iptables on your server applies NAT on your mail traffic, and the mail
>> routing table sends traffic to ISP1 (via 111.111.111.49).
>> - All other traffic is routed by the default routing table to
>> 192.168.1.15 which will presumably perform NAT by itself and send it to
>> ISP2 (via a unknown gateway).
>>
>> Is this correct?
>
> Yes, you have this correct.
>
>>
>> I don't understand this rule:
>>> #mark outgoing tcp/25 packets for routing
>>> -A OUTPUT -p tcp -m tcp --sport 25 -j MARK --set-mark 0x10
>>>
>
> You are correct yes, however it is also our main mail gateway, ie it
> revcieves mail as well, that is what the -sport is for.
>
> I just noticed where there may be a flaw, I will change and get back
> to you (after I ate lunch :) )

Ok after rethinking my strategy, I made some changes and now I am
logging packets, and this is what I saw

Dec 12 14:05:50 proxy kernel: [11676.519480] IN= OUT=eth0
SRC=192.168.1.3 DST=196.43.0.142 LEN=60 TOS=0x10 PREC=0x00 TTL=64
ID=60433 DF PROTO=TCP SPT=54928 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
MARK=0x10

Notice that it is marked, but still gets routed out through eth0....

I think my routing table is flawed :(

Regards
Ian

>
>> You are marking all outgoing traffic with source port 25, but smtp
>> traffic has destination port 25. Shouldn't this rule read:
>> -A OUTPUT -p tcp -m tcp --dport 25 -j MARK --set-mark 0x10
>> or even better:
>> -A OUTPUT -p tcp -m tcp -i eth0 --dport 25 -j MARK --set-mark 0x10
>>
>> If the above description is correct, then I think that the "-A
>> POSTROUTING -o eth1 -j SNAT --to-source 111.111.111.52" rule is breaking
>
> It is necessary so that the other end know what my IP is for back
> routing, else it sends all packet as if from 192.168.1.3.
>
>> stuff. It is NATing *all* traffic (where it should only NAT port 25
>> traffic). Possibly 192.168.1.15 will only NAT traffic from the
>> 192.168.1.0/24 subnet, but because of the iptables rule, traffic will
>> come from 111.111.111.52, which is thus ignored. This is remedied by
>> changing the rule to:
>> -A POSTROUTING -o eth1 -j SNAT --dport 25 --to-source 111.111.111.52
>>
>> I hope I understood your setup correctly, and this message is of any
>> help ;)
>>
>> BTW, for debugging these kind of problems, tcpdump and/or wireshark can
>> be of great help.
>>
>> Werner
>>
>> --
>> ubuntu-users mailing list
>> ubuntu-users at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>>
>