[ale] iptables issues with dual NIC'd hosts?

Discussion:

leam hall via Ale

2018-01-26 18:01:23 UTC

Using RHEL 6, two hosts (A, B) each with two NICs, Each host has one
NIC on each of two VLANs. Tomcat on Host_A rying to connect to MySQL
on Host_B, port 3306. iptables on Host_B looks open (0.0.0.0) for
TCP/3306.

Host_A_NIC_0 can connect to Host_B_NIC_0 TCP/3306
HOST_A_NIC_1 can NOT connect to HOST_B_NIC_1 TCP/3306.

They are 1 IP off and NIC_1 can ping NIC_1, but not connect TCP/3306.

Thoughts on how to figure out why when iptables looks open?

Leam
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Phil Turmel via Ale

2018-01-26 18:21:13 UTC

Permalink

Post by leam hall via Ale
Using RHEL 6, two hosts (A, B) each with two NICs, Each host has one
NIC on each of two VLANs. Tomcat on Host_A rying to connect to MySQL
on Host_B, port 3306. iptables on Host_B looks open (0.0.0.0) for
TCP/3306.
Host_A_NIC_0 can connect to Host_B_NIC_0 TCP/3306
HOST_A_NIC_1 can NOT connect to HOST_B_NIC_1 TCP/3306.
They are 1 IP off and NIC_1 can ping NIC_1, but not connect TCP/3306.
Thoughts on how to figure out why when iptables looks open?

I don't see any flaw in the iptables rules you posted. /-:
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Dustin Strickland via Ale

2018-01-26 18:26:43 UTC

Permalink

What interface is MySQL listening on

Post by leam hall via Ale

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Dustin Strickland via Ale

2018-01-26 18:28:03 UTC

Permalink

Disregard, I misread the issue. I'm stumped

Post by Dustin Strickland via Ale
What interface is MySQL listening on

Post by leam hall via Ale

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Lightner, Jeffrey via Ale

2018-01-26 18:56:51 UTC

Permalink

I donât see any iptables rules posted in this thread. Did I miss them?

Are you sure your outbound traffic to port 3306 on hostB NIC1 is going out hostA NIC1 and not out hostA NIC2? If you do tcpdump on hostB NIC1 do you see any attempts to reach the port? How are you verifying access is failing?

Is it possible MySQL itself has a security setting prohibiting hostA NIC2âs IP? On our Oracle installations we have to be sure Oracle has been configured to allow connections from specific servers or IPs on top of doing iptables rules.

From: Ale [mailto:ale-***@ale.org] On Behalf Of Dustin Strickland via Ale
Sent: Friday, January 26, 2018 1:28 PM
To: Phil Turmel; Atlanta Linux Enthusiasts
Subject: Re: [ale] iptables issues with dual NIC'd hosts?

Disregard, I misread the issue. I'm stumped

On Jan 26, 2018 12:26 PM, "Dustin Strickland" <***@gmail.com<mailto:***@gmail.com>> wrote:
What interface is MySQL listening on

I don't see any flaw in the iptables rules you posted. /-:
_______________________________________________
Ale mailing list
***@ale.org<mailto:***@ale.org>
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jim Kinney via Ale

2018-01-26 19:12:00 UTC

Permalink

Sounds like a routing problem. ip route will show the defaults. If BOTH
are not pointed at each other, nothing happens. Verify with tcpdump on
both ends - look for traffic to/from <host>
Host A has nics 1 & 2 (A1 & A2)Host B has nics 1 & 2 (B1 & B2)
Assumption is A1 and B1 are on network 192.168.0.0 and A2 and B2 are on
10.1.1.0. Assumption default route is 192.168.0.0.
To get those machines to talk on the 10.1.1.0 network, you will need to
use explicit IP address and adding a custom name in /etc/hosts is a
good idea.
Also need to verify that the database is listing on the correct IP -
ditto for tomcat.
I just spent _days_ trying to trace a multi-homed network FSCKUP
through iptables. Data in on port A never appears anywhere else.
tracing it through just showed where it vanished - between PREROUTING
RAW and PREROUTING NAT. I feel your pain.

Ed Cashin via Ale

2018-01-26 19:27:15 UTC

Permalink

By "tracing it through" do you mean looking at the counts for the iptables
rules, and noticing which rules incremented and which did not?

Tracing with tcpdump is great for debugging, but I don't see how that would
catch things getting stopped between chains inside the kernel---that's why
I ask.

Post by Jim Kinney via Ale
Sounds like a routing problem. ip route will show the defaults. If BOTH
are not pointed at each other, nothing happens. Verify with tcpdump on both
ends - look for traffic to/from <host>
Host A has nics 1 & 2 (A1 & A2)
Host B has nics 1 & 2 (B1 & B2)
Assumption is A1 and B1 are on network 192.168.0.0 and A2 and B2 are on
10.1.1.0. Assumption default route is 192.168.0.0.
To get those machines to talk on the 10.1.1.0 network, you will need to
use explicit IP address and adding a custom name in /etc/hosts is a good
idea.
Also need to verify that the database is listing on the correct IP - ditto
for tomcat.
I just spent _days_ trying to trace a multi-homed network FSCKUP through
iptables. Data in on port A never appears anywhere else. tracing it through
just showed where it vanished - between PREROUTING RAW and PREROUTING NAT.
I feel your pain.
Using RHEL 6, two hosts (A, B) each with two NICs, Each host has one
NIC on each of two VLANs. Tomcat on Host_A rying to connect to MySQL
on Host_B, port 3306. iptables on Host_B looks open (0.0.0.0) for
TCP/3306.
Host_A_NIC_0 can connect to Host_B_NIC_0 TCP/3306
HOST_A_NIC_1 can NOT connect to HOST_B_NIC_1 TCP/3306.
They are 1 IP off and NIC_1 can ping NIC_1, but not connect TCP/3306.
Thoughts on how to figure out why when iptables looks open?
Leam
_______________________________________________
See JOBS, ANNOUNCE and SCHOOLS lists athttp://mail.ale.org/mailman/listinfo
--
James P. Kinney III Every time you stop a school, you will have to build a
jail. What you gain at one end you lose at the other. It's like feeding a
dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark
Twain http://heretothereideas.blogspot.com/
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
Ed Cashin <***@noserose.net>

leam hall via Ale

2018-01-26 19:30:56 UTC

Permalink

The iptables rules were almost the default "wide open" wiht a specific
line for port 3306 as 0.0.0.0. accept.

The two machines are on the same vlan, no routing except the host.

Post by Ed Cashin via Ale
By "tracing it through" do you mean looking at the counts for the iptables
rules, and noticing which rules incremented and which did not?
Tracing with tcpdump is great for debugging, but I don't see how that would
catch things getting stopped between chains inside the kernel---that's why I
ask.

Post by Jim Kinney via Ale
Sounds like a routing problem. ip route will show the defaults. If BOTH
are not pointed at each other, nothing happens. Verify with tcpdump on both
ends - look for traffic to/from <host>
Host A has nics 1 & 2 (A1 & A2)
Host B has nics 1 & 2 (B1 & B2)
Assumption is A1 and B1 are on network 192.168.0.0 and A2 and B2 are on
10.1.1.0. Assumption default route is 192.168.0.0.
To get those machines to talk on the 10.1.1.0 network, you will need to
use explicit IP address and adding a custom name in /etc/hosts is a good
idea.
Also need to verify that the database is listing on the correct IP - ditto
for tomcat.
I just spent _days_ trying to trace a multi-homed network FSCKUP through
iptables. Data in on port A never appears anywhere else. tracing it through
just showed where it vanished - between PREROUTING RAW and PREROUTING NAT. I
feel your pain.
Using RHEL 6, two hosts (A, B) each with two NICs, Each host has one
NIC on each of two VLANs. Tomcat on Host_A rying to connect to MySQL
on Host_B, port 3306. iptables on Host_B looks open (0.0.0.0) for
TCP/3306.
Host_A_NIC_0 can connect to Host_B_NIC_0 TCP/3306
HOST_A_NIC_1 can NOT connect to HOST_B_NIC_1 TCP/3306.
They are 1 IP off and NIC_1 can ping NIC_1, but not connect TCP/3306.
Thoughts on how to figure out why when iptables looks open?
Leam
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Lightner, Jeffrey via Ale

2018-01-26 20:00:47 UTC

Permalink

When running "iptables -nL" Is your 3306 rule under "Chain INPUT (policy ACCEPT)" and above the end of that chain that reads:
"REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited"?

If you add the line to the INPUT chain but put it beneath the REJECT line it never gets there as the rules are read top to bottom.

-----Original Message-----
From: Ale [mailto:ale-***@ale.org] On Behalf Of leam hall via Ale
Sent: Friday, January 26, 2018 2:31 PM
To: Atlanta Linux Enthusiasts
Subject: Re: [ale] iptables issues with dual NIC'd hosts?

The iptables rules were almost the default "wide open" wiht a specific line for port 3306 as 0.0.0.0. accept.

The two machines are on the same vlan, no routing except the host.

Post by Ed Cashin via Ale
By "tracing it through" do you mean looking at the counts for the
iptables rules, and noticing which rules incremented and which did not?
Tracing with tcpdump is great for debugging, but I don't see how that
would catch things getting stopped between chains inside the
kernel---that's why I ask.

Post by Jim Kinney via Ale
Sounds like a routing problem. ip route will show the defaults. If
BOTH are not pointed at each other, nothing happens. Verify with
tcpdump on both ends - look for traffic to/from <host>
Host A has nics 1 & 2 (A1 & A2)
Host B has nics 1 & 2 (B1 & B2)
Assumption is A1 and B1 are on network 192.168.0.0 and A2 and B2 are
on 10.1.1.0. Assumption default route is 192.168.0.0.
To get those machines to talk on the 10.1.1.0 network, you will need
to use explicit IP address and adding a custom name in /etc/hosts is
a good idea.
Also need to verify that the database is listing on the correct IP -
ditto for tomcat.
I just spent _days_ trying to trace a multi-homed network FSCKUP
through iptables. Data in on port A never appears anywhere else.
tracing it through just showed where it vanished - between PREROUTING
RAW and PREROUTING NAT. I feel your pain.
Using RHEL 6, two hosts (A, B) each with two NICs, Each host has one
NIC on each of two VLANs. Tomcat on Host_A rying to connect to MySQL
on Host_B, port 3306. iptables on Host_B looks open (0.0.0.0) for
TCP/3306.
Host_A_NIC_0 can connect to Host_B_NIC_0 TCP/3306
HOST_A_NIC_1 can NOT connect to HOST_B_NIC_1 TCP/3306.
They are 1 IP off and NIC_1 can ping NIC_1, but not connect TCP/3306.
Thoughts on how to figure out why when iptables looks open?
Leam
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Leam Hall via Ale

2018-01-26 22:31:54 UTC

Permalink

Lots of good answers, thanks!

The question is on a work network that I don't have access to so I'll be
back at it Monday. Will let you know what I find out.

Leam
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jim Kinney via Ale

2018-01-26 19:58:16 UTC

Permalink

I actually setup an iptable trace target
example:
iptables -t raw -A PREROUTING -p tcp --destination <myexternalhostip>
--dport 80 -j TRACE #tweak the rules to select just traffic you want
iptables -t raw -A OUTPUT -p tcp --destination <myinternalhostip> --
dport 80 -j TRACE #uses internal IP now since after the DNAT rule
Now tail -f /var/log/messages
Oh. Need to set
kern.*;*.info;mail.none;authpriv.none;cron.none /var/log/messages in
/etc/rsyslog.conf and restart rsyslog.On Fri, 2018-01-26 at 14:27

Post by Ed Cashin via Ale
By "tracing it through" do you mean looking at the counts for the
iptables rules, and noticing which rules incremented and which did
not?
Tracing with tcpdump is great for debugging, but I don't see how that
would catch things getting stopped between chains inside the kernel
---that's why I ask.

--
James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo