Problem:
Connectivity problems caused by duplicate IP addresses
Background:
There are several ways to identify IP address conflict, and we address a few symptoms in this note:
Web Browser Connections
If your browser arbitrarily disconnects while in use, try reloading the page. If this results in an error, try returning to the previous page. Whatever was cached will generally load. New pages may work or fail, depending on changes in the ARP data.
SSH Connections
Sometimes after the SHH connection starts, it spontaneously ends in an error. The error can occur at any point during start up, but your SSH (or SCP) client will usually print, “network error: software cause connection abort.”
Telnet Connections
A Telnet connection can drop quickly and quietly, without printing a cryptic error message to let you know why the connection failed.
Special Notes about WebMux IP Adresses and Conflicts
IP address conflicts with the WebMux can happen on the administrative interface. They may also occur on the farm IP addresses or server IP addresses.
Administrative Access IP Address Conflict
Strange on-again, off-again access to the management interface is likely caused by an address conflict.
Remember, you can access your farm IP addresses for administrative access using SSH, Telnet, HTTP, and HTTPS ports. This is one reason why the defaults (77, 87, 24, and 35) are not the industry standards. Try to establish a connection by SSH to a farm on port 77, or HTTPS to a farm on port 35.
Farm IP Address Conflict
A Farm IP Address Conflict will exhibit similar behavior as the Administrative Access IP Address Conflict. Troubleshooting will be the same, but remember that the MAC address of the farm is the same as the WebMux.
Back-end Farm Server IP Address Conflict
If there is a conflict with the back-end farm servers, your load-balanced server will show their own MAC addresses in conflict. It is easy to lose track of addresses and accidentally assign a server to the same IP address as the farm. The Linux commands “arp” and “arping” will show the same MAC addresses for the WebMux and the server.
Confirming The Conflict
Once you have the symptoms and know which IP address has troubling connecting, you can determine which machines are using the same IP address. The arp command tells which IP address associates with which MAC address. Run arp several times to watch for IP addresses with changing MAC addresses. Unfortunately, arp returns a long list of machines, and the one of interest will be buried.
However if you use Linux, run “arping” multiple times with the parameter “-D.” The parameter will prevent duplicate IP addresses by detecting addresses in use before assigning a new address.
Run arp, or arping, until you see both MAC addresses involved. The first run will report one machine, and another run will show a different machine’s MAC address on the same IP address as the first machine.
Here is an example of how it works. We use 10.11.12.13 as the suspected duplicate IP address.
If you enter:
arping -I bro -D 10.11.12.13
the command will return something like:
ARPING 10.11.12.13 from 0.0.0.0 bro
Unicast reply from 10.11.12.13 [00:E0:81:42:21:41] 0.623ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)
The second run might report a second machine, or it may report the same machine as the first run:
arping -I bro -D 10.11.12.13
ARPING 10.11.12.13 from 0.0.0.0 bro
Unicast reply from 10.11.12.13 [00:E0:81:42:21:41] 0.621ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)
Eventually, a different MAC address will show up:
arping -I bro -D 10.11.12.13
ARPING 10.11.12.13 from 0.0.0.0 bro
Unicast reply from 10.11.12.13 [00:22:12:f0:02:3a] 0.624ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)
Resolving The Problem
If you have a record of the MAC addresses on your network, look up which machines are in conflict. Otherwise, you can look up the first half of the MAC address to identify the manufacturer. If your WebMux was made before 2013, the address will show up as CAI Networks or its NIC supplier. Since 2013, the manufacturer’s address translates to AVANU. The manufacturer information might give you a clue as to which machines are conflicting.
Once you determine the machines involved, use the appropriate process to change an IP address.
Changing An IP Address on A WebMux
To reset the IP address on the WebMux to resolve the conflict with 10.11.12.13, follow these steps:
- Turn on the WebMux, but do not connect to the network.
- Wait for it to boot. You will see the Webmux version information first, followed by its scrolling status information (connections, packets, bytes, RAM, CPU, etc.).
- Hold the Checkmark button for several seconds to transition to configuration mode.
- Hit the Checkmark button until you see the line with the address “11.12.13.”
- Press left or right to move the cursor to the field you want to change. (HINT: If you want to change the “13,” hit the left arrow one time and it should be at that octet. Press up once to change it to “14” then press the Checkmark button to continue.) Each octet is edited separately, so if you have to change more than one, just use the left-right keys to get there and the up-down to change the value.
- Once you have changed the IP address and hit the Checkmark button until you are at the “reboot” Hold the checkmark button until it says it is rebooting.
Give the WebMux a minute or two to reboot and you will be able to reach it across the network with the new IP address.