Recently at work, we had an issue where one of our guest networks wasn’t allowing new hosts onto the network. After some investigation, it appeared that we had simply run out of available IP addresses in the DHCP pool. We added 100 addresses and thought we were done until we noticed that within seconds those 100 addresses had
Unfortunately, within hours the new /24 was also consumed. At this point we began to suspect a Denial of Service attack, but we were wrong. Looking at the DHCP server for patterns or possibly spoofed MAC addresses we discovered something we weren’t looking for. Having exported the current DHCP lease table to Excel, we noticed that there were only 900 or so records, but the DHCP server was reporting 2000+ used IP addresses. This got us to distrust the GUI.
After some quick Google-Fu we found some Powershell commands to examine the DHCP database.
Get-DhcpServerv4Lease -scopeid 10.1.2.0 -AllLeases | Select-Object IPAddress,LeaseExpiryTime,AddressState
What we noticed when running the Powershell was that it showed all 2000+ addresses. The addresses that were missing from the GUI showed a state of Active, but their LeaseExpiryTime was old enough that the should have been in a state of Expired. In reading the documentation to figure this out, we learned a few things. First
To fix this situation we first tried running a PowerShell script to export the current database to CSV. We used a text editor to remove all of the entries where the LeaseExpiryTime was less than the current lease time. We then ran another PowerShell script to import the edited CSV to remove the old address leases.
Get-DhcpServerv4Lease -scopeid 10.1.2.0 -AllLeases | Select-Object IPAddress,LeaseExpiryTime,AddressState | Export-Csv -Encoding ascii -Path .\leases.csv Import-Csv -Path .\leases.csv | Remove-DhcpServerv4Lease
Unfortunately this still did not mark the addresses as Expired or Available. Some more Google-Fu led us to some indications that this might be related to issues with DHCP failover between this server and the standby. We removed the failover relationship and tried the Remove-DhcpServerv4Lease command again. This time all of the addresses that were older than the LeaseExpiryTime were marked as Expired. Four hours later they were all removed from the database and our IP pool was back to normal.
I would like to say that this all happened within an hour or so, but it was in actuality spread out over 48 hours with 5 engineers chasing our tails. Hopefully putting this out on the Internet will help someone else in the future.