Prevent Internal Access to Public IP From Going to Admin Pane

I have a server at home which I want accessible via my public IPv4. I set up port forwarding and I am able to access my server via SSH and http perfectly when connected to VPN or a hotspot. However, when I try to access the same domain or IPv4 address from within my local network, I get sent to the ARRIS SURFboard admin panel. Is there a way to disable this behavior and have it such that I can only access the admin panel via 192.168.0.1? ARRIS SBG8300