No it is not a hole in a line. When 4x4 is engaged no vacuum is applied to the IWEs so a hole would only affect you when trying to disengage.
More likely the solenoid has an issue and is not shutting off vacuum upon engagement of 4x4.
You need a vacuum gauge to do some diagnosis. They can usually be borrowed from an auto parts store on a loan a tool program.
Check the vacuum line done at the IWEs to ensure that when the switch is in 4x4 that there is NO vacuum. If there is vacuum I would replace the solenoid. While you have the gauge check to ensure that both IWEs hold vacuum as well as the lines.