One of the best things about programming in Python is the language’s sheer scope. The language is capable of advanced math, complex string manipulation, and even working with those concepts over the Internet. This has made Python an ideal choice for Web scraping.
Python’s libraries make it easy to load raw data from the web and extract the necessary information from it. Likewise, it’s easy to format raw HTML into a usable state with Python’s various functions. But this can bring along some unexpected error messages. One of the most common is a “no connection could be made because the target machine actively refused it” error. But you’ll soon find out what the error means and how to fix it.
What Does the Error Mean?
Many Python error messages point to the exact problem in your code. But the “no connection could be made because the target machine actively refused it” error can be a little misleading. At first glance, it implies that a program has actively refused a connection made by your client code. That can be the case. But the error message can also show up when a connection fails for any reason. The error doesn’t really imply active rejection in the way it suggests. It should instead be thought of a notification that your network connection has failed for a variety of different reasons.
A Closer Look at Python Networking
When Python’s socket library doesn’t receive a reply to its query it will usually assume that the connection was rejected. A server rejecting a connection, a firewall closing a port, and a total lack of a compatible server service will typically produce this error message. However, there are instances where server software might actively reject a connection. This is especially common with scraping, the process of extracting information from web pages for later processing.
Scraping is often done on dozens or even hundreds of pages on a single domain at the same time. And server software can interpret this as a potential DDoS attack. It’s not uncommon for server software to simply reject connections made in rapid succession. Because of the wide variety of possible causes, there’s an equally wide variety of solutions. But working through the most common fixes will take care of the vast majority of instances where the error appears.
Fixing the Error
Unfortunately, it’s essentially impossible to fully reproduce the error message in a way that’ll be applicable to most people’s individual situations. But we can consider hypotheticals. For the sake of argument, assume that we’re receiving this error while trying to scrape content from google. The actual content received from this process won’t be very useful as google is simply a useful service that won’t mind a little additional traffic while debugging code. But with that in mind, consider the following example.
import pandas as pd
while True:
myUrl= ‘https://www.google.com/’
pf = pd.read_html(myUrl)[0:10]
print(pf)
For the sake of the example, assume that Google didn’t like the fact that we were making constant requests and began rejecting the attempts. We could easily fix that problem with the following change.
import time
import pandas as pd
while True:
myUrl= ‘https://www.google.com/’
pf = pd.read_html(myUrl)[0:10]
print(pf)
time.sleep(5)
In this Python code, we set things up in a similar way to the original attempt. Our code loads google’s search page and then sends the result to Pandas. Pandas scrapes some basic information and then prints it to the screen. However, we then see the fix. The time module delays further requests by five seconds. This should be enough to ensure sites don’t mistake the scraping as a DDoS attack. And this will fix the “no connection could be made because the target machine actively refused it” error.
Now, let’s consider another common situation where connections might be rejected out of concern for automated attacks. Some sites check user agents to ensure that an actual human is browsing their content rather than a bot. This is often fairly easy to get around. Assume that, once again, Google is blocking our attempt to load the main search page. We’ll use a slightly different method to get around the problem. Consider the following code.
from bs4 import BeautifulSoup as bs
import requests
headers = {‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36’}
ourUrl = ‘https://google.com’
f = requests.get(ourUrl, headers=headers)
soup = bs(f.text,’html.parser’)
print(soup)
In this example, we create a fake browser user agent to spoof our scraping attempt. Google will now think that we’re loading up their page with the Chrome browser on Windows. If Google was blocking the attempt based on user-agent then we’ll now be able to cleanly load its content. The data is then sent to BeautifulSoup for processing and printed to screen.
Finally, there are also a few areas where the problem might not have anything to do with your code at all. Remember that this error can occur when a connection fails for any reason. And that includes instances where outgoing connections are simply blocked. This is especially common in office environments where Internet access is strictly limited.
You might simply be looking at a situation where browsers are set to use a proxy or similar system but your code isn’t. Some operating systems and routers might also be overzealous in what they allow as outbound traffic. You can often track this down by simply using ping on your system’s command line. We can use Google as an example again. You’d simply type the following.
ping HTTP://google.com
If the ping doesn’t go through successfully then it suggests overzealous blocking of outgoing data. This means that your system’s networking configuration will need to be reconfigured to provide you with broader access.