AnsweredAssumed Answered

Problem with my IP Address Being Blocked

Question asked by Burch Kealey on Oct 16, 2017
Latest reply on Oct 16, 2017 by Burch Kealey

For the second time that I am aware of you have blacklisted my home IP address.  Specifically when I try to visit some websites where I am a customer I get an Access Denied message.  The first time this happened I did not really understand the root cause and after replacing all of my hardware between the cable and the switch (modem and router) my ISP was kind enough to make sure I had a new IP address and then I was fine.  However it happened again this weekend.

 

I believe I was blacklisted because I run some web scraping from my home IP address.  However, my scraping is against the SEC EDGAR website - my behavior is allowed and expected by the SEC.  Here is some information from the SEC about using their website.  Notice that they ask we use efficient scripting etc.  Therefore they are aware that we use scripting tools to access their content. 

 


https://www.sec.gov/edgar/searchedgar/accessing-edgar-data.htm


Fair Access

To ensure that everyone has equitable access to SEC EDGAR content, please
use efficient scripting, downloading only what you need and please moderate
requests to minimize server load. The SEC reserves the right to limit
request rates to preserve fair access for all users. See Security and
Intrusion Detection <https://www.sec.gov/privacy.htm#security>.
I have never 'scraped' AA.COM or USPS.COM - rather I am a customer of
theirs and I am frustrated that your system is  using my activity to a
third-party unrelated site to block off my access to some critical services
that I rely on. 

 

I am reluctant to post my IP address in this public forum.  However if you need more information you can see the case I attempted to register yesterday  [AKAM-CASE #F-CS-2419181]

 

If you don't have access to the case easily I just tried to access AA.com again and the message returned was the following

 

You don't have permission to access "http://www.aa.com/" on this server.

Reference #18.54a75b68.1508167270.29b7015

 

I can't access USPS.com, MACYS.com or AA.com as well as a host of other sites.  While I understand that they are your customers and they set some levels of filtering it is based on your observations of my web activity.  My web activity is harmless and allowed.  I have been doing this scraping for more than a decade as it involves the collection of data for my research and for my customers. 

 

There are rumors that you use some AI process to evaluate/snoop on traffic.  If those are true you need to ignore activity against sites such as the SEC's website as those were designed for users like me to access using scripts.

 

Please remove my IP address from your blacklist.  I cannot wait for an interval when I am not scraping (I was heavily scraping this past weekend and while I usually scrape on the weekends I do sometimes work in the middle of the week).  I needed to prepare some shipping labels using the USPS website and I wanted to check some flight arrangements on AA.com and for both of these yesterday I had to borrow a neighbor's WiFi.

 

Thanks

Outcomes