Welcome to this special update on recent changes to eHive’s web security in response to recent outages.
Dear eHive users,
Thank you for being valued members of the eHive community and for your continued support of eHive. In this special edition of our newsletter, we would like to address recent issues with site performance and reliability. We’ll discuss what happened, why it happened and what we’ve done to address the issues at hand.
Summary
We have seen a huge increase in the number of eHive pages viewed by automated processes (bots) and in some cases the number of requests was so large that it has affected the performance of the website. On 28 August, we added a new layer of protection – Amazon’s Web Application Firewall – into the system to help us block these unwanted requests and get eHive running smoothly again. eHive is currently stable and working as expected.
What happened?
During late July and August we saw a few large spikes in internet traffic pointing at ehive.com. Traffic simply means how many times a computer tries to reach a page on eHive. When you go to the site or execute a search, you’re creating a tiny bit of traffic to the site. The spikes we saw during July involved hundreds of thousands of requests in a very short time period, many of which were complex searches for large sets of records. This caused the site to run slower and to be offline for brief periods (something we call an ‘outage’). We were able to address these issues by restarting a few services behind the scenes.
On 1 August, we received a spike in internet traffic significant enough to cause eHive to go offline for users. This spike was the biggest we had seen to date by some margin. Our development team worked on this and blocked a large number of IP addresses which were the origin of the increased traffic. At this point we came up with plan for how we want to address this issue moving forward.
On 27 and 28 August, we received an even larger spike in internet traffic causing eHive to go offline. Once back online, we experienced unreliable performance and further smaller outages. This outage occurred before we were able to execute the plan we had designed following the 1 August incident.
Why is this happening?
The nature of the internet is changing. Over the past twelve months the type of internet traffic all websites are receiving is altering. Current estimates are that only half of the traffic to an internet site is from genuine users (in our case, genuine researchers interested in your collections). The rest of the traffic is generally from something we call bots.
There are good bots and bad bots. The most obvious example of a good bot is the one Google use to index eHive. This allows them to know what content eHive has, and then show it to people searching on Google. Without it, people wouldn’t find your records. Similar bots exist for other search engines like Microsoft’s Bing and for other technologies that we like and are beneficial to us.
Bad bots are one of the internet’s current biggest issues. It’s difficult to tell in many cases what they’re being used for, but one example might be someone who wants to look at your website, scrape its information in bulk and use it to train their language models for an AI project. Typically a bad bot makes such a huge request for information from a website it creates a massive spike in traffic. Bad bots can also be used by people with malicious intent. The bots create such huge spikes in internet traffic to your site that its servers fail, causing your site to go offline. Such malicious attacks are referred to as Distributed Denial of Service attacks (DDOS attacks), though there are other ways malicious attacks can occur.
It is not possible to determine if the eHive outages on 1 and 27-28 August were bad bots or an explicit DDOS attack.
What have we done to address the issue?
The outage which occurred on 1 August highlighted the limitations we had in blocking the number of IP addresses necessary to thwart a spike in traffic caused by bots or bad actors. During that spike, our development team had to apply rules blocking certain IP addresses access to the site, which in turn prevented their traffic and the load to decrease.
The key point here is that our team had to manually select and block these addresses. We identified that we would like to use automated technology available from Amazon Web Services (AWS) called the Amazon Web Application Firewall (WAF) to help us block bad bots and bad actors quickly and at scale. We already recently implemented this technology on Vernon Systems’ sister web product Vernon Browser to great effect.
The plan we designed following the 1 August outage was to implement the same for eHive. The Web Application Firewall is critical as it maintains a live list of IP addresses which are the origin of bad bots and bad actors on the web. The move to using these services on eHive required some time to rewrite and reconfigure some key bits of eHive’s systems behind the scenes.
The outage on 27 and 28 August happened before we were able to execute this plan. The scale of the spike in traffic was such that our previous methods of blocking IP addresses was not sustainable. The team took the decision to immediately implement the change to using AWS tools and our developers worked on this collectively long into the night. As a result the new AWS tools are in place and are preventing bad actors impacting the site. eHive is currently stable and working as expected.
Our next steps are to spend more time carefully optimising these tools for eHive, as we had originally planned to do, in a more managed and measured manner. Work is ongoing in this area and will be for a little while. However, we now have the best tools available in place and working on eHive to try and prevent these issues from happening again.
Our commitment to you as users of eHive is always to do the best we can to provide you with a stable set of tools to look after your collections. We apologise for the issues we have seen over the past little while, but we are confident we’re now on the right track to continue supporting you in your work.
Thank you again for supporting eHive.
Kindest regards,
The eHive team
