PR Crisis Microsite Deployed in Cloud in Few Hours

At Česko.Digital community, we helped to create a crisis information website for Czech Ministry of Health about the coronavirus outbreak in the Czech Republic. The website has since become one of the main information channels for all Czech citizens. This article describes how we modified the website with the Ministry’s IT dept. to handle expected load.

During PR crises of corporations (e.g. faulty service or product), main company websites often become overloaded. This situation typically leads to an increased amount of inbound telephone calls and overloading the call centers. Changing the main company website usually takes some time (they are built on enterprise CMS) and scaling the customer telephone lines is not so easy (and really expensive). All of the above are reasons why most of companies choose to create a crisis-dedicated microsite or move communication to external platform (e.g. Medium).

A microsite solution (looks more professional) needs a decision where to run it. I don’t see any better scenario for a website to be deployed into the cloud – we need to have website scale-able and quickly deployed. I am sure there are even better tools you could use to build the crisis website, but for our solution we had to make a compromise – sometimes because of usability but most importantly because of the time constraint – what is easiest to set up, with what we have the most experience to be able to launch MVP (Minimum Viable Product) as soon as possible.

We wanted to have a solution which would be stable and we didn’t want to make the ministry’s PR department go through a Markdown language training, and so we decided to use WordPress. This CMS brings a friendly user interface but it’s a bigger challenge to configure it to be secure and ready for high performance.

The solution described below is designed to be deployed in Microsoft Azure. It can be also easily ported to AWS (CloudFront, S3, etc.). We selected Azure because most enterprise companies have Azure implemented in their organizations (usually because Azure Active Directory or Office 365).

Solution Architecture
Solution is built on idea, standard users don’t interact with CMS, only but with static HTML files generated by CMS. Thanks to this approach, we can easily achieve the desired performance and security.

WordPress (WP) is installed on a virtual server, which is isolated and accessible only via VPN (or IP filter at least). WP installation is configured for a different URL, than the destination URL of the website. Without this WP installation doesn’t work well. When the author generates static files (output is located on the server’s filesystem), our custom built script is triggered – this script syncs these generated files to Azure Blob Storage via Shared Access Signature (this could be simplified by mounting Blob storage as directory on filesystem).

Azure CDN is placed In front of Blob Storage. This service provides caching of content to the closest physical nodes (and in fact solves all our performance needs). When the author is satisfied with the exported version of articles, the author performs CDN cache invalidation and all readers will see the updated version in a few minutes. Thanks to saving SSL private key in Key Vault, the Azure CDN can behave as our website.

The diagram also shows a „registered app in AAD“. This application is granted to perform the CDN cache invalidation and provides access for our custom WP plugin – this plugin enables to trigger the invalidation from WP user interface via the Azure REST API.

Limitation of this solution is a lack of interactive content – e.g. search, poll etc. However, these features are usually not needed for a crisis website.

Solution Details and Usage
We used plugin WP2Static for static files export. The plugin works as a crawler. It crawls all pages and articles, generates static HTML files from them and downloads all other necessary files (e.g. images, CSS files, etc.). The plugin replaces URL of WP installation with production website URL.

Publishing the new content version was initially performed manually by running a bash script (in early stages of MVP).Few hours later we developed with Vladimir Smitka custom WP plugin (for cache invalidation) and script, which detects new export and uploads it on Blob Storage. This combination enables the authors to make a deployment by themselves from WP directly.

I also recommend setting up HTTPS-only redirection on Azure CDN (+redirection from a www. subdomain) by custom rules in the Rules engine feature.

Result
The website was pretty fast even with 11K concurrent active users. I believe Azure CDN capacity could handle a bigger amount but we haven’t needed it yet.

Despite this fact, the website was stable, according to Hlídač webů service (a monitoring service of the government websites). The graph below shows the moments when content was deployed. The website slowed down at those points.

Graph showing results on monitoring

Checklist Before You Start

• Ensure you have an Azure access with sufficient permissions – including all mentioned Azure services above are enabled (Azure Policy/de-registered service type in subscription)
• Ensure you have an access to your Azure AD – usually other department have to create application in AAD and assign role for given CDN instance
• Set up DNS records
• If you want to use a custom SSL certificate, ensure you have private key ready (Azure CDN is able to generate SSL certificate by itself, only needs to have set up DNS records to CDN)
• In case you have a custom SSL certificate, ensure you have an access to AAD to set up the Azure CDN so that it has permissions to access the private key in Key Vault.

General Tips for a Crisis Website

• Reduce TTL of DNS records to 5 minutes, in case you will need to change those quickly.
• Do the same change also for HSTS settings. In case you will misconfigured HTTPS, so you don’t want to cut off all your readers
• If your microsite will be subdomain crisis.acmecompany.com, be sure also to establish a redirection from www.crisis.acmecompany.com. The URLs are often preceded with www. by the press regardless of the canon URL form
• Set up Google Analytics. Even if you don’t want to evaluate results (what is searched by visitors), it’s good to know what the traffic was and for what traffic size you need to prepare your main website

High-availability Version

To get this solution ready also for the possibility of an Azure region outage (where your website is running), I also describe this solution in HA version. However, for the basic crisis websites the solution described above is sufficient.

The entire solution is duplicated also to another failover Azure region. An Azure Traffic Manager is placed in front of the Azure CDN. This service performs a health check and in case of failure of the primary region it automatically switches to the failover region. The Azure Site Recovery service is used for replication of VM with WP administration.

The script and the plugin requires a small modification to perform all changes on both Azure Storage and CDN.

Source Code and Settings
You can find the source code of WP plugin, the script and the necessary settings on my GitHub.

Thanks to Adam J. Sporka for text correction of the article.

Vaclav Jirovsky, Cloud Solution Architect, MSD IT