This guidance explains in detail how the system will be hosted when it is hosted by UNICEF. The term “hosting” refers to where the application lives. This is a very technical guidance of the implementation. The “hosting environments” described in the previous section are here explained in terms of their component parts. This “architecture”–the pieces used to build the system–was chosen for the benefits it provides, including data backups and security. This guidance also includes details of how the system will be secured from internal and external vulnerabilities, and the ongoing requirements for keeping the system secure.
It is assumed that the technical deployment activities are conducted by software vendors, supported by the Primero System Administrator and other IT staff as necessary, in collaboration with UNICEF.
The primary Primero instance is deployed to a hosted server that can be accessed via the internet. This setup will provide a consolidated repository of case management data.
The server is a single, central location where the application runs, making it easier to support, upgrade, and secure. Users will access this application via HTTPS from browsers on their desktops or laptops.
Each Primero instance (installation) must be maintained for the following:
- Application version
- Configuration
- User management
- Operating system security: fault monitoring, virus scanning, certificates, encryption
- Hardware security
Internet connectivity is a prerequisite for hosted Primero use. Before a site can start using the new case management system, connectivity at the site will have to be ensured. For use in field environments without strong connectivity, Primero is delivered as a progressive web application (PWA) with support for offline usage.
It is understood that consistent connectivity and electricity will not be always possible. In those situations, Primero may still be used as the progressive web app works offline and data can be saved on battery powered devices. Not all features are available when the application is offline. Any feature or functionality that requires access to the server will temporarily be disabled, and indicated on the user interface as greyed-out/unavailable. The user interface will also indicate “offline”. When connectivity returns, these features will automatically be reenabled.
If hosted by UNICEF, the live production instance of Primero will be hosted on UNICEF’s Microsoft Azure cloud subscription. This platform meets the following requirements:
- The ability to run Ubuntu 20.04 servers
- Server hardware support
- High availability of system services
- Data persistence and durability
- Ease of secure remote SSH access to a super user account (passwordless, secured by RSA keys) by Primero support staff
Deployment
At the core of the system is the Centrally Hosted Primero instance. The system will be accessible to end users via HTTPS, and optionally over an encrypted VPN connection. To be clear, the word “central” refers to the central location in the technical infrastructure and the instance’s role as the master repository of all data. Here the word “hosted” means that the central instance is hosted on server hardware and accessible over the internet. This instance can be cloud hosted. “Cloud” means that a third party (in this case Microsoft) is providing the physical server infrastructure and is responsible for the physical operation and maintenance of the data center and hardware.
The central instance will be accessible directly via the web.
The SSL certificate which is used to authenticate and encrypt communications between the central hosted Primero instance and the users accessing it via browser clients will be provided by the UNICEF procured Thawte USA (https://www.digicert.com/) certificate (that is embedded in Primero. This SSL certificate will need to be renewed, but as long as Primero is hosted by UNICEF this cost will be covered.
Building the central hosted Primero instance
The centrally hosted Primero instance relies on virtual application servers and supporting infrastructure (such as virtual machines, security access rules, private networks; see Deployment Architecture section below for details). This instance will be built, configured, and functional in advance of any on-the-ground deployment work.
During the Beta rollout, the primary instance will be hosted on UNICEF’s Azure subscription to Microsoft Azure.
This virtual server will be provisioned using Chef https://www.chef.io, a build automation tool. Chef codifies server configuration, allowing identical, repeatable system deploys to take place.
The Chef tool and associated Primero deployment configurations need to reside on a hosted machine (not the Primero server), which is remotely accessible by support staff. UNICEF is already running an existing Chef management server.
The following will take place to deploy the core application to this infrastructure:
- Primero source code is hosted in a private Git repository: https://github.com/primeroIMS/.
- The source code is versioned and tagged for specific releases.
- The core application will be deployed using Kubernetes.
- The target operating system will be Ubuntu Server 20.04 (64-bit).
- Chef deployment scripts (bundled along with the core application code) will deploy the code, build and deploy the application dependencies (such as Ruby, Postgres, Java, Solr, Nginx), and configure the central Primero instance.
- While Chef does the bulk of the application build and setup, it will need to be configured and triggered.
- The Chef configuration (which includes core system and database passwords)
When the system is up and running on the cloud hosts, the DNS records will need to be updated to point the domain name to the newly allocated IP address. Any DNS mail configuration will need to be performed at this time.
Access using RSA public/private keypairs via SSH will be granted to designated individuals within the software vendor, UNICEF, and other allocated IT staff to this server.
Although this portion of the deployment will be performed by UNICEF and/or the software vendor, detailed instructions for these deployment steps will be provided as supporting documentation. An operational understanding of Linux and cloud infrastructure will be required to perform this portion of the deployment.
System Architecture & Infrastructure
The recommended deployment architecture for hosted instances attempts to balance system performance, security, ease of use, and ease of distribution. It assumes that while the initial system usage will be relatively low, the usage will quickly grow and scale with the expansion of the national system.
The Hosted Primero Instance Infrastructure, the following considerations are addressed:
- Geographic location.The server will be located in a Microsoft Azure data center in Frankfurt, Germany.
- Performance and Virtual Hardware. While the initial set of users will be relatively small, the application will still run on a robust virtual machine. This machine should meet the following minimum requirements:
- 8+GB memory
- Roughly two CPU cores (2.5 GHz, Intel Xeon)
- Storage: 500GB or more. Primero v2 puts no physical cap on the number of image and PDF attachments. If attachments are heavily used by the program (signed documents, images), the server storage capacity should be able to grow
- Ubuntu 20.04
- Stack deployment. Primero is deployed as a monolithic stack: Application and database share the same server.
- Storage. The application and system files and the backing data are physically separated into two mounted volumes. Each volume is snapshotted nightly and stored for 15 days.
- Scalability. The initial instance roll out will take potential future system scalability into account. Each hosted central Primero instance is designed to be horizontally scalable in both numbers of application servers and Postgres nodes. If the situation calls for it, a separate database server could be created to sit alongside the existing servers. The server itself may also be strengthened to handle greater load.
- Security. Data exchanged with the internet will be encrypted using SSL/TLS.
Data Backups
The data backing the system is stored in Postgres data files and accompanying Lucene index files (used by Solr). It is recommended for automated disk snapshotting to take place on a regular basis. It is recommended that the backup be triggered nightly, and images from the last 15 days be preserved. If possible, the snapshots should be stored with the cloud service account, and are accessible only by those users granted access.
Note: that there is no special mechanism to back up data on mobile devices. If a mobile device breaks, and the data has not yet been synchronized to the server, all stored cases and incidents will be lost.
System Maintenance
The system will require continuing application support, including software and system updates. Updates will need to occur regularly, and will fall into the following categories:
Primero software new features and bug fixes. Primero source will be versioned and tagged. When a critical mass of desired functionality builds up, and the time is deemed right for a country instance to upgrade (perhaps during a relative lull in usage), a release will be scheduled in coordination with UNICEF.
Updates to system dependencies. These include security, performance, and bug fix updates for Primero dependencies. Security patches will need to be released as soon as possible, and a process must be determined for active monitoring and assessing of and responding to emerging security threats. Primero is deployed to Ubuntu 20.04. Automated system upgrades can be enabled to make sure the latest security patches are continually applied without external intervention.
Updates to virtual hardware. Cloud infrastructure allows us to treat hardware updates similarly to software updates. If a virtual server will be deemed to have less resources (CPU, memory, disk) allocated to it than necessary, the virtual resources will be upgraded. Downtimes can be minimized by careful failovers to prepared and provisioned new machines.
Updates to metadata. The form and field metadata drives the application. While Primero will support light configuration (adding new fields, updating possible lookup values), any field or form deletions or heavy field rearrangement must be handled as a Primero application update, complete with specialized and scripted data migrations to avoid unintentional loss of data
Cloud infrastructure allows for a reasonably painless upgrade process. Updates to Primero, cloud server instances, or even the cloud architecture will be handled during scheduled down-times. A detailed, step-by-step release plan will be drawn up by the development team and executed by a server administrator.
Technical Security Considerations
The security approach for the deployment architecture is to expose only the bare minimum to the outside world, to encrypt it when exposed, and to restrict resource access to specific actors. However, there are some points to keep in mind:
Use of HTTPS. Secure Primero communication over the internet is authenticated and encrypted via the SSL/TLS protocol. It relies on certificates issued by Certificate Authorities, usually private companies, subject to national laws. This may be considered a potential risk. Recently several security flaws have been identified in the SSL/TLS protocol itself. It will be very important to maintain the application infrastructure in order to run the latest patched and fixed versions of all security software. Basic HTTPS can be expanded to use client certificates in the future.
Disk Encryption. This is also known as at-rest encryption.
Explicit regulation of network traffic. Using firewalls or equivalent cloud services to lock down the types of traffic permissible on the system. Primero systems should only be able to communicate with other Primero instances, backup services, and be accessible for maintenance from trusted parties.
Application audit. The application logs are available for full review. They note each HTTP request for Primero with the requesting IP address, logged in user, and affected record id. In addition, Primero provides a simple UI to review actions performed by users.
User management and data ownership. Careful management of application users, and explicit ownership and permissions of data by users will ensure greater tightness of data security. It is important to be vigilant about disabling unused or compromised accounts, requiring strong passwords, and not over-assigning users to data. A single poorly-managed compromised account can put a lot of data at risk.
Timely security upgrades. Security upgrades must be regularly delivered both to the operating system and to the Primero application server and underlying components.
System Disaster Recovery
The system can suffer from failure on a number of fronts:
- Some malfunction can occur in the data center. This would make the central Primero application instances unavailable.
- Disk failure in the data center which would impact the Postgres data.
- Loss of internet connectivity in a location that was assumed to have solid connectivity. In the short term there might not be anything to do but wait until local connectivity is restored. In the long term, if the connectivity becomes less stable than was initially assumed, then the location may need to switch to using a local Primero instance.
- Malicious intrusion into the central instance machines by a hacker
Recovering the Central Instance
In the highly unlikely event that a failure occurs at a Microsoft data center with the primary Primero instance, the application server will need to be recreated from a recent ESB snapshot. Disk snapshotting technology is provided by Azure. The platform supports assignment of stable Azure DNS addresses which can in turn be used to sustain a stable public DNS name. If impossible to restore from a snapshot, the application instance can be recreated from scratch using Chef.
Malicious Intrusion
To mitigate impact from intrusion, the centrally hosted instance architecture disperses system functionality, and forces explicit permissioning for communication between these functional components. If an application or database server was compromised and perhaps infected, the best approach would be to rebuild the infected machines and recreate system user certificates and keys.