Scaling your Blue Prism Platform
The following demonstrates how to scale a Blue Prism installation from a small-scale environment to an enterprise-sized environment
Most Blue Prism customers grow their capability very quickly once the benefits of RPA are realized. This growth can often happen too quickly, without the correct forethought, preparation and infrastructure, which can lead to performance issues and/or a loss in service.
The advice given here is intended to provoke the thought and actions required to make a smooth transition from a small to a medium or large sized Version 6 Blue Prism implementation.
Many customers choose to create new infrastructure when jointly scaling and upgrading. This is generally a good tactic, as many customers fall into the trap of starting out with a small environment and scaling within that environment until they reach the infrastructure capacity. Much of the advice in this page is geared around a concurrent upgrade and scale action.
Given the broad range of factors that can affect performance of the Blue Prism software, Blue Prism does not warrant that the Blue Prism software will perform to a particular standard, or that a particular configuration will meet the performance expectations of a customer. Blue Prism recommends that customers ramp up their environments in a carefully controlled manner in order to identify and resolve any factors that may impact software performance.
Key Actions
Numerous aspects need to be considered before embarking upon an upscale exercise. Each will be explained in more detail later in the section, but to summarize, the following are most important
- Infrastructure specification - Does your infrastructure have the capacity to grow to the size you predict?
- SQL considerations - maintenance, tuning, sizing etc.
- Ensure levels of security are maintained if adding more hardware, more users, more content etc.
- Data Archiving - Is your session log archiving procedure set up correctly and apt for you needs?
- Public Cloud considerations - Do you have the correct configuration?
- Load Balancing - Will you require load balancing in your larger environment?
- Control Room - Do you have enough, or too many Control Room personnel?
- Are you on the right Blue Prism version? Blue Prism, like all software has end-of-life dates.
Before you start
Before embarking upon a scaling implementation, it is important to have a clear indication of how large, in terms of the number of Runtime Resources and Licences, you expect to grow to. Although the utilization of Runtimes can and does differ from installation to installation a Blue Prism implementation can be arbitrarily classified in size by the number of Runtime Resources. This number does not have to be exact, but it is important to predict to the nearest 50.
In order to be able to make this prediction it is important that a comprehensive Process Discovery exercise has been completed. Find out more about Process Discovery.
Infrastructure considerations
Any single application server needs only to have a minimum specification of 2 CPU and 4GB of RAM, but you should increase this to 4 CPU and 8GB of RAM if using the Data Gateways component (introduced in Version 6.5). As the Application Server has a main purpose of marshalling connections between the Runtime Resources and the Database, scaling needs to be done horizontally rather than vertically.
As a rule of thumb, for performance reasons, one Application Server should manage a maximum 100 Runtime Resources. Therefore, before implementing over 100 Runtime Resources ensure that you have a second functional Application Server. Likewise, 200+ Runtime Resources require three Application Servers and so on.
Best practice is to use a 1:100 + 1 formula, to ensure in the case of losing one Application Server, you still have enough capacity to run your environment adequately.
Multiple Application Servers should be load balanced to ensure the best results. More details on load balancing are below.
For multiple Application Servers all Application Servers require the same encryption key, and that the Blue Prism scheduler is enabled on all servers. Having the scheduler run on multiple servers requires those servers to have the same date \ time setting.
Always ensure the Latency between the Application Server and SQL Server is the lowest it can be. High latency between these two components can cause performance issues.
The Database server is the key component to your Blue Prism infrastructure. The Blue Prism application performance is highly dependent upon having a well maintained and lean Database.
Your Production Blue Prism database should be dedicated solely to the Blue Prism application.
Many customers have under-specified SQL Servers. This usually stems from starting with a small Blue Prism environment and neglecting to scale the SQL Server as the number of Runtime Resource and Processes grows. Below is a table of the minimum specification that a SQL Server should be for several given environment sizes.
Consider moving to a Physical SQL server if you are implementing a large (200+) amount of Runtime Resources. Physical may, generally speaking, perform better than virtual servers. If using virtualized SQL server then it is recommended to conduct a review of the best practice guides.
- The VMWare guide can be found here.
- The Microsoft guide to virtualizing SQL Server can be found here.
Once migrated ensure you carefully monitor the capacity of the SQL Server. Later versions of Blue Prism may have a higher performance overhead than the current older version that you are running.
Ensure the SQL server is neither CPU nor RAM bound by monitoring closely. Also monitor Services, Disk, CPU, Memory usage, Availability of specific windows services, Availability of Standard SQL services, Windows event viewer for the SQL Server. Configure alerts to be triggered depending on certain thresholds being met.
Storage performance for Blue Prism SQL Servers should be a significant consideration. Medium scale (up to 200 Runtime Resource) environments should aim to have at least 800-1000 IOPS. High scale (200+ Runtime Resource) should have over 1000 IOPS, the greater the better. For this reason, SSD Hard Drives are recommended in any Medium or High scale environment.
Employ Instant file initialization for better performance. This is a User Rights Assignment, set through the Local Security Policy on the SQL Server and is not a SQL setting. This feature typically benefits the speed at which data files can be grown automatically by removing the step of zero-initialization from the process, whilst also increasing the speed of recovery.
If migrating to a new server, ensure that the DB has the right sizing before the migration. This will prevent the unnecessary overhead of growing the Data file constantly with auto-growth.
Also consider any future expansion, for example adding new Runtime Resources and Processes, when either migrating, scaling or upgrading.
The biggest threat to performance of any Blue Prism database is mismanagement of the database size, in particular, letting Session Logs go unchecked and table sizes growing beyond what can be reasonably managed.
Read a detailed account on Database Maintenance here.
Whilst Runtime Resource specification is not particularly relevant to scaling, it is important to ensure that the timely running of Processes is not restricted by hardware specification of the Runtime Resources.
As a general guide the following can be used. However, it is important to note that Runtime Resources should always be above the minimum spec stated by the applications that are being automated. A digital workforce is always likely to work faster than a human.
For geographically dispersed Interactive Clients. Use RDP sessions on ICs hosted local to the Application Server. This means the RDP session takes the latency, not the Blue Prism Client. This should ensure the best performance.
Blue Prism is well suited to run in Public Cloud environments such as Azure, AWS and Google Cloud.
Specific Blue Prism Reference Architecture documents are available for each which provide much greater detail. In summary the following should be considered when planning a migration to Public Cloud
- Consider the use of newer technologies such as the PaaS Database offerings. Azure SQL provides an easily maintainable and scalable solution for a small (10 Runtime Resource) to medium (Up to 200 Runtime Resource) sized implementation.
- Make use of the inbuilt HA\DR capabilities, such as Availability Sets in Azure, Regions and Availability Zones in AWS, Managed Instance Groups in Google Cloud.
- Consider the best solution for authentication in your Active Directory if SSO is desired.
Load Balancing between Application Servers and Runtime Resources should always be implemented to improve High Availability and Disaster Recovery. Hardware or Software Load Balancers are equally effective.
At the time of writing, Azure Load Balancers work fine with the default settings, but AWS will not due to Blue Prisms incompatibility with cookie-based Load Balancers.
For Citrix NetScalar Load Balancers it has been noticed the best results are obtained by using ‘sourceIPhash’ persistence.
Minimize the amounts of Control Room users you have. Check out our Control Room Best Practices, but overall the fewer the number, the lesser the impact this has on the performance of the environment. Consider the use of a mirrored database which could be used as a read-only reporting tool if required.
Ensure you are moving to the latest Version of Blue Prism. Performance increases and bug fixes are always rolled up into the latest version and incrementally improve the user experience.
Further Considerations
Filling your database with unnecessary \ defunct \ deprecated Processes and Objects is bad practice. You should aim to keep the minimum amount of Processes and Objects and employing good housekeeping practices. A failure to do so may impact the performance of your Control Room experience.
If migrating to a new database, only export and import the Processes and Objects you require.
Implement a strong peer review process to ensure best practice for processes and objects are being followed. Also, the testing and proving of objects and processes should include the validation of negative outcomes throughout each execution stage and graceful recover in these scenarios.
Implement Resource Pools can be used to take the load off your environment. When you create a Resource Pool one machine becomes the 'leader' and therefore acts like a pseudo-Application Server.
Resource Pools should only contain 5-25 Runtime Resources
Utilize Multi-Team Environments Functionality to segregate the ability for users to view and interact with components (such as processes, objects and Runtime Resources) that they need access to only. This will reduce interactions with the database in views such as process studio and control room and will also reduce number of TCP connections established on launch of the Interactive Client to only the Runtime Resources that users need access to.