The Ugly, Hidden and Underestimated Costs of Building an On-Premise HPC System
Depending on your current and future HPC and organizational demands, each system offers benefits and limitations that need to be defined and compared. One of the main comparisons between systems is usually the Total Cost of Ownership (TCO). As I mentioned in a previous blog post, TCO not exactly a good fit for making buying decisions between fundamentally dissimilar alternatives. The TCO of on-premise HPC systems has been discussed for +30 years, even by our VP of Sales in his blog “The Real Cost of High-Performance Computing.” For people who are considering buying on-premise HPC systems, there are some hidden expenses that are often overlooked when calculating the TCO of and on-premise HPC system.
In this post, I intend to break down the TCO of an on-premise system and expose some expenses that may be overlooked.
A quick review on TCO
The broad definition of an on-premise HPC system’s TCO is that you sum the amount of all direct and indirect expenses correlated with your prospective system. The more obvious expenses are hardware, software, staffing, and power. For hardware, you need the following: servers, wiring, ToR switches, aggregation switches, server racks, power distribution units, etc. Then you must buy software that coordinates the communication between each node to solve complex problems. In addition, you must buy licenses for the software you plan on using. A resource that can be extremely variable and hard to estimate is the staffing required to develop, deploy, and maintain the on-premise HPC system. Finally, on-premise HPC systems require a lot of power and cooling capabilities: it is essential to calculate your energy consumption and how it will affect your operational expenses. Take the sum of the expenses for the items above and you have the basic TCO for your on-premise HPC system; however, there are some hidden costs that can heavily affect the TCO of your on-premise system.
Real-world, Hidden Costs
#1 The facilities hosting your HPC systems have cost dependencies that reach further than at first glance. Ensuring your facility has the proper cooling and power provisions necessary to support the current system and its potential scalability can save a lot of expenses down the road. Power is a major expense and can be extremely impactful on your overall operating expense. Depending on cluster location and utilization, your power costs can vary greatly. Due to your location, you may also see highly variable power prices that will heavily affect how you operate your HPC system to minimize expenses. In some cases, power can become over 1/3 of your operating expenses. Facilities and energy are important to consider when calculating your TCO and, for a large facility, should be considered a primary concern.
#2 Staffing will cost and vary more than you think, with performance and uptime suffering if neglected. One of the most variable and elusive expenses to define is the staffing for on-premise HPC systems. It can be very difficult to find, hire and train good Operations and IT Managers that can perform the development, deployment, and maintenance of an HPC system. Designing an HPC system requires expensive specialists to match the best hardware and software for your computing demands. The procurement of the system alone can cost as much as 5% of the total HPC system and takes at least 6 months. During this time, you must continue paying specialists to assemble the cluster while receiving no reward for the HPC system. Once deployed, the systems require very specific IT staffing to ensure its’ maintenance and operation. These employees require specialized skills to test and protect your HPC system’s longevity and performance. Finding the right employees to perform these functions can be cumbersome and costly, but is a priority when considering deploying an on-premise HPC system.
#3 Underutilization costs more than just the idle time, the associated overhead is substantial as well. An idle HPC system not only lowers your ROI, but can have devastating impacts on your product development cycle. Back-up systems can be overlooked because they are not considered necessary expenses to have an operating HPC system; however, the consequences for not having them can be dire. Generators, switches, gas, and maintenance of your backup energy system are all necessary to ensure that your systems are protected from power outages. Comparable to back-up energy provisions, back-up hardware is extremely important to mitigate an idle HPC system. Spare hardware is important to have on hand in case there is an issue; without backup hardware, you can find your system sitting idle while the part is repaired or bought. If you fail to plan, you should plan to fail; this is especially true for running an on-premise HPC system.
#4 Finally, on-premise technology is a constant uphill (and usually losing) battle. This is the harm caused by not utilized the best technology, and having to spend enormous efforts and capital to race to keep up. When comparing HPC systems, you have to acknowledge the costs and rewards, and their effect on each other. Not using the best technology can create expenses that stem from forfeiting rewards that are given by the best system. The expenses correlated to not using the best HPC solution are: lost productivity, missed innovation, longer time-to-solution, technology refresh cost, IT risk management, and increased IT debt and commitment. The most harmful forfeited reward is inefficiency in the research pipeline which creates a plethora of expenses correlated to the increase in time-to-market, delay in innovation, and increase in researcher idle time. The lack of HPC technology can cause your organization to have irreparable implications such as not being able to research larger problems and missing innovations that can make your organization uncompetitive. These expenses are often difficult to calculate because you have to assess how much more efficient your team will be with a better HPC solution and then work backwards to calculate the expenses correlated to inefficiency.
In summary, finding the true TCO of an on-premise HPC system can prove very difficult when considering all the hidden costs: staffing, facilities, power consumption, backup provisions, and forfeited rewards. I argue that one of the most important expenses to consider when comparing HPC systems is the expenses caused by forfeited rewards; however, these prove to be the most difficult to calculate and predict. The topic of TCO comparisons between cloud-enabled and on-premise HPC systems has been discussed regularly and is still not clearly defined. It is a comparison that we are working to improve, so if you have any comments or questions on this blog post or TCO, we would love to hear what you think.
Sara Jeanes. (2017, June 19). Cloud vs. Datacenter Costs for High Performance Computing (HPC): A Real World Example. Retrieved from: https://www.internet2.edu/blogs/detail/14114
Tony Spagnuolo. (2015, January). The Real Cost of High Performance Computing. Retrieved from: https://rescale.com/blog/the-real-cost-of-high-performance-computing/
Wolfgang Gentzsch. (2016, March 6). A Total Cost Analysis for Manufacturers of In-house Computing Resources and Cloud Computing. Retrieved from: https://community.theubercloud.com/wp-content/uploads/2016/04/TCO-Study-UberCloud.pdf
