Please send RFIs and RFQs to stan@securepower.io

APC White Paper 237: Digital Remote Monitoring and Dispatch Services’ Impact on Edge Computing and Data Centers

White Paper 237

Revision 1

by Victor Avelar 

Executive summary

Power and cooling infrastructure for edge computing and data center sites have roughly 3 times more data points / notifications today than it did 10 years ago. Traditional remote monitoring services have been available for over 10 years but were not designed to support this amount of data monitoring and the associated alarms, let alone extract value from the data. This paper explains how seven trends are re-defining remote monitoring and field service dispatch service requirements and how this will lead to improvements in operations and maintenance of IT installations.

Introduction

In this paper, we define remote monitoring as a service for checking the health of IT infrastructure equipment, performed by infrastructure vendors or third parties from afar. These services1 for IT power, cooling, micro data centers, and associated infrastructure systems have been around for over 10 years. But traditional offline services are limited compared to new digital2 services available today (see Table 1). These new services take advantage of cloud computing, data analytics, artificial intelligence (AI), and mobile apps. They also can directly link to maintenance (or “dispatch”) services for faster, more proactive equipment management.

A facility manager likely has no idea when they should replace a component in their UPS or cooling unit that might be about to fail. In contrast, a delivery truck driver gets an instant notification on their smart phone that their normal route is backed up 20 minutes with a recommended alternate route. This disparity has prompted us to look at how advancements and trends in IT are changing IT infrastructure monitoring and, in turn, how monitoring services will change data center and edge computing operations and maintenance.

The general concept of monitoring today is widely understood and anyone with a fitness tracker, continuous blood glucose monitor, or a learning home thermostat has had firsthand experience in how advances in IT have improved their lives. In particular, users benefit from immediate knowledge from their devices (e.g. calories burned, blood sugar level, etc.). However, most today are not benefiting from big data analytics and machine learning (AI). These and five other trends are poised to revolutionize how managers operate and maintain data centers.

This paper explains seven trends that are defining next-generation data center and edge computing remote monitoring services and its benefits. We describe the requirements to attain these benefits, and describe how operations and maintenance will evolve in the future.

Table 1 Comparison between traditional and digital remote monitoring

Trends influencing monitoring

Remote monitoring services available 10 years ago were desktop-based, limited in data output, and largely reactionary (i.e. depended on humans to interpret what was wrong). Digital remote monitoring has resolved these limitations through technology, and over the next few years more limitations will be addressed by technology. We see seven technology trends that are influencing data center monitoring.

  • Embedded system performance and cost improvements
  • Cyber security
  • Cloud computing
  • Big data analytics
  • Mobile computing
  • Machine learning
  • Automation for labor efficiency

We briefly describe these trends in this section and, in the next section, we describe the digital remote monitoring requirements needed to comprehend, mitigate, or take advantage of these trends.

Embedded system performance and cost improvements

Embedded systems are found in nearly all IT infrastructure devices including cooling units, PDUs, UPS, chillers, security cameras, etc. and basically control the operation of these devices. Without the outputs from these embedded systems, there would be nothing to monitor. Embedded systems have improved significantly over the years in terms of computing capability, data storage, communications, and pricing. This means data center devices today can provide much more data today than they could 10 years ago. We estimate that the total number of alarms and notifications available from power and cooling devices have increased over 300% over the last ten years. This increase comes from a combination of more sensors, more features, more algorithms, and higher sampling rates. The more data available, the more digital remote monitoring can infer helpful information from devices, as we describe later in the paper.

Cyber security

Cyber security is one of the biggest concerns5 among data center managers around the world. Not only are they concerned about IT equipment vulnerability, but also physical infrastructure equipment that has been exploited as “backdoors” into the IT network. Digital remote monitoring, as well as other cloud-based services, must comprehend cyber risks even before the product or service is created. Digital service providers need to demonstrate their secure development lifecycle (SDL) practices and policies. Ask for their SDL policy, and validate that the lifecycle includes phases that focus on training, security requirements, design, development (e.g. coding standards), verification, release, deployment, and response. In terms or architecture, there should be a single point of entry into your network using a gateway (usually software), and all devices communicate with the gateway. Figure 1 illustrates a recommended digital remote monitoring architecture.

There are several other factors that data center managers and security stakeholders must consider when evaluating a vendor and their digital remote monitoring service, therefore we discuss this topic further in White Paper 239, Addressing Cyber Security Concerns of Data Center Remote Monitoring Platforms.

Figure 1: Recommended digital monitoring architecture

Cloud computing

Cloud computing is a highly scalable method of storing and processing massive amounts of data. Cloud computing is what enables digital remote monitoring services. IT services such as predictive analytics and machine learning can run on a cloud computing platform to further increase the value of monitoring. Cloud-based remote monitoring platforms also provide a secure means by which managed service providers (MSPs) and infrastructure vendors can remotely monitor and manage equipment even when highly distributed across many sites. By merging the functions on the same digital platform, it also facilitates linking the monitoring service with maintenance dispatch services to make field servicing and replacement of equipment faster and less disruptive.

Big data analytics

Big data analytics may seem far from the mainstream but it applies to activities performed today such as condition-based maintenance (also referred to as predictive maintenance) for plane engines and predicting how many products manufacturers make for the holidays. A spreadsheet or database can only go so far to identify patterns in data. Big data analytics is required when6:

  • data volumes increase (e.g. petabytes of data)
  • data becomes unstructured (i.e. data variety like emails, free-form text fields, or trouble tickets)
  • data is processed in real-time (this is known as velocity)

Taking advantage of data analytics technologies available today will enable digital remote monitoring platforms to be more proactive and eventually predictive. This will reduce interruptions in IT service, and it will reduce your service costs overall.

Mobile computing

Global use of mobile phones to access the internet has grown year over year for the last several years while access through desktops has decreased year over year7. This trend applies also to data center managers who are increasingly asked to do more with fewer resources. Mobile computing helps alleviate this burden by allowing managers to float between locations without being disconnected from daily operations. A mobile app also enables you, the owner, to monitor the same data at the same time as the remote monitoring vendor.

Machine learning (AI)

Machine learning is related to data analytics in that it uses data to make predictions but it’s different in that it improves the model by using results from previous learning8. Machine learning can be used to drive an autonomous vehicle, recognize speech, recognize images, chose a Netflix movie, or accurately model the PUE of a very complex Google data centers. In all of these examples, the driving, the recognition, etc. improves over time.

Automation for labor efficiency

Automation for labor efficiency is not a “hot” trend but it’s particularly relevant to data center managers in an increasingly competitive business environment where they are being asked to do more with less. This is where automation through digital remote monitoring can help. Remote monitoring by the vendor or other 3rd party partner, in effect, automates management for you allowing you to focus labor resources elsewhere.

Digital monitoring benefits

The first trend in the previous section (Embedded system performance and cost improvements), creates an overarching challenge for data centers. The amount of data to track is increasing, rapidly making it harder for operations teams to interpret what it means and take the right actions. This is unsustainable, especially when you’re understaffed, or in the case of distributed edge computing sites, have no trained staff on site at all. Some other challenges managers face include:

  • A multitude of alarms from the same device when one alarm notification would have sufficed. This can actually cause alarm fatigue where the same repeated alarm will eventually be ignored due to human nature9.
  • Being understaffed or lacking onsite staff, managing maintenance and service activities - inspecting equipment, replacing batteries, swapping out failed equipment, etc. – becomes very burdensome, particularly if assets are highly distributed and geographically dispersed.
  • Calling customer support for help, dialing through a list of menus, waiting, getting someone who creates a trouble ticket but will likely have to escalate to resolve the problem.
A digital remote monitoring service that comprehends, mitigates, or takes advantage of the trends discussed above, can overcome these challenges and provide the following benefits. Digital remote monitoring requirements are provided for each benefit.
  • Reduced downtime / lower mean time to repair
  • Reduced operations overhead
  • Lowered cost of maintenance and services
  • Improved energy efficiency
  • Scalability

Reduced downtime / lower mean time to repair

Operation of IT equipment depends on stable electrical power, sufficient ventilation (or active cooling), as well as a secure location that is safe from unauthorized access or exposure to other physical and environmental threats. These dependencies mean that a highly resilient IT installation requires monitoring of the infrastructure equipment. If you are lacking the bandwidth or manpower to monitor, you risk downtime and will increase the time it takes to recover from incidents. Digital remote monitoring services can fill this need.

A review of downtime events typically reveals a series of state changes that collectively lead to downtime. In other words, a single failure event normally does not result in downtime. The whole point of monitoring data centers is to reduce the risk of downtime by identifying and addressing a state change before others occur. In this context, digital remote monitoring services should meet the following requirements.

  • Network operations center experts troubleshooting data center incidents should be screened and trained on cyber security. The more years of experience in offering digital remote monitoring, the more likely that an alarm, notification, or failure is resolved without causing downtime or making the problem worse. Experience in this case means that experts have learned through “near misses” over the course of their careers. Research in aviation and healthcare10 has shown that “near misses” are key to learning. Understanding and documenting why these incidents occurred reduce the risk of future errors.
  • Documenting all incidents must be part of any digital remote monitoring system.
  • The service should reduce break-fix resolution time through alarming, remote troubleshooting, and visibility into device lifecycle. This troubleshooting should be delivered by experts monitoring your data center 7x24.
  • Experts monitoring your sites should have a list of contacts to call in the event of a critical event. Managers should be able to update this list with the vendor at any time, ideally through a mobile app.
  • Compatibility with third-party devices improves the situational awareness of domain experts in the NOC. Knowing the status of all devices improves the chances of preventing or at least understanding the root cause of a problem or potential problem in order to provide faster resolution.
  • Predictive analytics and remote troubleshooting should reduce the number of times you need a service person working on your equipment. It’s all too common to hear about technicians showing up multiple times either because they needed help, didn’t have the right expertise, or didn’t have the right part. By understanding the problem fully, field service engineers can come prepared with the correct parts and tools thereby increasing the likelihood that something is repaired on the first visit.

Reduced operations overhead

The following are requirements that enable digital remote monitoring services to reduce operations overhead, leaving staff to focus on more important proactive tasks that add value to the business.
  • Vendor network operations center (Figure 2) staffed with the domain experts that support your data center(s) and/or distributed edge computing sites.
  • A service that connects the vendor’s field service operations with the monitoring service saves the subscriber time and effort. Vendor troubleshooting experts can directly and immediately connect and share site data with their field service personnel to schedule and dispatch for faster maintenance and unit replacement activities. By combining technical support and field services on to the same digital platform, operational efficiency is achieved.
  • A mobile app (Figure 3) allows data center managers and administrators immediate access to data and the status of their data center from anywhere at any time (not to mention peace of mind). Most people carry their phone therefore it’s logical that it be the primary means of receiving information related to the health of your data center. Logging into a desktop (sometimes requiring VPN) to troubleshoot a problem is time consuming and inconvenient. Vendors providing the remote monitoring service can have access to the same data at the same time.
  • Automatic trouble ticket generation should be provided through a mobile app. This can save a significant amount of time as it avoids tech support phone menus and explaining the same issue to multiple representatives. This aids significantly in reducing time to resolution. A related best practice is to track incidence via chats, messages, etc.

Fig 3 & 4: Example of a NOC and a digital monitoring app

  • Online chat via mobile app as a means to collaborate with the team as well as to gain instant access to domain experts in the NOC
  • Fast on-boarding means that in about 30 minutes you can install the gateway, auto discover devices, register the software, configure the smart phone app, and begin monitoring.
  • Manually entering devices to be monitored is time consuming and allows for human error. A digital remote monitoring system should auto-detect critical infrastructure devices using simple network management protocol (SNMP). Modbus TCP devices are not typically auto-detected because they need a device definition file (DDF). Gateways typically scan a range of IP addresses (user-specified), detect applicable devices, and present the data to the user.
  • Event processing is similar to how hospitals triage patients. The most critical alarms are prioritized in terms of notifications and actions. This practice reduces the burden on the data center operators knowing that the NOC experts will notify and guide them during an event that triggers multiple alarms.
  • Event correlation and root cause analysis evaluates multiple alarms and deduces possible causes and proposes possible solutions. This correlation process can be done by domain experts in a NOC or a combination of machine learning and experts. For example, one CRAH high temperature alarm may not be an issue, but six alarms on the same chilled water loop is likely a problem with the root cause being a closed supply water valve.
  • Alarm consolidation converts multiple alarms from the same device into a single incident. This practice avoids wasted time having to acknowledge multiple identical alarms. Furthermore, a workflow ticket should be automatically generated for this incident, to inform you of who is currently working on the issue, what’s been done so far, and to track its progress and eventual resolution.

Scalability

Scalability is the ability for the digital remote monitoring system to accept additional devices, or nodes, to monitor. Depending on how these systems are designed, monitoring may be limited to a few thousand devices. Scalability isn’t typically a problem for smaller data centers (e.g. 500kW IT load capacity) but is a serious problem for larger data centers or when there are a very large number of edge sites. Some data centers can have hundreds of thousands of devices to monitor and require polling every few seconds, therefore, a digital remote monitoring system should be designed using a horizontally-scalable, cloud-based architecture. This means that as more devices are monitored, the cloud service automatically adds more compute nodes to handle the monitoring. Operations managers need to identify their requirements and then understand the capabilities and limitations across the various monitoring services being evaluated.

The evolution of operations and maintenance

Use of embedded sensors in clothing, in watches, and in other “wearables” will allow doctors to predict when you’re getting sick or when you are at risk of a heart attack, and numerous other insights. By analyzing fuel consumption data, an airline can adjust its flight procedures like the position of its control surfaces to improve fuel efficiency. These are examples of the “Internet of Things” (IoT), where devices communicate with each other, through a gateway, micro data center, and or a cloud data center, ultimately adding value to our lives and our businesses.

With this backdrop, it’s easier to see how data centers of all shapes and sizes are fertile ground for improvement, made possible through the trends described in this paper and IoT in general. We see the following evolutions in operations and maintenance occurring over the coming years inside small edge computing sites and large data centers alike.

Evolution in operations

  • Just like autonomous cars are believed to experience less car accidents due to human error, so too will data centers experience less downtime due to human error. Reduction in downtime will be accomplished primarily through machine learning. As more data is collected on causes of downtime or near misses, digital remote monitoring systems will be able to predict when a data center is at risk of a downtime event occurring and provide data center operators appropriate steps to avoid it.
  • Remote monitoring services may evolve to include help with improving energy efficiency of the IT installations. It could be improved in two ways; more accurate device efficiency models and data center models. This accuracy will come as a result of data gathered from actual operation in different data centers, in different climates under different loads. The data center model, using machine learning, will eventually have enough data that it can suggest what cooling system settings will result in the lowest power consumption.
  • When the vendor’s NOC receives a data center alarm, the monitoring platform will be able to tell them what steps they need to take to correct whatever is wrong including scheduling and dispatch of appropriate field service personnel and replacement parts.
  • More complicated service procedures will be done on site with augmented reality (AR) technology where the vendor’s service technician wears a pair of special glasses and images appear instructing them on exactly what to do. AR will reduce errors and injuries.

Evolutions in maintenance

  • Traditional maintenance models charge customers for scheduled visits because manufacturers lack data and analytics to accurately predict when something will break or is running inefficiently. Data centers will move from calendar-based maintenance to condition-based maintenance. This will also encourage device manufacturers to use more sensors and algorithms that improve component failure prediction, improve contextual alarms, and ultimately reduce maintenance costs.
  • Manufacturers won’t need to rely on warranty cards and phone calls to track component failures. Instead, they will rely on a data lake and analytics that will provide them with rich insights, not only on component failures in the field, but how to improve the reliability of future products. The most compelling and valuable part of this evolution for data center managers is the speed at which this will occur. Today it takes much too long for manufacturers to gather enough data, to recognize a problem, then to understand what’s causing it, and finally to find a way to fix it.
  • The insights from field data and analytics will make field service visits more predictable. For example, there will be an increased likelihood that something is repaired on the first visit and lower risk of service defects (either during or after service is complete). Ultimately this translates into higher data center reliability and lower maintenance costs for data center managers.
  • Everything that field service technicians do will be logged and correlated with what has happened. By collecting enough of this data manufacturers will know that when they have a series of particular events, happen in a particular order, that it means a given action and or parts are required. This will evolve into a digital remote monitoring service automatically dispatching a field service technician with the correct work order and spare parts.
  • Particularly for those managing multiple, unmanned sites, modern digital remote monitoring services offer a field service dispatch option that simplifies

The value of the network

The term “network effect” gained widespread awareness during the rise of Facebook as a leading social network platform. The term basically means that as more people use a particular product or service, the more value users of that product or service will realize. The telephone is an often-used example of the network effect. If only one person in the world had a phone, there would be no value in it because they couldn’t talk to anyone else. But when millions of people have and use one, it becomes valuable. This is true of digital remote monitoring services.

If only one facility manager used a digital remote monitoring service like the one described in this paper, they wouldn’t get the value of data analytics and condition-based maintenance. That value is attained very quickly as more data centers and edge computing sites use the service and the collective data is analyzed to provide insights. For example, if 100,000 sites used the service, a large percentage of these data centers are likely to have an air-cooled packaged chiller cooling architecture. With this amount of data, analytics could suggest changes to their cooling system and the estimated savings these changes will have on the energy bill.

Conclusion

IT installations of all sizes are on a path to become more reliable and efficient through the use of digital remote monitoring services and condition-based maintenance made possible through technologies like big data and machine learning. However, this can only happen with platforms that take advantage of the data constantly generated by the physical infrastructure in a data center. Using vendor or 3rd party provided remote monitoring services can reduce downtime, reduce maintenance costs, and reduce staffing pressures. Using digital remote monitoring services that tie in automatic field service dispatch can reduce time to recover from equipment failures and minimize any service disruptions. Particularly for those managing distributed IT assets across multiple site with no trained onsite staff, service dispatch services become more valuable. Data center and IT operations managers should review the digital remote monitoring requirements provided in this paper as they begin to assess their own data center evolution and consider outsourcing remote management of their infrastructure equipment.

About the author

Victor Avelar is the Director and Senior Research Analyst at Schneider Electric’s Data Center Science Center. He is responsible for data center design and operations research, and consults with clients on risk assessment and design practices to optimize the availability and efficiency of their data center environments. Victor holds a bachelor’s degree in mechanical engineering from Rensselaer Polytechnic Institute and an MBA from Babson College. He is a member of AFCOM.