Drug safety alerts, a feature of electronic medical records (EMRs), are increasingly recognized as valuable tools for reducing adverse drug events and improving patient safety. However, there has also been increased understanding that alert fatigue, a state in which users become overwhelmed and unresponsive to alerts in general, is a threat to patient safety. In this paper, we seek to mitigate alert fatigue by filtering superfluous alerts. We design a method of predicting alert overrides based on past alert override rate, range in override rate, and sample size. Using a dataset from a large pediatric network, we retroactively test and validate our method. For the test implementation, alerts are filtered with 91-96% accuracy, depending on the parameter values selected. By filtering these alerts, we reduce alert fatigue and allow users to refocus resources to potentially vital alerts, reducing the occurrence of adverse drug events.