Article Critique – Binary Logistic Regression
Frontiers in Psychiatry | www.frontiersin.org 1 June 2018 | Volume 9 | Article 258
University of Applied Sciences and
Arts of Western Switzerland,
GGNet Mental Health Centre,
Universität Ulm, Germany
This article was submitted to
Public Mental Health,
a section of the journal
Frontiers in Psychiatry
Received: 08 November 2017
Accepted: 24 May 2018
Published: 12 June 2018
Hotzy F, Theodoridou A, Hoff P,
Schneeberger AR, Seifritz E, Olbrich S
and Jäger M (2018) Machine
Learning: An Approach in Identifying
Risk Factors for Coercion Compared
to Binary Logistic Regression.
Front. Psychiatry 9:258.
Machine Learning: An Approach in Identifying Risk Factors for Coercion Compared to Binary Logistic Regression
Florian Hotzy1*, Anastasia Theodoridou1, Paul Hoff1, Andres R. Schneeberger2,3,4,
Erich Seifritz1, Sebastian Olbrich1 and Matthias Jäger1
1 Department for Psychiatry, Psychotherapy and Psychosomatics, University Hospital of Psychiatry Zurich, Zurich,
Switzerland, 2 Psychiatrische Dienste Graubuenden, Chur, Switzerland, 3 Universitaere Psychiatrische Kliniken Basel,
Universitaet Basel, Basel, Switzerland, 4 Department of Psychiatry and Behavioral Sciences, Albert Einstein College of
Medicine, New York, NY, United States
Introduction: Although knowledge about negative effects of coercive measures in
psychiatry exists, its prevalence is still high in clinical routine. This study aimed at define
risk factors and test machine learning algorithms for their accuracy in the prediction of
the risk to being subjected to coercive measures.
Methods: In a sample of involuntarily hospitalized patients (n = 393) at the University
Hospital of Psychiatry Zurich, we analyzed risk factors for the experience of coercion
(n = 170 patients) using chi-square tests and Mann Whitney U tests. We trained machine
learning algorithms [logistic regression, Supported Vector Machine (SVM), and decision
trees] with these risk factors and tested obtained models for their accuracy via five-fold
cross validation. To verify the results we compared them to binary logistic regression.
Results: In a model with 8 risk-factors which were available at admission, the SVM
algorithm identified 102 out of 170 patients, which had experienced coercion and 174
out of 223 patients without coercion (69% accuracy with 60% sensitivity and 78%
specificity, AUC 0.74). In a model with 18 risk-factors, available after discharge, the
logistic regression algorithm identified 121 out of 170 with and 176 out of 223 without
coercion (75% accuracy, 71% sensitivity, and 79% specificity, AUC 0.82).
Discussion: Incorporating both clinical and demographic variables can help to estimate
the risk of experiencing coercion for psychiatric patients. This study could show that
trained machine learning algorithms are comparable to binary logistic regression and
can reach a good or even excellent area under the curve (AUC) in the prediction of
the outcome coercion/no coercion when cross validation is used. Due to the better
generalizability machine learning is a promising approach for further studies, especially
when more variables are analyzed. More detailed knowledge about individual risk factors
may help to prevent the occurrence of situations involving coercion.
Keywords: coercion, seclusion, restraint, coercive medication, involuntary hospitalization, machine learning
Hotzy et al. Machine Learning and Coercion
The use of coercive measures (e.g., seclusion, physical and mechanical restraint, forced medication) in psychiatric patients is a massive invasion in their integrity and freedom. As a result, the usage of coercion is controversially discussed since the beginning of modern psychiatry and certain approaches have tried to reduce its rates (1). Although some of those approaches were successful, there are still many patients in which coercion is used. Often the usage of coercion seems necessary when the patients are a danger for themselves or for others due to an underlying psychiatric disorder (2, 3). These situations are always associated with an
ethical dilemma. On one side coercion shall help to protect the patient’s or other’s integrity (2, 3). On the other hand it restricts the freedom of the person which is one of the basic human rights (4). Being a threat to oneself or others may have different reasons in psychiatric patients. In some situations patients are delusional and feel threatened by others which leads to the reaction to protect themselves and can result in threats to other patients or staff (5). Also in situations where the patients are threatening themselves or have suicidal ideations caused by the symptoms of their psychiatric disorder, coercive measures might become necessary to secure the patients survival.
The use of coercion distinguishes psychiatry from other medical disciplines where informed patients can decide to accept or reject a specific measure. Psychiatry at one hand aims to help
the patients to develop a self-determined life without burden of psychiatric symptoms. On the other hand psychiatry is legally determined to reject the patients freedom to move (involuntary hospitalization) but also the freedom to reject a specific measure (forced medication, physical or mechanical restraint, seclusion) if harm to self or others has to be disrupted.
It is obvious that such situations are challenging for the patients but also for the therapeutic team. Those challenges were topic of previous studies where it was shown that patients who experienced coercive measures often describe feelings of helplessness (6, 7), fear (8), anger (9, 10) and humiliation
(11). Due to that, some patients stated to avoid searching for psychiatric help in a crisis (12, 13). On the other hand there were some patients who retrospectively agree with the coercive measure (7, 9) and state that they would like to be forced into treatment again in the case of a future crisis (14). These contrary findings underline the controversy of this topic.
It was the goal of earlier studies to understand which patients experience coercion and to characterize their clinical, but also their socioeconomic features. Gaining better understanding of risk factors to experience coercion was thought to be helpful in the development of therapeutic strategies for patients at risk and thus, to reduce the prevalence of coercion.
During the last years specialized psychiatric intensive care units (PICU) had been the center of extensive research and it could be shown that some patient characteristics are associated with the transfer from a general psychiatric unit to a PICU and with the usage of coercion on these specialized wards (15). Furthermore psychotic disorders were shown to be frequently associated with coercion (16–24). Also personality disorders (25, 26), substance-use-related disorders (19) and
mental retardation (25) were found to be associated with coercion. A history of aggression (16–18, 22, 23, 25, 27– 29) was frequently found to be associated with coercion and violence/threats were described to be the second most frequent reasons after agitation/disorientation for the usage of coercion (30). Patients with a history of former voluntary and/or involuntary commitments (IC) and frequent hospitalizations (16–20, 24) and those with longer duration of hospitalizations (31) were also described to experience coercion more often. Those factors were described nearly uniformly throughout literature. Whereas other factors like male (20, 23–25, 32, 33) and female gender (22, 29) or younger (19, 20, 23, 25, 28, 29, 32, 33) and older age (22, 24) were controversially associated with coercion in different study sites. These inconsistent findings impede the definition of risk-factors which are independent of specific countries. The inconsistencies between study sites were discussed to be caused by cultural influences, organizational factors, societal factors, the clinic-culture or a combination (34, 35). Besides that, one has to bear in mind that prior studies followed different methodological approaches to analyze data which additionally limits the comparability between different study sites. Some studies used descriptive approaches (16, 32) or group comparisons with binominal, non-parametric tests or ANOVA (17–20, 22–24, 26, 29, 30). To describe risk factors regression analysis was frequently used (19–21, 23, 26, 28, 29, 31, 33) and some studies extended their findings with an estimation of the area under the curve (AUC) (23). One study used a latent class analysis (LCA) which is capable of detecting the presence of groups in individuals with relatively homogeneous clinical courses (25). Another study used Multilevel random effects modeling (27). Only a few studies tried to describe the potency of specific risk factors to affect the outcome coercion/no coercion. Furthermore, the description of the specificity and sensitivity of the statistical models is scarce. One study which followed this approach described an acceptable AUC for one model using bivariate analysis (23). Another study found that with the included parameters only a limited prediction of patients at risk was possible (31). Thus, besides the analysis of risk factors at our study site, the second aim of this study was to find statistical approaches with a good balance in their specificity and sensitivity and prediction accuracy for the outcome “coercion/no coercion” in psychiatric inpatients. Furthermore we wanted to analyze the risk factors for their weights in affecting the outcome coercion/no coercion.
In today’s psychiatric research machine learning is an emerging methodology. It is connoted with a great potential for innovation and paradigm shift as the algorithms facilitate integration of multiple measurements as well as allow objective predictions of previously “unseen” observations. We used this new approach to train and compare models with parameters available at admission and after discharge. To test for the hypothesis that machine learning algorithms are effective in the prediction of the outcome coercion/no coercion in psychiatric patients we compared binary regression analysis to the machine learning algorithms according to their sensitivity, specificity, accuracy, and AUC. Furthermore, we used machine learning to weight the included predictors for their potency in affecting the
Frontiers in Psychiatry | www.frontiersin.org 2 June 2018 | Volume 9 | Article 258
Hotzy et al. Machine Learning and Coercion
outcome coercion/no coercion. For the comparison of the two approaches we analyzed clinical data of involuntarily hospitalized patients at the University Hospital of Psychiatry Zurich and built two groups depending on the outcome Coercion/No Coercion.
Setting The study was reviewed and approved by the Cantonal Ethics Commission of Zurich, Switzerland (Ref.-No. EK: 2016-00749, decision on 01.09.2016). Commitment documents as well as the medical records of patients involuntarily hospitalized at the University Hospital of Psychiatry Zurich during a 6-month period from January first to June 30, 2016 were analyzed.
N = 16 wards of the University Hospital of Psychiatry Zurich with a total of 252 beds were included. The clinic provides mental health services for a catchment area of 485,000 inhabitants.
Study Sample No exclusion criteria were defined. We screened a comprehensive cohort of all patients admitted voluntarily and involuntarily to the University Hospital of Psychiatry Zurich during a 6-month period from January first to June 30, 2016 (n = 1,699 patients). For the analysis we included involuntarily committed patients (n = 577) and voluntarily committed patients who were retained at a later stage during their hospitalization and then changed to the legal status of involuntary hospitalization (n = 35).
Selection of Predictor Variables Selection of predictor variables for “training” an algorithm in machine learning is challenging. We used a recommended method and searched the literature databases for variables which were already described to be associated with the usage of coercion: Psychiatric diagnosis (16–24), aggressive behavior (16–18, 22, 23, 25, 27–30), former voluntary or involuntary commitment (IC) and frequent hospitalizations (16–20, 24), gender (20, 22–25, 29, 32, 33), and age (19, 20, 22, 23, 25, 28, 29, 32, 33) were identified as variables of interest. We searched the routine documentation in the electronic medical files of the patients for these variables. The medical files include documentation about the socio-demographic parameters, admission circumstances, prescribed medication, documentation of coercive measures, and treatment planning. As there was no standardized assessment for aggression we searched which indirect information could be used and found IC due to danger to others and involvement of police in the admission process as indirect markers for aggressive behavior. Furthermore we included the procedural aspects abscondence, appeal to the court, duration until day passes, duration of IC, duration of hospitalization into analysis. When patients are exposed to coercive medication mostly antipsychotics or benzodiazepines are used. We were interested if the patients, exposed to coercion differed from those without coercion according to their regular prescribed medication during hospitalization. Thus, we searched the medical files for the prescription of medication classes (antipsychotics, antidepressants, benzodiazepines, and others).
Analysis and Machine Learning We conducted analysis with MATLAB (MATLAB and Statistics Toolbox Release 2012b, The MathWorks, Inc., Natick, Massachusetts, United States.) and SPSS 23.0 (IBM Corp. Released 2011. IBM SPSS Statistics for Windows, Version 23.0. Armonk, NY: IBM Corp.) for Windows.
In a first step we compared patients with/without experience of coercion. We used cross-tabulation with chi-square tests for categorical variables. Due to the non-normal distribution we used Mann–Whitney tests for numeric variables. Variables that differed between both groups in bivariate analyses were included as potential risk factors in multivariate analysis. To analyze the impact of the risk factors on the outcome coercion/no coercion binary logistic regression analysis was used with coercion/no coercion as the dependent variable. The goodness of fit of the binary logistic regression model was assessed by the receiver operating characteristic (ROC) curve method. The AUC served as the criterion to determine the level of discrimination. Discrimination was deemed acceptable at AUC values between 0.7 and 0.79, excellent at values between 0.8 and 0.89, and outstanding at values over 0.9 (23). The specificity and sensitivity, positive predictive value (PPV) and negative predictive value (NPV) were calculated from the results of the different models.
Because of multiple comparisons Bonferroni’s adjustments were made to prevent Type I error inflation (α = 0.05/5 = 0.01).
In a second step we tested the hypothesis that machine learning algorithms can be used to predict the outcome. Again the outcome of coercion/no coercion was used as dependent variable. Because the outcome was already defined, supervised learning algorithms [Logistic regression, supported vector machine (SVM), and bagged trees algorithms] were used. We used cross-validation to test the trained model. The training set was divided in 5 equal sized subsets with one part being used to train a model and the other four subsets to evaluate the accuracy of the learnt model (five-fold cross validation). The error rate of each subset was an estimate of the error rate of the classifier. Cross-validation is used in machine learning to establish the generalizability of an algorithm to new or previously “unseen” subjects. The validity of the algorithms in predicting the outcome coercion from no coercion was evaluated using prediction accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). In this study, sensitivity and specificity represented correctly predicted occurrence of coercion (true positives) and correctly predicted lack of coercion (true negatives), respectively.
Logistic Regression The classifier models the class probabilities as a function of the linear combination of predictors. Logistic regression utilizes a typical linear regression formulation.
Support Vector Machines (SVM) This technique separates data by a hyperplane, trying to maximize the margin and creating the maximum distance between the hyperplane and the values which lie on each side. The higher this distance gets the better is the reduction of the expected generalization error.
Frontiers in Psychiatry | www.frontiersin.org 3 June 2018 | Volume 9 | Article 258
Hotzy et al. Machine Learning and Coercion
SVM are robust in dealing with large numbers of features included because only those features which lie on the margin of the hyperplane are included. If data are non-linear and separation is not possible on one hyperplane, SVM can create more dimensional hyperplanes in a higher dimensional feature space. SVM methods are binary. So in the case of this study where we compared the patient group with/without coercion no dummy-variables had to be created for the response-feature.