A Hybrid Brain-Computer Interface for Closed-Loop Position Control of a Robot Arm

Arnab Rakshit, Amit Konar, Senior Member, IEEE, and Atulya K. Nagar

Abstract—Brain-Computer interfacing (BCI) has currently added a new dimension in assistive robotics. Existing braincomputer interfaces designed for position control applications suffer from two fundamental limitations. First, most of the existing schemes employ open-loop control, and thus are unable to track positional errors, resulting in failures in taking necessary online corrective actions. There are examples of a few works dealing with closed-loop electroencephalography (EEG)-based position control. These existing closed-loop brain-induced position control schemes employ a fixed order link selection rule, which often creates a bottleneck preventing time-efficient control.Second, the existing brain-induced position controllers are designed to generate a position response like a traditional firstorder system, resulting in a large steady-state error. This paper overcomes the above two limitations by keeping provisions for steady-state visual evoked potential (SSVEP) induced linkselection in an arbitrary order as required for efficient control and generating a second-order response of the position-control system with gradually diminishing overshoots/undershoots to reduce steady-state errors. Other than the above, the third innovation is to utilize motor imagery and P300 signals to design the hybrid brain-computer interfacing system for the said application with gradually diminishing error-margin using speed reversal at the zero-crossings of positional errors. Experiments undertaken reveal that the steady-state error is reduced to 0.2%.The paper also provides a thorough analysis of the stability of the closed-loop system performance using the Root Locus technique.

I. Introduction

BRAIN-COMPUTER interfacing (BCI) is gaining popularity for its increasing applications in assistive robotics and rehabilitation engineering. BCI technology captures the human motor-intention to translate the thoughts into commands and actuates the robot to execute a mentally planned complex task. A BCI framework provides a nonmuscular channel of communication with the outer world to enhance the quality of life of people suffering from brainstem stroke, neuro-muscular disorder, and the like, with an aim to rehabilitate them back to their normal lives.

Hybrid BCI [1] is a widely used name in the BCI technology. Generally, it refers to multiple modalities of acquisition of brain activities, including functional near infrared spectroscopy (fNIRS), functional magnetic resonance imaging (fMRI), electroencephalography (EEG), electrocorticography (E-Cog), and the like. The second category of hybridization includes utilizing brain activity acquisition response in the presence of other physiological modality, such as muscle activity acquisition by electromyography (EMG),eye-movement acquisition by electro-occulography, and the like. In this paper, we, however, used the phrase “hybrid BCI”to refer to utilizing multiple brain signals within a single modality of brain-activity acquisition, i.e., EEG. Each brain signal refers to one specific cognitive functionality of the brain, such as motor movement planning, responding to rare/infrequent (error) stimuli, and communicating the subject’s choice (among alternatives) to the BCI system. It is,however, important to note that hybrid BCI may involve multiple EEG signals to decode a single cognitive activity of the brain [2] with an aim to improve reliability in the detection of the cognitive task undertaken.

EEG is preferred in BCI design for its non-invasiveness,faster temporal response, and low cost [3]. This paper employs EEG to capture the subject’s motor-intention.Existing works on mind-controlled external devices utilize a few selected brain signals, including steady-state visually evoked potential (SSVEP), P300, motor imagination (MI), and error- related potential (ErrP) for position control of artificial human appendages/external manipulators [4]–[8]. For example, P300 has successfully been used for goal/destination selection of a mobile robot/wheelchair [9], [10]. SSVEP has been utilized for direction control of wheelchairs [11]. Motor imagery is used mostly in position control applications to actuate an external device based on right/left hand-motor imagination [12], [13]. Lastly, ErrP is infrequently used to determine the occurrence of errors, particularly when the manipulator crosses a fixed (predefined) target position [14].

SSVEP has been proven to be the most promising brain pattern in the BCI technology [15]. It is elicited from the visual cortex of the occipital lobe as a response to some visual stimulus that flickers continuously at a certain frequency in the range of [6, 30] Hz. The SSVEP constitutes signal rhythms at the target frequency and its harmonics [16], and thus is useful in communicating the subject’s choice to the BCI system, when the options corresponding to the stimuli are flickered at selected frequencies.

When a subject undertakes MI, a signal called event-related de-synchronization (ERD) followed by event-related synchronization (ERS) originates from the motor cortex as mu and beta rhythms. ERD refers to a relative decrease in the signal power in the μ- and the β-bands during motor imagination in comparison to the resting state of the brain [17], [18]. On completion of the motor imagination task, the signal power of the β-band increases until the power level matches the average value of the resting potential. The latter signal is referred to as ERS.

Lastly, P300 is an important event-related potential (ERP)that appears in the subject’s EEG recording when they focus their attention to some significant but rare stimulus [19], [20].The subject must respond to the target stimulus in an either covert or overt manner in order for P300 to be evoked. This modality is characterized by a positive deflection in the EEG voltage waveform around 250–500 ms (on an average 300 ms)after the observation of the target stimulus. P300 can be recorded most prominently over the midline of the brain. This ERP is often used to monitor the subject’s attentiveness [21].

In closed-loop position control applications, we often come across a feedback signal for automatic detection of zero positional error with respect to a desired step input of the angular position command. Both P300 [22] and ErrP [23]have been employed by previous authors to address the problem. The ErrP used in the present context, often referred to as observational ErrP, is elicited from the medial frontal region with a negative deflection of approximately 250 ms after the subject observes a machine (or a person) to commit errors. Here, committing an error refers to crossing the desired angular position/set-point of the position-control experiment.ErrP, in general, is a reliable feedback signal. However, its magnitude is diminished in subjects with spinal cord injury and schizophrenic disorders [24], [25]. ErrP amplitude and latency also suffers from inter-subject and inter-trial variability [26], [27]. Lastly, ErrP is best elicited with discrete events [28]. Here the robot motion is continuous which makes the ErrP signal less suited to being used as feedback marker.Because of the above limitations of ErrP, we prefer to use P300, which is released approximately 300 ms from the onset of positional zero-crossing errors, experienced by the subject.

In this paper, SSVEPs corresponding to different flickering frequencies have been used to randomly select a link of the robotic manipulator. Motions of the manipulator links are then activated by the subject’s motor imagination. When the selected link reaches the desired target position, P300 is elicited in the subject’s EEG. The P300 is used to freeze the current motion of the robotic link. Since elicitation and detection of P300 waveform requires a finite (non-zero) time,the robotic link crosses the target position by a small angle before its motion is stopped. The link is then moved in the reverse direction of the last motion, i.e., towards the target location. Throughout the motion of the link, we maintain a gradually diminishing speed. When the link reaches the target again, P300 appears once again and the aforementioned process is repeated. This cyclic operation continues as long as the speed of the link remains above a pre-defined threshold value. Evidently, this process can align the link with the target with a high degree of accuracy. The BCI framework developed here consists of 4 successive stages. First, the acquired EEG signals are pre-processed to make them free from noise and artifacts. Second, the relevant features are extracted from the pre-processed signals. Third, the different classes of the experimental data points are determined by suitable pre-trained classifiers. Finally, control commands are generated to actuate the manipulator to serve the desired requirement.

The work presented in this paper is an extension of [14] by the following counts. First, it includes a provision for random ordered link-selection by SSVEP based BCI, instead of a fixed ordered link selection adopted in [14]. Second, the proposed hybrid BCI system takes into account MI and P300 signals for position control of a robot arm, instead of hybridization of MI and ErrP. Third, the paper provides a thorough analysis of stability using the Root Locus technique. Above all, the proposed method reduces the steady-state error drastically(0.2%) in comparison to the one presented in [14], thus justifying its scope in high precision rehabilitative appliances.

The original contribution of the paper lies in the architectural design of the hybrid BCI-based closed-loop position control system with special emphasis on the modeling of the position controller from the response analysis of the control system. The model presumes an expected time-varying response of the position-controller like that of a stable second order dynamics due to impulsive occurrence of error input at time t = 0, and thus determines the transfer function of the position controller in the Laplace-domain. Given the transfer function of an armature controlled DC motor (for one link of the robot), a stability analysis of the closed-loop position control system, involving both the motor and the controller, is performed using the Root-Locus analysis [29]. The analysis reveals an interesting observation that the stability margin of closed-loop system is determined by the initial choice of maximum speed of the robotic links. The proposed method of analytical formulation of controller is novel in the BCI-based position-control system. Secondly, a simple but elegant feature selection algorithm is proposed to automatically select the best feature-set from a pool of features with the aim to minimize the distance between pairs of intra-class data points and maximize the distance between pairs of inter-class data points for the selected feature set. An evolutionary algorithm is used to handle the said optimization problem.

The paper has 9 sections. Section II provides a general overview of the complete scheme. Section III deals with signal processing, feature extraction, and classification of used brain signals. Controller design and stability analysis are examined in Section IV. Experimental details are presented in Section V. Statistical analysis and controller performance analysis are given in Sections VI and VII, respectively.Comparison with similar works is undertaken in Section VIII.Finally, the concluding remarks are listed in Section IX.

II. System Overview

This section proposes a novel strategy to contr ol the position of a robot’s end-effecter in 3D space. The robot arm used here has 6-degrees of freedom (DOF) with the maximum reach of 580 mm in any arbitrary direction. The frame assignments for all the joints along with the directions of positive angular motions about the z-axis are shown in Fig.1.

Fig.1. Frame assignments of a 6-link Jaco robot arm.

The current work uses only the first three links, L1, L2, and L3, of the robot arm. Fig.2 illustrates the complete position control scheme. Three different brain signals, including SSVEP, ERD/ERS, and P300 are employed here to control the position of the robot arm in its workspace. The key steps of the process are elaborated next.

Fig.2. Basic block diagram of the proposed position control scheme.

A. Link Selection by SSVEP Detection

Existing research on BCI-controlled robotics utilizes a fixed order of link selection, i.e., the robot follows a fixed temporal sequence of link selection irrespective of the target position of the end-effecter [14]. In many real-world position-control applications, dynamic link selection based on the users’choice is required to improve the speed and accuracy of the position-control task. This paper aims at developing a dynamic order of link selection. One fundamental hindrance with dynamic link selection is the lack of communication to the robot about the choice of the current link by BCI means.This has been overcome here by attaching light emitting diodes (LEDs) to individual links, flickering at different frequencies. The subject needs to stare at one of the links carrying a LED, flickering at a fixed frequency, which in turn releases a special brain signal, called SSVEP. The SSVEP is frequency modulated at the flickering frequency and yields a large amplitude at the flickering frequency of the specific LED, which the subject pays attention to. The subjective interest of link-selection thus becomes apparent from the frequency spectra of the SSVEP.

B. Motion Activation by MI Decoding

Selection of a link by the subject indicates that the subject prefers to activate the link for clockwise/counterclockwise turning using motor imagery signals. The ERD/ERS motor imagery signal has been used to direct the motion of the link in the desired direction. The ERD/ERS signal exhibits a desynchronization in the μ- and the β-band power, followed by a synchronization in the β-band power, and thus takes a “v”-like wave-shape. Because of the contra-lateral connectivity between limbs and the brain, ERD/ERS is released by the left motor cortex for right-hand motor imagery (RHMI) and by the right motor cortex for left hand motor imagery (LHMI).Suppose the subject wants to move the link clockwise by RHMI and counter-clockwise by LHMI. Thus it is preferred to detect LHMI (RHMI) from the right (left) brain lobe for efficient decoding of the motor imageries.

Decoding of ERD/ERS requires determining the groundtruth and then identifying similar ERD/ERD traces from the experimental instances. One simple method to construct the ground truth is to take average of the available ERD/ERS traces from the experimental instances [30], where averaging is performed at the sample points over all instances. The averaged trace is defined as the ground truth ERD/ERS. The other ERD/ERS traces in the pool that satisfy the Gaussian characteristic at all the sample points are regarded as positive instances for the true ERD/ERS class. The Gaussian criterion is given below for the sake of convenience. Let Avibe the average value of all ERD/ERS traces at sample point i, and σibe the standard deviation of all the traces at sample point i.Then an unknown ERD/ERS trace is presumed to be close enough to the ground truth, if

After the 2 classes of the ERD/ERS traces are generated, we need to extract certain ERD/ERS features, which together with the class label represent a sample training instance.Hzorth parameters, adaptive auto-regressive parameters(AAR) and discrete wavelet co-efficient (DWT) are few useful features, which have received wide publicity in BCI research. Common spatial pattern (CSP) features have also shown promising performance in LHMI/RHMI classification tasks [31]. We would use CSP features for MI classification in this paper.

C. Alignment With the Target by P300 Identification

Once the MI signal is released by the subject’s brain, one selected link of the robot starts turning in the clockwise/counter-clockwise direction, and as a consequence, the end-effecter/link at some time point t, counted from the onset of the ERD/ERS signal crosses the fixed (pre-defined) target position. This phenomenon where the end effecter crosses the target position acts as an infrequent stimulus to the subject,causing him/her to release the P300 signal. In other words,release of the P300 signal from the subject’s brain in the present context, is a clear indication that the end-effecter/link crossed the target position, and thus needs a corrective action.Here, the release of P300 is used as corrective feedback to the BCI system to turn the robotic link in the reverse direction of its current movement direction with a gradual decay of its angular speed. The process is continued until the angular speed (or linear velocity) of the link goes below a user-defined threshold. The continued reduction in speed, and reversal of motion around the target-point, effectively results in a reduction in the peak overshoot and steady-state error. The pseudo-code for the proposed scheme of position control of a robotic arm is given below. The code is self-explanatory.

Algorithm Pseudo-code for the proposed position control scheme Begin ω0×e?λt,ω0 ε Initialize: Initialize angular speed of each link = where is the initial angular speed, λ (> 0) is the decay factor, and t =iteration; = lower limit of angular speed;Repeat 1. Use SSVEP decoding to select a link of the user’s choice;2. Use MI decoding to find the direction of motion (clockwise/counter-clockwise) of the selected link with a pre-selected angular velocity ω;While ω > ε do Begin 3. Continue moving the link until P300 is detected, implying that the selected link crossed the mentally-imagined target position;ω 5. After the link crosses the mentally imagined target position, stop it temporarily, and reverse the motion of the link with the magnitude of angular speed ω = where = the angular speed of the link just before zero-positional crossing;End-While;Until movement of all desired links are over;End.ω′×e?λt,ω′

III. SSVEP, ERD/ERS, and P300 Decoding

This section narrates the steps involved in processing the EEG signals with an ultimate aim to recognize the BCI signals: SSVEP, MI, and P300. The SSVEP signal originates from the visual cortex region of the occipital lobe. The electrodes O1 and O2 of the international 10–20 electrode placement system are nearest to that brain region used for the purpose of SSVEP decoding. The origin of the MI signals, on the other hand, is parietal cortex and sensory-motor cortex regions. The electrodes located nearest to this region are C4,C3, Cz, P3, P4, and Pz. Thus, these 6 electrodes are employed for MI signal classification. Lastly, the P300 signal appears with a relatively larger amplitude over the midline of the brain, thereby facilitating the use of Fz, Cz, and Pz electrodes for its identification. Thus in this paper a total of 9 electrodes,including Fz, C3, C4, Cz, P3, P4, Pz, O1, and O2, are employed for EEG signal acquisition.

A. Preprocessing

After acquisition, the EEG trials are filtered spatially by means of common average referencing (CAR) to remove common-mode noise, including thermal noise, power line interference, undesired physiological signals, etc., which appears uniformly across all the EEG electrodes [32]. Here,the sample-wise average of all the channels is subtracted from the signal samples of each channel at each time instant.Although there exists other sophisticated methods of noise and artifact removal, CAR has been chosen because of its low computational overhead in comparison to other existing filtering algorithms [33].

B. SSVEP Detection

1) SSVEP Preprocessing

The spatially filtered EEG signals are passed through a BPF of passband 0.1–30 Hz, realized with a 6th order elliptical filter of 1 dB passband ripple and 60 dB stopband attenuation.The reason behind the choice of the elliptical filter is that it provides sharp roll-off characteristics and good attenuation of ripples in both the pass and the stop bands.

2) Feature Extraction

For the purpose of SSVEP detection, power spectral density(PSD) estimates the three flickering frequencies and two harmonics of each of those frequencies that are used as the EEG signal features. In this paper, the Yule Walker method of auto-regressive (AR) spectral estimation is employed, where the AR model of the input signal is used to determine the PSD[34]. The AR based method is chosen over the conventional periodogram method primarily because of two significant advantages. First, for signal-to-noise ratio (SNR) greater than 0 dB, this method provides better frequency resolution than traditional periodogram methods. Second, this method is free from distortions due to side-lobe leakage effects which are inherent in the periodogram approach.

For previous j samples, y(n) can be represented as

where y(n) indicates the nth sample of the input signal and a(k) denote the AR parameters where k=[1,j], and j denotes the model order of the system. Here AR parameters are estimated using least mean square method (LMS). Rewriting(2) yields

For estimating the PSD of the original EEG signal, PSD of the white noise (error term) must be known, which is found to be its variance ( σ2). Therefore, the power spectral density estimate reduces to computation of (4) given below:

In the present context, three flickering frequencies (7 Hz,10 Hz, and 12 Hz) are considered. Spectral power of each of the 3 frequencies with their 2 harmonics is considered as a feature vector. For each frequency, a bandwidth of 0.5 Hz below and above the stimulus frequency is taken with a resolution of 0.1 Hz. Each stimulus frequency with its two harmonics generates 11×3=33 features for a single electrode. Total features generated in a single electrode for all the frequencies are 33×3=99. The study considers two electrodes for SSVEP detection, hence a total 99×2=198 features are generated in a single epoch. We reduce the set of features into a small set using the following feature selection algorithm.

3) Evolutionary Feature Selection

The motivation of feature selection is to identify the smallest possible set of features, which should ideally be independent of each other, but are sufficient to characterize the classes of the given training instances. Such characterization is necessary for recognition of the classes from the selected features/attributes of the training instances.Existing literature on feature selection is primarily aimed at selecting features based on their linear independence [35]. An alternative method, perhaps, is to select features so as to maximize inter-class separating distances and minimize intraclass separating distances. This requires construction of objective functions, ensuring the above requirements, and a time-efficient search algorithm that identifies the smallest possible set of features that satisfies the objective functions jointly.

where

and ||.|| denotes the Euclidean norm.

For the sake of simplicity and convenience, the objective functions presented in (5) and (6) are combined to form a single objective function (8), the minimization of which would serve the purpose.

Here, δ, a small positive number, is introduced in (8) to limit Obj3 to a finite value, particularly when Obj2 approaches zero. In the present circumstance, we select δ as a very small positive number (10–6), so that it has no influence on Obj3, as Obj2>> δ in (8), for such setting of δ.

Although there are several meta-heuristic algorithms to minimize (8), we select the well-known particle swarm optimization (PSO) algorithm, primarily for its small code,small convergence time, small run-time complexity and most importantly the authors’ familiarity with it [36] over a decade.The parameters of the PSO algorithm used include swarm confidence = 2.0, self-confidence = 2, and inertial weight =0.729 based on the authors’ experience [34].

4) Classification

A three-stage hierarchical linear support vector machine(LSVM) classifier has been employed for the purpose of SSVEP classification. Though any other standard pattern classifier could have served the purpose, LSVM is selected for its high classification accuracy and low computational overhead [37]. Moreover, LSVM requires smaller training time as compared to other classifiers such as, naive Bayesian or multi-layered Perceptron. Here SSVEP detection is performed in three distinct stages of binary classifications. In the first stage, the EEG trial is classified to check the presence of SSVEP in the selected time-window. On finding the presence of SSVEP, the second stage of classification is performed so as to determine if the trial corresponds to frequency f1. In the final stage, the non-f1trials are further classified into the two classes corresponding to frequencies f2and f3, respectively.

C. ERD/ERS Detection

1) MI Preprocessing

In this case, EEG trials filtered spatially by CAR, are again filtered by a BPF of passband 8–24 Hz. The BPF is designed by utilizing a 6th order elliptical filter of 1 dB passband ripple and 60 dB stopband attenuation.

2) Feature Extraction

For MI detection, we extract common spatial pattern (CSP)features. CSP is an optimized spatial filter, which aims at minimizing intra-class variance and maximizing inter-class variance of the filtered signals [38]. Let Xibe a (p×q) matrix representing band-pass filtered EEG data of class i,i=1,2,where p and q respectively denote the number of channels and time-slices used in a trial for data acquisition. Let C1and C2be the spatial covariance matrices for classes 1 and 2,respectively, where C1=X1X1Tand C2=X2X2T. CSP attempts to determine the optimal spatial filter vector w=[wi],where wiis the weight of the i-th channel, such that the ratio of variances of the spatially filtered signals wX1and wX2,given by JCSP(w) is optimized (maximized/minimized), where

The optimization of (9) is solved by the general eigen value decomposition (GEVD) technique. In fact, the principal components corresponding to the largest and the smallest eigen values of A=C2?1C1act as the desired spatial filters corresponding to the maximum and the minimum variances.The singular value decomposition (SVD) technique is employed next to obtain the CSP filter w=UTby representing the matrix AbyUDUT, where D is a diagonal matrix. The logarithm of the variance of CSP projections, i.e., log(wCwT)=log(var(wX)) for X=X1and X2are then used as CSP features of 2 classes.

The classical CSP algorithm outlined above works exceptionally well when the acquired EEG signals have large signal to noise ratio. However, because of non-stationarity of the EEG, the same algorithm may not work well universally across all subjects [31]. Particularly, it suffers from high sensitivity to noise, over-fitting and in-sensitivity to spectral information of the used EEG samples [31]. The sensitivity to noise and over-fitting are eliminated by adding suitable regularizing constraints [39] in the CSP objective function (8).To utilize discriminating wave-shapes and/or spectral information of RHMI and LHMI, there are 3 alternatives: i)using CSP features along with temporal [40] and spectral features [41] of EEG for classification; ii) undertaking CSP in narrow sub-bands of the useful frequency spectrum for MI,and then selecting the best set of features from the CSP features in b sub-bands using a mutual information based feature section [42] hereafter called filter bank CSP (FBCSP);and iii) considering both magnitude and phase of the EEG samples in the CSP formulation [43] to derive optimal CSP features. Here, we adopt both ii) FBCSP and iii) magnitudephase CSP (MPCSP) independently, and compare their relative performance with classical CSP in the experiment section. A brief outline to [43] is given in Appendix A.

3) Classification

This paper makes use of a 2-stage radial basis function kernelized support vector machine (RBF-SVM) classifier,where the first stage categorizes the feature vector of an EEG trial in the presence/absence of MI in the trial. The trials found to contain MI are classified in the second stage into LHMI and RHMI.

D. Signal Processing for P300 Detection

1) P300 Preprocessing

The CAR-filtered P300 signal is passed through a BPF of passband 0.1–10 Hz. The filter is implemented with the 6th order elliptical filter of 1 dB pass-band ripple and 60 dB stopband attenuation.

2) P300 Feature Extraction

Here, the adaptive autoregressive parameters (AAR) are utilized as the features of the EEG trials. The AAR model can efficiently represent the stochastic and non-stationary nature of EEG signals owing to the time-varying characteristics of the AAR coefficients [44] A jth order AAR model, AAR( j), is represented by (1) where the AR parameters are evolved with time using a recursive-least-squares (RLS) algorithm [45]with an update-coefficient set to a small number (= 0.008) to facilitate only small changes in consecutive iterations.

For offline sessions, here, P300 trial is captured for the duration of 2 s. Considering a sampling rate of 200 Hz, a total of 400 data samples are collected during a single trial. Here, a 6th order AAR model is considered, hence 6 × 400 = 2400 AAR features are obtained. AAR features are averaged with a moving window of length = 60 features (50 ms). So, the dimension of the feature vector for each electrode after window averaging is 40. Considering three electrodes,Fz,Cz,Pzwe thus have a total 40×3=120 features.

3) Feature Selection

The PSO-based feature selection algorithm introduced before is employed now to select the most discriminating features for 2 classes, representing P300 present or absent in a given time-window.

4) Classification

The selected features are submitted to an LSVM classifier to recognize the presence or absence of P300 in the selected window.

IV. The Controller Design

A. Control Strategy

Traditional BCI based position control paradigms realize open-loop control using MI. As a consequence, the performance of position control system cannot offer desired performance. A closed-loop feedback control system realization using BCI is a necessity to provide the desired performance in position control, such as reduced steady-state error and peak overshoot. The proposed control strategy can reduce both the steady-state error and peak overshoot than the ones obtained in open-loop position control. The peakovershoot is reduced by taking into account the occurrence of the P300 signal. Once the desired robotic link crosses the targeted position (target point in the case of translational movement along a line, target angle in the case of rotational movement on a plane, and target plane in the case of rotation of the link from one plane to the other), and then by reversing the motion of the link. The steady-state error here depends on the response time of the P300 signal and the time required for robotic motion. Since the P300 signal is released approximately 300 ms away from the onset of an oddball stimulus(here, crossing of the target position), the steady-state positional error due to P300 is negligible. The time delay due to motor activation time (or motor time constant) can be reduced by selecting the high speed motor.

The steady-state error can be reduced by gradually reducing the speed of the motor exponentially with time t, and reversing motor speed each time the robotic link crosses the target position. One important aspect that needs special mention here is that like traditional control, here the steadystate error can also be reduced at the cost of increased settling time [29].

Let the i-th link at time t be at position CPi(t) and S Pibe the visually fixed target position for the same link. Then error at time t is defined as follows:

A typical P300 response is generated by the brain whenever ei(t) crosses zero value, i.e., ei(t) is slightly positive or negative. It is important to mention here that in classical control theory [44], the control signal ui(t) for the i-th link is a function of error ei(t). However, in BCI-based position control, we do not have an absolute measure of the magnitude of error as the error is recognized visually by the brain from the zero-crossings of ei(t), and P300 can only ensure occurrence of zero-crossings without having any information about the magnitude of error. Thus, we need a different formulation of the controller. The control problem in the present context is formulated as follows.

1) The controller should allow an exponential decrease in speed throughout the motion of the targeted link.

2) For each zero-crossing of error, we switch off the linkmotor and then turn its motion in the reverse direction with the same speed as it had just before zero-crossing. The exponential decrease in speed is continued between each successive pairs of zero crossings.

3) Steps 1 and 2 are continued until the speed goes below a user-defined threshold. Once the speed goes below the threshold, simply stop the motion of the robotic link.

The threshold is estimated by a series of previous experiments to determine the smallest speed for which the steady-state error is negligible for the given moment of inertia of the selected link. Thus, the threshold speed for each link should be different due to a difference in the link moment of inertia. The variation of link velocity due to the occurrences of the P300 signal is given in Fig.3.

Let the P300 signal appears at time t = Ti, indicating one zero-crossing in (positional) error. We then allow an interval Ti+1–Ti, to generate a control action with an aim to reduce link speed by a factor of e?λtfor a real λ>0, and reverse its motion. The reversal of motion is synthesized by a factor of(?1)ifor the i-th control iteration. Let fi(t) be the controller response for t in [Ti, Ti+1] where v0is the start-up speed of the link at t = 0, (much before the first zero-crossing), which usually is high.

In (11), u(t) is a unit step function, and so u(t ?Ti)=1 for t ≥Tiand u(t ?Ti)=0, elsewhere.

Then for i = 0 to N zero-crossings of error ei(t), we model the controller response f(t) by (11).

Fig.3. Variation of link velocity due to the occurrences of the P300 signal.

Now, taking the Laplace transform of (12), we obtain

where Ti+1=Ti+?T.

Now, by Pade’s approximation [46], (14) can be re-written as

The e?(s+λ)T0term in the above equation reduces to unity as T0is taken as zero and the rest of the terms form a geometric progression whose common ratio is (?e?(s+λ)?T). Now by considering up to 4th order terms (i.e., N = 4) in the series, the approximate value of the sum of series appears as

Now, again applying Pade Approximation, we obtain

Considering a unit impulse as the error input at time point t = 0, we obtain the transfer function G1(s) of the controller as

Let G2(s) be the transfer function of the plant, which is taken as standard transfer function of an armature controlled DC motor, used in the robot actuator. The open loop transfer function of the system is defined by G(s)=G1(s)G2(s), where G2(s) takes the following form:

In simplifying the overall calculation, numerical values of the different parameters are assumed as

?T =1s,λ=0.5, and v0=K, the controller DC gain.

G1(s) and G2(s) take the following form after substitution of the numerical values of the parameters as given in Table I.

TABLE I Parameters of the Armature Controlled DC Motor

The root locus plot of G1(s)×G2(s) is shown in Fig.4 to determine the stability of the closed-loop system from its open-loop transfer function.

To determine the optimal choice of the controller parameter λ, we measured the time-domain parameters of the overall system response due to step input, and noted that the timedomain performance of the closed-loop system yields minimum peak overshoot and optimal settling time for λ=0.5. This is studied in the controller performance analysis section.

It is apparent from the plot, that system behavior is stable for a certain range of DC gain K. Stability margin of gain K for λ = 0.5, is found as 0

Fig.4. Root locus plot of the overall system transfer function G1(s) × G2(s).

V. Experiments and Results

This section presents the experimental protocol and the main results obtained, while conducting the experiments.Performance of the SSVEP, MI and P300 classifiers have been presented here in terms of four metrics viz. classification accuracy, true positive rate, false positive rate, and computation time.

A. The Experimental Framework

Subjects: Ten volunteers were chosen for the experiment.Medical history of the volunteers shows no evidence of any critical illness or any other surgery undergone in the near past.Among 10 subjects, 6 are male and 4 are female and all of them are in the age group of 24–30. A consent form was duly signed by them stating their willingness to participate in the experiment. All other safety and ethical issues were maintained according to Helsinki Declaration of 1970, revised in 2000 [47].

The EEG System Used: A 19 channel EEG device manufactured by Nihon-Kohden was used to acquire the EEG signal from the subjects. Electrodes were placed according to the standard 10–20 electrode placement system, which uses A1 and A2 as the reference electrodes and FPz as the common ground (Fig.5). A built-in notch filter of 50 Hz frequency eliminates the power-line disturbances. The device acquires EEG signal at a sampling rate of 200 Hz.

B. The Training Session

The training was offered in off-line mode with the help of a power-point (PPT) stimulator. The first slide includes a fixation cross to make the subject alert to the stimulator. The second slide includes a flickering source mounted on one link of a robot arm to help the subject reproduce the SSVEP at the source frequency. The simulator includes a 3 link robot arm,each with a provision for flickering at different frequencies,with the ultimate aim to select the link based on the frequency contained in the SSVEP produced by the subject. The subject is asked to gaze on the particular flickering source mounted on the robot link. Once the SSVEP is recognized, the corresponding link responsible for the SSVEP is highlighted with a green color. The third slide includes commanding the subject to produce left/right motor-imagery for clockwise/counter-clockwise rotation respectively of the link. The last slide contains a virtual scenario where a particular robot link crosses a fixed target. The subject on observing this is expected to yield a P300 signal. The above sequence of four slides is repeated for 60 times across one-week’s time. One typical instance of the training session, comprising a stimulus presentation followed by robotic actions in response to the stimuli is given in Fig.6.

Fig.5. Bold (blue) circles representing selected electrode positions in the international 10–20 electrode placement array.

C. The Test Session

The basic difference between the training and the test session lies in the phenomena that the training is imparted in

Fig.6. Stimulus description (header of the figure) with robotic actions in sequence (left to right of the first row followed by left to right of the second row) for a specific control task (to grip the ball).

simulation mode, while the test session is performed online with the real robot. Although, the difference is small, the test session usually is more complex, as the subject has to plan the three steps of operations: link selection by SSVEP, MI to move the selected robot arm, and P300 generation on observing a positional error (i.e., the link crosses the target position), without having a reference on time-limits/intervals.Link selection protocol and link movement direction protocol are illustrated in Tables II and III, respectively.

During the real-time testing session, each of the three brain signals are observed with a moving time window of 1 s.Hence all the signals are captured throughout length of the window but an exception is followed in case of MI detection.MI signal is observed thorough the entire length of the window but only last 0.2 s of the signal is taken into account[48].

D. Observed Waveforms/Traces

The acquired P300, SSVEP, and ERD/ERS traces for 5 distinct trials on a subject with their population average trace are given in Figs. 7, 8, and 9, respectively. In each case, the population average is obtained by taking the average of all available instances. It is evident from Fig.7 that positive peaks of P300 are generated around 250 ms to 350 ms,counted from the onset of the target stimuli (at the 0th second), whereas Fig.8 refers to the SSVEP response corresponding to a frequency of 7 Hz. It is interesting to note that band-power of SSVEP has shown a significant rise around 7 Hz. In Fig.9, the ERD plots show a sharp fall-off in magnitude at approximately 400 ms time-point, and signal power is restored approximately around 550 ms. Here ERD is

TABLE II SSVEP-Based Link Selection Protocol

TABLE III MI-Based Motion Activation Protocol

Fig.7. P300 waveform for 5 subjects represented by dotted lines and the population average of the signals represented by red solid line after acquisition from Cz electrode, and filtering by 6th order elliptical filter in band 0.1–10Hz.

Fig.8. PSD plot of SSVEP at 7 Hz frequency for 5 subjects represented by dotted lines and the average of the signals represented by black solid line after acquisition from channel O1.

Fig.9. Percentage ERD plot of 5 subjects taken at C3 electrode for right arm motor imagery with population average represented by a solid black line after filtering in [8–11] Hz.

quantified as a percentage change of power at each sample point relative to the average power in the reference interval[49].

E. Validation of the Decoders

In this paper 3 different methodologies are proposed to decode the three different EEG signals: SSVEP, MI, and P300. For the purpose of evaluating the performances of these decoders, 5 well-known performance metrics have been employed which are classification accuracy (CA), true positive rate (TPR), false positive rate (FPR) and computational time (CT) and Cohen’s kappa ( κ) co-efficient. The formal definitions of these metrics are presented below for ready reference

CA: The fraction of the total number of instances which are correctly identified by the decoder.

TPR: The ratio between the correctly detected positive instances and the total number of positive instances.

FPR: The fraction of the total number of negative instances which the decoders identify as positive ones.

CT: The time taken by the trained decoder in order to generate the inference about the presence/absence of the concerned signal in a single trial EEG.

Cohen’s kappa: The inter-rater reliability parameter for the categorical items and is a more robust parameter than percent classification accuracy [50].

For each subject, 7 instances are acquired of which 6 instances selected randomly are used for classifier training and the remaining one for classifier testing.

1) Validation of the MI Decoder

Table IV reports the performance evaluation results of the CSP based MI decoder which is required to identify the desired direction of link motion. It is apparent from the Table that MPCSP feature selection method along with radial basis function (RBF) kernelized SVM (RBF-SVM) classifier outperformed its nearest competitor FBCSP feature selection+ linear discriminant analysis (LDA) classifier by a margin of 1.3% in terms of classification accuracy. A higher inter rater reliability value (κ = 0.92) is obtained for MPCSP + RBF +SVM at the cost of increased training time. As the training session is performed offline, the relatively larger training time of MPCSP compared to FBCSP does not affect the real time performance of MPCSP-RBF-SVM classifier with its nearest competitor FBCSP-LDA.

TABLE IV Comparison of Different CSP-Based Classifiers

Now considering Table V, the best CA was attained in case of fourth subject which is 99.2%. The average metric values obtained are: CA = 98.1%, TPR = 0.93, FPR = 0.04, CT = 0.422 s,and kappa = 0.89. Here, the inter-subject variance of the results is represented as coefficient of variation (CV), which is

measured as ratio of observed mean and observed variance.The CV value for classification accuracies is found to be 0.01 whereas CV value of kappa over the different subjects is found to be 003.

TABLE V Classification Results of SSVEP, MI, and P300 (Standard Deviation is Given in Bracket Under Classification Accuracy)

2) Validation of the SSVEP Decoder

It is seen from Table V that the SSVEP decoder offered average CA, TPR, FPR, and CT of 95.9%, 0.92, 0.04, and 0.091 s, respectively. The average kappa value obtained is 0.91. The best CA of 96.8% was obtained for the fourth subject and the best kappa value is found to be 0.94 for the first subject. CV values of classification accuracy and kappa values over the different subjects are found to be 0.005 and 0.018, respectively. It is apparent that such values are significantly on the lower side.

3) Validation of the P300 Decoder

The average values of CA, TPR, FPR, kappa, and CT of P300 decoder obtained are 93.3%, 0.90, 0.03, 0.89, and 0.108 s,respectively with the best CA of 94.9% obtained for the 9th subject. CV values of classification accuracy are noted to be 0.01 and that of kappa is noted to be 0.02.

VI. Statistical Validation

ERD/ERS trials depicted in Fig.9 are statistically validated with the criterion (1) mentioned in Section II. Sample points of the depicted trials are compared with the sample points of the population mean trial (Ground truth) to check if they conform to Gaussian criteria.

P300 trials represented in Fig.7 are statistically validated with the population mean latency obtained from all trials of all subjects participating in the study. The mean latency of P300 is found to be 360 ms. The latency of 6 trials presented in Fig.7 is given in Table XII of Appendix B. The particular time instant when the highest peak of the P300 occurs is considered as the latency of the signal. Population mean latency is found to be 360 ms. One sample t-test [51] is used to statistically validate the trials with the population mean. The required null hypothesis is expressed as follows:

A confidence level of 95% with degrees of freedom (df = 5)are considered for obtaining p-value. Table VI provides the results of the One-sample t-test on P300 trials. The p value obtained here clearly indicates that assumed null hypothesis is true. Hence, the represented samples belong to the same population.

SSVEP trials presented in Fig.8 are also statistically validated using the One sample t-test against the population mean amplitude revealed by power spectral density.Amplitude of the highest peak occurring in 6.5–7.5 Hz frequency range is considered for comparison. The population mean amplitude is found to be 9.1 dB. Peak amplitudes of the trials are given in Table XIII in Appendix B.The null hypothesis is considered as same as considered in previous case. The details of results are given below in Table VII.

The p-value (two tailed, 95% confidence with df = 4)obtained in this case is 0.89, which clearly indicates that depicted trials are not statistically different; thus the assumed null hypothesis proves to be true in this case.

Performance of the classifiers is validated using the Friedman statistical test [52]. The Friedman test is a nonparametric statistical test. It ranks the classification algorithms for each dataset based on classification accuracy. The classifier with the highest classification accuracy gets the lowest rank of 1. For the i-th dataset and j-th algorithm, rijdesignates the relative rank of the classifier. Total rank of a classifier is evaluated by summing all the ranks it received for all the dataset. Rij, the total rank of the j-th classifier is given i n (21).

where N denotes the number of datasets. For each subject, we consider a dataset. The test considers the null hypothesis,which assumes that performances of all the classifiers are equivalent, so their rank sum should be equal. Under the null hypothesis Friedman statistics are distributed as χ with k–1 degrees of freedom. Here k denotes the number of classifiers used in the study.

TABLE VI Result of One-sample t-test on P300 Trials

TABLE VII Result of t-test Applied on SSVEP Traces

Table VIII provides the rank of the classifiers used for SSVEP, MI, and P300, respectively. The Freidman statistic is calculated by

where N = number of databases, and k = no. of competitive classifiers. Now using N = 10, k = 5 and ranks obtained from Tables XIV, XV, and XVI (given in Appendix B), the value of χ2Fis determined separately for each of three categories of signal and compared with the critical value of the chi-square obtained with a 95% confidence level and 4 degrees of freedom. The χ2Fvalues obtained along with the critical value are presented in Table VIII.

TABLE VIII Comparison Table of chi-square Values

It is evident from the table that in each case value obtained from the Friedman test exceeds the critical value, so the null hypothesis that all the classifiers are equivalent is discarded.Hence the performance of the classifiers is evaluated by their cumulative ranks. The classifier with the smallest rank has the best performance.

It is apparent that the RBF-SVM classifier has lowest cumulative rank sum in each case and performs best in the study.

VII. Controller Performance Analysis

The performance evaluation of the proposed method of robot arm position control is done based on four popularly used metrics, i.e., steady-state error, peak overshoot, settling time, and success rate. The formal definitions [14] of these metrics are given below for ready reference.

1) Steady-State Error (ess): The difference between steady state position of the end effector of the robot and visually fixed position of the subject, in the limit as time goes to infinity.

2) Peak Overshoot (MP): The maximum positional shift in response with respect to the (desired) steady-state position of the end-effector. It is expressed as percentage of final value. It is the first maximum peak.

where CPpis the final response, and CPssis the steady-state response of the system for step input.

3) Settling Time (ts): It is the time taken by the response to reach and stay within 2% of the steady-state value or the desired value.

4) Success Rate (SR): It is defined as the ratio between number of successful attempts by the subject to the total number of attempts. A trial is considered as successful when subject is able to reach the target within the positional tolerance of 2%.

Performance metrics given in Table IX, have been averaged over 10 subjects over 100 online sessions. A qualitative comparison is also included in the same table. Three different control strategies have been compared here.

TABLE IX Relative Performance Analysis

The first strategy uses only motor imagery, whereas the second strategy uses MI and P300 to move a particular arm and the third approach using SSVEP, MI, and P300 jointly.

As observed from Table IX, the proposed scheme performs better than the scheme employing the motor imagery detectors. The results show how the inclusion of the P300 based error detector has led to a drastic improvement in the results. The steady-state error has improved by 7.53% with the error being closer to zero for the proposed scheme. In addition, the settling time has also considerably been reduced from 31 s to 24 s as initial speed for the proposed scheme is relatively higher, and so first zero-crossing occurs much earlier. In the MI + ErrP scheme, the initial speed of the links has moderate value in avoiding peak overshoot. The success rate of the proposed scheme also is improved with respect to that of MI + ErrP.

Another experimental study reveals that choice of λ has an influence on controller performance. In fact λ = 0.5 yields optimal settling time of 24 s and a near-optimal peak overshoot of 4.2%, which is 4.18 for λ = 1.0 and also optimal steady-state error of 0.2%. In fact steady-state error does not decrease for λ ≥ 0.5. Result is presented in Table X.

TABLE X Closed Loop Response of the System With Unit Step Input

VIII. Comparison With Existing Literature

There exist quite a few papers involving BCI where EEG based robot manipulation has been used successfully. Most of the cited references use an open-loop control strategy to control the position of the end-effecter. Those were achieved using different BCI signals like SSVEP [53], MI [13], [54].All of them use a controller which essentially works in on/off control mode, and no feedback path is introduced between the robot and the human. Very recent works have used the P300 brain pattern for mobile robot navigation [8] and for the movement of a rehabilitative external agent [10]. All of the works are exposed to high positional error and a large value of peak overshoot because of the absence of any feedback mechanism from the robot to human subject. The above approaches also need a rigorous amount subject training to achieve a satisfactory calibration.

This paper proposes a novel idea of minimizing bidirectional error and peak overshoot of the controller by incorporating a closed loop control strategy. It also reduces the scope of subject-dependency by using Event Related Potential P300 for stopping further movement of the arm. For a particular link operation, the subject has to perform the MI task once, and position alignment with target will be achieved automatically by means of P300, for which the subject only has to concentrate on the object he has visually fixed. Such use of a closed-loop control strategy for BCI based robot control is novel in the literature. The flexible link selection scheme also provides some definite advantages over endeffecter based control scheme of manipulator. First, it is observed that linear movement of the end-effecter (translation along three co-ordinate axes) leads to the alignment singularity [55] condition when operated in a large range. It is not possible for the subject to resolve the singularity condition without any external intervention. Second, due to the lack of flexibility in controlling each degree of freedom, an overall movement of six joints becomes necessary even when the task can be achieved by turning only a single link. It is evident from relative performance analysis that the steady-state error in the case of the proposed approach is reduced drastically.

The comparison of the work with Hybrid BCI systems designed for position control by robots is presented in Table XI.It is clear from the Table that the proposed study and the study by Bhattacharyya et al. [14] only address the problem of aclosed-loop control strategy. The proposed study has improved steady-state error, settling time and success rate, and so outperforms [14] significantly.

TABLE XI Comparison With Other Hybrid BCI Based Position Control

IX. Conclusion and Future Directions

This paper introduced a new approach for position-control of a robot’s end-effecter by judiciouly controlling the positions of the individual links of the arm. The choice of individual link selection and their position control is left to the user. The individual link selection is performed by the user by noticing the flickering LED mounted on the link. In fact, each link has one LED mounted over it to flicker at a fixed frequency. If the subject releases a P300 from one of the links,it’s inferred that the subject prefers to use the link in the next time-slot for position control.

Apart from BCI-based link selection, the other important attributes of our work lie in ERD/ERS based motor planning of the previously selected link, and a P300 induced automatic stopping and speed-reversal, where at, the target position is reached by an individual link.

A thorough modeling and analysis of the controller performance undertaken in the paper reveals that the proposed BCI based control is stable with low steady-state error (0.2%),low peak-overshoot (4.2%), and relatively lower settling time(24 s).

An analysis of the root locus of the overall system reveals that the stability margin of the proposed system is contingent on the initial choice of the maximum speed of the robotic links. The classifiers chosen have high classification accuracy and their ability to work in presence of noise proves their elegance in the present study.

Above all, the proposed system outperforms all existing and reported works utilizing BCI-based position control with respect to both classifier and controller performance, thus justifying its utility in rehabilitative aids for people with neuro-motor disability. Future works may involve i) designing alternative control strategies to reduce subjective cognitive load; ii) improving clasifier design, particularly MI classifiers following [48], [59]; and iii) removing ocular artifacts following [60] to develop robust, noise-insensitive BCI based control systems.

Acknowledgments

The authors gratefully acknowledge the RUSA 2.0 funding to Jadavpur University, UKRI’s Global Challenges Research Funds (GCRF) to support the collaborative work between Jadavpur University and Liverpool Hope University, and also the CSIR, Govt of India financial aid for doctoral fellowship to the first author.

Appendix A

Amplitude and Phase-Sensitive CSP (AP-CSP)

The classical CSP formulation takes into account of the amplitudes of the EEG time samples, disregarding the phases of the EEG signals. In [40], the authors considered both amplitude and phase of the EEG signals to obtain more reliable CSP features, responsible for improving the classification accuracy for the 2-class classification problem.

Appendix B

Tables Prepared for Statistical Tests

TABLE XII Latency of the Represented P300 Trials

TABLE XIII Amplitude of the Represented SSVEP Trials

TABLE XIV Rank Table of Classifiers Used In SSVEP Detection

TABLE XV Rank Table of Classifiers Used in Motor Imagery Detection

TABLE XVI Rank Table of Classifiers Used in P300 Detection