Enhancing the Prediction of User Satisfaction with Metaverse Service Through Machine Learning

Seon Hong Lee,Haein Lee and Jang Hyun Kim

1Department of Applied Artificial Intelligence/Department of Human-Artificial Intelligence Interaction,Sungkyunkwan University,Seoul,03063,Korea

2Department of Interaction Science/Department of Human-Artificial Intelligence Interaction,Sungkyunkwan University,Seoul,03063,Korea

Abstract: Metaverse is one of the main technologies in the daily lives of several people, such as education, tour systems, and mobile application services.Particularly, the number of users of mobile metaverse applications is increasing owing to the merit of accessibility everywhere.To provide an improved service, it is important to analyze online reviews that contain user satisfaction.Several previous studies have utilized traditional methods, such as the structural equation model (SEM)and technology acceptance method(TAM)for exploring user satisfaction,using limited survey data.These methods may not be appropriate for analyzing the users of mobile applications.To overcome this limitation, several researchers perform user experience analysis through online reviews and star ratings.However,some online reviews occasionally have inconsistencies between the star rating and the sentiment of the text.This variation disturbs the performance of machine learning.To alleviate the inconsistencies,Valence Aware Dictionary and sEntiment Reasoner(VADER), which is a sentiment classifier based on lexicon, is introduced.The current study aims to build a more accurate sentiment classifier based on machine learning with VADER.In this study,five sentiment classifiers are used,such as Na?ve Bayes,K-Nearest Neighbors(KNN),Logistic Regression,Light Gradient Boosting Machine (LightGBM), and Categorical boosting algorithm(Catboost)with three embedding methods(Bag-of-Words(BoW),Term Frequency-Inverse Document Frequency (TF-IDF), Word2Vec).The results show that classifiers that apply VADER outperform those that do not apply VADER,excluding one classifier(Logistic Regression with Word2Vec).Moreover,LightGBM with TF-IDF has the highest accuracy 88.68%among other models.

Keywords:Metaverse;ubiquitous computing;user satisfaction;online review;big data;VADER;machine learning;natural language processing

1 Introduction

Metaverse is one of the most promising technologies which actualize virtual reality.Considering the Consumer Electronics Show (CES)held in 2022, metaverse was evaluated as one of the most important technologies of the future along with Non-Fungible Token (NFT)[1].Regarding the emergence of metaverse,various platforms and applications are increasingly attempting to utilize the technology in their service [2].Because this new trend is occurring in various fields (e.g., learning system, tour system), ubiquitous computing infrastructure in relevant fields are being established[3,4].Moreover,considering the introduction of metaverse,several users can use the diverse metaverse applications in mobile and PC environments [5].However, few studies have been conducted in the area of mobile metaverse applications.Considering the increasing number of users of metaverse, it is important to analyze user experience to provide the metaverse application users with improved services[6].

Regarding the process of analyzing user experience, user satisfaction can be obtained by online reviews [7].User satisfaction on the internet is strongly related to the performance of a service [8].Moreover,several scholars generally analyze user satisfaction to identify the factors which positively influence the fulfillment of service[9-11].Considering these aspects,several researchers are trying to relate user satisfaction to predict the reuse intention of service[12,13].For instance,by predicting user satisfaction in vehicle sharing systems,unnecessary resources can be reduced,and an efficient service can be provided[14].

However,most studies adopt few sample questionnaires to obtain user satisfaction,using conventional methods such as Structural Equation Model(SEM)or Technology Acceptance Model(TAM).To overcome the limitation of conventional methodologies,resources such as big-size online data(e.g.,online reviews)from ubiquitous computing are getting more pervasive[15].Big data are collected and analyzed in various fields[16,17].Considering such a trend,this study aims to predict user satisfaction using big data from online reviews.Particularly,the authors use various machine learning models and embedding methods to find the optimal model that predicts user satisfaction.Furthermore,to remove biased text data,Valence Aware Dictionary and sEntiment Reasoner(VADER)is applied.

In the study,4,783,669 online reviews of metaverse application(i.e.,Roblox)were collected from Google play store.Roblox is one of the most well-known metaverse applications.Considering a successful metaverse introduction, Roblox had 47 million daily users in 2021 [18].Subsequently,to predict user satisfaction, five sentiment classifiers based on machine learning (i.e., Na?ve Bayes,Logistic Regression, K-Nearest Neighbor (KNN), Light Gradient Boosting Machine (LightGBM),and Categorical boosting algorithm (Catboost))with three methods of embedding (Bag-of-Words(BoW),and Term Frequency-Inverse Document Frequency(TF-IDF),Word2Vec)were implemented.This study provides the first model to predict user satisfaction using various machine learning approaches and big data in the metaverse domain.In addition,using VADER,the authors developed classifiers having more higher performance than the models that did not remove inconsistencies between user’reviews and user satisfaction.

2 Related Studies

2.1 Metaverse and Ubiquitous Computing

The metaverse and ubiquitous computing are closely related in different fields.Tang[3]suggested that metaverse could be used in educational systems accessible everywhere to provide beneficial experiences for students.Erdem and?Seker[4]discovered that smart tourism was developing along with the expansion of ubiquitous computing and metaverse systems.This is because metaverse technology provides a personalized experience.In addition, these technologies have helped increase satisfaction through interconnected connections between tourists and community members.Therefore, both tourists and community residents have improved their lives and have experienced sustainability.Regarding the metaverse technology, in museum exhibition contents, the audience can have a good sight from any place [19].Therefore, metaverse technology is one of the main factors that make ubiquitous computing possible.

2.2 Previous Studies on User Experience(Satisfaction)and Online Review

Predicting user satisfaction is an important task for various businesses [20].Considering the importance of users’feedback, several researchers frequently used questionnaires to analyze user satisfaction [21-23].Zviran et al.[21] employed exploratory factor analysis (EFA)to identify the factors influencing user satisfaction,using survey data from 359 respondents.Alawneh et al.[22]used factor analysis and multiple linear regressions with survey data from 400 staff members to evaluate the factors related to satisfaction.By using UMUX-Lite and SUS models,Korableva et al.[23]evaluated user satisfaction with data from 60 respondents.To predict user satisfaction,Xu et al.[24]introduced Technology Acceptance Method(TAM)based on 426 survey questionnaires.Alalwan[10]also utilized online questionnaire surveys from 500 smartphone users,proposing a UTAUT2 model that analyzed customer satisfaction.In previous studies, user satisfaction was mainly analyzed and predicted by survey instruments,which did not consider the big data approach.

To overcome previously mentioned limitations,online reviews in customer evaluation can consider quality data as big data [16].Nwakanma et al.[25] indicate that online reviews are used to predict user satisfaction.This can help when receiving feedback from users for products and services.Online reviews contain the emotions and intentions of customers about products and services [26].Zhou et al.[27] suggest that an online review is related to users’assessment, which helps future consumers to purchase products or services.Moreover,marketers in various fields(e.g.,health care,restaurant, and hotel service)understand that analyzing online reviews is important for increasing products or services’success[17,28-30].Furthermore,Chen et al.[31]demonstrated that online reviews could be used to predict user satisfaction and to improve service quality.Therefore, it can be an appropriate approach because online reviews contain a viewpoint of user satisfaction.

2.3 Machine Learning and User Experience

Considering online review data, the user satisfaction classifier model (e.g., Logistic Regression)was suggested by Nwakanma et al.[25].Moreover, several machine learning models were proposed as approaches to the prediction of user satisfaction.Furthermore,accurate results have been obtained by employing machine learning technology when investigating user satisfaction.Tab.1 summarizes the studies on user satisfaction, which have applied machine learning to numerous service areas.Considering e-commerce, Bansal et al.[32] adopted machine learning models (e.g., Na?ve Bayes,Logistic Regression, Support Vector Machine, and Random Forest)as user satisfaction classifiers.Considering their study,Random Forest had the highest accuracy(90.66%)and Na?ve Bayes had the lowest accuracy (54.84%).Similarly, Lin [33] experimented with XGBoost and LightGBM machine learning and various models(e.g.,Support Vector Machine,Random Forest,and Logistic Regression)in the Women’s clothing online market.LightGBM had the highest accuracy(98%).Shah et al.[34]compared machine learning models (Support Vector Machine, Decision Trees, Na?ve Bayes, and Logistic Regression)in Amazon Product reviews.The accuracy of Random Forest (93.17%)and Logistic Regression(90.88%)had better performances than the other models.Cao et al.[35]developed prediction models(Catboost,Random Forest,Support Vector Machine)for users’online purchasing behavior.Catboost had the best accuracy (98.38%), compared to other models.Sharma and Shafiq[36]utilized online reviews for building models that are able to predict purchase intentions on the basis of information related to service experience.As a result,LightGBM had a better performance(96.1%)than Logistic Regression(95.8%).Applying machine learning by considering user experience can be useful for predicting user satisfaction and for identifying the appropriate development direction in several domains.

Table 1: Summary of the existing studies applying machine learning to user satisfaction

3 Method

3.1 Data Collection

The authors collected 4,783,669 user review data from the RoBlox application in Google play store.The dataset is composed of three metadata that have textual reviews, star ratings, and dates(from June 19,2014 to November 23,2021).

3.2 Preprocessing

To compose sentiment classifiers based on machine learning,star ratings were chosen as a feature as offered by previous study [37].Moreover, previous study has shown a tendency of considering reviews with four or five out of five stars as positive reviews and those with less than three stars as negative reviews.Following Li et al.[37], the reviews that have three-star ratings were eliminated,analyzing the data with three stars out of five as neither positive nor negative.

However,the sentiments revealed in online reviews are sometimes not congruent with star ratings.There are two kinds of such cases.The first is where the reviews in writing show saliently positive sentiment;nonetheless,the reviewer awards a low star rating.The second is where the reviews in writing have negative sentiments;however,the reviewer assigns a high star rating(Tab.2).

Table 2: Examples of disagreement between the content of reviews and star ratings

These cases can hinder the performance of sentiment classifiers based on the machine learning models[38].To remove these cases,the authors used the VADER by Hutto et al.[39].

The VADER is introduced by Hutto et al.[39]as a rule-based model for broad sentiment analysis.Moreover, its efficiency can be measured by comparing it to classical methods such as Affective Norms for English Words (ANEW), Linguistic Inquiry and Word Count (LIWC), SentiWordNet,and machine learning algorithms.Elbagir et al.[40] classified tweets on the 2016 US election.The results showed great accuracy in distinguishing classes.Furthermore, Borg et al.[41] investigated sentiment analysis using e-mails from customers.Because the dataset did not have any sentiment information, they utilized VADER for the labeling of emails.They indicated predictable features in email communication.Therefore, VADER has a discrimination against LIWC or previous lexiconbased methods because it is more sensitive to emotions in social media contents.Therefore,the authors adjusted the VADER for examining the sentiment of user reviews and eliminating disagreement cases between texts and star ratings.Consequently,it is possible to build an honest and unbiased predictable machine learning model.

Considering this process,SentimentIntensityAnalyzer in the Natural Language Toolkit(NLTK)of Python library was used to examine positive and negative indices and to calculate the total sentiment index(i.e.,compound).Positive,negative,and neutral sentiments are determined by a compound score.This study labels the positive (compound value greater than or equal to 0.05), negative (compound value less than or equal to-0.05),and neutral(compound value between-0.05 and 0.05)reviews.

Subsequently,regular expressions using the Python library were applied to delete special symbols.In addition, NLTK was used to eliminate the stop-words and to extract lemmatization.extract lemmatization and to remove the stop-words in NLTK.

Additionally, the authors assigned different training dataset for the data group which applied VADER(Group 1)and for the other data group which did not apply VADER(Group 2).The training set for Group 1 consisted of the same values of star rating and VADER.Data with incongruent values were excluded from the training set.Group 2 did not have any exclusion.To evaluate the results,the authors assigned sentiment labels(positive:1,negative:0)to the 21,440 test dataset(Fig.1).

Figure 1:Diagram of the total process

3.3 Imbalanced Class

Considering a balanced distribution of the training set with VADER (positive: 314,627 and negative:155,084)and without VADER(positive:4,306,104 and negative:384,767),down-sampling was performed as suggested by Liu et al.[42].Consequently, the count of the positive and negative review data became identical.The training set with VADER(positive:155,084 and negative:155,084)and without VADER(positive:384,767 and negative:384,767)had a balanced distribution.

3.4 Text Feature Vectorization

In the process of applying classification algorithms, it is important to perform appropriate embedding processing in various fields.For example,watermark embeddings are used for analyzing medical audio data or to analyze image data[43,44].In this study,the authors utilized the traditional embedding(BoW,TF-IDF)that have been employed in the field of natural language processing and the recently developed embedding(Word2Vec).

Expression methodologies for processing natural languages such as bag-of-words (BoW), term frequency-inverse document frequency(TF-IDF)[45],and Word2Vec[46],were adjusted for selecting the representation features from the text data as proposed by Krouska et al.[47].These methods are performed to find the appropriate features of the classifiers.

Bag-of-Words transforms text data in a digitization expression method for adjusting machine learning models.These features were achieved by employing the count vectorizer module in scikitlearn,a Python library.

Frequency-inverse document frequency expresses the conciseness of a word in a certain document when a document group combines numerous documents[45].This is the value obtained by multiplying TF and IDF.Term-frequency (TF)indicates the value that informs the frequency of a certain word within a document,and IDF is the relative frequencies applicable to a set of documents.

The vector W based on a document(d)is set by a product of the statistics TF(t,d)and DF(t).The term frequency(TF)is a word count(t)that occurs in document(d).The document frequency(DF)is the number of documents where the particular word(t)comes.The inverse document frequency(t)is obtained by document frequency.|D|is the total count of documents.

When inverse document frequency about a word(IDF)emerges in a small number of documents,it has a high value.Moreover,it has low value when it appears in large documents.The word feature W is computed as a dot product[48].Therefore,TF-IDF features are attained by applying the Tfidf Vectorizer module in scikit-learn.

Word2Vec was developed by Mikolov et al.[46].Word2Vec is based on the distribution hypothesis that words appearing in similar contexts have similar meanings.When words in a text are vectorized based on the distribution hypothesis, vectors of similar words have similar vector values.Thus,Word2Vec is a way to learn to express the meaning of words by dispersing words in multiple dimensions of a vector.The Word2Vec is classified into two learning methodologies: Continuous Bag of Words(CBOW)and Skip-gram.The Skip-gram has a better performance than CBOW overall [49,50].Therefore,the authors used a Skip-gram model.The Skip-gram is a method of predicting surrounding words using a central word:

whereQis a training complexity of the Skip-gram model,andCis the maximum distance of the words.Moreover,Vis the size of the vocabulary,andDis the word representations.Word2Vec features were obtained from the gensim module in a Python library.

3.5 Machine Learning for Classification

The authors used five machine learning classifiers (Na?ve Bayes, K-Nearest Neighbor, Logistic Regression,LightGBM,and Catboost)and three embedding methods(BoW,TF-IDF,and Word2Vec)with 5-fold cross validation.Furthermore, the grid search, which is supported in the scikit-learn package,was utilized for optimizing the hyperparameters[51].

3.5.1 Na?ve Bayes

The Na?ve Bayes classifier is a conditional probability model universally used for text classification.Considering the Na?ve Bayes classifier based on Bayes’theorem,each attribute is expected to be independent[52]:

where P(A|B)is the posterior probability,P(B)is the evidence,P(A)is the prior probability,and P(B|A)is the likelihood.The Na?ve Bayes model was applied with the scikit-learn package in Python.

3.5.2 K-Nearest Neighbor(KNN)

The K-Nearest Neighbor(KNN)is a supervised learning and non-parametric algorithm.Because of KNNs’simplicity and effectiveness, KNN is one of the most widely used classifiers [53,54].Moreover,KNN employs a similarity function that arranges data points based on their closeness to their neighbors[55].To calculate the distance between data points,the Euclidean distance is operated in the KNN algorithm.The related formula is as follows:

where d indicates the distance,and x and y are separate data points.The KNN model was built using the scikit-learn package in Python.

3.5.3 Logistic Regression

Logistic Regression was introduced by Cox [56].Logistic Regression expresses the relationship between dependent and independent variables as a specific function.This is used for the predictive models.Particularly, it is applied to a binary problem [57].Logistic Regression is a continuous and increasing function and has a value ranging from zero to one.

The Logistic Regression model was set up using the scikit-learn package in Python.

3.5.4 Light Gradient Boosting Machine(LightGBM)

The LightGBM is an advanced machine learning model for improving the computational complexity and out of memory[58].Regarding the training process,when creating trees,the LightGBM classifier uses a leaf-wise method as it moves into the deeper layers.Therefore,low memory is used,and high performance was achieved.The LightGBM classifier was constructed using the lightgbm library in Python.

3.5.5 Categorical Boosting Algorithm(Catboost)

Catboost is a kind of gradient boosting algorithms which can deal with categorical data.Catboost could perform a random permutation of the data and compute the mean of label value for the data with identical category value.Catboost alleviates overfitting and lowers the loss function in the process of iteration[35,59].The Catboost classifier was from the catboost library in Python.

4 Result s

Considering the instruction of Visa et al.[60],the prediction consequences of the classifiers were effectuated by implementing a confusion matrix(Tab.3):precision[Eq.(7)],recall[Eq.(8)],accuracy[Eq.(9)],and F1-score[Eq.(10)].Consequently,the LightGBM classifier using the TF-IDF generated the highest accuracy with 88.68%.The Logistic regression with TF-IDF(88.35%)achieved a relatively high accuracy.Moreover, the TF-IDF method appeared to have a relatively superior performance,compared to BoW and Word2Vec.Excluding one (Logistic Regression with Word2Vec)of the 15 models, all the models to which VADER was applied showed a high performance.The summary of the occurrence results is described in Figs.2,3,and Tab.4.

Table 3: Confusion matrix

Figure 2:Accuracy of the models(embedding)with VADER applied

Figure 3:Accuracy of models(embedding)without VADER applied

Table 4: Confusion matrix results of the models(embedding)

5 Conclusion

The ubiquitous market is developing with the expansion of the mobile application market and metaverse technology [61].Considering these tendencies, various fields try to improve the user experience [62].This improvement of user experience has become an important factor in increasing user satisfaction[7].To predict user satisfaction,previous studies approached the traditional method(e.g.,SEM)with a limited number of samples[12,13].This study considers this drawback by providing the optimal machine learning with big data.Five sentiment classifiers based on machine learning with three embedding methods were utilized.In addition, to remove the inconsistency between the sentiment of the star ratings and the review text,VADER was used.Therefore,the optimal machine learning model(LightGBM with TF-IDF)with VADER had an 88.68%of accuracy.

This study contributes to relevant fields in two aspects.Methodologically,it utilized VADER for eliminating disagreement between start ratings and review contents and compared its results with traditional approaches.In addition, the current study comprehensively reviewed extant studies on predicting user satisfaction from online reviews and enhanced the performance of related models.

This study has the following practical implications.Considering the results,LightGBM had a good performance both in the models without text disagreement and those with it.These outcomes can be used to develop an online review analysis platform in the field of metaverse systems such as Roblox.Additionally, in this study, TF-IDF and BoW are more appropriate than Word2Vec in representing words.Therefore, it is recommended to exclude Word2Vec embedding from the standpoint of a company that uses limited resources and human resources for building the sentiment classifier.

Theoretically,this study presents that user satisfaction can be predicted through user-based texts such as online reviews.Because online reviews and user satisfaction are closely related to application service[31],this perspective is important.Therefore,our methodologies can help service providers to select the right service for users even when only reviews without star ratings are available.Particularly,our methodologies provide insights into how reviews on applications can be used for data analysts.Therefore,the results of this study are helpful in understanding users’perceptions of metaverse-related services.

The current study has some limitations that need to be resolved in the future.First, Google play store does not offer users’personal information.Thus,the current study cannot examine age or nationality in its analysis.Second,this study analyzes one mobile metaverse application for observing user satisfaction.Therefore,it is required to confirm the methodology in relation to other metaverse applications.Third,this study employs English data.Hence,future studies can examine data in various languages.

Acknowledgement:We would like to thank Editage(www.editage.co.kr)for English language editing.

Funding Statement:This study was supported by a National Research Foundation of Korea (NRF)(http://nrf.re.kr/eng/index)grant funded by the Korean government(NRF-2020R1A2C1014957).

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.