In the post-COVID-19 era, which features technological innovation in the fourth industrial revolution, as well as "Untact," non-face-to-face, and non-contact, digital transformation is accelerated across library services, and non-face-to-face and online based services are expected to be expanded. As COVID-19 brings a new paradigm beyond just a pandemic event and is expected to strengthen its non-face-to-face and online based services, a survey conducted by the Public Library Association (PLA) found that public libraries in the U.S. are pushing for new services such as extending return deadlines online (76%), online services such as e-books and streaming media (74%), and virtual programming (61%). In addition, various new activities such as reallocating budgets for collecting printed data to purchase digital data or providing telephone services to users who have difficulty connecting digitally are being promoted.

For users who cannot visit the library due to COVID-19, many libraries continue to communicate with local residents through online chat, text, telephone, and e-mail. Among them, the key changes in library services can be found in "expanding e-book services" and "increasing budgets for digital data purchases," reflecting changes in living patterns changed by COVID-19. In other words, the prolonged spread of COVID-19 is accelerating the digital transformation as the preference and demand for free e-books in electronic libraries are sharply increasing. This is not a phenomenon limited to libraries, but the overall use of online content and streaming services is increasing, and the so-called untact reading culture is also expanding in the book market, and more people are practicing reading through digital contents such as e-books and audio books. As an example, the e-book usage rate of the Jongno-gu e-library site increased 5.6-fold year-on-year, while that of the Gangnam-gu office e-library increased by more than 10,000 books (75%) compared to last year, and that of the Gyeonggi Cyber Library nearly doubled (Park, 2020; Park, 2020; Seo, 2020). YES24, a leading online bookstore, also reported that e-book sales jumped 41% year-on-year in May 2020, while audiobook platform StoryTell reported a 3.5-fold increase in new domestic subscribers compared to the first quarter before the outbreak of COVID-19 (Lee, 2020).

Under these circumstances, libraries are quickly responding to users' demands by increasing their share of e-books. At a time when the utilization rate of electronic libraries is increasing, users' issues related to integrated viewers, system quality, and readability can also be increased and raised in proportion to the amount of use. In the post-COVID-19 era, users' information acquisition and usage behavior, which features the Internet, online, and non-face-to-face, is expected to change, and it seems that it is time for a qualitative assessment while meeting the quantitative aspects of the provision of electronic books. Therefore, this study aims to analyze users' assessment of electronic libraries in mobile App markets and to improve the services based on them.

The e-book service provided by libraries is divided into two types, general e-books provided semi-permanent, and subscription-type e-books whose list is updated each year, depending on the type of purchase. However, from the user's point of view, both are recognized as the same e-books (Baek, 2018), and the way they use e-books provided by libraries can be largely divided into using computers in the library, electronic library websites, and electronic library applications. Problems arising in connection with e-book services that are relatively free from time and space constraints include rapid changes in subscription methods and copyright issues (National Library of Korea, 2019). Among them, the biggest issue for library users is that they have to install a designated viewer program separately by each e-book distributor (Baek, 2018; Kim, 2020).

In lending e-books, there is also the problem that integrated e-book sites and subscription-type e-book sites do not work together. Although certain books have been borrowed from a subscription-type e-book site, the integrated e-book site cannot verify the lending history. When reading non-subscribed e-books, the user has to install a designated viewer by each book distributor. In other words, books distributed by A company can only be read on A company's viewer, while books distributed by B company can only be read on B company's viewer. Due to these problems, reading e-books on mobile devices requires a more complex process. For example, when users of the Seoul Library want to use e-books provided by the library, they will basically install the Seoul Library App and the integrated e-book library App. If they don't install the subscription-type e-book viewer App in advance, the search for the subscription-type e-book itself is virtually blocked. To read general e-books, the user needs to install a distributor-specific viewer. Therefore, if four distributors are supplying e-books to the Seoul Library, a total of five viewers (four for the distributors and one for subscription-type e-books) must be installed on the user's phone in order to read e-books freely on the device, and a total of seven apps, adding the Seoul Library App and e-book App, must be installed to use the entire Seoul Library e-book Service (Baek, 2018). Therefore, in theory users can read all kinds of e-books on the integrated e-book websites, but they are dissatisfied with having to go to another homepage or install a new app to read them. In response to the complaints from the users, libraries explain that they provide semi-permanent e-books and subscription-type e-books, which are updated every year, to secure as much content as possible, which in turn inevitably leads to the increase in the number of viewer Apps. Fundamentally solving these problems are likely to be difficult due to the conflicting interests of various distributors. Nevertheless, both types are recognized as the same e-books from the user's point of view, serving as factors that make them feel uncomfortable and dissatisfied with the service.

In order to evaluate e-library or cyber library-related applications from the user's point of view, this study looked at the relevant prior studies divided into the evaluation and analysis of e-books, the evaluation and analysis of specific applications, and the research conducted in relation to applications or e-books of existing libraries. First, the evaluation and analysis of e-books are conducted from the perspectives of interface usability (Yoon, Kim, & Kim, 2013), terminal usability (Kwak & Bae , 2011), and provider performance evaluation (Lee, 2011). Second, evaluation and analysis of specific applications are basically conducted on usability (Kwon & Kim, 2015; Kang et al., 2019), similar to studies on e-book assessment and analysis, and further focused on UI design usability (Yoon, 2018; Chun, 2020). In addition, research was conducted to analyze the functions and utilization of public Apps. In the latter case, the activities of the App were compared and analyzed using the type of service, number of downloads, ratings, and number of downloads over population provided by public apps in the disaster safety area (Kim et al., 2014). Third, studies conducted in relation to applications or e-books of existing libraries can be organized as follows: First, analysis of library mobile applications was conducted on the basis of the Ministry of Public Administration and Security's mobile application accessibility guidelines, on the accessibility of domestic library mobile applications (Jang & Nam, 2014), and on the user satisfaction level of university library e-book services (Nam & Choi, 2011).

This study aims to explore how users who are reading e-books on mobile devices actually evaluate the applications in each library and find them uncomfortable based on ratings and reviews in the App markets.

In this study, first, the basic background and purpose of the study, research method, and the scope were set, and the relevant literature and empirical prior studies were analyzed. Next, users' evaluations of electronic libraries were collected and analyzed from Google Play Store. Lastly, based on the analysis, measures to improve the quality of e-libraries were discussed.

If the word of mouth in the past was mainly spread face-to-face, word of mouth online is mainly disseminated through users' reviews. While word of mouth online is not so different from traditional word of mouth in that it is an interactive communication process that exchanges experiences and information related to products and services, it has different characteristics compared to traditional word of mouth because of the addition of the characteristics of the Internet. Namely, first, communication is done in words written rather than spoken, and second, in the case of traditional offline word of mouth, the subject of word of mouth was limited to close connections, such as friends, relatives, and coworkers, while word of mouth over the Internet are subject and scope-free (Christiansen & Tax, 2000). Third, the amount of word of mouth communication online is greater than that of traditional offline word of mouth, and it is characterized by the simultaneous acquisition of information from various sources (Chatterjee, 2001). The most distinct characteristic of communication done online is the interaction, which is manifested in comments (Kim & Yoon, 2010). Comments collectively can have a tremendous impact (Kim & Sun, 2006). This study paid attention to user review data in the mobile app markets because the comment section provides an opportunity for people to express their opinions and plays a role in providing additional information (Kim & Yoon , 2010). Most of the research on users' reviews online have been done on the impact of online comments and ratings on consumers' willingness to purchase products (Shin, Kim, & Park, 2009; Park, Wang, & Lee, 2019; Hwang & Kim, 2007). They often showed that online reviews had a significant impact on consumers' product evaluation and willingness to purchase products and that if the reviews were negative, they tended to more influential than positive ones (Park & Yoo , 2006; Hwang & Kim, 2007; Lee & Park, 2006). This showed that consumers were more likely to become sensitive to negative word of mouth considering uncertainties online lacking clues in their purchasing environment compared to offline (Lee & Park, 2006; Kim & Yoon, 2010).

Through these findings, this study noted that ratings and reviews were channels for expressing users' opinions, and it was intended to evaluate current e-library services based on users' comments and ratings and to suggest future improvement directions.

In this study, Google Play, the most popular App market, was used as a research tool. According to IGA Works, a mobile big data company, Google Play topped the list with 1.646 trillion won (78.6%) in the first half of 2019, One Store ranked second with 249.2 billion won (11.9%), and Apple App Store ranked third with 119.9 billion won (9.6%) (Lim, 2019). Google dominates the domestic App market with its Google Play, and in response, Korea's three main mobile carriers, KT, SK Planet, and LG U+ launched One Store, an integrated app store. Apple App Store is another popular App market used by many consumers (Park, 2020). It was reasonable to use Google Play Store as a tool for this research, given that 225 results were turned when electronic libraries were searched on Google Play Store whereas 82 for Apple App Store.

E-library applications provided by libraries available to general users were targeted for research. Libraries operated for students and institutional staff, the Bank of Korea, and universities were excluded. Search keywords were "Electronic Library." As of September 4, 2020, Google Play Store turned 225 search results. Among them, 51 applications that were not provided by public libraries were selected for analysis (see Table 1).

[Table 1.] Analysis Target

Analysis Target

In the case of A14, it has not been updated since 2015 and has not been utilized, but it has been provided continuously, causing inconvenience to users.

Currently, libraries are introducing VR, AI, and robot services in various ways to prepare for the 5G and Fourth Industrial Revolution, but it is necessary to improve the quality of services currently being provided as much as new services. The user's experience forms the image of the library, and it is fixed through one experience, so having them experience the service again after a bad experience can be harder than having them use it for the first time.

Looking at a result of analyzing the current reviews on electronic libraries, 1,068 out of 4,532 were satisfied with the application as a whole. The remaining 76.43% of users' reviews, excluding the 23.57% of the reviewers, were found to have negative and double meanings. Users who were satisfied with their overall use had positive opinions on the convenience of searching or booking, the availability of free e-books, the convenience of reading books anytime, anywhere, and the availability of various information and services without having to physically visit the library. It was shown that users who were satisfied with their use had positive assessments of the various advantages and current availability, but at the same time hoped that the application would stabilize over time, while understanding that it might only be natural that there were many errors when the application tried to integrate and manage so many libraries and book data across the country (see Table 2).

[Table 2.] Evaluation on the Overall use of the Library App

Evaluation on the Overall use of the Library App

key words such as "Sign up," "Log in," and "Log out" frequently appeared when user reviews of electronic libraries were classified by key words. This suggests that users were experiencing many problems from the Sign-up, and errors related to Sign-up are shown in Table 3. The most common errors were found to be "impossible to search for postal codes, addresses, etc.,"Which must be entered when signing up, or "Not authenticated by telephone numbers."

[Table 3.] User Reviews on Sign-up

User Reviews on Sign-up

In particular, the inconvenience that needs to be improved in relation to sign-up for the membership is the complicated process. For example, "as a procedure for children under the age of 14 to sign up, the user refers to the following procedure.

Next, Log-in-related errors can be distinguished as follows: First, users were experiencing situations such as an error that the correct ID/Password did not match the membership information, or that they were not registered, so they were asked to sign up again. In addition, users were found to be experiencing additional inconveniences in carrying out ID/Password authentication procedures several times, primarily judging that the member information they have subscribed to their own misspelling.

Regarding users' reviews that they could not log in after signing up, the service managers collectively said, "The log-in ID is an e-mail address. Please enter your e-mail address to the end when entering your ID." It is believed that this might not be the right answer because the system already indicated "ID (E-mail)."

The inconvenience associated with logging in includes applications that must be logged in at every access and errors that cause automatic log-in to be disabled, and users demanded that the system should be stabilized or required automatic log-in or log-in to be retained.

System malfunction cases include: App could not be found, blank screen, white screen, not working, not subscribed, not connected, not running, not opened, not turned on, unable to load pages, http 550, creating the library app, and failure to initialize the app. Key words for system instability include: forced termination, bounced off, stopped, turned off, unstable, and stabilization. System malfunction and instability issues are shown in Table 4.

[Table 4.] System Malfunction and Instability

System Malfunction and Instability

"System malfunction" includes the inability to connect to servers and the cases where the reasons are not specified and the system itself is not running. In particular, as the server itself was not operating/running, many users were either unable to access the app itself or experiencing unstable servers during use. The manager's answer included: "It could be due to network problems with the place where the application operating server is located, library computer system shutdown, and problems with user's network environment or device."

It is deemed necessary to check the satisfaction and quality of the basic e-book service. In particular, it is necessary to visualize how electronic libraries provided by general libraries differ from systems provided by commercial entities, thus showing differences at the system level.

There were also problems with slow App response, not loading at all, long loading time, and delays.

[Table 5.] Frequent Errors and Bugs

Frequent Errors and Bugs

Users pointed out that frequent errors and bugs were occurring and noted the need for systematic improvement. Rather than pointing out specific errors, they made a comprehensive appeal for the inconvenience of using them due to frequent errors.

Meanwhile, users tried to solve the problems they encountered by deleting and reinstalling to the point where they labelled it a "Disposable App."

It was been shown that the lend-download-reading process was not working smoothly (see Table 6). There was a problem with shortened lending period because download-related errors occurred after the books were lent. In that case, the administrator would need to extend the lending period. It seemed that there were not enough staff to handle such matters that require immediate response, and it is necessary to provide real-time chat and Q&A support to the users.

[Table 6.] Errors in Lending and Downloading e-books

Errors in Lending and Downloading e-books

A user commented in his/her review, "There was a notice that if there was a download error, you could reinstall the application." The problem was that it offered an ironic solution that required the user to go through the cumbersome process of removing and reinstalling the app whenever the frequent download errors occur. The fundamental solution to this problem is the stabilization of the system, and it is expected that there should be no basic problems in the system.

In addition, additional information on contact, e-mail address, Q&A bulletin board, or real-time chat links will be needed, along with a notice saying "Ask the administrator" in case of an error.

It is necessary to improve the overall search function because "Search functions" are often inconvenient or not working in many aspects from the user's perspective.

The key word "Update" was frequently found in reviews for electronic libraries, and the related issues are shown in Table 7.

[Table 7.] Update related Issues

Update related Issues

Some users were dissatisfied with the app update and left comments that the previous version was better or that the upgraded app was rather inconvenient. Users who were familiar with an existing interface felt uncomfortable or referred to poor readability. In some cases, the diversification of functions made it more complex. It seems that mobile devices and tablets, unlike computers, need to selectively and concisely provide key and important information and information that is widely used, considering that the scope of their displays are limited.

The reasons for these "not satisfied after an update" comments were that the administrators or operators did not have a good grasp of the users, did not fully consider the direction of improvement in consideration of convenience to the user/citizen, and did not know the direction the users wanted. In response, regular surveys seem to be needed to develop and update the services from a short, medium, and long-term perspective. This is considered as important as reflecting the individual needs of the users. On the other hand, there were comments that expressed satisfaction, pointing out existing problems were solved as well.

Another problem was errors after an update. This could be largely divided into cases where users who had not previously experienced errors experienced system instability, including errors related to log-in, book search, forced shutdown, etc., and errors where personal contents, books contained in the personal library section, preselected information, and personal information after an update disappeared. Furthermore, comments referring to "not working after an update" were easily found. It seems urgent to discuss systematizing the update and testing the system to stabilize it after the update.

Reasons for frequent updates were, first, the system itself was fundamentally unstable, and second, many updates were required in the process of improving these unstable systems. Third, as additional errors occurred frequently after updates, subsequent updates for the updates were needed, creating a vicious cycle. Since there were many inconveniences after updates, careful approaches and tests seem essential.

In terms of usability, some commented that they did not know how to use the basic functions, so it's necessary to provide a guide screen (tutorial) on the app screen as well as a simple instruction manual to use e-books.

As for readability, some users wanted to add and improve settings in font, letter spacing, background color, and margins. In particular, in the case of fonts, characters were cut off from the screen at a certain size, and users wanted the most suitable font for the e-book environment, conspicuous fonts, etc. Users also wanted to be able to import and select fonts from specific folders like other e-book Apps, in addition to fonts in the system. For letter sizes, it is necessary to raise the upper limit of maximum letter size in consideration of different age groups.

On the other hand, a scroll-type book-viewing format was requested to resolve the lag/stumbling that was appearing on every page turn.

[Table 8.] User Interface

User Interface

For the "Poor management" sector, users pointed out a series of errors and mentioned the need for basic checks and tests, such as whether the apps work properly. They also mentioned insincere responses, problems that could not be solved for a long time. Despite the absence of feedback and inconvenience affecting users in real time, the problems were not solved in time, which needs to be improved. Therefore, the library should first have the perception that reviews are the opinions of users and that proper responses to them are essential. Second, it seems necessary to open real-time response channels. Third, advance notice of updates or check is required.

Above all, only 862 (19.02%) administrators' responses were registered to 4,532 user comments collected in this study. In addition, the biggest issue of poor management was the fact that the same problems were repeated and they did not fundamentally improve. It appeared that some users stopped using the service because the same problems were not resolved even though they kept brought up.

Next, in "personal information" Issues, there were comments that pointed out that contact information (Address book) and camera access rights were unnecessary, and there were inconveniences that the libraries did not provide an explanation, especially as additional access authority was requested when updating the App.

The administrators mentioned that access to users device functions such as phone calls, camera, and location were to set the library service functions, but they still need to clear guidance and manual for users to download for both users and the libraries. In addition, it seems to be necessary to carefully check areas where personal information including information provided for sign-up and membership are involved.

[Table 9.] Service Management

Service Management

With the use of multi-device by users becoming common in recent years, library applications are required to provide a stable environment for reading on various devices such as PCs, e-book readers, mobile phones, and tablets. This shows that general availability is required not only for the types of devices such as tablets and e-book readers, but also for the general availability of individual models, versions (old and latest versions). In particular, as a way to use it in Crema d, there were comments like "I extracted and installed APK because Google Play Store could not be installed, but it doesn't work, is it difficult to distribute APK without depending on Google or Naver's app store ?" So, various aspects of consideration are required to ensure general availability.

Various devices have a negative impact on reading, depending on the level of optimization for each device. For example, there are problems such as the letters on the tablet being cut off or the icons that set the font size, spacing, background color might not be visible. Therefore, it seems that the compatibility of each device should be stabilized, developed, and tested.

Application errors were occurring in various cases including network problems. It is feared that space the content takes up also causes problems in the overall operation of the service.

For compatibility, users in each device environment were found to feel uncomfortable when using Apps and visiting web pages. In response to this, the administrators admitted that the improvement of problems such as lack of compatibility of mobile devices and restriction of the search function and the attempt to provide library information and various functions of several libraries through one search were mixed, causing inconveniences to users.

[Table 10.] Other Issues

Other Issues

Various errors are occurring in electronic library applications, and 301 (6.64%) cases of system malfunction (library service connection errors) occurred among 4,532 total error cases. The basic system deployment itself is unstable, and this instability is accompanied by frequent updates and additional errors due to updates. Therefore, improving the quality of electronic library services based on current applications requires, among other things, systematic improvement and stabilization.

The need to install viewers for each publisher or company according to the copyright of e-books is a problem that cannot be solved in a short period of time, but it can be operated based on a wide range of compatibility, such as device-specific and version-specific offerings. Also, screen ratio and all functions should be implemented smoothly regardless of each device environment such as tablets, e-book readers, and PCs as well as mobile phones.

Review analysis shows that as additional errors occur frequently after updates, another update for updates ensues, leading to a vicious cycle. Since there are many inconveniences after updates, it seems that tests are essential. Also, users need to be informed before the update to ensure that it does not interfere with their use and for expected problems that occur frequently in the system after the update so that they can prepare for problems from the perspective of all parties involved.

The inconveniences that need to be improved in relation to sign-up for the membership are the complicated process. For example, "as a procedure for children under the age of 14 to sign up, the user refers to the following procedure.

It is necessary to provide real-time chat, Q&A bulletin boards, contact information, or e-mail where users can easily find them, and to provide a window for users to immediately solve problems that occur to them in real time. Active management of applications and enhanced user support are essential, given that only 19.02% of all reviews have been answered.

Users should be guided and supported so that they can conduct a detailed evaluation by dividing them into systems, services, convenience, usability, amount of content, management aspects, etc. In addition, managers need to not only check this regularly, but also seek feedback and practical changes.

Currently, it is necessary to provide various functions and user manuals in the application from the basics, such as how to provide e-books in electronic libraries, how to subscribe and provide e-books, the library's efforts to provide e-books, and why users are forced to install multiple apps. These basic guides and user manuals are expected to help users understand the operation of electronic libraries and support efficient use. In the case of the user manual, it seems that the content of e-books and other additional services can be provided through the library, YouTube, etc. so that anyone can easily follow them.

Currently, additional functions requested by users include settings that allow them to further tune and maintain the application, mainly in their own use environment. After basic functions are provided, it is necessary to focus on how more convenient they will be for individual users and on providing services that enable them to create a customized reading environment through setup and selection functions.

In addition, it is necessary to stabilize the service so that users do not feel that integration and linked apps, which are promoted to provide more information and services, are not a disadvantage, and to actively promote various online services provided by libraries, including electronic libraries, in the post-COVID-19 era to promote the use of the services.

In addition to the electronic libraries provided by the regular libraries, users have high expectations on mobile applications that have already been developed on other platforms such as YES24 and Kyobo Bookstore and those that are familiar to users, and they are familiar with the interface, design, etc. If one wants to further develop a mobile application that has already been developed, one should develop it to run the functions of the existing application smoothly and at the same time differentiate it.

E-book, which can be read using smart devices such as tablets and smartphones, provides the pleasure of reading free of time and space constraints. Domestic e-book subscription services include e-libraries provided mainly by regular libraries and services such as Millie, RidiSelect, Yes24 Book Club, and Kyobo Bookstore Sam. In addition to providing free reading opportunities, it seems time for libraries to prepare for the leap in electronic libraries through quantitative and qualitative improvements.

Among the ways in which users choose an application, application reviews by other users play an important role in making purchase or download decisions (Kim et al., 2012). Therefore, the significance of this study can be found in that it provides a future direction for improvement that reflects the user's opinion. Nevertheless, the limitation of this study is that it was measured based on user reviews that could be found in a specific mobile App market. In addition to reviews, related research can be materialized through surveys on users.