Quality and Readability of Online Information Available for the General Public on Orofacial Granulomatosis

INTRODUCTION

In the early 20th century discovery of a disease characterized by temporary facial oedema and recurrent facial palsy was first documented and reported by two authors- Hubschamann and Rossolimo.1 Many authors have reported cases since then. Luscher2 described the presentation of lip swelling accompanied by features of lower motor neuron lesion in 1949. Weisenfield described a similar presentation.3 Melkersson reviewed a case of short-lasting oedema of the face with recurrent facial palsy, supplementing a link between the two.4 Rosenthal described a triad of clinical features; facial oedema, facial palsy and a fissured tongue.5 This clinical triad came to be known as 'MelkerssonRosenthal syndrome' (MRS).

Although orofacial granulomatosis remains a relatively rare disease, the number of individuals that have OFG is increasing, leading them to seek information on several aspects of the disease on the World Wide Web.5 However, based upon some previous studies of oral diseases,7,8 it is possible that information available to the public will vary qualitatively and may possibly be difficult to read and/or comprehend. Since there are no previous studies on the quality of information on the web pertaining to OFG, there is a requirement to determine how likely it is for patients and/or their carers to find easily readable information without compromising the accuracy of the material on the disorder.

METHODOLOGY

On the 22nd of March, 2018 four key terms; 'OFG disease', 'orofacial granulomatosis', 'cheilitis granulomatosa' and 'Melkersson Rosenthal Syndrome' were searched on the web using the search engine google.co.uk. They yielded 253000, 111000, 21,200 and 146 results respectively. No advanced sorting or refinement was done on these results at this stage and only the top 100 consecutive sites were shortlisted from each of these searches. These top 100 websites were screened and assessed. The irrelevant and repeated sites were not included. Furthermore, exclusion criteria were applied. Scientific research articles, book excerpts or reviews, websites that demanded authenticated password access and links that contained information in a language other than English were excluded. This resulted in 58 links that were then characterized as done in the Ni Riordain and McCreary (2009) paper based on 4 criteria, namely, affiliation (if it was commercial, belonged to a nonprofit organisation, government or a university or medical centre), specialisation (exclusively confined to orofacial granulomatosis or partly related to orofacial granulomatosis), content type (whether the content contained medical facts, clinical trials, human interest stories or question and answers) and content presentation (image, video and audio)

Three instruments were used to assess the quality of information in the shortlisted websites. These were the Journal of the American Medical Association (JAMA) benchmarks for website analysis (Silberg et al, 1997), the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) (Shoemaker et al, 2013) and the presence of Health on the Net (HON) seal. An excel spreadsheet was created to aid in systematic collection of data

The aim of the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) is to determine understandability as well as actionability of education material available to patients. Understandability refers to materials being easily understood when people from different backgrounds (health related or non-health related) and of variable knowledge on the topic of health can understand and interpret messages from the text. Actionability refers to the individuals identifying and acting upon the information provided (Shoemaker et al, 2013). The PEMAT-P for printable materials (e.g., brochures, pamphlets, PDFs) contains a total of 24 questions-17 pertaining to understandability and 7 relevant to measuring actionability. Each question has a response action of 0 or 1 (0=Disagree, 1=Agree). Additionally, some questions can also be answered with a 'not applicable' (N/A) response depending on available information (e.g., tables or diagrams being absent). The material's scores are then calculated individually for understandability and actionability. A higher score is representative of higher understandability and actionability. For example, an understandability score of 90% is more understandable than a material with a score of 50%. The same rule applies for actionability. Although the PEMAT cannot be used for podcasts or to evaluate friendliness of websites, it is the only tool that measures actionability of a material. The questionnaire used in PEMAT is shown in Table 1. The material being assessed by PEMAT may be highly understandable but it cannot be relied upon entirely for accuracy or comprehensiveness. For this reason, supplemental quality assessments are essential. The JAMA benchmarks were hence used to evaluate the quality of each website analysed by PEMAT. This instrument requires fulfilment of 4 criteria; display of authorship (authors and contributors

along with their credentials), display of attribution (citations), disclosure (enlistment of ownership of medical content, sponsorship, commercial funding availability and conflict of interest) and currency (dates on which the material was posted or updated) (Silberg et al,1997). The HON seal was founded in 1995 by a non-profit organisation. This Swiss-based seal deems medical information as a source of quality information for health professionals, patients and laymen alike (Hon.Ch, 2015). This seal (Figure 1) is displayed on websites that abide by ethical conduct set down by HON. Eight outlined principles are required to fulfil the criteria to be accredited by HON. These are authority, privacy, attribution, justifiability, transparency, complementarity, financial disclosure and advertising policy (Table 2). The explanation of each principle

is listed below in the table. It must be kept in mind, however, that the HON seal is neither an indicator of accuracy nor of comprehension. To evaluate the readability of health content online, two assessment tools were used. These were the Flesch Reading Ease Score (FRES) and the Simple Measure of Gobbledygook (SMOG).

The Flesch Reading Ease Score (FRES) was developed by Rudolph Flesch in 1948. It uses a formula to indicate the reading ease of a given text, thereby grading it from very easy to very difficult. The formula it uses is: 206.835 - (1.015 x average sentence length) - (84.6 x average number of syllables per word)

A higher score correlates to a more readable passage and vice versa. In the current study, the readability score was calculated using the online programme (https://readability-score.com) by pasting a text of up to 500 words from the website onto the calculator. The readability ease is indicated in the Table 3

The SMOG Readability Formula was created by McLaughlin (1969). It is a simple method used to determine reading level of written materials. It was enhanced by Harold C. McGraw in 2008. It makes use of sentences and words with 3 or more syllables in a text and a conversion table to assess the readability. It is available for short as well as long materials. The SMOG conversion tables are shown as Table 4.1 and Table 4.2. The site used was http://www.readability formulas.com/free-readability-formula-tests.php.

RESULTS

The search from the four different keywords yielded a total of 385,346 results. The results were narrowed down to 400 the top 100 consecutive websites from each of the term searched. Among these 400 results, a total of 32 links were repeated and were excluded. Two links were non-operational and hence excluded. Out of the remaining 366 links, further 261 websites were removed as they were links to scientific articles or research papers, 10 links were removed because they were links to book pages, 7 links were removed because they were blogs or discussion panels, 5 required a password for access, 11 had information in a language other than English and 14 contained either just pictures or no information on the disease and were deemed irrelevant.

This process resulted in 58 websites (12 from OFG disease, 15 from orofacial granulomatosis, 15 from cheilitis granulomatosa and 16 from Melkersson Rosenthal Syndrome) that were then assessed and reviewed for further analysis. The process of selection and exclusion is illustrated below in Figure 1.

Categorisation of the 58 websites was based on affiliation, specialisation, content type and content presentation. Figure 2 shows the categorised data. Majority of the websites represented data from non-profit organisations (60%), followed by commercial websites (26%). All websites except one exclusively contained information regarding the search term used (e.g. orofacial granulomatosis). Twenty-one websites (36%) contained at least one image to aid understanding. All of the sites had medical facts. There were

neither clinical trials nor human interest stories.

Quality of information was variable. The results of the questions answered and their respective percentages are recorded according to the PEMAT in Table 5. The total understandability score was just 55%, making it difficult for the reader to understand and comprehend on most websites. The total actionability score was only 22% for the 58 websites and hence poor.

Only one website (https://www.guysandstthomas.nhs.uk/ resources/patient-information/nutrition-and-dietetics/ cinnamon-and-benzoate-free-diet-OFG.pdf) had the highest understandability and actionability score of 84.6% and 83% respectively, whereas the lowest score for understandability was 20% scored by two websites. The lowest actionability score was 0, scored by 17 (29%) websites.

Out of the 58 websites, 42(72%) made their purpose evident. Nine (16%) websites contained a distraction in the form of advertisements. Only eighteen (31%) used understandable language and even fewer (24%) defined and explained medical terminology used on the website. Only one website required calculations and explained how to do so. Information was broken into shorter sections in fortyseven (81%) sites. Text was presented in logical sequence in forty-four (76%) websites. Only one (2%) website provided a summary. Approximately half (48%) used highlights and bullet points to direct to key points. Forty (69%) websites identified at least one action to be taken, however, instructions were directly addressed only in eight (14%) websites. None of the web-links made the use of visual aids to make instructions regarding taking action easier.

With regard to JAMA benchmarks fifty-one (88%) websites fulfilled the benchmark for the disclosure, whereas attribution benchmark was met by only nineteen (33%) websites. The authorship and currency benchmarks were fulfilled by twenty-four (41%) and thirty-seven (64%)

websites. There were seven (12%) websites that achieved all four benchmarks. Table 6 shows the percentages of benchmarks achieved. A minimum of one benchmark was achieved by all 58 websites. Only 5 (9%) websites had the HON seal displayed on their page. The Flesch Reading Ease score ranged from a minimum of 0 on multiple websites to a maximum of 69.9 on one website indicating that available information online ranged from being very difficult to standard in comprehension. The mean score was 22.7. Fifty-four (93%) of the websites contained material that was difficult to very difficult. Only one (2%) website had fairly easy content. The Simple Measure of Gobbledygook (SMOG) index ranged from 7.6 to 22.2. The mean score was 13.4. According to the SMOG index, all the material available online on the 58 websites was readable for a 6th-8th Grader as illustrated below.

DISCUSSION

Quality of life is severely affected in individuals with chronic conditions who have poor health literacy.9 In this modern day and age, accessibility to information has become easy and readily available, mainly due to technological advancements.10,11 Health information online has enriched patients with knowledge to bridge the gap between themselves and the physician, prior to their appointment with a health consultant.12 Many individuals report conducting successful online searches related to a disease before visiting their physician.13

The health related information on the internet is certainly advantageous but the drawbacks must also be considered. Immediate access, personal privacy, perceptual variety from various sources, reduced appointment time and convenience of the patient are some of the advantages highlighted in the literature.14 Some identified disadvantages include excessive information, complicated medical terminology, lack of credibility and misleading support groups.15 It is essential that health information available online should be easily readable and understandable for the target population i.e. patients/general population.16 A website should be an effective communication tool directed towards the patient which is only possible when the information enlisted is easily understood.17

This means that in addition to assessing the quality of online readable health information, assessment of readability is also important.

Orofacial granulomatosis is a rare disease and hence, information on it is scarce and scattered on the internet. Prior to this study, there have been no quality and readability assessments of online information on OFG. The details of epidemiology are limited with variable knowledge on clinical features. The treatment strategies vary according to different authors, ranging from diet modifications to use of systemic immunosuppressants and surgery.

The objective of this study was to investigate online information comprehensively which could be easily accessible to patients and laypersons alike. In the current study, the Google search engine (google.co.uk) was used as Google remains the most commonly used search engine according to statistics from the Statista website (www.statista.com) in 2018. A recent study suggests that information found online can mislead individuals from non-medical backgrounds.18 Although online information plays a role in increasing knowledge of an individual regarding the disease and its treatment, younger and more educated people benefit from these websites more than the aged and less educated.19

In the current study, it is evident that patients would find information challenging or carers might find it difficult to gather information regarding orofacial granulomatosis. The search was narrowed down to 58 links out of 385,346 hits from the four search terms initially used. After excluding the non-functional and duplicated webpages, it was found that only 24% of the websites had easy language that did not require a medical background to understand. This indicates that individuals seeking information on OFG might fail to find the relevant comprehensive answers. Sources of information were classified broadly into four different categories-university/hospital, non-profit organizations, commercial and governmental.20 Majority of the websites (60%) belonged to non-profit organizations and therefore one would hope that the information provided would not be distorted by a hidden commercial agenda. However, the material contained therein could be considered too complicated for a lay person.

It was seen that at least one action was mentioned in many websites (69%) to help resolve the symptoms. However, all 58 websites focused heavily on the aetiology of OFG. Contrary to a study by Houts et al.,21 which highlighted that the use of graphic content such as pictures along text can benefit the reader, none of the 58 websites included pictures of treatment options to make understanding of the reader better. Levels of health literacy, defined as the ability of a person to retrieve, process, understand and apply basic health information, is known to be low. The assessment of the quality of online information is therefore key to determine whether patients of questionable health literacy levels can benefit from their online read. The public faces a challenge to access accurate health care information especially due to low levels of health literacy.22 Many tools have been devised to assess quality which include the CDC Clear Communication Index but they do not consider the audience (www.cdc.gov) and hence are not as effective as the PEMAT. The Knee Osteoarthritis Patient Education Questionnaire (KOPEQ) is another disease specific tool which has been used to assess validity of a patient education material. In addition to possessing all the advantages of PEMAT it also evaluates the pedagogic value of information available.23

COMDQ has also shown to be a valid and reliable outcome measure for patients with chronic oral mucosal diseases in a UK population.24 As this is the first study on patient information in OFG, no disease specific tool was available for use. The quality of online information can be assessed via The Patient Education Materials Assessment tool (PEMAT) which is considered a consistent and reliable quality assessment tool. The PEMAT for printable material was used in this study as it demonstrates good inter-rater reliability (IRR) and adds objective value to patient education materials.25 A second reviewer assessed the results conducted using the PEMAT tool in this study to increase the reliability of the findings.

The mean understandability score of the study using PEMAT was 7 (55%) with a standard deviation of 2.8. The mean actionability score was 1 (22%) with a standard deviation of 1.1. The threshold value for PEMAT to deem a material qualitatively valid is 70%22,hence in this case the quality was poor overall. These results were influenced by websites having the right content but unable to present it in a simple, understandable manner.

In 1997, Silberg et al. published the JAMA benchmarks as a proposed quality standard for health information online. It mentions four criteria that must be clearly apparent on a website. These are authorship (author name and credentials), currency (date of publishing and updating), attribution (references and citations) and disclosure (conflict of interest).  

With regard to JAMA benchmarks less than half (41%) of the websites clearly stated the author, however, even fewer (33%) listed the credentials and qualifications they possessed. Only a small proportion of these websites (12%) mentioned the author, their credentials, date of publishing and declared that there was no conflict of interest. The fulfilled benchmarks serve as a reliable source of knowledge whereas lack of credentials makes the information less conclusive and hard to rely on by the general public.

Health care workers should be aware of patients using the World Wide Web as a means of gaining medical and dental information and be ready to assist patients in browsing through available medical information online.27 The health on the HON code is such a seal that indicates reliability of online health information to its users. Due to its long certification processing time, it is not commonly used by many authors since its initiation in 1995 (www.hon.ch). There is also a monetary contribution required for any website to be accredited by HON. The fees are £142 and £288 for a non-profit organisation and commercial website respectively to acquire the HON seal. In the current study only five (9%) of the online sources had the HON seal. This is lower than a study conducted on online information on treatment of leukoplakia which had 17% of the websites displaying the HON code.7 Another indicator of highly reliable information-The Information Standard-was recently made available by NHS but is under review to develop an improved and sustainable model (www.england.nhs.uk) and was not used in this study.

The question arises whether the information accessed is easy to read for patients seeking online material about their condition or treatment. The readability of a text is defined as the level of comprehension an individual should have to understand and comprehend the material (Weiss et al 2003). The FRES is a commonly used readability tool and was supplemented by the SMOG in this study. Only one website had a readability score interpreted as fairly easy and it was simple to comprehend the information listed, whereas, the readability of the remaining websites varied from difficult to very difficult. The National Institute of Health (NIH) and American Medical Association recommend that information meant for general public or patients should not surpass the sixth-grade level. The mean reading score was above sixth grade in this study, showing that most websites were hard to understand for the general population. Long words may make a sentence complicated.28 It is also suggested that simplified information and shorter words in sentences may fail to inform individuals of the disease properly if they are aware of commonly used medical/scientific terminology.29

SUMMARY AND CONCLUSION

This study is the first of its kind, assessing the quality and readability of online information on orofacial granulomatosis. OFG is a rare, yet increasingly apparent disease entity. The information pertaining to it is scarce and difficult to understand for the general public. This study has demonstrated that even using four different search terms the Google search engine failed to deliver good quality, easily readable text for patients.

It would be reasonable to conclude that improvements need to be made in order to make information regarding OFG on the internet simplified yet comprehensive, so modern day patients can understand it, irrespective of the education level they possess. This process of betterment can be made by information providers and healthcare workers on the web for people who wish to study their symptoms online. Emphasis should however, be made by professionals warranting patients to visit a dentist or other healthcare providers for effective diagnosis and relevant treatment.

SOURCE OF FUNDING

Nill

CONFLICT OF INTEREST 

There are no conflicts of interest

REFERENCES

1. Mignogna MD, Fedele S, Russo LL, Muzio LL. The multiform and variable patterns of onset of orofacial granulomatosis. J Oral Pathol Med. 2003;32:200-5. https://doi.org/10.1034/j.1600-0714.2003.00106.x
2. Luscher E. Syndrom von Melkerson-rosenthal. Schweiz Med Wschr. 1949;79:1-3.
3. Wiesenfeld D, Ferguson M, Mitchell D, MacDonald D, Scully C, Cochran K, et al. Oro-facial granulomatosis-a clinical and pathological analysis. QJM:1985;54:101-13.
4. Melkersson E. Ett fall av recideverande facial spares. Samband Med Angioneurotisk Odem Hygeia (Stockholm). 1928;90:737-41.
5. Gale G. Disorders of the Orofacial and Gastrointestinal Tract2014.
6. Rana AP. Orofacial granulomatosis: A case report with review of literature. J Indian Soc Periodontol. 2012;16:469-74. https://doi.org/10.4103/0972-124X.100934
7. Wiriyakijja P, Fedele S, Porter S, Ni Riordain R. Web‐based information on the treatment of oral leukoplakia – quality and readability. J Oral Pathol Med. 2016;45:617-20. https://doi.org/10.1111/jop.12459 
8. Alnafea S, Fedele S, Porter S, Ni Riordain R. Online Information on the Treatment of Burning Mouth Syndrome: Quality and Readability. J Oral Facial Pain Headache. 2017;31. https://doi.org/10.11607/ofph.1717
9. Panagioti M, Skevington SM, Hann M, Howells K, Blakemore A, Reeves D, et al. Effect of health literacy on the quality of life of older patients with long-term conditions: a large cohort study in UK general practice. Qual Life Res. 2018;27:1257-68. https://doi.org/10.1007/s11136-017-1775-2
10. Fiksdal AS, Kumbamu A, Jadhav AS, Cocos C, Nelsen LA, Pathak J, et al. Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives. J Med Internet Res. 2014;16:e224. https://doi.org/10.2196/jmir.3341
11. Jing Z, Tao W, Xiucheng F. Patient value co-creation in online health communities: Social identity effects on customer knowledge contributions and membership continuance intentions in online health communities. J Serv Mana. 2015;26:72-96. https://doi.org/10.1108/JOSM-12-2013-0344
12. Osei-Frimpong K, Wilson A, Lemke F. Patient co-creation activities in healthcare service delivery at the micro level: The influence of online access to healthcare information. Technol Forecasting and Social Change. 2018;126:14-27. https://doi.org/10.1016/j.techfore.2016.04.009
13. Cotten SR, Gupta SS. Characteristics of online and offline health information seekers and factors that discriminate between them. Soc Sci Med. 2004;59:1795-806. https://doi.org/10.1016/j.socscimed.2004.02.020
14. Skinner H, Biscope S, Poland B. Quality of internet access: barrier behind internet use statistics. Soc Sci Med. 2003;57:875-80. https://doi.org/10.1016/S0277-9536(02)00455-0
15. Cline RJW, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res. 2001;16:671-92. https://doi.org/10.1093/her/16.6.671
16. Daraz L, Morrow AS, Ponce OJ, Farah W, Katabi A, Maajzoub A, et al. Readability of Online Health Information: A Meta-Narrative Systematic Review. Am J Med Qual. 2018;0(0):1062860617751639. https://doi.org/10.1177/1062860617751639
17. Ahmed OH, Sullivan SJ, Schneiders AG, McCrory PR. Concussion information online: evaluation of information quality, content and readability of concussion-related websites. Br J Sports Med. 2012;46:675-83. https://doi.org/10.1136/bjsm.2010.081620
18. Wathen CN, Harris RM. "I Try to Take Care of It Myself." How Rural Women Search for Health Information. Qual Health Res. 2007;17:639-51. https://doi.org/10.1177/1049732307301236
19. Jacobs W, Amuta AO, Jeon KC. Health information seeking in the digital age: An analysis of health information seeking behavior among US adults. Cogent Social Sciences. 2017;3:1302785. https://doi.org/10.1080/23311886.2017.1302785
20. Hanif F, Read JC, Goodacre JA, Chaudhry A, Gibbs P. The role of quality tools in assessing reliability of the Internet for health information. Informa Health Social Care. 2009;34:231-43. https://doi.org/10.3109/17538150903359030
21. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: A review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61:173-90. https://doi.org/10.1016/j.pec.2005.05.004
22. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96:395-403. https://doi.org/10.1016/j.pec.2014.05.027
23. Huber EO, Bastiaenen CH, Bischoff-Ferrari HA, Meichtry A, de Bie RA. Development of the knee osteoarthritis patient education questionnaire: a new measure for evaluating preoperative patient education programmes for patients undergoing total knee replacement. Swiss Med Wkly. 2015;145:w14210. https://doi.org/10.4414/smw.2015.14210
24. Ni Riordain R, Hodgson T, Porter S, Fedele S. Validity and reliability of the Chronic Oral Mucosal Diseases Questionnaire in a UK population. J Oral Pathol Med. 2016;45:613-16. https://doi.org/10.1111/jop.12425
25. Vishnevetsky J, Walters CB, Tan KS. Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT). Patient Educ Couns. 2018;101:490-6. https://doi.org/10.1016/j.pec.2017.09.003
26. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. J Am Med Assoc. 1997;277:1244-5. https://doi.org/10.1001/jama.1997.03540390074039
27. Diaz. AJ, A. GR, J. NJ, E. RS, D. FP, W. MA. Patients' Use of the Internet for Medical Information. J Gen Intern Med. 2002;17:180-5. https://doi.org/10.1046/j.1525-1497.2002.10603.x
28. Khushabu K, Poonam M, R. HD, Nitin A, Soly B, Michael S, et al. Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol. 2013;3:325-33. https://doi.org/10.1002/alr.21097
29. Matthew E, K. DA, Kristien B, A. FJ, M. DO. Patient Information in Graves' Disease and Thyroid-Associated Ophthalmopathy: Readability Assessment of Online Resources. Thyroid. 2014;24:67- 72. https://doi.org/10.1089/thy.2013.0252