Research Article - Biomedical Research (2018) Volume 29, Issue 11

Validation of educational material for diabetes self-management education: Judgemental and criterion validity

Kisokanth G^1*, Prathapan S², Indrakumar J³ and Ilankoon IMPS⁴

¹Department of Supplementary Health Sciences, Faculty of Health-Care Sciences, Eastern University, Sri Lanka

²Department of Community Medicine, Faculty of Medical Sciences, University of Sri Jayewardenepura, Sri Lanka

³Department of Medicine, Faculty of Medical Sciences, University of Sri Jayewardenepura, Sri Lanka

⁴Department of Allied Health Sciences, Faculty of Medical Sciences, University of Sri Jayewardenepura, Sri Lanka

*Corresponding Author:

Kisokanth G
Department of Supplementary Health Sciences
Faculty of Health-Care Sciences
Eastern University

Accepted date: April 17, 2018

Abstract

Objective: To describe the validation process of an educational material for diabetes self-management education among patients with type 2 Diabetes Mellitus (T2DM).

Methods: Methodological study, developed in three phases: construction of the educational material, validation of the educational material by judges, and by patients with T2DM (criterion validity). The validation process was conducted by 7 judges and 30 patients with T2DM. Criterion validity was assessed against gold standard by field expert. A 70% or more total disagreement of any of the items by the experts was decided to remove from the educational material. Criterion validity was measured by sensitivity, specificity, as well as the area under receiver operating characteristic (ROC) curve. The expected primary outcome was the HbA1c level to be decreased by 0.5% or more after the implementation of intervention guide among patients with T2DM.

Results: The changes suggested by judges were accepted and included in the final version of the educational material. All experts (100%) agreed that illustrations and texts in the education material motivate the patient with T2DM to understand the important of lifestyle modifications. The sensitivity of the developed intervention guide was 81.5% (95% CI=61.9-93.7%) with a specificity of 100.0% (95% CI=15.8-100.0%) and the area under ROC curve of 0.907. Intervention had a statistically significant reduction (p<0.05) in HbA1c [8.60% (IQR 2.60) vs. 7.40% (IQR 2.10)] and FBS level [159.00 mg/dl (IQR 77.50) vs. 134.00 mg/dl (IQR 40.50)] at the end of 3 months. Further, the mean BMI at baseline was higher compared to 3 months of intervention [24.88 Kg/m2 (SD ± 3.06) vs. 24.19 Kg/m2 (SD ± 2.79)] which was statistically significant (p=0.000).

Conclusion: The educational material was validated in terms of judgment and criterion validity. It would be used by nurse health educators for diabetes self-management education.

Keywords

Educational materials, Validation, Self-management, Diabetes mellitus

Introduction

Educating the patients includes providing information and support, correcting misconceptions and helping them to understand the roles and responsibilities of self-care [1]. Further, education can help them and their families to recognize and minimize fear and anxiety, cope with stress and avoid complications [2] and increase self-efficacy improving self-esteem [3]. The use of printed educational material can be a complement to verbal guidance helping patients to achieve empowerment and easy the education process [4]. Educational materials are identified as the single most important component positively impacting the health status of patients with T2DM.

Before using the educational material, it has to be validated. Validity of a tool is the extent to which it measures what it is expected to measure in the target population [5]. There are many types of validity namely judgment and criterion validity. The judgemental validity included face, content and consensual validity. Face validity is how the variables appear in order for the investigator to measure what he/she wants to measure. Content validity is to check if all aspects of the measures were covered. Consensual validity is to assess the agreement of the experts on the tool [6]. Criterion validity is the best and most obvious way of appraising to find a criterion (gold standard) that is known or believed to be close to the truth and to compare the results of the measure with this criterion [6]. Further, criterion validity is used to validate an instrument when investigator attempts to develop a shorter or more convenient version of an instrument by correlation to a “gold standard” [7]. The educational material included actions to promote healthy eating, engaging in regular physical activities, guidance of stress management, factors for good glycemic control among patients with T2DM based on baseline study. Thus, this study was aimed to validate the educational material for diabetes self-management education using judgmental and criterion validity.

Methodology

Study design

This was methodological study consisted of three phases: construction of the educational material, validation of the educational material by judges (judgmental validity), and by patients with T2DM (criterion validity).

Construction of the educational material

Educational material was developed by principal investigator (PI). This educational material is a guide with lifestyle modifications instructions including a dietary and physical activity which was prepared along with the results of the baseline study [8-10] which includes knowledge regarding T2DM, attitudes of patients and culturally acceptable practices for self- management of T2DM among Tamils with T2DM and factors for good glycemic control among patients with T2DM.

The educational material had twenty two (22) pages including cover page and content page. The seven pages (5-11) presented text and diagram displays on the knowledge aspects of T2DM. Other seven pages (12-18) demonstrated the health instruction of healthy eating (food that are recommended and those that should be avoided), physical activities and stress management for successful management of T2DM. Next two pages (19, 20) included details on factors associated with good glycemic control and the last page displays the checklist of patients’ aim to achieve better glycemic control. Initially, education material was prepared in English language.

Judgemental validity

This intervention guide was validated judgementally by judges who experts in the field of Medicine, Community Medicine, Nutrition and Nursing. They were invited to check the relevant and suitable guide for the Sri Lankan setting. The experts consisted of the following specialists; The Consultant Community Physician (1), Consultant Physicians (2) with special experience in diabetic care, Nutritionist (2) with special experience in diabetic care and Nursing professional (2) with special experience in diabetic care.

The intervention guide was sent to the above experts for their comments and opinion. The PI explained the purpose and objectives of this step to the experts. They were instructed to read the intervention guide carefully and voice any difficulties on testing it and to check for the following; ambiguity-if the items are ambiguous or poorly worded, double barreled statements-if one item asks two or more questions at the same time, each of which can be answered differently and jargon. The most appropriate medical words in the guide as it was to be used by medical professionals.

The experts were requested to give their feedback to the PI. The PI analyzed the comments of experts for each item as follows: Percentages of total agreement, percentage of agreement with minor changes, percentage of agreement with major changes and percentage of total disagreement for each item. A 70% or more total disagreement of any of the items by the experts was decided to remove from the educational material. The items with agreement with minor and major changes were highlighted to all the experts. With their approval, many of the minor changes were corrected and some of the major changes were changed after majority of the experts agreed to that particular change. An evaluation form was designed to evaluate the content and appearance of the educational material by the judges, with seven items. The judges performed the corresponding evaluation of the agreement and relevance of each item (agrees or disagrees). Further, the evaluation form consisted of open questions for comments and suggestions. At the end of the evaluation, the recommendations of the judges were accepted and incorporated.

Translation of educational material

The educational material was prepared in English and the first step was to translate the guide to Tamil language. Translation, back translation method was used during the translation process. The intervention guide was translated to Tamil by two independent language experts. They were requested by the PI to retain the original structure and content as much as possible. The PI discussed the variations with the two experts and consensual alterations were made. The agreed Tamil version of the intervention guide was back translated to English by other two English-Tamil language experts. The back translated version was again rechecked with the original intervention guide by the PI for consistency. Any discrepancy was corrected again after discussions with the expert translators.

Criterion validity

A group of Tamil patients with T2DM were randomly selected and intervention guide was implemented among them individually at medical clinic, Base Hospital, Kaluwanchikudy after obtaining permission from Medical Superintendent of the hospital. Before implementing the intervention guide to the study sample, baseline HbA1c, FBS and body mass index (BMI) were measured and recorded. As intervention guide consists of two major components such as dietary modifications and physical activities (lifestyle modification), were implemented in a selected group of Tamil patients with T2DM for the period of 3 months. After completion of 3 months, HbA1c, BMI and FBS were again measured and recorded. Informed consent was obtained after explaining the purpose and the nature of the study.

The minimum sample size calculation was based on the testing of criterion validity. The required minimum sample size was calculated at the required absolute precision level for sensitivity and specificity based on Buderer’s formula [11].

N1 sample size based on sensitivity=Z2Sn (1-Sn)/ L2 x Prevalence

Z=Standard normal deviation for the selected level of confidence, which is 1.96 for a confidence level of 95%.

Sn=Expected sensitivity as (80%)

L=Required level of precision (0.2)

Prevalence (good glycemic control)=proportion of patients having good glycemic control was considered to be 50% to obtain the maximum sample size

N1=1.96 × 1.96 × 0.8 × 0.2/ 0.2 × 0.2 × 0.5

N1=30.7 ≈ 30

N2 based on specificity=Z2Sp (1-Sp)/ L2 × (1- Prevalence)

Z=Standard normal deviation for the selected level of confidence, which is 1.96 for a confidence level of 95%.

Sp=Expected specificity as (80%)

L=Required level of precision (0.2)

N2=1.96 × 1.96 × 0.8 × 0.22/0.2 × 0.2 × 0.5

N2=30.7 ≈ 30

N1 ≈ N2, therefore N=30.

The post-hoc sample size calculation was performed with the sample size of 30 (at alpha 0.05) and power was 100%.

The expected primary outcome was the HbA1c level to be decreased by 0.5% or more after the implementation of intervention guide among Tamil patients with T2DM. The minimum significant reduction of HbA1C level was observed as 0.5% in self-management education intervention in several studies [12,13].

Every patient was assessed by a consultant physician (gold standard) at the baseline and at the end of 3 months after assessing the history, examination, BMI and FBS. The primary outcome of tool, HbA1c was measured after assessment by the consultant physician (gold standard) to avoid bias. The HbA1c improvement was cross checked with the gold standard.

Further, criterion validity was measured by sensitivity, specificity, predictive values and likelihood ratios [14] as well as the area under receiver operating characteristic (ROC) curve [15]. Receiver operating characteristic (ROC) curve is defined as a plot of sensitivity versus 1-specifity and the area under the curve (AUC), are the other measures of assessing the accuracy of test [15].

Overall improvement was assessed on a primary outcome (HbA1c) and secondary outcomes (FBS and BMI) between baseline and after 3 months of intervention by paired t-test or Wilcoxon signed-rank test based on normality distribution of the differences between the two related groups. The p value <0.05 was considered statistically significant.

Results

Assessment of Judgment validity

The recommendation were given by the judges are listed in the Table 1.

Table 1: Recommendations of judges.

Majority of judges (6/7, 85.7%) were agreed that the content covered in educational material regarding on healthy eating and physical activities are relevant to the patients with T2DM. Further, all experts agreed that illustrations and texts in the education material motivate the patient with T2DM to understand the important of lifestyle modifications as well as the applicability of the educational material in everyday clinical nurse health education practice can be possible. The overall agreement on content of the educational material by the judges is shown in Table 2.

Table 2: The overall agreement on content of the educational material by the judges.

Assessment of criterion validity

The total of 30 patients was included and 1 patient dropped out. The respondent rate was 96.7%. Majority of them were females (n=25, 86.2%).

Overall improvement of primary and secondary outcome

A Wilcoxon signed rank test showed that intervention through intervention guide had a statistically significant change in HbA1c and FBS level at 3 months of intervention. Further, the mean BMI at baseline was higher compared to 3 months of intervention [24.88 Kg/m² (SD ± 3.06) vs. 24.19 Kg/m² (SD ± 2.79)] which was statistically significant (p=0.000). Majority of participants (n=22, 75.9%) had improved their HbA1c level by ≥ 0.5% in 3 months (Tables 3 and 4).

Table 3: Mean/median of primary and secondary outcome variables between baseline and after 3 months of intervention.

Table 4: Overall HbA1c improvement among all participants.

Individual improvement of primary outcome

Gold standard of the intervention guide was assessed by the consultant physician based on the clinical improvement of each patient at the end of 3 months compared with baseline. The sensitivity of the developed intervention guide was 81.5% (95% CI=61.9-93.7%, p=0.009, r=0.482), with a specificity of 100.0% (95% CI=15.8-100.0%) (Tables 5 and 6) and the area under ROC curve was 0.907 (Figure 1).

Figure 1: ROC curve between sensitivity and specificity.

Table 5: HbA1c improvement by the intervention guide and gold standard.

Table 6: Statistic results of sensitivity, specificity and other values of the test with 95% confidence interval.

Discussion

The educational material was developed based on the results of baseline of this study which included the knowledge aspects of disease and lifestyle modifications instructions (dietary modifications and daily physical activities). A Sri Lankan study had recommended that findings of knowledge on foot care among T2DM have to be used as a guideline for diabetes health education programme which would help to focus on deficit areas [16]. The newly developed educational material was implemented among T2DM patients to assess the effectiveness of self-management of T2DM. The effectiveness of intervention was assessed by the primary outcome of HbA1c. The UKPDS revealed that HbA1c is being used as the most reliable test of assessing glycemic control and risk of complications [17].

According to Wagner et al. [18] self-management health education in T2DM is an important strategy for good glycemic control where patients receive periodic health education session to meet their needs. In order to improve the primary outcome of HbA1c in self-management of T2DM, the patient education becomes an integral part [19]. Thus, the printed educational material was developed and validated before implement. The educational material was validated judgementally by experts in the field. In the process of analyzing the content and the appearance of the educational material, contributions were included judges who were experts in diabetes care. The judges provided information relevant to modification of information provided in the educational material. Hundred percent of the judges agreed on the applicability of the educational material for clinical nursing practice. At the same time, judgemental validity has its own limitations where there is no statistical test to determine whether a measure adequately covers or represents the content area and it totally depends on judgement of experts [20].

Sensitivity and specificity measures were used to assess the intrinsic validity of educational material against a gold standard [21]. The sensitivity and specificity of the intervention using educational material was 81.5% and 100% respectively. The measure is of high validity when the both sensitivity and specificity are high [6]. This intervention was found to have an AUC of 0.907 and it revealed that educational material would be acceptable. The maximum of AUC=1 interprets that the test is perfect in the differentiation between the test and gold standard [22]. As there is no true gold standard, assessing the clinical improvement of glycemic control by the consultant physician would be affected by internal and external factors that might influence the judgement [23].

Further, in this study, case manager implemented the intervention. Case Management offers an intervention to successful self-management of DM [24] through diabetes health education. The Case manager coordinated the activities performed by a nurse to provide the health care services required by patients [25]. In addition, case manager provides and coordinates health care services to the population and which is a participative process to identify and facilitate options and services for meeting individual’s health needs [26]. Further, case manager work closely with patients with DM on lifestyle modifications and day to day self-management practices which would improve the clinical outcomes [27]. Furthermore, case manager plays an important role in providing holistic individualized care by developing rapport with patients, facilitating communication between patients and care provider and empowers patients with adequate knowledge necessary for self-management of DM. In addition, diabetes case manager provides high quality care for patients by emphasizing the importance of self-management and promoting patients’ choice and self-directed decision-making by through education [27]. It is providing wide range of intervention to improve diabetes self-management through patient education, counseling and support on self-management [28].

On implementation of printed educational material among patients with T2DM, the level of HbA1c was significantly reduced in study participants by 1.33% (± 1.57) in 3 months. The mean HbA1c level of 8.91% (± 2.22) at pre-interventional level was declined to 7.58% (± 1.36) at the end of 3 months of intervention. Further, nearly 76% of participants had improved their HbA1c level by ≥ 0.5%. In addition, FBS and BMI were also reduced significantly in study participants. This shows that study participants were able to reach the target HbA1c level ≤ 7.0% of Sri Lanka [29]. Further, it revealed that implementation of the new educational material had led to significant improvement in self-management of T2DM among patients in relation to glycemic improvement by adaptation on self-care activities such as appropriate diet, regular exercise, appropriate use of medications and regular follow-up. Based on the results, the printed educational material might help the patient to be independent with regards to self-management of T2DM.

The limitations: Unfortunately the study is limited to only 30 diabetic subjects with one drop out as it is a pilot study. The only biochemical parameter HbA1C was used to assess the glycemic control. In addition, change in self-care behaviors of patients with DM due to intervention was not measured objectively.

Conclusion

The educational material was developed and validated for its content and relevance. This educational material appears to be relevant, being considered as a new teaching aid for health education activities to provide information about T2DM and its self-management which motivates the patients to adhere to lifestyle modification to improve outcome.

Acknowledgement

Authors wish to extend their gratitude to University of Sri Jayewardenapura, Sri Lanka (Grant No-ASP/01/RE/MED/ 2016/62) for providing me financial assistance to carry out and complete the study. The authors would like to thank all patients who participated in this study. Also, we express our gratitude to the judges who have given their valuable input to develop and validate the new educational material for diabetes self-management of education.

References

1. Rothrock JC. Concepts basic to Perioperative Nursing. In Rothrock JC. (ed). Alexander?s Care of the Patient in Surgery. USA: Mosby, 2007.
2. Seth S. How readability of patient materials affects. J Vasc Nurs. 2000; 18: 97-101
3. Fox VJ. Postoperative patient care and pain management. In Rothrock JC. (ed). Alexander?s Care of the Patient in Surgery. USA: Mosby; 2007.
4. Sousa CS, Turrini RNT, Poveda VB. Educational intervention in patients undergoing orthognathic surgery: Pilot study. J Nurs Educat Practice 2015; 5: 126-134.
5. Jayawardene D, Gunawardena N, Dissanayake M. RASCH: a new approach in tool validation. J Coll Commun Physic Sri Lanka 2017; 23: 39-40.
6. Abramson JH, Abramson ZH. Research methods in community medicine: Surveys, epidemiological research, programme evaluation, clinical trials, 6th Edition, John Wiley Ltd, 2008.
7. Stewart AL, Hays RD, Ware JE. The MOS short-form general health survey. Reliabil Valid Patient Populat 1988; 26: 724-735.
8. Kisokanth G, Prathapan G, Indrakumar J, Joseph J. Patients with diabetes mellitus in a tertiary care hospital; their Knowledge on disease, treatments and complications. Int J Curr Res 2014; 6: 7487-7492.
9. Kisokanth G, Prathapan G, Indrakumar J, Joseph J. Self-management of diabetes mellitus among tamils in the batticaloa district, sri lanka: A qualitative study. Pakistan J Nutrit 2016; 15: 455-460.
10. Kisokanth G, Prathapan G, Indrakumar J, Joseph J. Factors associated with good glycemic control among Tamil patients with type II diabetes mellitus: a case control study. Diabetologie Stoffwechsel 2016.
11. Buderer NM. Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity. Acad Emerg Med 1996; 3: 895-900.
12. Wilson C, Curtis J, Lipke S, Bochenski C, Gilliland S. Nurse case manager effectiveness and case load in a large clinical practice: Implications for workforce development. Diabetic Med 2005; 22: 1116-1120.
13. Sperl-Hillen J, Beaton S, Fernandes O, Von Worley A, Vazquez-Benitez G, Parker E. Comparative effectiveness of patient education methods for type 2 diabetes: a randomized controlled trial. Arch Intern Med 2011; 171: 2001-2010.
14. Chien PFW, Khan KS. Evaluation of a clinical test. II: Assessment of validity. Br J Obstetrics Gynacol 2001; 108: 568-572.
15. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29-36.
16. Wickramasinghe IP, Siritunga TLSS. Type 2 diabetes mellitus related foot problems and knowledge, practices related to foot care among type ii diabetes mellitus patients who attend to diabetes clinics in General Hospital Kurunegala. Int J Sci Technol Res 2016; 5: 138-142.
17. United Kingdom Prospective Diabetes Study (UKPDS) Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet 1998; 352: 837-853.
18. Wagner EH, Glasgow RE, Davis C, Bonomi AE, Provost L, McCulloch D, Carver P, Sixta C. Quality improvement in chronic illness care: a collaborative approach. Jt Comm J Qual Improv 2001; 27: 63-80.
19. https://www.nice.org.uk/Guidance/NG28
20. Kimberlin CL, Winterstein AG.Validity and reliability of measurement instruments used in research. Am J Health Syst Pharm 2008; 65: 2276-2284.
21. Malhotra RK, Indrayan AA. Sample nomogram for sample size for estimating sensitivity and specificity of medical tests. Indian J Ophthalmol 2010; 58: 519-522.
22. Hajian-Tilaki K. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Internal Med 2013; 4: 627-635.
23. Hurst H, Bolton J. Assessing the clinical significance of change scores recorded on subjective outcome measures. J Manipulative Physiol Ther 2004; 27: 26-35.
24. Stuckey HL, Dellasega C, Graber NJ, Mauger DT, Lendel I, Gabbay RA. Diabetes nurse case management and motivational interviewing for change (DYNAMIC): Study design and baseline characteristics in the chronic care model for type 2 diabetes. Contemp Clin Trials 2009; 30: 366-374.
25. Zeng Z, Shuai T, Yi LJ, Wang Y, Song G. Effect of case management on patients with type 2 diabetes mellitus: a meta-analysis. Chin Nurs Res 2016; 3: 71-76.
26. Llewellyn A, Leonard M. Nursing case management review and resource manual (3rd ed.). Silver Spring, MD: American Nurses Credentialing Center, 2009.
27. Watts SA, Lawrence RH, Kern E. Diabetes nurse case management training program: enhancing care consistent with the chronic care and patient-centered medical home models. Clin Diabetes 2011; 29: 25-31.
28. Loveman E, Royle P, Waugh N. Specialist nurses in diabetes mellitus. Cochrane Database Syst Rev 2003; 2: CD003286.
29. Somasundaram NP, Wijeyaratne CN, De Silva S, Siribaddana S, Illangasekera U, Rajaratnam H. Diabetes mellitus: Glucose control. Sri Lanka J Diabetes Endocrinol Metabol 2013; 3: 45-57.