Measuring the Effectiveness of Holistic Approach to Teaching and Assessment used in Teaching a Decision Support Systems (DSS) Course

This paper investigates application of holistic approach to teaching and assessment and applies selected strategies to measure the effectiveness of such approach using results from teaching a Decision Support Systems (DSS) Course. Assessment is an integral part of the learning cycle and for any teaching strategy to succeed, teaching should be aligned with assessment strategies. The strategies used in this paper for measurement of the effectiveness of the holistic approach are student ratings, self-evaluation, and learning outcome measures.


Introduction
This paper focuses on the effectiveness of holistic approach to teaching and assessmentas part of the learning cycle-, as illustrated by a practical case of applying such approach to a Decision Support Systems (DSS) course.Gorry and Scott-Morton (1971) defined Decision Support Systems as "interactive computer-based systems, which help decision makers utilize data and models to solve unstructured problems".umsl.edu(2018) define Decision Support Systems as "an interactive computer-based system that help decision-makers use data and models to solve ill-structured, unstructured or semi-structured problems".The author has been teaching DSS course over several semesters and noticed the complex nature of this course as compared to other courses.That makes the course an ideal candidate for adoption of holistic approach to teaching and assessment to ease part of the pressures that the students are facing in understanding the numerous concepts and techniques taught throughout the course.The DSS course is an elective course taught to third year students in the Management Information Systems department of a business school.This study covers four academic semesters.

Importance of the Study
Available literature has limited cases of applying holistic approach to teaching and assessment in the field of management information systems in general, and DSS.Documenting a practical case for such application, with details of the theoretical framework developed and how to measure the effectiveness of the teaching and assessment strategies could be valuable for those considering the adoption of similar approaches.

Review of related literature
This part of the paper will include review of literature relating to measurement of effectiveness of teaching strategies, and holistic approach to assessments.

Measuring the effectiveness of teaching strategies
Past researches have discussed ways of evaluating the effectiveness of teaching strategies (Berk,2005;National Research Council, 2003;Biggs andTang,2011). Berk (2005) emphasized the importance of measuring the effectiveness of teaching for two reasons: to improve the teaching quality, and to be used as the basis for performance-based promotions.Berk (2005) listed 12 strategies to measure teaching effectiveness: (a) student ratings, (b) peer ratings, (c) self-evaluation, (d) videos, (e) student interviews, (f) alumni ratings, (g) employer ratings, (h) administrator ratings, (i) teaching scholarship, (j) teaching awards, (k) learning outcome measures, and (l) teaching portfolio (p.49); he proposed using multiple sources of evidence to perform such measurement.Svinicki (2001) discussed the downside of collecting feedback from students using traditional surveys; students fear centered around "lack of faith in the response received, their fear of being targeted because of negative feedback and, most important, their lack of knowledge about how to give proper feedback " (Svinicki (2001) as quoted in Fathelrahman,2019, p.4). Reflection from teachers on survey results will help build confidence among students in such surveys (Fathelrahman,2019).Biggs and Tang ( 2011) believe that one of the direct sources to know about how effective our teaching is our students; they noted that " this is quite a different issue from the usual student feedback questionnaire, which is about how you teach particular courses" and recommended open-ended rather than structured type of questionnaires (p.53).Researchers have considered the importance of teacher reflection and coined the term reflective teacher as an important skill for modern teachers (Kolb, 1999;Biggs and Tang, 2011;Gibbs, 1988).Biggs and Tang, (2011) particularly noted that reflection and feedback from students are correlated which could be attribted to their theory of constructive alignment.In a broader term , as noted by Berk(2005), teacher can reflect on their teaching strategies in ways other than just using feedback from their students.

Holistic Approach to Assessments
Adult education and skills sectors such as nursing, health, and social care has long considered applying holistic approach to assessments (SkillsRecognition ,2016;Scie-socialcareonline.org.uk, 2018;Taylor, 2018).SkillsRecognition (2016) states that holistic approach to assessment "focuses on the assessment of whole work activities rather than specific elements.When assessing a unit of competence holistically, the assessment activities integrate several aspects of the competency.Holistic assessment, underpinned by professional judgement, can reduce the time for assessment, and can promote greater equity in the assessment process".SkillsRecognition (2016) has listed the benefits of holistic assessment of being the chance to integrate number of assessed areas, reduction in assessment time, and prompting a greater equity in the assessment process".In planning a holistic approach to assessment, the teacher needs to ensure that his assessment type can cut across several units.The author noted that the principles of holistic approach to assessment as indicated by TheCollegeofSocialWork (2016) share common assessment principles such as validity and reliability.The only difference is that holistic assessments should be used when "objectives are inter-related and complex".Brookes.ac.uk (2018) listed the principles of assessments as reliability, relevance, transferability and validity.These are important measures to ensure the quality of an assessment.Holistic assessment makes evidence taking and documenting more easier by obtaining such evidence holistically rather than in segments.Though having diverse number and type of assessments is advocated by past research for the aim of catering for different skills and learning styles of the learner (Chamillard and Braun '2000) that have no contradiction with the holistic approach to assessment as the difference is in how evidence is collected.One criticism for comprehensive assessment noted by Beemt, Watering, Meeuwen, Lemmens, and Brok (2016) is that lower self-confidence and more stress had negatively impacted the relationship between grades and intrinsic motivation.

Methodology
This paper will use selected strategies to measure how effective is the experience of applying a holistic approach to teaching and assessment in a DSS courses.The strategies used are student ratings, selfevaluation, and learning outcome measures.(Berk,2005).The selection of these strategies is based on the availability of data and the relevance to the context in which the experiment is conducted.Berk (2005) noted that as national standards gives guidelines on "how teaching effectiveness or performance should be measured" … "they can guide the development of the measurement tools, the technical analysis of the results, and the reporting and interpretation of the evidence for decision making"(p.49).Table 1 bellow illustrates the conceptual framework of the DSS course.The framework is developed using the National Qualification Framework (Mu.edu.sa ,2018) which bears a great similarity to the Bloom's taxonomy (Vanderbilt University, 2018).The National Qualification Framework comprises of five learning domains: knowledge, cognitive skills, interpersonal skills & responsibility, communication, information technology, numerical, and psychomotor where the last domain applies only to professions such as medical studies.Alignment of course learning outcomes, assessment methods, and teaching strategy is documented to jointly represent "a consistent agreement between student learning, assessment, and teaching" (Mu.edu.sa ,2018).As noted by Biggs and Tang (2011), theoretical framework for the course facilitates holistic assessment by enabling the person to see "the relationship between the parts and the whole" (p.2150).In the DSS course conceptual framework, the intended learning outcomes are clearly stated and aligned with the assessment strategies suggested.This makes it easier to have a consolidated view of the course and facilitates taking holistic approach to teaching and assessments.
Within the course undertaken, the teacher integrated three types of assessments into one holistic assessment.The integrated assessments were case study, report, and exercises.One of the incentives used to sell the idea of the holistic approach to students is that it can reduce the number of assessments in an assessment-intensive type of course (13 assessment).One more advantage of this integration is that the teacher has been able to embed writing and presentation skills (such as PowerPoint presentations and APA style formatting for the references) with minimal efforts from students compared to isolated assessments.The data presented is an aggregated data and does not disclose any participant details , and presents authors teaching experience and therefore no consent was requested by the author.For each experiment of the holistic approach to teaching and assessment, the teacher started by creating a common theme in each semester that most of the teaching and assessments will be based on and this theme is communicated to students early in the course intake.The business environment (climate) is constantly changing, and it is becoming more and more complex.Organizations, private and public, are under pressures that force them to respond quickly to changing conditions and to be innovative in the way they operate.Such activities require organizations to be agile and to make frequent and quick strategic, tactical, and operational decisions, some of which are very complex.Making such decisions may require considerable amounts of relevant data, information, and knowledge.Processing these, in the framework of the needed decisions, must be done quickly, frequently in real time, and usually requires some computerized support (Turban et al, 2011).Within this theme, the students investigated the use of DSS systems in different business functional areas and what types of computational tools are used.
Another theme addressed by students is business Analytics (BA) and predictive analytics, exploring commercial software that support each concept and how these two terms are related as well as architecture, features and type of decision-making models that the product supports.
One theme in which students worked on was simulation software, what type of problem does this software supports, how testing and validation, conduct the experimentation, and evaluation of results are performed in the selected software.Students were also asked to summarize the technologies used to develop and run the selected software.

Measuring the Effectiveness of Holistic Approach to Teaching and Assessment
This part will analyses data related to the strategies used for measurement of the effectiveness of teaching and assessment in our DSS course delivery.

Students Ratings
One strategy used for measurement of the effectiveness of teaching and assessment in our DSS course is how students rated their learning in different aspects of the course.The survey was conducted in two of the 4 semesters where teaching took place.In the 2018 DSS course students survey, the overall rating by students was high (84%).It is noted that all questions that are directly related to teacher performance received an excellent rating (92% -100%) : ▪ My instructor(s) were fully committed to the delivery of the course.
▪ My instructor(s) had thorough knowledge of the content of the course.
▪ My instructor(s) were available during office hours to help me.
▪ My instructor(s) were enthusiastic about what they were teaching.
The overall student's satisfaction rate for the whole program was78.5%.In the 2019 DSS course students survey, the overall rating by students was good (68%).It is noted that all that are directly related to teacher performance received a good-to-high (84%) rating.The overall student's satisfaction rate for the whole program was 68.2%.There are external factors that could have impacted the students' evaluation for the course in this semester.The move to the new female campus was delayed for several weeks and this has impacted delivery of the course and caused students frustration.One area that has received lower rating is related to the course materials.This has been common across all courses and a team has been assigned to work on updating all textbooks to the latest version, recommendations have been received at the time of writing this paper and are planned for implementation next semester.
Table3.DSS course student's evaluation survey statistics.When asked about whether the amount of work they had to do in this course was reasonable for the credit hours allocated, the majority of students agreed about that , an indication that using holistic assessment helped easing the pressure on the students and the time needed for completing course requirements.The teacher has realized the time and efforts saving when applying the holistic approach to teaching and assessment as compared to other non-holistic approach followed in other courses (e.g.preparing lessons and assessments, marking assessments, and giving feedback to students).

Self-Evaluation
Fathelrahman (2019) discussed the importance of reflection for teacher and how that can improve the quality of learning in future cohorts.Boyce (2009) listed few factors to consider in self-assessment of teaching: identify strengths and areas in need of improvement; impact of recent changes; and plans for future improvements and development activities.At the personal level, the teacher has tried to improve the course resources to compensate for the complaint from students about course textbook as well as recommending upgrade to textbook.Main strength of the teacher is his experience of teaching the course for several years and the industry knowledge he brings to this course.The course can benefit a lot from having a practical session.Though the course includes sessions on using Excel as a DSS tool, a dedicated tutor and a practical session on commercial DSS tools would be more beneficial.The teacher may also consider other teaching strategies such as experiential learning.

Learning Outcome Measures
The program applies specific mechanisms and tools for measuring the construction and achievements of the learning outcomes at both program and course levels.This is applied through measurement of agreed performance indicators and benchmarking.The effectiveness of teaching and assessments are measured through mechanisms such as peer-review, assessment moderation and exams moderation.Each teacher prepares a course report every semester and these are consolidated into annual program report, both are approved in courses / program presentation workshops.Any recommendations are addressed, and action plan is drawn for future cohort.The performance indicators showed the stability at both the program and course levels and no major deviation has been observed over several semesters.The overall pass rate for the course was 98% and 93% for the whole program (Table 4).

Conclusion and Recommendations
Holistic approach to teaching and assessments, if planned correctly can results in saving the time of both the student and the teacher.This provides opportunity for integration of learning and assessment of students understanding of the whole course rather than in-integrated parts of the course.
The author noted that applying the holistic approach requires marketing to in a smart way, as some students tend to resist any change in the original planned specifications of the course.The author suggest that such concern can be addressed by communicating any changes to students early at the beginning of the course.As the case with any other assessment, holistic assessment should be quality assured and checked for reliability, relevance, transferability and validity (Brookes.ac.uk ,2018).

Table 1 .
Conceptual Framework of the DSS Course.

Table 2 .
Schedule of Assessment Tasks for Students During the Semester.
Table 5 lists grades distribution for the DSS over two sample semesters.Considering the relatively small number of students in each cohort, distribution of grades appears normally distributed and no inflation is observed in the higher categories (A+ and A) or fail grade.