University Quality Activities 2006
Report to Council

2.2       The Vice-Chancellor’s Group (Quality) continued its major role in conceiving and planning quality across all areas and aspects of the university.  As the peak body for quality, VCG (Quality) paid particular attention to planning for the 2006 quality audits in Australia by AUQA and in South Africa by the Higher Education Quality Committee (HEQC).  Further detail on the audits is provided in paragraph 5.2 and Attachment 3.

 

3.  Act

3.1    The VCG (Quality) has major institution-wide responsibility for ensuring action to embed the Monash approach to quality assurance and improvement in all areas of the university’s operation. http://www.adm.monash.edu.au/execserv/committees/vcg-quality/

3.2    The Senior Deputy Vice-Chancellor initiated a restructure of the committees of Education Committee in mid 2006 and the Learning, Teaching and Quality sub-committee was established in June. This committee subsumes the role of the Committee of Associate Deans Teaching (CADeT) and has reporting to it the Transnational Quality Assurance and Improvement Committee (formerly a committee of Academic Board) and the new Educational Technology Committee.  The Learning, Teaching and Quality sub-committee, chaired by the Pro Vice-Chancellor (Quality), focuses on ensuring high and consistent quality in the University’s educational activities.  In particular, it monitors and reports to Education Committee data on unit evaluations, student performance, progression, attrition, academic experience and graduate outcomes, within faculties and across faculties, disciplines and campuses.  Unit and course profiling (see paragraph 4.4 and Attachment 2) will significantly enhance the ease of monitoring of such data.

3.3    The Monash Quality Network (MQN) came into being in 2005 as the amalgam of separate faculty and support service quality networks that had been operating for a number of years.  MQN provides a forum for discussing quality issues and sharing good practice, and has been acting as the formal reference group for audit preparations.  A self-evaluation undertaken by MQN members at the end of 2005 showed that 88% of respondents felt the network was operating effectively.  Further information on MQN is available at: www.adm.monash.edu.au/cheq/mqn/index.html

3.4    Within faculties, oversight of quality takes place either in quality management groups with input from the Centre for Higher Education Quality (CHEQ; eg Business and Economics; Medicine, Nursing and Health Sciences; Information Technology; Engineering; Pharmacy) or within other faculty structures.  Some divisions have also formed dedicated structures for the consideration of quality matters.

3.5    As reported to Council in 2005, a discussion paper applying the Monash Quality Cycle to committees of all kinds and at all levels was developed by CHEQ during 2005.  This paper was based on University Council’s experience of monitoring its own effectiveness over seven years.  During 2006, the paper was included in the induction provided to committee secretaries, has been adopted by a number of committees within the university and proved to be very useful to committees preparing themselves for AUQA and HEQC audit interviews.

3.6    During 2006, the PVC (Quality) accepted appointment as Chief Executive Officer of Monash College Group and a review of Monash College Group quality assurance and improvement was commissioned to be undertaken by the Acting Director, CHEQ.  The report will be presented to Monash College Group Board by the end of 2006 but progress has already been made in aligning the quality systems of the university and Monash College Group (for example in terms of unit evaluation, MEQ and reviews) and in establishing a compliance, accreditation and quality office for Monash College Group.

4. Evaluate: Monitor

The monitoring aspect of ‘evaluate’ is defined by shorter term measures to provide formative feedback undertaken by internal personnel and for internal purposes. 

 4.1    A systematic approach to unit evaluation was implemented at Monash in 2005.  All units are evaluated by students at least annually using a survey with ten common items used across all faculties.  Results are analysed at various levels and reported publicly (see http://www.adm.monash.edu.au/cheq/evaluations/unit-evaluations/index.html).

Analysis of unit evaluation data shows that there has been improvement in each of the eight common quantitative items in Semester 1, 2006 compared with Semester 1, 2005 (Figure 1).

Figure 1: Comparison of mean scores in unit evaluation common items, 2005 and 2006

4.2    The Monash Experience Questionnaire (MEQ) was administered for the second time at the end of 2005 and various levels of reports prepared (see Attachment 1). In brief, results showed improvement on each scale, compared with MEQ03 (Figure 2).  This is a particularly pleasing result, although there remain areas for improvement.  Statistical analyses to determine the most poorly performing items for MEQ03 and MEQ05 show a similar pattern, with four of the bottom five items in both surveys relating to the provision of timely and appropriate feedback to students on assessed work.  Unit evaluation data supports this as an area requiring further attention.  The new Learning, Teaching and Quality sub committee of Education Committee is investigating ways in which unit evaluation and MEQ data is used to effectively deliver improvements. 

Figure 2: Comparison of means for the major scales and global items, MEQ03 and MEQ05

For the first time in 2005, Monash College and Monash Foundation Year (MUFY) participated in MEQ, with a slightly modified version of the survey.  Monash College students reported positively on all items, with 88% broadly satisfied with their study experience. MUFY students reported positively on all items, with 96% broadly satisfied with their study experience.
Monash has signed Memoranda of Understanding with the University of Sydney and the University of Queensland for the sharing of student experience information from their respective evaluations of current students.  Comparison of results (Figure 3) show that Monash has surpassed University B on all scales and University A on generic skills.  Monash is making improvements towards University A levels on all other scales.

Figure 3: Comparison of performance in good teaching, generic skills, learning community scales and overall satisfaction for MEQ05 with benchmark partners, 2005

MEQ is a lead indicator of the national Course Experience Questionnaire (CEQ) which is used by DEST as an important measure in the Learning and Teaching Performance Fund (LTPF) from which Monash received $4.6m in 2006.

 4.3    The Monash Support Experience Questionnaire (MSEQ) gauges student satisfaction with central and faculty-based support services and is administered in alternate years with MEQ. Results of the first MSEQ are currently being analysed and will be available at the end of 2006.

4.4    Unit and Course Profiling has developed from the unit evaluation project and will represent a major breakthrough in terms of the routine provision of data to support Education Committee and the Learning, Teaching and Quality sub-committee in their educational quality monitoring roles. Profiling refers to routine reports that will provide unit convenors, course coordinators, Heads of Department, Associate Deans (Teaching) and Deans with a consolidated single-page report of an agreed set of indicators, with three-year time series, on variables such as enrolment, attrition, progression, grade point average, unit evaluation, CEQ, GDS, student cohort data and financial data.  Each variable will have ‘red and green light’ flagging to denote areas worthy of further consideration.  Consolidated university-level reports will be produced for Education Committee and the Senior DVC.  The purchase of Cognos software to facilitate this project also increases the University’s capacity more generally to achieve better leverage from its institutional data.  Examples of the profiles are provided in Attachment 2.

4.5    The Monash Questionnaire Series on Teaching (MonQueST) is a suite of 11 separate questionnaires, each of which allows student feedback to be obtained on a different teaching activity such as lecturing, field-work teaching, problem-based teaching, project supervision and workshop teaching. MonQueST has been in use at Monash since 1994.  CHEQ administers the MonQueST system with evaluations requested online by, and reported back to, individual teachers. Reports are accompanied by a guide to their interpretation, including information on university and faculty aggregates.

It is not anticipated that a staff member would conduct a MonQueST evaluation more frequently than every two to four years and therefore the University monitors usage in three-year rolling cycles.  As shown below, there has been a gradual increase of staff undertaking a MonQueST evaluation from 31% to 36%, over the three-year rolling cycles from 2000–2004.  Highest rates of participation in the period 2000–2004 have been on the Malaysian campus followed by the South African campus. Academic staff at levels A, B and C participate in MonQueST evaluations more frequently than staff at higher levels (D and E).

Figure 4:  Percentage of staff undertaking a MonQueST survey for three-year rolling cycles, 2000–2004

4.6    Other important monitoring surveys include the Course Experience Questionnaire (CEQ), the Graduate Destination Survey (GDS), and the Postgraduate Research Experience Questionnaire (PREQ), all of which are required by DEST.  The Learning and Teaching Performance Fund will distribute over $82 million in 2007 according to performance on CEQ, GDS and other measures.  Monash has placed increased attention at increasing response rates for the CEQ and GDS surveys which in the current round has lifted responses to above 50%.

4.7    The Service Level Agreement project saw a number of developments in 2006 including ongoing work on an agreement between the Gippsland campus and faculties in support of the new financial arrangements for this campus.  Discussions commenced with both the Malaysian and South African campuses about the introduction of a service level agreement at these locations.  Divisions continued to report publicly every six months on their performance against agreed indicators.

5.  Evaluate: Review

The reviewing aspect of ‘evaluate’ is defined as a longer term, more formal process than monitoring.  It provides both formative and summative feedback and typically involves an external component.

5.1    VCG (Quality) is charged with planning and monitoring responses to the recommendations of institutional self-reviews and in 2006 this activity centered around follow-up to the research and research management review (2004) and the international self-review (2005).  Steady progress is being made with the implementation of recommendations from both reviews.  VCG (Quality) will oversee the implementation of recommendations from the AUQA and HEQC quality audits.

5.2    The Australian Universities Quality Agency (AUQA) and Higher Education Quality Committee (HEQC) quality audits have constituted significant projects for Monash with work continuing throughout 2006.  Reports on planning and preparations for the audits were made to many groups including VCG (Quality), Council, Academic Board, Senior Management Committee and Monash Quality Network, with the latter acting as the formal audit reference group.  A summary report on the audits is attached (Attachment 3).

5.3    There is a requirement that each ‘element’ (for example, department, school, centre, faculty, division, as appropriate) undertake an Academic or Support Service Review at least every five years.  As reported to Council in 2005, there has been an increase in review activity in recent years which indicates that the level of review activity is beginning to approach that necessary for compliance with university policy (see also Figure 5).  It is anticipated that the AUQA audit will affirm the University’s recognition of the need for improvement in this area.

Figure 5:  Number of academic and support service reviews, 2002–2006 (note that 2006 data is as at 10 October 2006)

6. Improve

6.1    Much of the improvement activity undertaken is at operational faculty and divisional level.  However, a central ‘improvements’ website is maintained to allow students and staff to see improvements made as a result of surveys.  At this site, the actual survey can be accessed, together with analyses and reports, and improvements that have been recorded as a consequence. http://www.adm.monash.edu.au/cheq/reports/improvements/index.html

6.2    Formation of the Centre for the Advancement of Learning and Teaching (CALT) at the end of 2005 was partially driven by the need to better configure staff development towards the strategic requirements of the University as evidenced through the quality information generated by CHEQ.   Both centers report to the PVC (Quality) and have been working closely throughout 2006 on strategic improvement projects based on MEQ and unit evaluation data. This is evidence of the increasing level of sophistication in the University’s management response to strategically important evaluative information.  Joint activities such the recent combined CHEQ/CALT planning retreat are supporting a close working relationship between the two centers.

6.3    The Systematic Improvement Strategy in Response to Unit Evaluation Data project involves faculties, collaboratively with CHEQ and CALT, identifying those units not performing as well as they could be (based primarily on unit evaluation data) and implementing and evaluating strategies for improvement.

The main principles of this approach are summarised as follows:

• working from data sources;

• targeting poorly performing units as a priority;

• establishing academic support teams consisting of CHEQ, CALT and faculty staff;

• linking staff and student development support;

• focusing initially on aspects which are perceived as being relatively easy to improve;

• providing concentrated support over a short term to achieve high impact;

• documenting and demonstrating improvement as a consequence of the actions taken.

The Victorian College of Pharmacy (VCP) has been involved in the first pilot of this approach throughout 2006. Academic staff in the VCP were aware of the College’s aim to improve unit evaluation data through its various internal committees and educational governance structures. The Senior Executive, in particular the Associate Dean (Teaching) played an important and proactive role in this project. The Dean and the Faculty’s Quality Management Group were also highly supportive and the Dean has been visible in the project through his attendance at the academic staff development sessions.  Ten units were prioritised with detailed analysis then undertaken of the main areas requiring improvement and strategies developed and implemented to focus on these areas.  As well as the specific unit changes, the project has led to some generic changes to learning and teaching practice within the faculty.  Those academic staff responding to a brief evaluation survey reported that they considered the project to be valuable. A detailed evaluation of the project is underway. Importantly, unit evaluation data in Semester 2, 2006 will be analysed to determine the effect of interventions on student satisfaction.

Throughout 2006, targeted systematic improvement projects have also commenced within the Faculty of Information Technology.  A similar approach, using MEQ data, is being taken with a project underway at the Gippsland campus.

 

Attachments:

Attachment 1:  Monash Experience Questionnaire 2005 (MSWord, 906KB)
Attachment 2:  Unit and Course Profiling (MSWord, 294KB)
Attachment 3:  Quality Audits Report (MSWord, 67KB)

Download this report in PDF (96.4KB) format