Review of the Centre for Higher Education Quality The CHEQ Response 1/9/04 Introduction The Centre for Higher Education thanks the Review Panel for the time that they devoted to this review and for delivering a Report that is both supportive and helpful in terms of planning for the future. Confirmations and Commendations CHEQ wishes to thank the panel for recognising the contribution it has made to the University as follows (direct quotation from the Review Report is signified by italics throughout this response document). The panel concludes that, given the relatively short time it has existed, CHEQ has made impressive progress in effectively addressing its eight stated aims. The panel would like to highlight the dedication with which the staff of CHEQ have taken on the challenges of making a new centre work. Their commitment and responsiveness to stakeholder needs and to a ‘client service philosophy’ was consistently identified during panel interviews. The effectiveness of the work of the Director in building relationships with senior staff across the university and in senior committees and policy formation was also consistently noted. The panel is of the view that CHEQ is having the significant impact on the culture change it is expected to facilitate at Monash University. During interviews those who have had contact with CHEQ consistently indicated their support for its work. Many indicated that they have started to embed continuous monitoring and improvement into their ongoing practice, rather than relying only on more periodic, externally initiated reviews. When suggestions for improvement were made it was always within the context of making an effectively operating unit even better.” In a number of ways CHEQ represents a ‘good practice’ model for the many quality units that are now emerging in Australian universities, as national and international pressures for effective quality management in higher education grow. The following achievements of CHEQ over the first three years of its existence were confirmed during interviews and are seen as being particularly significant: * The high level of understanding and ownership of key quality concepts by the staff interviewed by the panel, in particular the ‘fitness for purpose’ definition of quality, and the principles and values outlined in the key CHEQ overview document ‘Quality at Monash: Values & Principles’; * CHEQ’s initiation of the idea of a whole of institution self-review (Still Learning: The Report of our Self-Review) which helped set the organisational development priorities pursued by the Monash Quality Development Committee over the past two years; * The reinvigoration of the review processes at Monash, underpinned by a wide range of new training and support materials and the establishment of a distinctive review schedule database; * The dramatic expansion of the university’s quality tracking and improvement processes, from a focus on teaching evaluations when the Centre was first constituted to the currently extensive suite of instruments and their associated administration, processing, analysis and reporting strategies as outlined in Section 3.7 of the CHEQ Self Review Report. Two of the many recent developments in this area which are noteworthy are the Monash Experience Questionnaire (the results of which are provided to faculties on a CD to enable further analysis) and the Monash Employer Survey. The panel also endorses the use of ratings of importance as well as performance in the relevant CHEQ surveys – as was the case, for example, in the Caulfield PG Facilities & Services Survey. * A number of those who met with the panel, such as people associated with residential services, noted that their collaboration with CHEQ in developing a survey-feedback system had demonstrably improved daily practice and key outcomes; * The use of the ‘CHEQ Standard’ to assure the quality of surveys developed under CHEQ auspices (Self Review pg 36) is commended. Some review participants asked if there was a way in which the quality and staging of local surveys developed without reference to CHEQ might be coordinated across Monash to avoid duplication and the possibility of ‘oversurveying’; * CHEQ’s use of benchmarking in key development areas. One example is the benchmarking being undertaken with the University of Sydney on the teaching-research nexus, an area now attracting significant attention in AUQA audits and identified in Still Learning (pgs 24-5). Other examples include the process benchmarking being undertaken with La Trobe, RMIT and the University of Melbourne, the SLA project with UK institutions and an MEQ project with the University of Queensland and the University of Sydney. Further development of this work is identified in the Self Review Report and is endorsed by the Review Panel; * The initiation of the ‘mapping Monash’ project and the development and delivery of a Service Level Agreement System to support strategic cost management. This development is distinctive, has attracted significant external interest (CHEQ Self Report pg 44) and is well in advance of most universities. The fact that this complex and challenging initiative has been taken up and was consistently supported during the panel interviews confirms this as significant achievement for CHEQ, as noted on pg 31 of the Self-Review Report; * The initiation and implementation of quality networks in both the academic and support areas of Monash was generally endorsed as a successful CHEQ initiative by the members of those networks interviewed by the panel; * The development of the CHEQ Quality Advisers Team which provides custom-tailored assistance to those involved in key quality activities across the university. The hard work and the quality of assistance of these advisers were providing was highlighted by many of the users of their services who met with the Review Panel; * The systematic focus on quality development at all Monash sites, in particular ensuring that there are equivalent quality management systems in place at the university’s two overseas campuses in South Africa and Malaysia. This positions Monash particularly well for the Offshore QA component of its forthcoming AUQA audit; * The positive promotion of Monash at the local, national and international level through the CHEQ Director’s participation as chair of AUQA audit panels, his membership of the original development team for the Australian Universities Quality Forum, staff presentations of papers at key local and international conferences, their participation in higher education quality consultancies in countries like South Africa where Monash has campuses; the range and quality of Centre publications and use of the CHEQ website (Self Report pg 20). Evidence of the positive reputation which CHEQ has developed is seen in the number of overseas delegations which want to visit the Centre, its profile in the higher education quality networks of Australia and New Zealand, the number of hits its website is attracting (Self Review pg 21) and the international consultancies it is invited to participate in; * the use of the CHEQ website as a resource base and an instrument to deliver online evaluation instruments (Self Review Report, pg 21). Potential Areas for Development CHEQ will attempt to implement the suggestions of the Review Panel where ever possible, and as outlined below. Throughout the document, direct quotation from the Review Report is in italics (numbered) and the CHEQ response and action is in normal (bullet point). 1. Actioning the long term aim assisting the university to ‘develop routine monitoring mechanisms which, when in place, will provide information to inform both planning and review. Some of this relates to evaluation tools (MEQ, Employer Survey, Support Services Survey, Staff Survey) but it goes beyond these to the routine aggregation of specific data into regular and readily usable reports’ (Self Review Report pg 49). In this regard the development of power play cubes being used in a number of universities which bring together relevant data bases in student enrolments, progression, graduation, employability, salaries, CEQ scores and CEQuery results may be worth exploring. Investigating ways to link tracking reports into the annual planning and budgeting process was also notes in discussions with CHEQ and with other review participants. * As noted by the panel, ‘routine monitoring mechanisms’ at institutional level such as MEQ, Student Services Survey and Employer Survey have been delivered. A Staff Survey has also been developed. * A proposal to develop Unit and Course Report Cards (the result of ‘power play’ cubes or their equivalents) was developed by CHEQ during 2004 and has been put forward to the University as an ITS Strategic Innovation Project Proposal for funding in 2005. * CHEQ is also exploring the possibility of using CEQuery software for analysis of qualitative responses from PREQ, CEQ and the Student Services Survey. 2. The panel was asked to comment on ‘formal vs. informal evaluation with respect to CHEQ’s support of senior management’ (Self Review Report, pg 22). The panel endorses the continued use by CHEQ of informal feedback on its services via the various Monash quality networks which the Centre is involved; and its consistent use of participant feedback on its workshops and satisfaction surveys like those identified in the Self Review Report (pg 48) along with the performance data cited in its annual reports to the Monash executive. The panel does suggest, however, that it may also be useful to use a more comprehensive user feedback survey in which the importance and performance of all its services are rated at the one time, but only biennially. In this regard it is noted that CHEQ has already identified a wide range of KPIs for its services and activities. * CHEQ will continue to gather feedback on an activity basis as endorsed by the panel. * CHEQ supports the intent of the recommendation concerning a comprehensive user feedback survey. CHEQ services and activities cover a wide range of individuals, many of whom only know and use CHEQ for a particular purpose (eg MonQueST). CHEQ will therefore develop satisfaction surveys tailored to the various individuals and groups that use its service and which may be merged to provide overall and therefore more comprehensive user feedback information. 3. Pursuing and further refining the Monash Graduate Attributes Project (Still Learning pg 21 and CHEQ Self Review, pg 25) was endorsed by a range of those who met with the panel. It may be worth considering studies which seek to identify, by surveying successful early career graduates, key capabilities central to effective performance in a particular profession or discipline. This will help situate and validate the more generic attributes. The results can also be used to ‘backward map’ towards enhancements in the curriculum and assessment of the profession/ discipline concerned. In this regard studies of successful early career graduates undertaken by UTS and AUT may be of interest. * CHEQ will ensure that the results and recommendations of the Employer Survey and the Monash Experience Questionnaire (especially with regard to graduate attributes) are passed on to the body that has responsibility for this area: the Graduate Attributes Working Party of Education Committee. 4. Assisting the university to focus greater attention on quality management in the research area (Self Review pg 49). This is distinct from the research training area where CHEQ is already active through its coordination of PREQ and PG Research Supervisor Survey Reports; * CHEQ acknowledges the good working relationship with the research training area and looks forward to further collaboration, especially with regard to the use of CEQuery for analysing PREQ and PGRSS qualitative comments. * The Director of CHEQ will discuss with the DVC (Research) the appropriate support of quality assurance and improvement in the research portfolio. 5. Proceeding to identify the optimum way to ‘close the loop’ on the various quality tracking reports being now produced by the CHEQ evaluations team; * CHEQ has opened a website (“Improvements”) for posting improvements made as a consequence of institutional surveys. The first survey tracked on this site is the Caulfield Services and Facilities Survey. http://www.adm.monash.edu/cheq/academic/feedback_loop.html * At the same website (“Closing the Feedback Loop”) CHEQ has also provided guidelines for others on how to feedback improvements to survey participants. * CHEQ will also develop guidelines for implementing recommendations resulting from such evaluation tools. 6. The intention to promote further collaboration and cross-fertilisation between the Quality Adviser (Academic Programs) and the Faculty Quality Coordinator’s Network is endorsed. It was also suggested during the panel’s interviews with members of the FQCN and the parallel Support Services Quality Network that some of the strategies that are already being used to link both networks be further pursued; * Both groups have reported to VCG(Q) on how they collaborate. Presentations at each other’s meetings have also been made and the possibility of developing joint workshops or a Quality Forum is being investigated. 7. Further enhancement and use of the CHEQ website in the ways identified in the CHEQ Self Review Report (pg 21), including refinement of its navigation was endorsed. This could include actioning the findings from the SLA Project Evaluation 2003 and feedback from participants in the SLA project which indicated that material can sometimes be difficult to locate. The Centre’s commitment to rationalising and continuing to enhance the many support materials and guidelines on the website was endorsed. * This refers to the 2003 SLA project evaluation in which one respondent indicated that ‘it was a little hard to find the documents on the CHEQ website’. The other comments received were that materials on the CHEQ website are: ‘thorough’, ‘clear’, ‘to the point’ and ‘easy to understand’, with stakeholders reporting that they are ‘more than happy’ with the materials provided. * Usability of the CHEQ website will be further improved as a major revamp of the website is undertaken to accommodate adoption of the new Monash branding guidelines including expert advice from the ITS web design team on improving usability of the website. 8. Refining the KPIs used as part of the SLA project and careful monitoring of the next stages of its roll-out have been notes in the CHEQ Self Review (pg 33) and were endorsed by a number of those interviewed. Given the workload involved, focusing on high priority services nominated by faculties and support services may be a useful way to ensure that the project is cost-effective as is ensuring that it becomes more embedded in the university’s planning and budget cycles; * CHEQ thanks the panel for endorsing this point which was made by in the CHEQ Self Review Report. The Group 1 SLA services have been reporting for some time and so possibilities for the development of KPIs are now being identified. * Reference groups, including the Faculty Managers, indicate areas of high priority for review of the SLA. * Strategic review of priority areas is identified in the University’s Annual Budget Forum (eg Recruitment of International Students project 2003 for reporting 2004). 9. Investigating ways of using scanning equipment and/or online systems for national surveys and other surveys in order to optimise the efficiency and speed of reporting the results. It is noted that CHEQ already has digital character readers. As the Self Review Report notes: ‘The challenge ahead is to develop technical systems to support the area and to this end, a strategic initiative project of $103K will be delivered during 2004’. A number of Australian universities are currently investigating the same issue and this opens up the possibility of further process benchmarking; * CHEQ uses optical mark readers not digital scan readers (OMRs are more accurate which has advantages for reliability and disadvantages for use with a variety of inputs). As part of the Unit Evaluation Strategic Initiatives project, which has now grown to $263K and is to be delivered in part in late 2004 and more fully in 2005, CHEQ is investigating the use of a number of commercial software packages to replace the ageing custom built system. CHEQ investigated systems in use at other Australian Universities prior to developing the Strategic Initiative. 10. CHEQ’s continued participation in the qualitative analysis of student comments and benchmarking of results made possible through the development of CEQuery (Self Report pg 43) is endorsed. It is noted that, subsequent to the Review visit, Monash has agreed to join a group of Australian universities in a HEIP proposal to further develop the use of these data for quality assurance purposes; * As noted by the panel, CHEQ has been a participant in the development of CEQuery and continues to be involved the latest CEQuery HEIP project. Use of CEQuery will be undertaken on a trial basis with PREQ qualitative data, then rolled out to CEQ and Student Services Survey to demonstrate its use in different environments. 11. Continuing to promote the work of CHEQ nationally and internationally, including development of articles which document its most significant achievements in relevant journals in conjunction with the strategies outlined on pgs 43-44 of the Self Review Document; * CHEQ will continue to lead and participate in international fora and to publish and promote its work as outlined in the Self Review Report. 12. Continued refinement of the internal communications strategies in CHEQ is encouraged to further develop use of the operational plan which identifies key areas of action, timelines, responsibilities and outcomes. The regular fortnightly staff and quality advisor meetings and its annual planning 2-3 day retreat are commended and their continued refinement is endorsed; * The Operational Plan has been considered routinely at staff meeting. In future it has been agreed that three staff meetings per year will focus exclusively on progress with regard to the Operation Plan and all CHEQ staff members will be involved in these meetings. Otherwise the Director will continue to meet on a fortnightly basis with each quality adviser individually, and in addition, with the Evaluations Manager. Additional areas for further investigation 1. Promoting the many achievements of CHEQ both internally and externally. For example, a number of people who met with the panel suggested that, in order to (re) promote the work of CHEQ, it might be appropriate to update as appropriate and redistribute the CHEQ Information and Services pamphlet; * Reprinting of ‘out of stock’ CHEQ pamphlets and other materials has been held up for approximately a year by the new branding requirements. 2. Investigating the development of common survey instruments (or items) with other universities which have a similar mission and profile with a view to supporting benchmarking for improvement, along the lines currently pursued by the ATN; * CHEQ is developing benchmarking of MEQ with the University of Sydney and the University of Queensland. Benchmarking of the Student Services Survey and Staff Survey will be explored when these have been undertaken. 3. Assisting Monash to ensure that students are informed about what is being done to address the key improvement messages in their feedback and that these messages are addressed promptly and wisely at the local level. The panel is aware that work in this area is already underway, including plans to hold a CHEQ workshop for faculties and support services staff after the MEQ results have been considered by the Senior Management Committee; * The MEQ workshop was conducted on 24 June 2004 and a follow up workshop concerning the use of qualitative data is planned. * As indicated at 5 above, CHEQ has opened a website (“Improvements”) for posting improvements made as a consequence of institutional surveys. The first survey tracked on this site is the Caulfield Services and Facilities Survey. http://www.adm.monash.edu/cheq/academic/feedback_loop.html * At the same site (“Closing the Feedback Loop”) CHEQ has also provided guidelines for others on how to feedback improvements to survey participants. * CHEQ will also develop guidelines for implementing recommendations resulting from such evaluation tools. 4. Tracking the changing workloads and ensuring that the profile of staff meets the emerging needs of CHEQ and that the Centre’s accommodation and infrastructure align with these changes. * Workloads will continue to be monitored and where workload becomes acute, issues will continue to be discussed with staff and plans made for renewal after the workload pressure has passed. * We believe that the panel formed an incorrect impression regarding the distribution of administrative support staff within the Centre. The Administrative Officer is responsible for the provision of support to all Quality Advisers and other senior staff. The suggestion that evaluations staff form part of the support for the QA Research and Evaluations is incorrect. * The profile of CHEQ will continue to be monitored with regard to meeting the strategic objectives of the University. * As detailed during the Review, CHEQ has expended approximately $15K developing office space for staff. All staff of the Evaluations area are collocated as are all of the Quality Advisers and their support. All parts of CHEQ are within a minute’s walk of each other. 5. Reviewing the staff development strategy for CHEQ outlined in the Self Review Report (pg 44) to ensure that this focuses on key, agreed gaps in expertise identified by staff, as well as supporting the implementation of new areas of operation and the refinement of existing processes; * As outlined in the CHEQ Self Review Report, there are various strategies to accomplish staff development and staff have been strongly supported (including financially) to undertake development opportunities. The whole CHEQ team has undertaken training where a need has been identified i.e. Calendar. Staff have the opportunity to identify gaps in expertise and investigate and nominate additional training opportunities at any time and during their individual performance reviews. The Director also identifies and financially supports areas of staff development that will be of benefit to the university. 6. Developing quality management for the university’s Community Engagement activities; * The Quality Adviser (Support services) will review the Community Engagement Plan and prepare a briefing for the DV-C (now that MAPA is in his portfolio) on Monash’s Community Engagement Plan with the intention of building the quality cycle into the plan. 7. Exploring ways of ensuring effective representation of the broader student view in the review process. For example, it was suggested that student representation on review committees would be more effective if that person had demonstrably canvassed a broad range of fellow student view prior to participation; * A section on gaining student input to the review process will be included in the revision of the review guidelines to be completed later this year. 8. Identify additional ways of working with separate, but co-located University Planning and Research Unit; * The Deputy Vice-Chancellor has recently instituted meetings with the Directors of CHEQ, Audit and Risk Management and University Planning and Research. The Library and ITS have also been added to this list. The issue of internal institutional research (for example in terms of MEQ analyses) will be discussed further. 9. As CHEQ develops, a number of review participants noted that it will be important for the university to monitor, and when necessary, to adjust the profile of the Centre’s roles. One example of this is deciding how an anticipated scale up of work in the area of quality management for research will be managed and coordinated with the Office of the DVC (Research). * This point was covered at 4 (initial) above, as follows: the Director of CHEQ will discuss with the DVC (Research) the appropriate support of quality assurance and improvement in the research portfolio. Conclusion CHEQ wishes to thank the panel for its concluding section which includes the following. The panel is of the view that an effective CHEQ is a key ingredient in positioning Monash successfully in the increasingly challenging quality-focused operating environment the university now faces. The Centre has emerged from this Review as a widely supported and core component in ensuring that Monash is ‘Still Learning.' As Leading the Way: Monash 2020 concludes: ‘Most critically, success will depend upon an uncompromising commitment to Monash ’s institutional autonomy and to quality, measured by international standards and assurance processes’ (pg 10).