Development of a Comprehensive Evaluation System for Multidisciplinary Team-based Supported Employment Services for People with Severe Mental Disorders

doi:10.12114/j.issn.1007-9572.2020.00.533

Abstract

Abstract: Background　Given a global consensus that the employment of people with severe mental disorders is beneficial to their mental rehabilitation，multidisciplinary involvement in supported employment services has become a research hotspot worldwide in recent years. However，since the service model is still at its infancy in China，the country is in urgent need of an evaluation index system that can help set up the service system. Objective　To develop a scientific，standard and comprehensive evaluation system for multidisciplinary team-based supported employment services for people with severe mental disorders and determine the weight coefficients of its indicators. Methods　Literature review and Delphi method were applied to establish an evaluation framework and develop a questionnaire for Delphi survey. A total of 10 experts were invited to complete two rounds of email-based questionnaire surveys from July to September 2019. According to the degrees of coordination and concentration of the experts' opinions in the survey，the indicators of the evaluation system were added，subtracted or modified to develop a comprehensive evaluation system and determine the weight coefficients of its indicators. Results　The finally developed comprehensive evaluation system consists of 3 first-level indictors（governmental and resource input，service process and outcome），13 second-level indictors and 60 third-level indictors with determined weight for each indicator. The positive coefficient of experts was 100.0%. The coefficients of experts' familiarity，judgment and authority were 0.930，0.850 and 0.890，respectively. The Kendall's concordance coefficients for the two rounds of surveys were 0.327 and 0.621，respectively，suggesting that the opinions of experts were very consistent（P<0.001）. Conclusion　The system was developed according to the two-round survey results with high positive coefficient of experts，high level of experts' authority and coordination and consensus，may provide a reliable basis for the evaluation of multidisciplinary team-based supported employment services for people with severe mental disorders. And it is conductive to further application and popularization of the supported employment services through the government platforms.

Key words: Multidisciplinary team, Mental disorders, Supported employment, Index system, Delphi method

摘要： 背景　严重精神障碍残疾人参与就业有利于精神康复已成为全球共识，多学科参与的支持性就业服务项目近年来成为全球范围内的研究热点，这种模式在我国尚处于探索阶段，急需一套评价指标体系，以为服务项目的建立提供指导与评价。目的　为多学科服务团队介入严重精神障碍残疾人支持性就业建立科学、规范的综合评价指标体系。方法　运用文献资料法和德尔菲法，建立评价指标框架，编制专家函询问卷，2019年7—9月邀请10名专家，通过前后两轮函询，根据专家意见的协调程度和集中程度，对指标进行增减和修改，建立综合评价指标及权重系数。结果　通过两轮函询，构建了一个包括3个一级指标（组织与资源投入、服务过程和工作结果）、13个二级指标和60个三级指标的综合评价指标体系，并确定了各指标的权重。专家积极系数为100.0%，熟悉系数、判断系数、权威系数分别为0.930、0.850、0.890，两轮的协调系数分别为0.327和0.621，意见有较好的一致性（P<0.001）。结论　专家的积极程度高，权威性和协调性较好，两轮咨询后专家意见趋于一致，筛选出的综合评价指标体系可为多学科服务团队介入严重精神障碍残疾人支持性就业服务的评价提供依据，有利于进一步借助政府平台应用推广。

关键词: 多学科服务团队, 精神障碍, 支持性就业, 指标体系, 德尔菲法

DUAN Libo,ZHENG Hong,ZHUANG Jianlin. Development of a Comprehensive Evaluation System for Multidisciplinary Team-based Supported Employment Services for People with Severe Mental Disorders　[J]. Chinese General Practice, 2021, 24(16): 2072-2076. DOI: 10.12114/j.issn.1007-9572.2020.00.533.
段李博,郑宏,庄建林. 多学科服务团队介入严重精神障碍残疾人支持性就业综合评价指标体系的构建研究[J]. 中国全科医学, 2021, 24(16): 2072-2076. DOI: 10.12114/j.issn.1007-9572.2020.00.533.

References

［1］石丹淅，王琦．残疾人就业状况变动趋势分析［J］．残疾人研究，2013，3（4）：77-80.
SHI D X，WANG Q．Trends on employment situation of the disabled in China［J］．Disability Research，2013，3（4）：77-80.
［2］YIP K S．The community care movement in mental health service：implication for social work practice［J］．International Social Work，2000，43（1）：33-48．DOI：10.1177/002087280004300104.
［3］符大伟．促进智力和精神残疾人辅助性就业的实践探索——以广州市康园工疗服务机构为例［J］．残疾人研究，2015，29（3）：23-27.
FU D W．Practices to promote supported employment for persons with intellectual or mental disabilities：takes Guangzhou Kangyuan Vocational Rehabilitation Center as an example［J］．Disability Research，2015，29（3）：23-27.
［4］CIMERA R E．The economics of supported employment：what new data tell us［J］．Journal of Vocational Rehabilitation，2012，37（2）：109-117.
［5］郑宏，鞠康．严重精神障碍残疾者“医院-社区一体化”职业康复规范化建设研究［J］．中国全科医学，2018，21（35）：47-52．DOI：10.12114/j.issn.1007-9572.2018.00.293.
ZHENG H，JU K．Standardization building of occupational rehabilitation based on "hospital-community integration" service model for patients with mental disability due to severe mental disorder［J］．Chinese General Practice，2018，21（35）：47-52．DOI：10.12114/j.issn.1007-9572.2018.00.293.
［6］王春枝，斯琴．德尔菲法中的数据统计处理方法及其应用研究［J］．内蒙古财经学院学报（综合版），2011，9（4）：92-96．DOI：10.3969/j.issn.1672-5344.2011.04.022.
WANG C Z，SI Q．A study of data statistical processing method of Delphi method and its application［J］．Journal of Inner Mongolia Finance and Economics College，2011，9（4）：92-96．DOI：10.3969/j.issn.1672-5344.2011.04.022.
［7］袁树华，师鉴，高伟，等．应用德尔菲法选择疾病预防控制机构公共卫生应急反应能力评价指标［J］．环境与健康杂志，2007，24（9）：732-733．DOI：10.3969/j.issn.1001-5914.2007.09.028.
［8］张博．精神病患者职场受歧视的原因、危害及对策［J］．广州市公安管理干部学院学报，2010，20（3）：53-55．DOI：10.3969/j.issn.1673-6915.2010.03.015.
［9］杜林，李伦，雷江华．美国残疾人支持性就业的发展及对我国的启示［J］．中国特殊教育，2013，20（9）：14-20．DOI：10.3969/j.issn.1007-3728.2013.09.003.
［10］BROWN B B．Delphi process：a methodology used for the elicitation of opinions of experts［J］．California：the Rand Corporation，1968：3925-3942.
［11］曾照云，程安广．德尔菲法在应用过程中的严谨性评估-基于信息管理视角［J］．情报理论与实践，2016，39（2）：64-68．DOI：10.16353/j.cnki.1000-7490.2016.02.014.
［12］MACDONALD E B，RITCHIE K A，MURRAY K J，et al．Requirements for occupational medicine training in Europe：a Delphi study［J］．Occupational and Environmental Medicine，2000，57（2）：98-105.
［13］王栋，肖水源．精神卫生服务评估指标的筛选［J］．中南大学学报（医学版），2012，37（11）：1129-1134．DOI：10.3969/j.issn.1672-7347.2012.11.009.
WANG D，XIAO S Y．Selection of assessment indicators for mental health service［J］．Journal of Central South University（Medical Science），2012，37（11）：1129-1134．DOI：10.3969/j.issn.1672-7347.2012.11.009.
［16］邵爽，金光辉，陈超，等．2003—2012年全科医学学科相关科研论文的分布情况研究［J］．中国全科医学，2015，18（1）：116-120．DOI：10.3969/j.issn.1007-9572.2015.01.38．
SHAO S，JING G H，CHEN C，et al．Research on the distribution of scientific research papers related to general medicine from 2003 to 2012［J］．Chinese General Practice，2015，18（1）：116-120．DOI：10.3969/j.issn.1007-9572.2015.01.38．
［17］赵家义，韩一平，沈璐，等．上海市杨浦区基层医疗机构科研能力现状及对策研究［J］．中国全科医学，2017，20（16）：1941-1945．DOI：10.3969/j.issn.1007-9572.2017.16.008.　
ZHAO J Y，HAN Y P，SHEN L，et al．Status and solutions for the scientific research capabilities in primary care providers in Yangpu District，Shanghai［J］．Chinese General Practice，2017，20（16）：1941-1945．DOI：10.3969/j.issn.1007-9572.2017.16.008.
［18］张晓玲，阎渭清，陈欣.全科医师规范化培训考核指标体系的构建［J］．中国全科医学，2016，19（34）：4225-4229．DOI：10.3969/j.issn.1007-9572.2016.34.016.　
ZHANG X L，YAN W Q，CHEN X．Establishment of appraisal index system of standardized training of general practitioners［J］．Chinese General Practice，2016，19（34）：4225-4229．DOI：10.3969/j.issn.1007-9572.2016.34.016.
［19］温忠麟，侯杰泰，马什赫伯特．结构方程模型检验：拟合指数与卡方准则［J］．心理学报，2004，36（2）：186-194.　
［20］王芳．全科医生综合素质评价指标体系的构建研究［D］．青岛：青岛大学，2017：12-13.
［21］潘莹，陈宇革，王朝昕，等．基于人才培养的上海市某区全科医生综合能力评价研究［J］．中国全科医学，2020，23（25）：3230-3233．DOI：10.12114/j.issn.1007-9572.2020.00.072.
PAN Y，CHEN Y G，WANG Z X，et al．Personal development-oriented comprehensive ability evaluation for general practitioners：a survey from a district of Shanghai［J］．Chinese General Practice，2020，23（25）：3230-3233．DOI：10.12114/j.issn.1007-9572.2020.00.072.
［22］王朝昕，于德华，石建伟，等．社区卫生服务中心科研能力综合评价体系特异性分析与优化研究［J］．中国全科医学，2019，22（28）：3401-3405．DOI：10.12114/j.issn.1007-9572.2019.00.503.　
WANG Z X，YU D H，SHI J W，et al．Comprehensive scientific research performance evaluation system for community health centers：a specificity analysis-based optimization study［J］．Chinese General Practice，2019，22（28）：3401-3405．DOI：10.12114/j.issn.1007-9572.2019.00.503.
［23］陈万思.知识员工胜任力：理论与实践［M］．上海：上海财经大学出版社，2007：34-35.
［24］张利平，郑文贵，李望晨，等．多属性决策技术在全科医生核心胜任力综合评价中的应用研究［J］．中国卫生事业管理，2016，33（2）：137-141.　
ZHANG L P，ZHENG W G，LI W C，et al．Discussing the application of multi-attribute decision in the comprehensive evaluation of general practitioners' core competency［J］．Chinese Health Service Management，2016，33（2）：137-141.
［25］叶腾飞，潘小炎．基于层次分析法的广西城市社区全科医生能力评价指标体系的构建研究［J］．中国全科医学，2014，17（7）：752-755．DOI：10.3969/j.issn.1007-9572.2014.07.006.　
YE T F，PAN X Y．Construction of evaluating index system based on the AHP on the competence of urban general practitioner in Guangxi［J］．Chinese General Practice，2014，17（7）：752-755．DOI：10.3969/j.issn.1007-9572.2014.07.006.

[1]	LIU Hongya, GUO Aizhen, JIN Hua, FU Qiangqiang, SHI Jianjun, ZHANG Hanzhi, YANG Sen, YU Dehua. The Necessity and Training Strategies for Expanding the Responsibilities in General Practice Niche in China: a Study on the Continuous Development of General Practitioners' Sub-specialties Based on the Delphi Method [J]. Chinese General Practice, 2026, 29(20): 2781-2789.
[2]	CHEN Cunchuan, ZHANG Huifang, FAN Boyang, SUN Wenning, WANG Yingjie, ZHANG Ao, ZHAO Yang, WANG Haipeng. Construction of Evaluation Indicator System for Primary Care Doctors' Service Capability in Integration of Medical and Preventive Care [J]. Chinese General Practice, 2026, 29(19): 2705-2710.
[3]	LIU Xiaoyu, ZHU Rongxin, ZHANG Bo, SU Hang, REN Xianqing. Defining Clinical Questions and Outcome Indicators in Guidelines for Diagnosis and Treatment of Nocturnal Enuresis in Children with Integrated Traditional Chinese and Western Medicine Based on Modified Delphi Method [J]. Chinese General Practice, 2026, 29(15): 2050-2055.
[4]	ZHAO Jingru, JU Yvfei, XU Xiaotong, LI Dan, MENG Jie, LI Hui. A Systematic Review of the Current Situation and Influencing Factors of Medication Management for Patients with Severe Mental Disorders in China [J]. Chinese General Practice, 2026, 29(13): 1745-1752.
[5]	SHEN Yong, ZHOU Jingying, WANG Xueping, JIA Meixiang, LIU Jiajia, ZHAN Haojian, WANG Lifang, YAN Hao, PENG Danyuan, YUE Weihua. Construction of Support Services List for Parents of Children with Autism Based on "Timing It Right" Theory [J]. Chinese General Practice, 2026, 29(13): 1688-1698.
[6]	LIU Haiyan, WANG Haitang, DU Zhaohui. The Evaluation Indicators for the High-quality Development of Family Doctor Contract Services Using the Delphi Method [J]. Chinese General Practice, 2026, 29(10): 1277-1285.
[7]	PANG Ting, LI Wanxin, ZHANG Haoran, LIN Zhiying, XU Xin. Construction of the Evaluation System of Cognitive Impairment Screening Service in the Community Based on RE-AIM Framework [J]. Chinese General Practice, 2026, 29(08): 1044-1050.
[8]	HUANG Zichen, XIA Zhongyuan, ZHI Congcong, CHEN Ying, LIU Ningyuan, LIU Dianlong, ZHENG Lihua. Defining Clinical Queries and Construction of Outcome Indexes in Guidelines on the Diagnosis and Treatment of Anal Fistula by Integrated Traditional Chinese and Western Medicine [J]. Chinese General Practice, 2025, 28(32): 4095-4101.
[9]	ZHI Congcong, CHENG Yicheng, HUANG Zichen, WANG Xiaolong, LI Xue, ZHENG Lihua. Collection and Determination of Clinical Questions and Outcome Indicators Based on the Guidelines for Integrated Traditional Chinese and Western Medicine Diagnosis and Treatment of Hemorrhoids (2025 Edition) [J]. Chinese General Practice, 2025, 28(32): 4089-4094.
[10]	MA Huaping, LI Hongpei, HU Wenyue, HU Yuli, MOU Lei, WU Yue, CHEN Runming, HAN Zhenyun. The Construction of the PROs Scale and Its Theoretical Model for Patients with Perimenopausal Depressive Disorder Treated with Traditional Chinese Medicine [J]. Chinese General Practice, 2025, 28(29): 3703-3710.
[11]	ZHENG Caiyun, WU Shuanger, YU Ganquan, WANG Xin. Construction of Evaluation Index System Inter-professional Collaboration in Family Doctor Team [J]. Chinese General Practice, 2025, 28(28): 3518-3524.
[12]	ZHAO Wenwen, LI Nuoya, ZHANG Yali, ZHANG Jinjia, ZHANG Min, LIU Hualei, XI Biao, WANG Rongying. Research on Evaluation of Quality Evaluation System of General Practice Faculty in Standardized Clinical Residency Training Bases (General Hospitals) [J]. Chinese General Practice, 2025, 28(25): 3187-3192.
[13]	ZHAO Yali, LU Xiaoqin, LIU Jue, ZHANG Yifan, ZHU Zuyi, CHEN Kaiyuan, LIU Min, LIANG Wannian. The Construction of Assessment Index System of Artificial Intelligence General Practitioner [J]. Chinese General Practice, 2025, 28(22): 2705-2711.
[14]	WEI Xu, YIN Yuhui, WANG Xu, YU Ruowen, ZHANG Yili, SUN Kai, WANG Hui, XIE Shiming, LI Yan, QIN Xiaokuan, YIN Xunlu, LI Linghui, ZHU Liguo. Study on the Assessment System of Bone Health Service Capacity of Primary Healthcare Institutions [J]. Chinese General Practice, 2025, 28(19): 2354-2362.
[15]	ZENG Yongtian, CHEN Riling, NONG Xueyan, LIU Zhou, LIANG Lizhong, ZHU Zijian. Clinical Research Progress and Challenges of Digital Therapeutics from Screening to Intervention in Autism Spectrum Disorder [J]. Chinese General Practice, 2025, 28(14): 1702-1708.