隨著機器學習模型的發展，越來越多的模型被運用在各式領域，資料科學家也不斷提高預測準確度要求。但在追求高準確度的同時，我們往往忽略了了解預測結果提高的原因或規則。為了應對這一挑戰，本研究探索將大型語言模型（LLMs）整合進因果發現框架中，以提升模型的準確性和解釋性。我們利用LLMs替代人工專家標注因果規則，並系統化轉化人類直覺為機器可讀的規則。方法包括LLMs不斷標注並精煉因果規則，然後將其重新整合到數據集中提升模型性能。研究結果顯示，整合LLMs顯著提升了模型的準確性和穩健性，同時改善了解釋性。儘管此方法存在規則複雜性和計算成本等限制，但它為未來研究提供了寶貴見解，突顯了LLMs在醫學研究中的潛力，支持更個人化和有效的治療計劃，並為進一步優化LLMs在因果發現中的應用奠定基礎，增強醫學決策。

With the advancement of machine learning models, they are increasingly being applied across various domains, while data scientists continuously strive to enhance accuracy. However, as prediction accuracy improves, we often overlook the reasons or rules behind these improvements. To address this challenge, this study explores integrating LLMs into the causal discovery framework to enhance accuracy and interpretability. We use LLMs to replace human experts in annotating causal rules and systematically transform human intuition into machine-readable rules. Our method involves an iterative process where LLMs annotate and refine rules, then reintegrate them into the dataset to improve performance. Results show that integrating LLMs significantly enhances accuracy and robustness while improving interpretability. Despite limitations like rule complexity and computational costs, this methodology offers valuable insights for future research, highlighting the potential of LLMs in medical research. It supports more personalized and effective treatment plans by identifying patient subgroups with altered risks from medications, laying the groundwork for further optimization of LLMs in causal discovery and enhancing medical decision-making.

論文審定書 i
摘要 ii
ABSTRACT iii
LIST OF FIGURES v
LIST OF TABLES vi
1. INTRODUCTION 1
2. BACKGROUND AND RELATED WORK 5
2.1 OVERVIEW OF CAUSAL INFERENCE 5
2.2 CAUSAL TREE 7
2.3 CONTROLLING CONFOUNDING VARIABLES 11
2.4 HUMAN-IN-THE-LOOP IN DATA ANNOTATION 14
2.5 LLMS IN CAUSAL DISCOVERY 18
3. CAUSAL RULE DISCOVERY WITH LLMS-IN-THE-LOOP LEARNING 21
3.1 FIRST STEP: CONTROLLING CONFOUNDING VARIABLES 22
3.2 SECOND STEP: RULES REFINEMENT USING LLMS-IN-THE-LOOP 23
3.3 THIRD STEP: ITERATION 28
3.4 FOURTH STEP: EVALUATION 29
4. EXPERIMENT RESULTS 31
4.1 DATA SOURCE 31
4.2 EXPERIMENT RESULTS 32
5. CONCLUSION 43
6. REFERENCE 47
APPENDIX 54

Angrist, J. D., & Imbens, G. W. (1995). Identification and Estimation of Local Average Treatment Effects (Working Paper 118). National Bureau of Economic Research. https://doi.org/10.3386/t0118
Babic, B., Nesic, N., & Miljković, Z. (2008). A review of automated feature recognition with rule-based pattern recognition. Computers in Industry, 59, 321–337. https://doi.org/10.1016/j.compind.2007.09.001
Bangalore, S., Messerli, F. H., Kostis, J. B., & Pepine, C. J. (2007). Cardiovascular Protection Using Beta-Blockers. Journal of the American College of Cardiology, 50(7), 563–572. https://doi.org/10.1016/j.jacc.2007.04.060
Bargagli-Stoffi, F. J., Cadei, R., Lee, K., & Dominici, F. (2024). Causal Rule Ensemble: Interpretable Discovery and Inference of Heterogeneous Treatment Effects (arXiv:2009.09036). arXiv. https://doi.org/10.48550/arXiv.2009.09036
Bergmark, B. A., Mathenge, N., Merlini, P. A., Lawrence-Wright, M. B., & Giugliano, R. P. (2022). Acute coronary syndromes. The Lancet, 399(10332), 1347–1358. https://doi.org/10.1016/S0140-6736(21)02391-6
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Brown, N. J., & Vaughan, D. E. (1998). Angiotensin-Converting Enzyme Inhibitors. Circulation, 97(14), 1411–1420. https://doi.org/10.1161/01.CIR.97.14.1411
Brownlee, J. (2018, July 17). The Role of Randomization to Address Confounding Variables in Machine Learning. MachineLearningMastery.Com. https://machinelearningmastery.com/confounding-variables-in-machine-learning/
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15(3), 39:1-39:45. https://doi.org/10.1145/3641289
Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. https://doi.org/10.1145/2939672.2939785
Dai, S.-C., Xiong, A., & Ku, L.-W. (2023). LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis (arXiv:2310.15100). arXiv. https://doi.org/10.48550/arXiv.2310.15100
Egami, N., Hinck, M., Stewart, B. M., & Wei, H. (2024). Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models (arXiv:2306.04746). arXiv. https://doi.org/10.48550/arXiv.2306.04746
Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5), 1189–1232.
Greifer, N., & Stuart, E. A. (2021). Matching Methods for Confounder Adjustment: An Addition to the Epidemiologist’s Toolbox. Epidemiologic Reviews, 43(1), 118–129. https://doi.org/10.1093/epirev/mxab003
Grimmer, J., Messing, S., & Westwood, S. J. (2017). Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods. Political Analysis, 25(4), 413–434. https://doi.org/10.1017/pan.2017.15
Holzinger, A. (2016). Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Informatics, 3(2), 119–131. https://doi.org/10.1007/s40708-016-0042-6
Huber, C., Benda, N., & Friede, T. (2022). Subgroup identification in individual participant data meta-analysis using model-based recursive partitioning. Advances in Data Analysis and Classification, 16(3), 797–815. https://doi.org/10.1007/s11634-021-00458-3
Jager, K. J., Zoccali, C., MacLeod, A., & Dekker, F. W. (2008). Confounding: What it is and how to deal with it. Kidney International, 73(3), 256–260. https://doi.org/10.1038/sj.ki.5002650
Kıcıman, E., Ness, R., Sharma, A., & Tan, C. (2023). Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (arXiv:2305.00050). arXiv. https://doi.org/10.48550/arXiv.2305.00050
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024). BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (arXiv:2402.10373). arXiv. https://doi.org/10.48550/arXiv.2402.10373
Li, J., Ma, S., Le, T., Liu, L., & Liu, J. (2017). Causal Decision Trees. IEEE Transactions on Knowledge and Data Engineering, 29(2), 257–271. https://doi.org/10.1109/TKDE.2016.2619350
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9), 195:1-195:35. https://doi.org/10.1145/3560815
Liu, X., Xu, P., Wu, J., Yuan, J., Yang, Y., Zhou, Y., Liu, F., Guan, T., Wang, H., Yu, T., McAuley, J., Ai, W., & Huang, F. (2024). Large Language Models and Causal Inference in Collaboration: A Comprehensive Survey (arXiv:2403.09606). arXiv. https://doi.org/10.48550/arXiv.2403.09606
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., & Ge, B. (2023). Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models. Meta-Radiology, 1(2), 100017. https://doi.org/10.1016/j.metrad.2023.100017
McGrath, L. J., Ellis, A. R., & Brookhart, M. A. (2015). Controlling Time-Dependent Confounding by Health Status and Frailty: Restriction Versus Statistical Adjustment. American Journal of Epidemiology, 182(1), 17–25. https://doi.org/10.1093/aje/kwu485
Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023). Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review, 56(4), 3005–3054. https://doi.org/10.1007/s10462-022-10246-w
Pearl, J. (2010). An Introduction to Causal Inference. The International Journal of Biostatistics, 6(2), 7. https://doi.org/10.2202/1557-4679.1203
Polikar, R. (2012). Ensemble Learning. In C. Zhang & Y. Ma (Eds.), Ensemble Machine Learning: Methods and Applications (pp. 1–34). Springer. https://doi.org/10.1007/978-1-4419-9326-7_1
Pourhoseingholi, M. A., Baghestani, A. R., & Vahedi, M. (2012). How to control confounding effects by statistical analysis. Gastroenterology and Hepatology From Bed to Bench, 5(2), 79–83.
Richardson, M., Garner, P., & Donegan, S. (2019). Interpretation of subgroup analyses in systematic reviews: A tutorial. Clinical Epidemiology and Global Health, 7(2), 192–198. https://doi.org/10.1016/j.cegh.2018.05.005
Rubin, D. B. (2005). Causal Inference Using Potential Outcomes: Design, Modeling, Decisions. Journal of the American Statistical Association, 100(469), 322–331. https://doi.org/10.1198/016214504000001880
Ruggieri, S. (2002). Efficient C4.5 [classification algorithm]. IEEE Transactions on Knowledge and Data Engineering, 14(2), 438–444. https://doi.org/10.1109/69.991727
SONG, Y., & LU, Y. (2015). Decision tree methods: Applications for classification and prediction. Shanghai Archives of Psychiatry, 27(2), 130–135. https://doi.org/10.11919/j.issn.1002-0829.215044
von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Pfrommer, J., Pick, A., Ramamurthy, R., Walczak, M., Garcke, J., Bauckhage, C., & Schuecker, J. (2021). Informed Machine Learning—A Taxonomy and Survey of Integrating Knowledge into Learning Systems. IEEE Transactions on Knowledge and Data Engineering, 1–1. https://doi.org/10.1109/TKDE.2021.3079836
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022). A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135, 364–381. https://doi.org/10.1016/j.future.2022.05.014
Wu, Y., Jiang, A. Q., Li, W., Rabe, M., Staats, C., Jamnik, M., & Szegedy, C. (2022). Autoformalization with Large Language Models. Advances in Neural Information Processing Systems, 35, 32353–32368.
Yazdani, A., & Boerwinkle, E. (2015). Causal Inference in the Age of Decision Medicine. Journal of Data Mining in Genomics & Proteomics, 6(1). https://doi.org/10.4172/2153-0602.1000163
Yuan, Y., Wu, L., & Zhang, X. (2021). Gini-Impurity Index Analysis. IEEE Transactions on Information Forensics and Security, 16, 3154–3169. https://doi.org/10.1109/TIFS.2021.3076932