估算腎小球過濾率是臨床實踐中評估腎功能的重要指標。雖然傳統方程式和使用臨 床及實驗室數據的機器學習模型可以估算估算腎小球過濾率，但準確預測未來的估算腎小 球過濾率水平仍然是腎臟病學家和機器學習研究人員面臨的重大挑戰。最近的研究表明， 大型語言模型和大型多模態模型可以作為各種應用的強大基礎模型。本研究通過一個包含 50 名患者的實驗室和臨床數據集，探討了大型多模態模型在預測未來估算腎小球過濾率 水平方面的潛力。通過整合各種提示技術和大型多模態模型的集成，我們的研究發現，這 些模型在結合精確提示和估算腎小球過濾率軌跡的可視化表示時，其預測性能可與現有的 機器學習模型媲美。本研究拓展了基礎模型的應用，並建議未來的研究可以利用這些模型 來解決複雜的醫學預測挑戰。

The estimated Glomerular Filtration Rate (eGFR) is an essential indicator of kidney function in clinical practice. Although traditional equations and Machine Learning (ML) models using clinical and laboratory data can estimate eGFR, accurately predicting future eGFR levels remains a significant challenge for nephrologists and ML researchers. Recent advances demonstrate that Large Language Models (LLMs) and Large Multimodal Models (LMMs) can serve as robust foundation models for diverse applications.
This study investigates the potential of LMMs to predict future eGFR levels with a dataset consisting of laboratory and clinical values from 50 patients. By integrating various prompting techniques and ensembles of LMMs, our findings suggest that these models, when combined with precise prompts and visual representations of eGFR trajectories, offer predictive performance comparable to existing ML models. This research extends the application of foundation models and suggests avenues for future studies to harness these models in addressing complex medical forecasting challenges.

論文審定書 .................................................................................................................................. i
摘要 ............................................................................................................................................. ii
Abstract ..................................................................................................................................... iii
Table of Content ....................................................................................................................... iv
Table of Figures ......................................................................................................................... v
Table of Tables ......................................................................................................................... vi
1. Introduction .................................................................................................................... 1
2. Background ..................................................................................................................... 3
2.1. Importance of eGFR and Traditional Estimation Methods ................................... 3
2.2. Transforming Time Series Data into Images for Superior Model Performance .. 6
2.3. The Promise of Large Multimodal Models in Healthcare ...................................... 8
2.4. Diverse Prompting Techniques for Enhancing Performance................................. 9
2.5. Advanced Techniques: Prompt Ensemble and LLM Ensemble .......................... 11
3. Methods ......................................................................................................................... 16
3.1. Overview of the Series-to-Image LMM Framework ............................................. 16
3.2. Prompt Ensemble and LMM Ensemble ................................................................. 19
4. Materials and Experiments ......................................................................................... 21
4.1. Data Collection and Preprocessing ......................................................................... 21
4.2. Model Implementation and Prompt Design ........................................................... 24
4.3. Machine Learning Models for Comparison ........................................................... 26
4.4. Results and Evaluation Metrics............................................................................... 27
5. Conclusion and Discussion .......................................................................................... 31
Reference .................................................................................................................................. 34
Appendix: Prompt Templates ................................................................................................ 42

Albawi, S., Mohammed, T. A., & Al-Zawi, S. (2017). Understanding of a convolutional neural network. 2017 International Conference on Engineering and Technology (ICET), 1–6. https://doi.org/10.1109/ICEngTechnol.2017.8308186
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Chen, Y.-C., Weng, S.-C., Liu, J.-S., Chuang, H.-L., Hsu, C.-C., & Tarng, D.-C. (2017). Severe Decline of Estimated Glomerular Filtration Rate Associates with Progressive Cognitive Deterioration in the Elderly: A Community-Based Cohort Study. Scientific Reports, 7(1), 42690. https://doi.org/10.1038/srep42690
Fan, Z., Yang, Q., Xu, Z., Sun, K., Yang, M., Yin, R., Zhao, D., Fan, J., Ma, H., Shen, Y., & Xia, H. (2022). Construct a classification decision tree model to select the optimal equation for estimating glomerular filtration rate and estimate it more accurately. Scientific Reports, 12(1), 14877. https://doi.org/10.1038/s41598-022-19185-6
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., ... Vinyals, O. (2024). Gemini: A Family of Highly Capable Multimodal Models (arXiv:2312.11805). arXiv. https://doi.org/10.48550/arXiv.2312.11805
Gramopadhye, O., Nachane, S. S., Chanda, P., Ramakrishnan, G., Jadhav, K. S., Nandwani, Y., Raghu, D., & Joshi, S. (2024). Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering (arXiv:2403.04890). arXiv. http://arxiv.org/abs/2403.04890
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2023). A Survey of Large Language Models for Healthcare: From Data, Technology, and Applications to Accountability and Ethics (arXiv:2310.05694). arXiv. http://arxiv.org/abs/2310.05694
Inaguma, D., Hayashi, H., Yanagiya, R., Koseki, A., Iwamori, T., Kudo, M., Fukuma, S., & Yuzawa, Y. (2022). Development of a machine learning-based prediction model for extremely rapid decline in estimated glomerular filtration rate in patients with chronic kidney disease: A retrospective cohort study using a large data set from a hospital in Japan. BMJ Open, 12(6), e058833. https://doi.org/10.1136/bmjopen-2021-058833
Isaza-Ruget, M. A., Yomayusa, N., González, C. A., H., C. A., De Oro V., F. A., Cely, A., Murcia, J., Gonzalez-Velez, A., Robayo, A., Colmenares-Mejía, C. C., Castillo, A., & Conde, M. I. (2024). Predicting chronic kidney disease progression with artificial intelligence. BMC Nephrology, 25(1), 148. https://doi.org/10.1186/s12882-024-03545-7
Jager, K. J., Kovesdy, C., Langham, R., Rosenberg, M., Jha, V., & Zoccali, C. (2019). A single number for advocacy and communication—Worldwide more than 850 million individuals have kidney diseases. Nephrology Dialysis Transplantation, 34(11), 1803–1805. https://doi.org/10.1093/ndt/gfz174
Jiang, D., Ren, X., & Lin, B. Y. (2023). LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (arXiv:2306.02561). arXiv. http://arxiv.org/abs/2306.02561
Jiang, W., Zhang, D., Ling, L., & Lin, R. (2022). Time series classification based on image transformation using feature fusion strategy. Neural Processing Letters, 54(5), 3727–3748.
Lees, J. S., Welsh, C. E., Celis-Morales, C. A., Mackay, D., Lewsey, J., Gray, S. R., Lyall, D. M., Cleland, J. G., Gill, J. M. R., Jhund, P. S., Pell, J., Sattar, N., Welsh, P., & Mark, P. B.
(2019). Glomerular filtration rate by differing measures, albuminuria and prediction of cardiovascular disease, mortality and end-stage kidney disease. Nature Medicine, 25(11), 1753–1760. https://doi.org/10.1038/s41591-019-0627-8
Levey, A. S., Coresh, J., Greene, T., Marsh, J., Stevens, L. A., Kusek, J. W., Van Lente, F., & for Chronic Kidney Disease Epidemiology Collaboration. (2007). Expressing the Modification of Diet in Renal Disease Study Equation for Estimating Glomerular Filtration Rate with Standardized Serum Creatinine Values. Clinical Chemistry, 53(4), 766–772. https://doi.org/10.1373/clinchem.2006.077180
Levey, A. S., Coresh, J., Tighiouart, H., Greene, T., & Inker, L. A. (2020). Measured and estimated glomerular filtration rate: Current status and future directions. Nature Reviews Nephrology, 16(1), 51–64. https://doi.org/10.1038/s41581-019-0191-y
Levey, A. S., Stevens, L. A., Schmid, C. H., Zhang, Y. (Lucy), Castro, A. F., Feldman, H. I., Kusek, J. W., Eggers, P., Van Lente, F., Greene, T., Coresh, J., & for the CKD-EPI (Chronic Kidney Disease Epidemiology Collaboration). (2009). A New Equation to Estimate Glomerular Filtration Rate. Annals of Internal Medicine, 150(9), 604–612. https://doi.org/10.7326/0003-4819-150-9-200905050-00006
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023). LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (arXiv:2306.00890). arXiv. http://arxiv.org/abs/2306.00890
Li, H., Wang, H., Sun, X., He, H., & Feng, J. (2024). Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM (arXiv:2404.11209). arXiv. http://arxiv.org/abs/2404.11209
Li, J., & Wang, Q. (2023). Comparison of the representational ability in individual difference analysis using 2-D time-series image and time-series feature patterns. Expert Systems with Applications, 215, 119429. https://doi.org/10.1016/j.eswa.2022.119429
Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024). More Agents Is All You Need (arXiv:2402.05120). arXiv. http://arxiv.org/abs/2402.05120
Li, L., Astor, B. C., Lewis, J., Hu, B., Appel, L. J., Lipkowitz, M. S., Toto, R. D., Wang, X., Wright, J. T., & Greene, T. H. (2012). Longitudinal Progression Trajectory of GFR Among Patients With CKD. American Journal of Kidney Diseases, 59(4), 504–512. https://doi.org/10.1053/j.ajkd.2011.12.009
Li, W., & Liao, J. (2017). A comparative study on trend forecasting approach for stock price time series. 2017 11th IEEE International Conference on Anti-Counterfeiting, Security, and Identification (ASID), 74–78. https://doi.org/10.1109/ICASID.2017.8285747
Li, X., Kang, Y., & Li, F. (2020). Forecasting with time series imaging. Expert Systems with Applications, 160, 113680. https://doi.org/10.1016/j.eswa.2020.113680
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023). Making Large Language Models Better Reasoners with Step-Aware Verifier (arXiv:2206.02336). arXiv. http://arxiv.org/abs/2206.02336
Lin, M.-Y., Cheng, L.-J., Chiu, Y.-W., Hsieh, H.-M., Wu, P.-H., Lin, Y.-T., Wang, S.-L., Jian, F.- X., Hsu, C. C., Yang, S.-A., Lee, H.-L., & Hwang, S.-J. (2018). Effect of national pre-ESRD care program on expenditures and mortality in incident dialysis patients: A population-based study. PLOS ONE, 13(6), e0198387. https://doi.org/10.1371/journal.pone.0198387
Matsushita, K., Mahmoodi, B. K., Woodward, M., Emberson, J. R., Jafar, T. H., Jee, S. H., Polkinghorne, K. R., Shankar, A., Smith, D. H., Tonelli, M., Warnock, D. G., Wen, C.-P.,
Coresh, J., Gansevoort, R. T., Hemmelgarn, B. R., & Levey, A. S. (n.d.). Comparison of Risk Prediction Using the CKD-EPI Equation and the MDRD Study Equation for Estimated Glomerular Filtration Rate.
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems (arXiv:2303.13375). arXiv. http://arxiv.org/abs/2303.13375
Oh, S. W., Byun, S.-S., Kim, J. K., Jeong, C. W., Kwak, C., Hwang, E. C., Kang, S. H., Chung, J., Kim, Y.-J., Ha, Y.-S., & Hong, S.-H. (2024). Machine learning models for predicting the onset of chronic kidney disease after surgery in patients with renal cell carcinoma. BMC Medical Informatics and Decision Making, 24(1), 85. https://doi.org/10.1186/s12911-024- 02473-8
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., ... Zoph, B. (2024). GPT-4 Technical Report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole- Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P., Arcas, B. A. y, Webster, D., ... Natarajan, V. (2022). Large Language Models Encode Clinical Knowledge (arXiv:2212.13138). arXiv. http://arxiv.org/abs/2212.13138
Sonoda, Y., Kurokawa, R., Nakamura, Y., Kanzawa, J., Kurokawa, M., Ohizumi, Y., Gonoi, W., & Abe, O. (2024). Diagnostic Performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” Cases. https://doi.org/10.1101/2024.05.26.24307915
Stevens, P. E., Ahmed, S. B., Carrero, J. J., Foster, B., Francis, A., Hall, R. K., Herrington, W. G., Hill, G., Inker, L. A., Kazancıoğlu, R., Lamb, E., Lin, P., Madero, M., McIntyre, N., Morrow, K., Roberts, G., Sabanayagam, D., Schaeffner, E., Shlipak, M., ... Levin, A. (2024). KDIGO 2024 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease. Kidney International, 105(4), S117–S314. https://doi.org/10.1016/j.kint.2023.10.018
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., Islamaj, R., Kapoor, A., Gao, X., & Lu, Z. (2023). Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings in Bioinformatics, 25(1), bbad493. https://doi.org/10.1093/bib/bbad493
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (arXiv:2305.04388). arXiv. http://arxiv.org/abs/2305.04388
Velasco-Gallego, C., & Lazakis, I. (2022). Development of a time series imaging approach for fault classification of marine systems. Ocean Engineering, 263, 112297. https://doi.org/10.1016/j.oceaneng.2022.112297
Wang, H., Bowe, B., Cui, Z., Yang, H., Swamidass, S. J., Xie, Y., & Al-Aly, Z. (2022). A Deep Learning Approach for the Estimation of Glomerular Filtration Rate. IEEE Transactions on NanoBioscience, 21(4), 560–569. https://doi.org/10.1109/TNB.2022.3147957
Wang, Z. M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Yang, J., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Huang, S. W., Fu, J., & Peng, J. (2024). RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models (arXiv:2310.00746). arXiv. http://arxiv.org/abs/2310.00746
Wang, Z., & Oates, T. (n.d.). Encoding Time Series as Images for Visual Inspection and Classification Using Tiled Convolutional Neural Networks.
Woillard, J.-B., Salmon Gandonnière, C., Destere, A., Ehrmann, S., Merdji, H., Mathonnet, A., Marquet, P., & Barin-Le Guellec, C. (2021). A Machine Learning Approach to Estimate the Glomerular Filtration Rate in Intensive Care Unit Patients Based on Plasma Iohexol Concentrations and Covariates. Clinical Pharmacokinetics, 60(2), 223–233. https://doi.org/10.1007/s40262-020-00927-6
Xiang, W., Li, C., Zhou, Y., Wang, B., & Zhang, L. (2023). Generative Action Description Prompts for Skeleton-based Action Recognition (arXiv:2208.05318). arXiv. http://arxiv.org/abs/2208.05318
Yadkori, Y. A., Kuzborskij, I., György, A., & Szepesvári, C. (2024). To Believe or Not to Believe Your LLM (arXiv:2406.02543). arXiv. http://arxiv.org/abs/2406.02543
Yang, H., Li, M., Zhou, H., Xiao, Y., Fang, Q., & Zhang, R. (2023). One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering. https://doi.org/10.1101/2023.12.21.23300380
Yang, J., Sun, Y., Chen, Y., Mao, M., Bai, L., & Zhang, S. (2024). Time series-to-image encoding for saturation line prediction using channel and spatial-wise attention network. Expert Systems with Applications, 237, 121440. https://doi.org/10.1016/j.eswa.2023.121440
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015). Visual Madlibs: Fill in the blank Image Generation and Question Answering (arXiv:1506.00278). arXiv. http://arxiv.org/abs/1506.00278
Zhang, L., Richter, L. R., Kim, T., & Hripcsak, G. (2024). Evaluating and Improving the Performance and Racial Fairness of Algorithms for GFR Estimation (p. 2024.01.07.24300943). medRxiv. https://doi.org/10.1101/2024.01.07.24300943
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., Hua, Y., Mao, C., You, C., Wu, X., Zheng, Y., Clifton, L., Li, Z., Luo, J., & Clifton, D. A. (2024). A Survey of Large Language Models in Medicine: Progress, Application, and Challenge (arXiv:2311.05112). arXiv. http://arxiv.org/abs/2311.05112
Zsom, L., Zsom, M., Salim, S. A., & Fülöp, T. (2022). Estimated Glomerular Filtration Rate in Chronic Kidney Disease: A Critical Review of Estimate-Based Predictions of Individual Outcomes in Kidney Disease. Toxins, 14(2), 127. https://doi.org/10.3390/toxins14020127