博碩士論文 etd-0612121-152838 詳細資訊


[回到前頁查詢結果 | 重新搜尋]

姓名 蘇芳沂(FANG-YI SU) 電子郵件信箱 E-mail 資料不公開
畢業系所 資訊管理學系研究所(Department of Information Management)
畢業學位 碩士(Master) 畢業時期 109學年第2學期
論文名稱(中) 建構基於專家迴路的人工指導機器學習
論文名稱(英) Human-guided Machine Learning by Enabling Expert-in-the-Loop
檔案
  • etd-0612121-152838.pdf
  • 本電子全文僅授權使用者為學術研究之目的,進行個人非營利性質之檢索、閱讀、列印。
    請遵守中華民國著作權法之相關規定,切勿任意重製、散佈、改作、轉貼、播送,以免觸法。
    論文使用權限

    紙本論文:立即公開

    電子論文:校內校外完全公開

    論文語文/頁數 英文/48
    統計 本論文已被瀏覽 81 次,被下載 27 次
    摘要(中) 隨著科技的發展,機器學習已經被廣泛運用在各個領域當中,但是大多時候人們只把重點放在模型的正確率上。雖然開發出許多強大的演算法,但是這些模型的架構也變得愈來愈複雜,使得人類無法理解模型的邏輯及判斷因素。因此我們無法得知模型是否從資料中學習對觀測現象合理的解釋與關係,所以在特定領域中(例如:醫療、工業)這樣的模型沒有可解釋性讓人信任,甚至因此無法被運用。此外,模型準確度不光受到模型複雜程度的影響,還會受到訓練資料影響,其中最常見的問題就是缺少資料量或資料特徵與預測目標間並沒有相關性的問題。
    在本研究中,我們提出新的 human-in-the-loop machine learning 架構, 利用混合效應模型樹(Generalized Linear Mixed-Effects Model Tree)這種可解釋性模型讓專家可以透過模型產生的規則清楚了解模型是如何做預測,並且根據專家自身的知識及經驗給予模型更合適的要預測值,其目的是希望透過人的介入來改善機器學習模型潛在的捷徑學習問題,其次藉由規則的呈現也能減少專家標註資料的時間成本。實驗結果顯示,該方法可以從專家回饋的資料中學習新的符合實務應用場景的資料規則,並且能夠找到可解釋並更適合用來預測目標變數的特徵,以提升模型整體表現。此外標註者也可以從模型規則中找出以往沒有思 考過的判斷邏輯、消除思考的偏見,進而使得模型及人類雙方在決策判斷上都可以更加完備。
    摘要(英) Along with the development of technology, machine learning has been widely used in various fields, but most of the time people only focus on the accuracy of the model. Although many powerful algorithms have been developed, the architecture of these models has become more and more complex, which makes it impossible for humans to understand predictive factors in models. Therefore, it is impossible to know whether the models accurately learn the appropriate relationship, so in some certain fields, (e.g., medical, industrial), such models are not explanatory enough to be trusted or even be used. In addition, the accuracy of the model is not only affected by model complexity, but also by the training data. The most common problem is the lack of data, or there is no correlation between the target variable and the input features.
    In this study, we propose a novel human-in-the-loop machine learning architecture, which uses an interpretable algorithm, the Generalized Linear Mixed-Effects Model Tree, so that experts can clearly understand how the model makes predictions through the rules generated by the model, and give more appropriate predicted values to the model according to their knowledge and experience. The purpose of our method is to improve the potential shortcut learning problems of machine learning models through human intervention. Secondly, to reduce the time cost of annotating data by experts through the representation of rules. Experimental results show that the proposed method can learn new data rules in line with practical application scenarios from the feedback of experts, and can find the features that can be interpreted and are more suitable for predicting target variables, so as to improve the overall performance of the model. In addition, the annotator can also find out the judgment logic that has not been considered before and eliminates the thinking bias from the model rules, so that both the model and human beings can be more complete in decision-making and judgment.
    關鍵字(中)
  • 人機迴路機器學習
  • 規則學習
  • 捷徑學習
  • 混合效應模型
  • 可解釋性
  • 使用者信任
  • 關鍵字(英)
  • Human-in-the-Loop Machine Learning
  • Rule Learning
  • Shortcut Learning
  • Mixed-effects Model
  • Interpretability
  • User Trust
  • 論文目次 論文審定書 i
    摘要 ii
    Abstract iv
    List of Figures vii
    List of Table viii
    1. Introduction 1
    2. Background and Related Work 2
    2.2 Mixed Effects Model 2
    2.2 Generalized Linear Mixed-Effects Model Tree 6
    2.3 Shortcut Learning 8
    2.4 Data Annotation and Active Learning 10
    2.5 Human-in-the-loop Machine Learning (HitL–ML) 14
    3. Methodology 17
    3.1 Mixed Effects Model for Longitudinal Data 17
    3.2 Rules Assistance for Data Annotation 19
    3.3 Krippendorff's Alpha Coefficient 20
    3.4 Retraining Model with Experts' Feedback 21
    4. Experimental Result and Discussion 22
    4.1 Data Description 22
    4.2 Experiments 26
    5. Discussion 31
    6. Conclusion 31
    7. References 32
    參考文獻 Andriluka, Mykhaylo, Jasper R. R. Uijlings, and Vittorio Ferrari. “Fluid Annotation: A Human-Machine Collaboration Interface for Full Image Annotation.” In 2018 ACM Multimedia Conference on Multimedia Conference - MM ’18, 1957–66. Seoul, Republic of Korea: ACM Press, 2018. .
    https://doi.org/10.1145/3240508.3241916
    Aroyo, Lora, and Chris Welty. “Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation.” AI Magazine 36, no. 1 (March 25, 2015): 15-24. .
    https://doi.org/10.1609/aimag.v36i1.2564
    Bates, Douglas, Martin Maechler, Ben Bolker [aut, cre, Steven Walker, Rune Haubo Bojesen Christensen, Henrik Singmann, et al. Lme4: Linear Mixed-Effects Models
    32 Using “Eigen” and S4 (version 1.1-27.1), 2021.
    project.org/package=lme4
    .
    https://CRAN.R-
    Branley-Bell, Dawn, Rebecca Whitworth, and Lynne Coventry. “User Trust and
    Understanding of Explainable AI: Exploring Algorithm Visualisations and User Biases.” In Human-Computer Interaction. Human Values and Quality of Life, edited by Masaaki Kurosu, 382–99. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2020.
    https://doi.org/10.1007/978-3-030-49065-
    2_27
    .
    Breiman, Leo. “Bagging Predictors.” Machine Learning 24, no. 2 (August 1, 1996):
    123–40.
    https://doi.org/10.1007/BF00058655
    .
    Bryk, Anthony S., and Stephen W. Raudenbush. Hierarchical Linear
    Models: Applications and Data Analysis Methods. Hierarchical Linear Models: Applications and Data Analysis Methods. Thousand Oaks, CA, US: Sage Publications, Inc, 1992.
    Endert, Alex, M. Shahriar Hossain, Naren Ramakrishnan, Chris North, Patrick Fiaux, and Christopher Andrews. “The Human Is the Loop: New Directions for Visual Analytics.” Journal of Intelligent Information Systems 43, no. 3 (December 1, 2014): 411–35. .
    https://doi.org/10.1007/s10844-014-0304-9
    33 Fishbane, S., and A. R. Nissenson. “The New FDA Label for Erythropoietin Treatment:
    How Does It Affect Hemoglobin Target?” Kidney International 72, no. 7 (October 1, 2007): 806–13. .
    https://doi.org/10.1038/sj.ki.5002401
    Fokkema, M., N. Smits, A. Zeileis, T. Hothorn, and H. Kelderman. “Detecting
    Treatment-Subgroup Interactions in Clustered Data with Generalized Linear Mixed-Effects Model Trees.” Behavior Research Methods 50, no. 5 (October 1, 2018): 2016–34. .
    https://doi.org/10.3758/s13428-017-0971-x
    Fokkema, Marjolein, and Achim Zeileis. Glmertree: Generalized Linear Mixed Model Trees (version 0.2-0), 2019. .
    https://CRAN.R-project.org/package=glmertree
    Geirhos, Robert, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland
    Brendel, Matthias Bethge, and Felix A. Wichmann. “Shortcut Learning in Deep Neural Networks.” Nature Machine Intelligence 2, no. 11 (November 2020): 665–73. .
    https://doi.org/10.1038/s42256-020-00257-z
    Hilgard, Sophie, Nir Rosenfeld, Mahzarin R. Banaji, Jack Cao, and David C. Parkes.
    “Learning Representations by Humans, for Humans.” ArXiv:1905.12686 [Cs,
    Stat], October 9, 2020.
    http://arxiv.org/abs/1905.12686
    .
    Holzinger, Andreas. “Human-Computer Interaction and Knowledge Discovery (HCI-
    KDD): What Is the Benefit of Bringing Those Two Fields to Work Together?” In Availability, Reliability, and Security in Information Systems and HCI, edited by
    34 Alfredo Cuzzocrea, Christian Kittl, Dimitris E. Simos, Edgar Weippl, and Lida Xu, 319–28. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2013. .
    https://doi.org/10.1007/978-3-642-40511-2_22
    Holzinger, Andreas. “Interactive Machine Learning for Health Informatics: When Do We Need the Human-in-the-Loop?” Brain Informatics 3, no. 2 (June 1, 2016): 119–31. .
    https://doi.org/10.1007/s40708-016-0042-6
    Holzinger, Andreas, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell.
    “What Do We Need to Build Explainable AI Systems for the Medical Domain?” ArXiv:1712.09923 [Cs, Stat], December 28, 2017. .
    http://arxiv.org/abs/1712.09923
    Holzinger, Andreas, Markus Plass, Michael Kickmeier-Rust, Katharina Holzinger, Gloria Cerasela Crişan, Camelia-M. Pintea, and Vasile Palade. “Interactive Machine Learning: Experimental Evidence for the Human in the Algorithmic Loop.” Applied Intelligence 49, no. 7 (July 1, 2019): 2401–14. .
    https://doi.org/10.1007/s10489-018-1361-5
    Honeycutt, Donald R., Mahsan Nourani, and Eric D. Ragan. “Soliciting Human-in-the-
    Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy.” ArXiv:2008.12735 [Cs], August 28, 2020. .
    http://arxiv.org/abs/2008.12735
    35 Huang, Sheng-jun, Rong Jin, and Zhi-Hua Zhou. “Active Learning by Querying
    Informative and Representative Examples.” In Advances in Neural Information Processing Systems 23, edited by J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, 892–900. Curran Associates, Inc., 2010.
    http://papers.nips.cc/paper/4176-active-learning-by-querying-informative-and-representative-examples.pdf
    .
    Kieseberg, Peter, Edgar Weippl, and Andreas Holzinger. “Trust for the ‘Doctor in the Loop,’” n.d., 2.
    Krippendorff, Klaus. “Computing Krippendorff’s Alpha-Reliability.” Departmental
    Papers (ASC), January 25, 2011.
    https://repository.upenn.edu/asc_papers/43
    .
    Lage, Isaac, Andrew Slavin Ross, Been Kim, Samuel J. Gershman, and Finale Doshi-Velez. “Human-in-the-Loop Interpretability Prior.” ArXiv:1805.11571 [Cs, Stat], October 30, 2018. .
    http://arxiv.org/abs/1805.11571
    Liu, Ce, William T. Freeman, Edward H. Adelson, and Yair Weiss. “Human-Assisted Motion Annotation.” In 2008 IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008. .
    https://doi.org/10.1109/CVPR.2008.4587845
    Ngai, Grace, and David Yarowsky. “Rule Writing or Annotation: Cost-Efficient
    Resource Usage for Base Noun Phrase Chunking.” ArXiv:Cs/0105003, May 2, 2001. .
    http://arxiv.org/abs/cs/0105003
    36 Nguyen, Dung H. M., and Jon D. Patrick. “Supervised Machine Learning and Active Learning in Classification of Radiology Reports.” Journal of the American Medical Informatics Association: JAMIA 21, no. 5 (October 2014): 893–901. .
    https://doi.org/10.1136/amiajnl-2013-002516
    Nourani, Mahsan, Joanie T. King, and Eric D. Ragan. “The Role of Domain Expertise
    in User Trust and the Impact of First Impressions with Intelligent Systems.” ArXiv:2008.09100 [Cs], August 20, 2020. .
    http://arxiv.org/abs/2008.09100
    Papenmeier, Andrea, Gwenn Englebienne, and Christin Seifert. “How Model Accuracy and Explanation Fidelity Influence User Trust.” ArXiv:1907.12652 [Cs], July 26, 2019. .
    http://arxiv.org/abs/1907.12652
    “R: A Language and Environment for Statistical Computing.” Accessed July 7,
    2021.
    https://www.gbif.org/zh-tw/tool/81287/r-a-language-and-environment-for-
    statistical-computing
    .
    “R Interface to Keras.” Accessed July 7, 2021.
    https://keras.rstudio.com/
    .
    Ristoski, Petar, Dmitry Yu Zubarev, Anna Lisa Gentile, Nathaniel Park, Daniel
    Sanders, Daniel Gruhl, Linda Kato, and Steve Welch. “Expert-in-the-Loop AI for Polymer Discovery.” In Proceedings of the 29th ACM International Conference on
    37 Information & Knowledge Management, 2701–8. Virtual Event Ireland: ACM, 2020. .
    https://doi.org/10.1145/3340531.3416020
    Settles, Burr. “Active Learning Literature Survey.” Technical Report. University of
    Wisconsin-Madison Department of Computer Sciences, 2009. .
    https://minds.wisconsin.edu/handle/1793/60660
    Therneau, Terry M, Elizabeth J Atkinson, and Mayo Foundation. “An Introduction to Recursive Partitioning Using the RPART Routines,” n.d., 60.
    Wang, Meng, and Xian-Sheng Hua. “Active Learning in Multimedia Annotation and
    Retrieval: A Survey.” ACM Transactions on Intelligent Systems and Technology 2, no. 2 (February 2011): 1–21. .
    https://doi.org/10.1145/1899412.1899414
    Ware, James H. “Linear Models for the Analysis of Longitudinal Studies.” The
    American Statistician 39, no. 2 (May 1985): 95–101.
    https://doi.org/10.1080/00031305.1985.10479402.
    “Welcome · Human-in-the-Loop Machine Learning MEAP V10.” Accessed November
    14,2020.
    https://livebook.manning.com/book/human-in-the-loop-machine-learning/chapter-10/v-10/
    .
    Wright, Marvin N., and Andreas Ziegler. “Ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R.” Journal of Statistical Software 77, no. 1 (2017). .
    https://doi.org/10.18637/jss.v077.i01
    38 Yu, Kun, Shlomo Berkovsky, Ronnie Taib, Dan Conway, Jianlong Zhou, and Fang
    Chen. “User Trust Dynamics: An Investigation Driven by Differences in System Performance.” In Proceedings of the 22nd International Conference on Intelligent User Interfaces, 307–17. Limassol Cyprus: ACM, 2017. .
    https://doi.org/10.1145/3025171.3025219
    Zanzotto, Fabio Massimo. “Viewpoint: Human-in-the-Loop Artificial
    Intelligence.” Journal of Artificial Intelligence Research 64 (February 10, 2019): 243–52. .
    https://doi.org/10.1613/jair.1.11345
    “Shiny.” Accessed July 7, 2021.
    https://shiny.rstudio.com/
    .
    Marcus, Gary. “Deep Learning: A Critical Appraisal.” ArXiv:1801.00631 [Cs,
    Stat], January 2, 2018.
    http://arxiv.org/abs/1801.00631
    .
    Zeileis, Achim, Torsten Hothorn, and Kurt Hornik. “Model-Based Recursive
    Partitioning.” Journal of Computational and Graphical Statistics 17, no. 2 (June 2008): 492–514. .
    https://doi.org/10.1198/106186008X319331
    LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep Learning.” Nature 521, no.
    7553 (May 2015): 436–44.
    https://doi.org/10.1038/nature14539
    .
    Hughes, John. “Krippendorffsalpha: An R Package for Measuring Agreement Using
    Krippendorff’s Alpha Coefficient.” ArXiv:2103.12170 [Stat], March 22,
    2021.
    http://arxiv.org/abs/2103.12170
    .
    口試委員
  • 林耕霈 - 召集委員
  • 李珮如 - 委員
  • 康藝晃 - 指導教授
  • 口試日期 2021-07-02 繳交日期 2021-07-12

    [回到前頁查詢結果 | 重新搜尋]


    如有任何問題請與論文審查小組聯繫