Bayesian Analysis in Natural Language Processing

Natural language processing (NLP) went through a profound transformation in the mid-1980s when it shifted to make heavy use of corpora and data-driven techniques to analyze language. Since then, the use of statistical techniques in NLP has evolved in several ways. One such example of evolution took place in the late 1990s or early 2000s, when full-fledged Bayesian machinery was introduced to NLP. This Bayesian approach to NLP has come to accommodate for various shortcomings in the frequentist approach and to enrich it, especially in the unsupervised setting, where statistical learning is done without target prediction examples.

We cover the methods and algorithms that are needed to fluently read Bayesian learning papers in NLP and to do research in the area. These methods and algorithms are partially borrowed from both machine learning and statistics and are partially developed "in-house" in NLP. We cover inference techniques such as Markov chain Monte Carlo sampling and variational inference, Bayesian estimation, and nonparametric modeling. We also cover fundamental concepts in Bayesian statistics such as prior distributions, conjugacy, and generative modeling. Finally, we cover some of the fundamental modeling techniques in NLP, such as grammar modeling and their use with Bayesian analysis.

自1980年代中期以來，自然語言處理（NLP）經歷了一次深刻的轉變，從重度使用語料庫和數據驅動技術來分析語言。自那時以來，NLP中統計技術的使用已經以多種方式演變。其中一個演變的例子發生在1990年代末或2000年代初，當時完整的貝葉斯機制被引入到NLP中。這種貝葉斯方法對NLP進行了各種不足的彌補，特別是在無監督設置下，統計學習是在沒有目標預測示例的情況下進行的。

我們介紹了閱讀NLP中貝葉斯學習論文所需的方法和算法，以及在該領域進行研究所需的方法和算法。這些方法和算法部分來自機器學習和統計學，部分是NLP內部開發的。我們介紹了推理技術，如馬爾可夫鏈蒙特卡羅抽樣和變分推理，貝葉斯估計和非參數建模。我們還介紹了貝葉斯統計中的基本概念，如先驗分布，共軛性和生成建模。最後，我們介紹了NLP中一些基本的建模技術，如語法建模及其與貝葉斯分析的應用。