STAT 479 Lecture 16

Plackett-Luce Models

Motivation: NFL Mock Drafts

• Projection for which players taken in what order
• Huge industry of fans & media releasing mock drafts regularly

• Can we derive any “wisdom from the crowd” to assess prospects?
  • CBS vs Draft Network
  • Consensus “big board” (i.e., overall ranking of players)
• \(\mathbb{P}(\textrm{available at pick 11} \vert \textrm{not taken in top-3})\)?

Setup

• \(p\) items (indexed by \(j = 1, \ldots, p\))
• \(n\) raters (indexed by \(i = 1, \ldots n\))
• Each rater will report a rank ordering of their top \(n_{i}\) items
• Observations: \(i(1) \succ i(2) \succ \ldots \succ i(n_{i})\)
• \(i(k)\): index of item that rater \(i\) ranked \(k\)-th overall
• Raters may rank \(n_{i} < p\) items

Synthetic Data Example

• 4 items (\(A\), \(B\), \(C\), and \(D\)) & 6 raters

  • Rater 1: \(A \succ B \succ D \succ C\)
  • Rater 2: \(B \succ A\)
  • Rater 3: \(A \succ D \succ C\)
  • Rater 4: \(B \succ D \succ C\)
  • Rater 5: \(A \succ B \succ D \succ C\)
  • Rater 6: \(A \succ D \succ C\)

• Can we deduce a global ranking?

A Generative Model

• Rater \(i\) first decides how many items to rank (i.e., draw \(n_{i}\))
• Rater \(i\) sequentially selects items without replacement
  • Each step, pick one item from available options
  • First item selected is \(i(1)\), second item selected is \(i(2) \ldots\)
  • Stops after selecting \(i(n_{i})\)

Generative Example I

• Rater 1: \(A \succ B \succ D \succ C.\)
• Decides to rank all 4 items
• First step: draws \(A\) from available items \(\{A, B, C, D\}.\)
• Second step: draws \(B\) from \(\{B, C, D\},\)
• Third step: draws \(D\) from \(\{C,D\}.\)
• Fourth step: draws \(C\) from \(\{C\}\)

Generative Example II

• Rater 2: \(B \succ A\)
• Decides to rank only 2 items
• First step: draws \(B\) from \(\{A,B,C,D\}\)
• Second step: draws \(A\) from \(\{A, C, D\}\)

Plackett-Luce Model

• Item \(j\) has latent parameter \(\lambda_{j}\) (“desirability”) \[ \mathbb{P}(\textrm{select } j \textrm{ from } \mathcal{S}) = \begin{cases} \frac{e^{\lambda_{j}}}{\sum_{s \in \mathcal{S}}{e^{\lambda_{s}}}} & \textrm{if } j \in \mathcal{S} \\ 0 & \textrm{otherwise}. \end{cases} \]
• Bradley-Terry: special case where \(\mathcal{S}\) contains 2 items

Example I

• Rater 1: \(A \succ B \succ D \succ C\)
• First select \(A\) from \(\mathcal{S} = \{A, B, C, D\}\) \[\mathbb{P}(\textrm{select } A) = \frac{e^{\lambda_{A}}}{e^{\lambda_{A}} + e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}}\]

• Then select \(B\) from \(\mathcal{S} = \{B, C, D\}\) \[ \mathbb{P}(\textrm{select } B \vert \textrm{selected } A) = \frac{e^{\lambda_{B}}}{e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}} \]

Example (cont’d)

• Then select \(B\) from \(\mathcal{S} = \{B, C, D\}\) \[ \mathbb{P}(\textrm{select } D \vert \textrm{selected } A, B) = \frac{e^{\lambda_{D}}}{e^{\lambda_{C}} + e^{\lambda_{D}}} \]

• Can multiply these probabilities to obtain \[ \begin{align*} \mathbb{P}(A \succ B \succ D \succ C) &= \frac{e^{\lambda_{A}}}{e^{\lambda_{A}} + e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}} \\ ~ &\times \frac{e^{\lambda_{B}}}{e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}}\times \frac{e^{\lambda_{D}}}{e^{\lambda_{C}} + e^{\lambda_{D}}} \end{align*} \]

Fitting Plackett-Luce Models in R

• PlacketLuce available on CRAN

install.packages("PlackettLuce")

• Like BradleyTerry2, syntax is a bit idiosyncratic
• Data must be formatted as ranking object

Ranking Matrix I

• Encode \(n\) rankings of \(p\) items in an \(n \times p\) matrix \(\boldsymbol{\mathbf{R}}\)
• \(r_{ij} = k\): rater \(i\) ranks item \(j\) \(k\)-th overall
• \(r_{ij}\) is NA if rather \(i\) does not rank item \(j\)

• Rater 1 (\(A \succ B \succ D \succ C\)) and Rater 4 (\(B \succ D \succ C\))
  • \(r_{1,A} = 1\), \(r_{1,B} = 2\), \(r_{1,D} = 3\) and \(r_{1,C} = 4\)
  • \(r_{4,B} = 1,\) \(r_{4,D} = 2,\) and \(r_{4,C} = 3\). \(r_{1,A}\) is NA

Creating a ranking object

• Creating \(\boldsymbol{\mathbf{R}}\)
• Using as.rankings()

R <- matrix(
  c(1,2,4,3,
    2, 1, NA, NA,
    1, NA, 3, 2,
    NA, 1, 3, 2,
    1, 2, 4, 3,
    1, NA, 3, 2),
  nrow = 6, ncol = 4, dimnames = list(c(), c("A", "B", "C", "D")),
  byrow = TRUE)
R

      A  B  C  D
[1,]  1  2  4  3
[2,]  2  1 NA NA
[3,]  1 NA  3  2
[4,] NA  1  3  2
[5,]  1  2  4  3
[6,]  1 NA  3  2

R_rank <- PlackettLuce::as.rankings(R)
R_rank

[1] "A > B > D > C" "B > A"         "A > D > C"     "B > D > C"    
[5] "A > B > D > C" "A > D > C"    

Estimating Plackett-Luce Parameters

fit <- PlackettLuce::PlackettLuce(rankings = R_rank)
lambda_hat <- coef(fit)
lambda_hat

         A          B          C          D 
 0.0000000 -0.5006691 -4.9505260 -2.5287292 

Estimating \(\mathbb{P}(A \succ B \succ D \succ C)\) I

• Rater 1 must first select \(A\), then \(B,\) then \(D\)

\[ \begin{align*} \mathbb{P}(\textrm{select } A) &= \frac{e^{\lambda_{A}}}{e^{\lambda_{A}} + e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}} \\ \mathbb{P}(\textrm{select } B \vert \textrm{selected } A) &= \frac{e^{\lambda_{B}}}{e^{\lambda_{B}} + e^{\lambda_{C}} + e^{\lambda_{D}}} \\ \mathbb{P}(\textrm{select } D \vert \textrm{selected } A, B) &= \frac{e^{\lambda_{D}}}{e^{\lambda_{C}} + e^{\lambda_{D}}} \end{align*} \]

• Multiply these probabilities to get \(\mathbb{P}(A \succ B \succ D \succ C)\)

Estimating \(\mathbb{P}(A \succ B \succ D \succ C)\) II

pA <- exp(lambda_hat["A"])/sum(exp(lambda_hat[c("A", "B", "C", "D")]))
pB <- exp(lambda_hat["B"])/sum(exp(lambda_hat[c("B", "C", "D")]))
pD <- exp(lambda_hat["D"])/sum(exp(lambda_hat[c("C", "D")]))

cat("P(A) = ", round(pA, digits = 3), "\n")

P(A) =  0.591 

cat("P(B | A) = ", round(pB, digits = 3), "\n")

P(B | A) =  0.875 

cat("P(D | A,B) = ", round(pD, digits = 3), "\n")

P(D | A,B) =  0.918 

cat("P(A > B > D > C) = ", round(pA * pB * pD, digits = 3))

P(A > B > D > C) =  0.475

Example: Beyond Ranking Probabilities

• \(\mathbb{P}(B \textrm{ selected second})\)
• \(\mathbb{P}(B \textrm{ selected second} \vert A \textrm{ selected first})\)

pA <- exp(lambda_hat["A"])/sum(exp(lambda_hat[c("A", "B", "C", "D")]))
pAB <- exp(lambda_hat["A"])/sum(exp(lambda_hat[c("A", "B", "C", "D")])) * exp(lambda_hat["B"])/sum(exp(lambda_hat[c("B", "C", "D")]))
pCB <- exp(lambda_hat["C"])/sum(exp(lambda_hat[c("A", "B", "C", "D")])) * exp(lambda_hat["B"])/sum(exp(lambda_hat[c("A", "B", "D")]))
pDB <- exp(lambda_hat["D"])/sum(exp(lambda_hat[c("A", "B", "C", "D")])) * exp(lambda_hat["B"])/sum(exp(lambda_hat[c("A", "B", "C")]))
cat("P(B selected second) = ",round(pAB + pCB + pDB, digits = 3), "\n")

P(B selected second) =  0.536 

cat("P(B selected second | A selected first) = ", round(pAB/pA, digits = 3), "\n")

P(B selected second | A selected first) =  0.875 

NFL Mock Draft Data

• CSV file three_round_mock.csv available on Box (link): https://go.wisc.edu/7b7du4

raw_data <- readr::read_csv(file = "three_round_mocks.csv")
set.seed(129)
raw_data |> dplyr::slice_sample(n = 5)

# A tibble: 5 × 7
  site                         date       name         pick position type  url  
  <chr>                        <date>     <chr>       <dbl> <chr>    <chr> <chr>
1 Pacific Takes - Follman      2018-03-19 R.J. McInt…    55 DT       Fan   http…
2 Walter Football - Alex9299   2018-02-14 Sam Darnold     1 QB       Fan   http…
3 Walter Football - Lucasmd3   2018-03-30 Mike Hughes    41 CB       Fan   http…
4 Baltimore Beatdown - Lericos 2018-02-13 Maurice Hu…    18 DT       Fan   http…
5 Fox 59 - Joe Hopkins         2018-02-26 Josh Jacks…    14 CB       Media http…

Forming the Ranking Matrix

• Data is in long format
• Must reshape it into wide format

     Sam Darnold Derwin James Arden Key Tarvarus McFadden Mike McGlinchey
[1,]           1            2         3                 4               5
[2,]           1           16         5                12               6
[3,]           1            4         6                NA               8
[4,]           1            2         9                17               7
[5,]           4           10         9                 6               8

Estimating \(\lambda_{j}\)’s

mock_rankings <- PlackettLuce::as.rankings(x = ranking_matrix)
fit <- PlackettLuce::PlackettLuce(rankings = mock_rankings)
lambda_hat <- coef(fit)
round(sort(lambda_hat, decreasing = TRUE)[1:10], digits = 3)

       Sam Darnold     Saquon Barkley         Josh Rosen      Bradley Chubb 
             0.000             -0.561             -0.997             -1.455 
    Quenton Nelson Minkah Fitzpatrick     Baker Mayfield   Josh Allen (WYO) 
            -1.963             -1.963             -2.277             -2.440 
      Roquan Smith       Derwin James 
            -2.627             -2.986 

Softmax

• Plackett-Luce probabilities computed using softmax \[ \sigma(x_{1}, \ldots, x_{n}) = \left( \frac{e^{x_{1}}}{\sum{e^{x_{i}}}}, \ldots, \frac{e^{x_{n}}}{\sum{e^{x_{i}}}}\right) \]
• Available in R using mclust::softmax
• Must install mclust

install.packages("mclust")

Draft Simulation

• Loop over all picks in the first rounds (1–32)
• In each iteration (i.e., pick)
  • Remove previously selected players from \(\mathcal{S}\)
  • Estimate selection probabilities \(e^{\lambda_{j}}/\sum_{j' \in \mathcal{S}}{e^{\lambda_{j'}}}\)
  • Use sample() to select one available player
• Will maintain running list of available & selected players

Simulating the 2nd pick

• Suppose Saquon Barkley was picked first overall

players <- names(lambda_hat)
selected_players <- c("Saquon Barkley") 
available_players <- players[!players %in% selected_players] 

probs <- mclust::softmax(x = lambda_hat[available_players])
names(probs) <- available_players
sample(x = available_players, size = 1, prob = probs)

[1] "Sam Darnold"

Simulation

• \(\mathbb{P}(\textrm{Saquon available at 11} \vert \textrm{Saquon not picked top-3})\)
• Helper function determines whether player picked in top-\(k\)