Skip to main content

Probability and Cumulative Dice Sums

Finding the Best Book Quotes: Power Ranking Goodreads

My friend Jordan Ellenberg wrote an article for the Wall Street Journal recently describing a metric to roughly measure the least read books, which he calls the Hawking Index (HI). As he describes it, take the page numbers of a book's five top highlights, average them, and divide by the number of pages in the whole book.

On a discussion thread on Facebook, this led to a proposal from me for measuring the general quality of a quote. Assume a user has some level of discrimination \(D\); the higher the value of \(D\), the more likely they are to quote a passage. Now assume each passage has some measure of quality \(Q\); the higher the value of \(Q\), the more likely a passage is to be quoted. Let's try a classic Bradley-Terry-Luce model - if a user with discrimination \(D\) quotes at least one passage from a particular work, the probability \(p\) that they'll quote a given passage with quality \(Q\) from that same work is roughly \[ p = \frac{Q}{Q+D}.\]
Nice, but where can we find data to actually test this theory? It turns out an excellent source is Goodreads. Users are numbered sequentially starting with Goodreads founder Otis Chandler's ID of 1; a user's associated quote page has a simple URL structure. Furthermore, every user's quote list has an associated RSS/Atom URL that provides an easy to parse XML file.

The steps:
  1. Write a simple Go scraper to uses a goroutine to quickly and sequentially get user quote pages, grab the count of quotes shown on that page (truncated at 30), then grab the RSS/Atom URL. This is able to process about 2500/users minutes over a slow internet connection.
  2. Write a simple Ruby program to fetch quote XML files for every user that has at least one quote.
  3. Write a simple Ruby program to fetch the author URL and work URL for every quote cited by at least one user.
  4. Load all this data into a PostgreSQL database.
  5. Write a simple R program to pull the data from the database and fit our model. We assign a value for the outcome "quoted" of 1 if the user quoted a passage; we assign "quoted" the value 0 if the user quote a passage from that work, but not that particular passage. Both user "discrimination" and quote "quality" are treated as factors and modeled as random effects; the R package "lmer" was used to fit the model.
If there is interest, I can make all of this code publicly available through my GitHub account. None of it is proprietary.

As a test I grabbed the data from 333760 users, 10724 had at least one quote, there were 83160 total quotes and 32621 unique quotes. The model took about 3.5 minutes to fit using an optimized BLAS library and 4 hyperthreaded CPU cores.

The top 10 quotes on Goodreads, ranked by quality:
  1. “No one can make you feel inferior without your consent.”
    Eleanor Roosevelt, This is My Story
  2. “Outside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.”
    Groucho Marx, The Essential Groucho: Writings For By And About Groucho Marx
  3. “Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.”
    H. Jackson Brown Jr., P.S. I Love You
  4. “I am so clever that sometimes I don't understand a single word of what I am saying.”
    Oscar Wilde, The Happy Prince and Other Stories
  5. “Insanity is doing the same thing, over and over again, but expecting different results.”
    Narcotics Anonymous, Narcotics Anonymous
  6. “He has achieved success who has lived well, laughed often, and loved much;Who has enjoyed the trust of pure women, the respect of intelligent men and the love of little children;Who has filled his niche and accomplished his task;Who has never lacked appreciation of Earth's beauty or failed to express it;Who has left the world better than he found it,Whether an improved poppy, a perfect poem, or a rescued soul;Who has always looked for the best in others and given them the best he had;Whose life was an inspiration;Whose memory a benediction.”
    Bessie Anderson Stanley, More Heart Throbs Volume Two in Prose and Verse Dear to the American People And by them contributed as a Supplement to the original $10,000 Prize Book HEART THROBS
  7. “The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.”
    Terry Pratchett, Diggers
  8. “Love all, trust a few, do wrong to none.”
    William Shakespeare, All's Well That Ends Well
  9. “The Paradoxical Commandments
    People are illogical, unreasonable, and self-centered.
    Love them anyway.
    If you do good, people will accuse you of selfish ulterior motives.
    Do good anyway.
    If you are successful, you will win false friends and true enemies.
    Succeed anyway.
    The good you do today will be forgotten tomorrow.
    Do good anyway.
    Honesty and frankness make you vulnerable.
    Be honest and frank anyway.
    The biggest men and women with the biggest ideas can be shot down by the smallest men and women with the smallest minds.
    Think big anyway.
    People favor underdogs but follow only top dogs.
    Fight for a few underdogs anyway.
    What you spend years building may be destroyed overnight.
    Build anyway.
    People really need help but may attack you if you do help them.
    Help people anyway.
    Give the world the best you have and you'll get kicked in the teeth.
    Give the world the best you have anyway.”
    Kent M. Keith, The Silent Revolution: Dynamic Leadership in the Student Council
  10. “She is too fond of books, and it has turned her brain.”
    Louisa May Alcott, Work: A Story of Experience

Comments

  1. as you are using Pairwise comparison, i think the order of comparison would effect the result. there is a famous Condorcet Paradox..

    it would be great, if you have the code published on github :)

    ReplyDelete
  2. Awesome. Also, post a psql dump of the DB to github so we don't have to abuse goodreads' servers.

    ReplyDelete
  3. Did you mean "the higher the value of D, the _less_ likely they are to quote a passage"? Because the formula gives p = 1 for D=0 and p->0 for D-> infinity.

    ReplyDelete

Post a Comment

Popular posts from this blog

Notes on Setting up a Titan V under Ubuntu 17.04

I recently purchased a Titan V GPU to use for machine and deep learning, and in the process of installing the latest Nvidia driver's hosed my Ubuntu 16.04 install. I was overdue for a fresh install of Linux, anyway, so I decided to upgrade some of my drives at the same time. Here are some of my notes for the process I went through to get the Titan V working perfectly with TensorFlow 1.5 under Ubuntu 17.04. Old install: Ubuntu 16.04 EVGA GeForce GTX Titan SuperClocked 6GB 2TB Seagate NAS HDD + additional drives New install: Ubuntu 17.04 Titan V 12GB / partition on a 250GB Samsung 840 Pro SSD (had an extra around) /home partition on a new 1TB Crucial MX500 SSD New WD Blue 4TB HDD + additional drives You'll need to install Linux in legacy mode, not UEFI, in order to use Nvidia's proprietary drivers for the Titan V. Note that Linux will cheerfully boot in UEFI mode, but will not load any proprietary drivers (including Nvidia's). You'll need proprietary d

Mixed Models in R - Bigger, Faster, Stronger

When you start doing more advanced sports analytics you'll eventually starting working with what are known as hierarchical, nested or mixed effects models . These are models that contain both fixed and random effects . There are multiple ways of defining fixed vs random random effects , but one way I find particularly useful is that random effects are being "predicted" rather than "estimated", and this in turn involves some "shrinkage" towards the mean. Here's some R code for NCAA ice hockey power rankings using a nested Poisson model (which can be found in my hockey GitHub repository ): model <- gs ~ year+field+d_div+o_div+game_length+(1|offense)+(1|defense)+(1|game_id) fit <- glmer(model, data=g, verbose=TRUE, family=poisson(link=log) ) The fixed effects are year , field (home/away/neutral), d_div (NCAA division of the defense), o_div (NCAA division of the offense) and game_length (number of overtime

A Bayes' Solution to Monty Hall

For any problem involving conditional probabilities one of your greatest allies is Bayes' Theorem . Bayes' Theorem says that for two events A and B, the probability of A given B is related to the probability of B given A in a specific way. Standard notation: probability of A given B is written \( \Pr(A \mid B) \) probability of B is written \( \Pr(B) \) Bayes' Theorem: Using the notation above, Bayes' Theorem can be written:  \[ \Pr(A \mid B) = \frac{\Pr(B \mid A)\times \Pr(A)}{\Pr(B)} \] Let's apply Bayes' Theorem to the Monty Hall problem . If you recall, we're told that behind three doors there are two goats and one car, all randomly placed. We initially choose a door, and then Monty, who knows what's behind the doors, always shows us a goat behind one of the remaining doors. He can always do this as there are two goats; if we chose the car initially, Monty picks one of the two doors with a goat behind it at random. Assume we pick Door 1 an