Skip to main content

Why does Kaggle use Log-loss?

Finding the Best Book Quotes: Power Ranking Goodreads

My friend Jordan Ellenberg wrote an article for the Wall Street Journal recently describing a metric to roughly measure the least read books, which he calls the Hawking Index (HI). As he describes it, take the page numbers of a book's five top highlights, average them, and divide by the number of pages in the whole book.

On a discussion thread on Facebook, this led to a proposal from me for measuring the general quality of a quote. Assume a user has some level of discrimination \(D\); the higher the value of \(D\), the more likely they are to quote a passage. Now assume each passage has some measure of quality \(Q\); the higher the value of \(Q\), the more likely a passage is to be quoted. Let's try a classic Bradley-Terry-Luce model - if a user with discrimination \(D\) quotes at least one passage from a particular work, the probability \(p\) that they'll quote a given passage with quality \(Q\) from that same work is roughly \[ p = \frac{Q}{Q+D}.\]
Nice, but where can we find data to actually test this theory? It turns out an excellent source is Goodreads. Users are numbered sequentially starting with Goodreads founder Otis Chandler's ID of 1; a user's associated quote page has a simple URL structure. Furthermore, every user's quote list has an associated RSS/Atom URL that provides an easy to parse XML file.

The steps:
  1. Write a simple Go scraper to uses a goroutine to quickly and sequentially get user quote pages, grab the count of quotes shown on that page (truncated at 30), then grab the RSS/Atom URL. This is able to process about 2500/users minutes over a slow internet connection.
  2. Write a simple Ruby program to fetch quote XML files for every user that has at least one quote.
  3. Write a simple Ruby program to fetch the author URL and work URL for every quote cited by at least one user.
  4. Load all this data into a PostgreSQL database.
  5. Write a simple R program to pull the data from the database and fit our model. We assign a value for the outcome "quoted" of 1 if the user quoted a passage; we assign "quoted" the value 0 if the user quote a passage from that work, but not that particular passage. Both user "discrimination" and quote "quality" are treated as factors and modeled as random effects; the R package "lmer" was used to fit the model.
If there is interest, I can make all of this code publicly available through my GitHub account. None of it is proprietary.

As a test I grabbed the data from 333760 users, 10724 had at least one quote, there were 83160 total quotes and 32621 unique quotes. The model took about 3.5 minutes to fit using an optimized BLAS library and 4 hyperthreaded CPU cores.

The top 10 quotes on Goodreads, ranked by quality:
  1. “No one can make you feel inferior without your consent.”
    Eleanor Roosevelt, This is My Story
  2. “Outside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.”
    Groucho Marx, The Essential Groucho: Writings For By And About Groucho Marx
  3. “Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.”
    H. Jackson Brown Jr., P.S. I Love You
  4. “I am so clever that sometimes I don't understand a single word of what I am saying.”
    Oscar Wilde, The Happy Prince and Other Stories
  5. “Insanity is doing the same thing, over and over again, but expecting different results.”
    Narcotics Anonymous, Narcotics Anonymous
  6. “He has achieved success who has lived well, laughed often, and loved much;Who has enjoyed the trust of pure women, the respect of intelligent men and the love of little children;Who has filled his niche and accomplished his task;Who has never lacked appreciation of Earth's beauty or failed to express it;Who has left the world better than he found it,Whether an improved poppy, a perfect poem, or a rescued soul;Who has always looked for the best in others and given them the best he had;Whose life was an inspiration;Whose memory a benediction.”
    Bessie Anderson Stanley, More Heart Throbs Volume Two in Prose and Verse Dear to the American People And by them contributed as a Supplement to the original $10,000 Prize Book HEART THROBS
  7. “The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.”
    Terry Pratchett, Diggers
  8. “Love all, trust a few, do wrong to none.”
    William Shakespeare, All's Well That Ends Well
  9. “The Paradoxical Commandments
    People are illogical, unreasonable, and self-centered.
    Love them anyway.
    If you do good, people will accuse you of selfish ulterior motives.
    Do good anyway.
    If you are successful, you will win false friends and true enemies.
    Succeed anyway.
    The good you do today will be forgotten tomorrow.
    Do good anyway.
    Honesty and frankness make you vulnerable.
    Be honest and frank anyway.
    The biggest men and women with the biggest ideas can be shot down by the smallest men and women with the smallest minds.
    Think big anyway.
    People favor underdogs but follow only top dogs.
    Fight for a few underdogs anyway.
    What you spend years building may be destroyed overnight.
    Build anyway.
    People really need help but may attack you if you do help them.
    Help people anyway.
    Give the world the best you have and you'll get kicked in the teeth.
    Give the world the best you have anyway.”
    Kent M. Keith, The Silent Revolution: Dynamic Leadership in the Student Council
  10. “She is too fond of books, and it has turned her brain.”
    Louisa May Alcott, Work: A Story of Experience

Comments

  1. as you are using Pairwise comparison, i think the order of comparison would effect the result. there is a famous Condorcet Paradox..

    it would be great, if you have the code published on github :)

    ReplyDelete
  2. Awesome. Also, post a psql dump of the DB to github so we don't have to abuse goodreads' servers.

    ReplyDelete
  3. Did you mean "the higher the value of D, the _less_ likely they are to quote a passage"? Because the formula gives p = 1 for D=0 and p->0 for D-> infinity.

    ReplyDelete

Post a Comment

Popular posts from this blog

A Bayes' Solution to Monty Hall

For any problem involving conditional probabilities one of your greatest allies is Bayes' Theorem. Bayes' Theorem says that for two events A and B, the probability of A given B is related to the probability of B given A in a specific way.

Standard notation:

probability of A given B is written \( \Pr(A \mid B) \)
probability of B is written \( \Pr(B) \)

Bayes' Theorem:

Using the notation above, Bayes' Theorem can be written: \[ \Pr(A \mid B) = \frac{\Pr(B \mid A)\times \Pr(A)}{\Pr(B)} \]Let's apply Bayes' Theorem to the Monty Hall problem. If you recall, we're told that behind three doors there are two goats and one car, all randomly placed. We initially choose a door, and then Monty, who knows what's behind the doors, always shows us a goat behind one of the remaining doors. He can always do this as there are two goats; if we chose the car initially, Monty picks one of the two doors with a goat behind it at random.

Assume we pick Door 1 and then Monty sho…

What's the Value of a Win?

In a previous entry I demonstrated one simple way to estimate an exponent for the Pythagorean win expectation. Another nice consequence of a Pythagorean win expectation formula is that it also makes it simple to estimate the run value of a win in baseball, the point value of a win in basketball, the goal value of a win in hockey etc.

Let our Pythagorean win expectation formula be \[ w=\frac{P^e}{P^e+1},\] where \(w\) is the win fraction expectation, \(P\) is runs/allowed (or similar) and \(e\) is the Pythagorean exponent. How do we get an estimate for the run value of a win? The expected number of games won in a season with \(g\) games is \[W = g\cdot w = g\cdot \frac{P^e}{P^e+1},\] so for one estimate we only need to compute the value of the partial derivative \(\frac{\partial W}{\partial P}\) at \(P=1\). Note that \[ W = g\left( 1-\frac{1}{P^e+1}\right), \] and so \[ \frac{\partial W}{\partial P} = g\frac{eP^{e-1}}{(P^e+1)^2}\] and it follows \[ \frac{\partial W}{\partial P}(P=1) = …

Behind the Speadsheet

In the book "The Only Rule Is It Has to Work: Our Wild Experiment Building a New Kind of Baseball Team", Ben Lindbergh and Sam Miller recount a grand adventure to take command of an independent league baseball team, with the vision of trying every idea, sane or crazy, in an attempt to achieve a winning edge. Five infielders, four outfielders, defensive shifts, optimizing lineups - everything.

It was really an impossible task. Professional sports at every level are filled with highly accomplished and competitive athletes, with real lives and real egos. Now imagine walking in one day and suddenly trying to convince them that they should be doing things differently. Who do you think you are?

I was one of the analysts who helped Ben and Sam in this quest, and I wanted to write some thoughts down from my own perspective, not as one of the main characters, but as someone more behind the scenes. These are some very short initial thoughts only, but I'd like to followup with some…