Search 5 Million Print Books in Microseconds

23 Comments

Damn that John Durant. Just as I was beginning to get a handle on all my tasks he comes up with something that’s just too fun not to play with.

It’s Google’s search engine for the millions of books they’ve digitized over the years. John explains.

Google has been digitizing books for years now, and a team of researchers just published some amazing findings based on the corpus so far — over 5 million books. That’s 4% of all books. Ever. Printed. You can read about the paper here. The full paper is easy to read, and incredible (free with registration). Google also released this tool to view the frequency of any n-gram since 1720. (An n-gram is a set of characters separated by a space. 1-gram = technology, 1942, R2D2; 2-gram = yellow fever, John Wayne. And so on.) I wish they had tools like this when I was writing history papers back in college.

Careful, ’cause it’s addicting. But you can essentially conduct your own mini studies for associations as much as you like. Here’s my stab: whole grains, saturated fat and obestity epidemic. You can click on it to see the larger version or, simply click here.

Whole Grains Saturated Fat and the Obesity Epidemic
Whole Grains, Saturated Fat and the Obesity Epidemic

What I find particularly interesting is how both mentions of whole grains and saturated fat track in lockstep with one-another beginning in about 1970, and then you have mentions of the obesity epidemic taking off in 2000, 30 years later. Could that have anything to do with that 10 pounds per year gain (= 30 pounds and you’re obese)? I don’t know. Of course, this is only valuable for getting hints of what you might want to study further.

What would be interesting is to plug those same terms into a tool that does the same thing for all published research. Whaddya want to bet it looks about the same?

Alright, one more, for fun. How about Atkins diet vs. low fat?

Atkins diet vs low fat
Atkins diet vs low fat

Interesting how the Atkins diet came out right at the dip in low fat mentions. Not sure how to hypothesize an interpretation on that.

Alright, go have fun. If you find interesting associations, drop the link in comments here, at John’s or both.

(Can’t wait to see what Ned Kock come up with).

Share6
Tweet14
Pin
20 Shares

The Fabulous Gumroad Store

PATREON SUPPORT

My own on-the-scene expat photos, stories, podcasts, and video adventures, currently from the island of Phuket, Thailand. Plus, anything else I write or video about.

Elixa Probiotic is a British biotech manufacturer in Oxford, UK. U.S. Demand is now so high they've established distribution centers in Illinois, Nevada, and New Jersey.

Still, sell-outs happen regularly, so order now to avoid a waiting list.

Reader Interactions

Comments

  1. julianne December 17, 2010 at 16:13

    This is fun – thanks for pointing it out.

    Just did hyperactivity and ADHD, then Vitamin D and codliver oil

    Reply
    • Juan December 19, 2010 at 07:40

      Good point, Steve. There are copyright issues involved with anything newer unless they have permission from the copyright owner. As you may know, before a book can become public domain, the copyright owner –usually the author– has to be dead for 70 years. Google Books does not always include entire books; sometimes it’s just snippets of the volume or a virtual index card, and they can thus present a book and not infringe copyright laws.

  9. julianne December 18, 2010 at 00:05

    Vitamin C, Vitamin D, Vitamin E
    http://ngrams.googlelabs.com/graph?content=vitamin+C%2Cvitamin+D%2CVitamin+E&year_start=1900&year_end=2008&corpus=0&smoothing=3

    Then pick an early year to look at and see what has been writen about each – fascinating eg vit c http://www.google.com/search?q=%22vitamin%20C%22&tbs=bks:1,cdr:1,cd_min:1938,cd_max:1942&lr=lang_en
    Life article
    “How to hold on to youth and retard old age” …

    Reply
  12. eco December 19, 2010 at 02:54

    Will be even better if you put actual obesity statistics superimposed over the mentions of “saturated fat” and “whole grains”… 😉

    Reply

Leave a Reply to eco Cancel reply

Your email address will not be published. Required fields are marked *