Monday, July 20, 2009

Example 7.6: Find Amazon sales rank for a book

In honor of Amazon's official release date for the book, we offer this blog entry.

Both SAS and R can be used to find the Amazon Sales Rank for a book by downloading the desired web page and ferreting out the appropriate line. This code is likely to break if Amazon’s page format is changed (but it worked as of October, 2010). [Note: as of spring 2010 Amazon changed the format for their webpages, and the appropriate text to search for changed from "Amazon.com Sales Rank" to "Amazon Bestsellers Rank". We've updated the blog code with this string. As of October 9, 2010 they added a number of blank lines to the web page, which we also now address.]

In this example, we find the sales rank for our book. Some interesting information about interpreting the rank can be found here or here.

Both SAS and R code below rely on section 1.1.3, ”Reading more complex text files.” Note that in the displayed SAS and R code, the long URL has been broken onto several lines, while it would have to be entered on a single line to run correctly.



In SAS, we assign the URL an internal name (section 1.1.6), then input the file using a data step. We exclude all the lines which don’t contain the sales rank, using the count function (section 1.4.6). We then extract the number using the substr function (section 1.4.3), with the find function (section 1.4.6) employed to locate the number within the line. The last step is to turn the extracted text (which contains a comma) into a numeric variable.

SAS

filename amazon url "http://www.amazon.com/
SAS-Management-Statistical-Analysis-Graphics/
dp/1420070576/ref=sr_1_1?ie=UTF8&s=books
&qid=1242233418&sr=8-1";

data test;
infile amazon truncover;
input @1 line $256.;
if count(line, "Amazon Bestsellers Rank") ne 0;
rankchar = substr(line, find(line, "#")+1,
find(line, "in Books") - find(line, "#") - 2);
rank = input(rankchar, comma9.);
run;

proc print data=test noobs;
var rank;
run;



R

# grab contents of web page
urlcontents <- readLines("http://www.amazon.com/
SAS-Management-Statistical-Analysis-Graphics/
dp/1420070576/ref=sr_1_1?ie=UTF8&s=books
&qid=1242233418&sr=8-1")
# find line with sales rank
linenum <- suppressWarnings(grep("Amazon Bestsellers Rank:",
urlcontents))

newline = linenum + 1 # work around October 2010 blank spaces
while (urlcontents[newline] == "") {
newline = newline + 1
}

# split line into multiple elements
linevals <- strsplit(urlcontents[newline], ' ')[[1]]

# find element with sales rank number
entry <- grep("#", linevals)
# snag that entry
charrank <- linevals[entry]
# kill '#' at start
charrank <- substr(charrank, 2, nchar(charrank))
# remove commas
charrank <- gsub(',','', charrank)
# turn it into a numeric opject
salesrank <- as.numeric(charrank)
cat("salesrank=",salesrank,"\n")


The resulting output (on July 16, 2009) is

SAS

rank

23476


R

salesrank= 23467

7 comments:

Mike said...

Would your R function work if with this command

linenum <- suppressWarnings(grep("Amazon.com Sales Rank:", urlcontents))

we found that the line ended at "Rank:"? Suppose that Amazon put a new line right before the rank integer?

Ken Kleinman said...

Hi Mike--

Thanks for reading!

Short story: yes, that's the right code, and the blog should now show it.

Longer story: the Blogger software preview shows longer lines than are displayed, in some cases.

Ken

Anonymous said...

So why do SAS and R return different sales ranks in your example above?
Never fiddle with the original output...

SAS rank: 23476
R rank: 23467

Ken Kleinman said...

LOL. You got me. And my lousy typing skills. FTR, I was copying from a screenshot.

http://preview.tinyurl.com/ncdzbw

So the better rank was correct...

Nick Horton said...

Note that Amazon recently changed the format of their pages, so the relevant string to search for is "Amazon Bestsellers Rank". I've updated the blog with this change.

Lamar Hunt III said...

Amazon has changed the format again to: "Amazon Best Sellers Rank".

However, I tried to run the SAS program and it was not reading the line with the rank info onto the same line as the one that contains "Amazon Best Sellers Rank". So I just changed the relevant string to "in Books" in the code and it works.

Lamar Hunt III said...
This comment has been removed by the author.