Diving deep into Amazon Web Services

From storage to payment, Amazon is dangling an array of low-cost services – but will customers bite?

Wading into Web Information Services

Amazon's Web Information Services are essentially query interfaces into extensive databases generated by a mixture of Web crawlers and Web traffic monitors. Data-mining organization can tap into the crawler-produced data to sift through information that is as wide-ranging as the Web itself. The utility of Web traffic data is self-evident to any company or individual interested in user visitation trends to their sites -- as well as to related or competing sites.

AlexaWeb Search: Amazon's Alexa Web Search is the result of partnering between Amazon and Alexa, and it lets you query the information gathered by Alexa's Web crawler bots. The quantity of information available is difficult to gauge; Alexa has been crawling the Web for over a decade, and the Internet is in nonstop growth. Alexa's site says that, while its bots are working constantly, it takes about two months for a complete cycle through the Internet.

When Alexa adds a new Web site document to its database, it indexes about 50 attributes associated with that document. Attributes include the document's language, its Open Document Category, various parsed components of the URL, geographic location of the hosting server, and more. Also available is the document's text, the first 20KB of which is text-indexed. All this is available for searching.

Naturally, searches on such a large database can take time. The Alexa Web Search service is architected so that when you issue a search, the service returns a request ID. You use this ID to track the status of your search's progress. When the search is complete, results are stored in a (possibly gigantic) text file. The text file can be downloaded and "mined" locally.

AlexaWeb Information Service (AWIS): The Alexa Web Information Service lets you dip into traffic data gathered by various Alexa tools deployed about the Internet. You can query information for a specific URL, such as site contact information, traffic statistics (going back five years), and more. You can also discover how many links are on a given page, how many URLs are embedded in JavaScript, or the more interesting statistic of how may other sites link to the target ("inward-pointing" links). You can also use AWIS to fetch a thumbnail image of a Web page, useful for displaying pop-ups in response to a cursor hovering over links.

The accuracy of Alexa's data is unclear. The Alexa Web site states that the "traffic data are based on the set of toolbars that use Alexa's data, which may not be representative of the global internet population." Meanwhile, an Amazon Web services representative informed me that Amazon "aggregate[s] data from multiple sources to give you a better indication of Web site popularity." In any case, the ability to scour the text content of whole swaths of the Internet makes the Alexa Web service a profitable vein for Web data spelunkers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about ACTFedoraGigabyteLinuxMySQLOracleUbuntuVIA

Show Comments