Using Yahoo! Search Service

Yahoo!, just like many other websites, is allowing developers to access its content through Web Services. In this blog, I show you how to create a simple JSP page to access Yahoo's Search service.

Web services are broadly of two types - SOAP based and RESTful. Yahoo provides RESTful service for its search. Accessing any REST (Representational State Transfer) web service is as simple as constructing a URL and invoking it. I am using URL class of java.net package to make a request to Yahoo's RESTful web service.

Search service returns an XML document (XML Schema) that contains Result elements. So, my program reads all Result elements and displays value of Title element using value of URL element as hyperlink.

To know more about how Yahoo Search service works, see Yahoo Web Search.

Steps to use Yahoo! Search Service

Take the following steps to use Yahoo's search service using Yahoo's demo application id - YahooDemo. You can as well obtain a unique application id for yourself. Obtain Application ID.

Create a new web application using any IDE of your choice (I use NetBeans 6.0).
Web container could be anything of your choice. I am using Tomcat.

Create a .jsp file to take input from user using HTML form and search Yahoo's service for results.

<%@page contentType="text/html"%>
<%@page pageEncoding="UTF-8"%>
<%@page import="java.io.*,java.net.URL,javax.xml.xpath.*,org.w3c.dom.*,org.xml.sax.InputSource,java.net.URLEncoder"%>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
   "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Yahoo Search</title>
    </head>
    <body>
    <h1>Yahoo Search</h1>
    
    <form action="index.jsp">
        Enter search string : <input type=text name=pattern value="${param.pattern}">
        <input type=submit value="Search"/>
    </form>
    
    <%
     String pattern = request.getParameter("pattern");
     if ( pattern == null) return;
     
     String appid = "";
     // use URLEncoder to encode search pattern - it is required.
     String yrequest = "http://search.yahooapis.com/WebSearchService/V1/webSearch?appid=YahooDemo&query=" +  URLEncoder.encode(pattern,"UTF-8") + "&results=5";
     try {
          URL url = new URL(yrequest);
          InputStream in = url.openStream();
                      
          XPathFactory x = XPathFactory.newInstance();
          XPath path = x.newXPath();
          // take data that comes from service invocation
          InputSource source = new InputSource(in);
          NodeList nl = (NodeList) path.evaluate("//urn:yahoo:srch:Result", source, XPathConstants.NODESET);
          
          if ( nl.getLength() == 0 )
                 out.println("<h3>Sorry! No Match Found!</h3>");
          else
             for ( int i = 0 ; i < nl.getLength(); i ++) {	
                Element e = (Element) nl.item(i);
                out.println("<a href="  + e.getChildNodes().item(2).getTextContent() + ">" +
                              e.getChildNodes().item(0).getTextContent() + "</a><p/>");
             } // end of for
                     
     } 
     catch (Exception e) {
         out.println("<h3>" + e.getMessage() + "</h3>");
     }
    
%>
    
</body>
</html>

Important Points About .JSP

JSP starts with a HTML Form to take search string from user.
If JSP is called directly (not from itself - postback) then it does NOT call service as there is no pattern to search for.
Expression Language - EL is used to get request parameter pattern (value="${param.pattern}") and display its value in textbox to retain the value entered by the user even after postback.
If called from itself, it constructs URL to call Yahoo's RESTful Search Web Service as mentioned by Yahoo's documentation.
You must encode search pattern using URLEncoder.encode() to ensure pattern is encoded.
Create a URL object and take input stream to receive data that comes from URL.
The output is an XML document. You can see its schema here.
As I want to take only Result element from the document, I am using XPATH.
The namespace urn:yahoo.srch is associated with all elements, so we have to use that namespace as the prefix for elements. I am using 0th element, which is Title and 2nd element, which is URL.
If no element is found then display error message.
If search yields some values then display title of each result using URL of the result.
If you enter RESTFul Web Service as the search string, the following output shows the result.

So, that's all you have to do to access Yahoo's Search Service. I have used URL class to call the service and XPATH to process the XML document returned by service.