javahtmlhtml-content-extractionweb-content

reading web page source code in java Differs from the orginal webpage source code


I am trying to implement program to read webpage source code and save it in text file then do some operations in it but the problem when I read web page source code , there is difference between the orginal web page source code and the output of java program web page source code.

my program :

        String inputLine;
        URL link = new URL("http://www.ammanu.edu.jo/English/Articles/newsArticle.aspx?id=2935");
        BufferedReader in = new BufferedReader( new InputStreamReader(link.openStream(),"UTF-8"));

        while ((inputLine = in.readLine()) != null){
                System.out.println(inputLine);
        }
        in.close();

and my problem in this line of source code

orginal webpage source code at line 1156 :

<img id="img" src="http://www.ammanu.edu.jo/AdminImages/20652935/_DSC0246.jpg" style="height:310px;width:400px;border-width:0px;display:block;float:left; padding-right:5px;" />

Actual output of java program :

<img id="img" src="http://www.ammanu.edu.jo/AdminImages/20652935/_DSC0246.jpg" height="310" width="400" border="0" style="display:block;float:left; padding-right:5px;" />

any suggested solution for this problem ?


Solution

  • You have to set the user the User-Agent:

    con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
    

    Full code:

        String inputLine;
        URL link = new URL("http://www.ammanu.edu.jo/English/Articles/newsArticle.aspx?id=2935");
        URLConnection con = link.openConnection();
        con.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");
    
        BufferedReader in = new BufferedReader( new InputStreamReader(con.getInputStream(),"UTF-8"));
    
        while ((inputLine = in.readLine()) != null){
            System.out.println(inputLine);
        }
        in.close();
    

    Result

    <img id="img" src="http://www.ammanu.edu.jo/AdminImages/20652935/_DSC0246.jpg" style="height:310px;width:400px;border-width:0px;display:block;float:left; padding-right:5px;" />