I am trying to scrape some content form a website but you must be logged in order to view specific content. I want make a login using user id & password and keep session cookies on: m.amway.com i tried using Jsoup.... however after using the code below i realize that Jsoup cannot read javascript which is what the website is based on.... Does anyone have a method i could use to login, keep session cookie, and scrape content, using something other than Jsoup? Thanks in advance.
public String Jlogin(String User, String Pass) throws Exception{
String title = "didnt work";
Response logRes = Jsoup.connect(AmwayURL)
.data("userid", User)
.data("userpswd", Pass)
.method(Method.POST)
.execute();
// get all cookies
Map<String, String> cookies = logRes.cookies();
Document doc1 = logRes.parse();
String sessionId = logRes.cookie("JSESSIONID");
Document doc2 = Jsoup
.connect("https://m.amway.com/business/volume/pvbv/inquiry.ashx")
.cookie("jsessionid", sessionId).get();
System.out.println(doc2);
title = doc2.toString() + "................." + sessionId;
return title;
}
You can use a much larger API called HttpClient.
has the following classes: - HttpGet - HttpPost - HttpEntity - HttpResponse
HttpResponse reads Javascript from any page, as follows: EntityUtils.toString(HttpResponse.getEntity());
for more details on how to use the API, check this link (Extremly helps): http://www.codeblues.in/blog/?p=5