Consulting

Results 1 to 7 of 7

Thread: How to do web scraping from java based website

  1. #1
    VBAX Regular
    Joined
    Dec 2014
    Posts
    67
    Location

    How to do web scraping from java based website

    Hello all,


    I need to pull a report from a java based website into excel. The website asks for start date and end date and generates the report based on the dates supplied, but the problem is, the source of the page don't show any details wherein i can use the details to pull the report through VBA. Is there any way we can do this?

    Any help on this will be great.



    sorry, I'm not able to attach source file here...


    Thanks!


    JayeshG

  2. #2
    Moderator VBAX Regular
    Joined
    May 2010
    Location
    Dayton, OH
    Posts
    15
    Location
    Fill in all the information, but before hitting Submit press F12 to bring up the web console. In your web console (depending upon the browser, the names and icons are different), look for the network tab. Then press enter. If Java is sending a request to a server, you should be able to capture it here. You should also be able to see what the response is. You need to figure out if it's using REST or SOAP. Once you know that, you can attempt making the server request and capturing the response in VBA.

  3. #3
    VBAX Regular
    Joined
    Dec 2014
    Posts
    67
    Location
    Hi jpo645,

    Thanks for your reply. Let me try this out. Will get back on this.

  4. #4
    VBAX Regular
    Joined
    Dec 2014
    Posts
    67
    Location
    Hi Jpo,

    It uses SOAP, as its using the POST method. So, now how to go about this. What info I'll be needing to get the report?

  5. #5
    VBAX Regular
    Joined
    Dec 2014
    Posts
    67
    Location
    Hi,

    Can I get a sample code for this? Or the code works the same for what is used for normal site scraping?

    Thanks!

  6. #6
    Moderator VBAX Regular
    Joined
    May 2010
    Location
    Dayton, OH
    Posts
    15
    Location
    Well this part is more art than science. Hopefully, in the post request, you can see the breakdown of the request to know what you should send in. Usually, the values are in the URL or in the payload. Alternatively, if you are familiar with the system you are requesting information from, there may be an API that explains exactly what's required in the request. If it's SOAP, that means the response is going to come back as XML. This is good news because you can scrape the XML response as you would HTML (indeed, arguably you can scrape it more easily).

    The problem is, SOAP requests are a little old. Here are a few conversations about it, i've found:

    http://community.spiceworks.com/topi...a-vba-in-excel
    http://scn.sap.com/community/epm/blo...ustom-vba-code
    http://www.soapuser.com/client4.html

    But outside of that, I couldn't give you any sample code.

  7. #7
    VBAX Regular
    Joined
    Dec 2014
    Posts
    67
    Location
    Okay kool. Thanks Jpo. Will check it out!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •