Executing html from the command line?

gregmcc

gregmcc

Honorary Master
Joined
Jun 29, 2006
Messages
29,427
Reaction score
10,492
Location
Somewhere in the world
I've got a linux server where I want to execute some html. The html is a form where I've populated a username and password - then clicking on submit and all works as expected in Chrome/FF. The form is submitted to a external web site and data is returned.

I now want to screen scrape the output so need to run this from command line - tried wget/curl etc but it just returns the source code.

Any ideas on what I can use to "run" this code.

Code: 
<html lang="en">
    <head>
    <title></title>

    </head>
    <body>
        <form method="POST" action='https://www.someorothersite.com/login.fcc'>



                <input name="TARGET" id="TARGET" value='HTTP://www.someorothersite.com/usage/getUsage.do' type="hidden" />
                <input value="xxxxx" id="USERNAME" name="USER" type="text">
                <input value="xxxxx" style="width: 175px;" id="PASSWORD" name="PASSWORD" size="25" maxlength="100" type="password" autocomplete="off" >

        <input alt='Log in' title='Log in' type='image'>
                </form>

    </body>
</html>
 
Last edited:
Think you are approaching the problem from the wrong side, predefining a html form and trying to post that to a site is different from scraping the output

look at wget/curl.
wget --post-data 'USERNAME=&PASSWORD='' 'https://www.someorothersite.com/login.fcc'

The website probably has some safeguards for robots, so you need to set the useragent

e.g.
--user-agent='Mozilla/4.8 [en] (Windows NT 6.0; U)'
 
Last edited:
maybe try a scripting language like Perl or Python?

here is a Perl example ...

Code: 
#!/usr/bin/perl -w
use LWP::UserAgent;
use HTTP::Cookies;
use strict;

my $cookie_jar = HTTP::Cookies->new();
my $ua = LWP::UserAgent->new;
push @{ $ua->requests_redirectable }, 'POST';
$ua->cookie_jar($cookie_jar);
$ua->timeout(10);

my %form = (
	'TARGET' => "HTTP://www.someorothersite.com/usage/getUsage.do",
	'USER' => "xxxx",
	'PASSWORD' => "xxxx"
);

my $response = $ua->post( "https://www.someorothersite.com/login.fcc", \%form );
if ($response->is_success) {
	my $cont = $response->decoded_content;
	print "$cont\n";
} else {
	die $response->status_line;
}
 
I'm busy using tamper data for firefox and will have a look at the headers. Hopefully one of the headers is just missing.
 
You can also use tools like fiddler and Postman to compose your needed request headers and body visually. When you get that working, use cURL or any command line tool to perform the request.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X