Download file from WWW

HMG Samples and Enhancements

Moderator: Rathinagiri

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Re: Download file from WWW

Post by edk » Sun Mar 12, 2017 9:18 am

hbtip library itself does not support https.
You need to add support for SSL to be able to use https protocol.

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Post by edk » Sun Mar 12, 2017 10:20 am

mol wrote:
Sun Mar 12, 2017 6:06 am
Can you add to your code lines, which changes http AGENT to Mozilla or IE?
Marku add after oSoap:open

Code: Select all

oSoap:setRequestHeader ("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)")
edk.

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Post by edk » Tue Mar 14, 2017 11:58 am

Hi RPC.
RPC wrote:
Sat Mar 11, 2017 6:49 pm
(...) Could you please send a program to download the info in a CSV/text file. I would be very grateful. (...)
Here is a program that retrieves data from the NSEIndia.
NSE.7z
(1.82 MiB) Downloaded 40 times
Unfortunately I can't download the file "CSV" (from the link in the corner), because the link executes the script on the server, I can't run it.
However, I'm thinking that the downloaded html table by my program will allow you to easily obtain the datas.
nse.png
nse.png (19.83 KiB) Viewed 474 times
PS. Especially in the program I used two methods to obtain data to show (teach) how this can be done.
The first method uses OLE, the second uses hbTIP.

Edward.

RPC
Posts: 95
Joined: Fri Feb 10, 2017 4:12 am
Has thanked: 22 times
Been thanked: 3 times

Post by RPC » Tue Mar 14, 2017 5:56 pm

Hi Edward,
Many thanks for the program. You are just great !. :ugeek:
I can download the index information from nseindia website now. :D
The information is in html format. Can you please tell me a way to read this file and import the information in preferably CSV file. :?:
Thanks
RPC

User avatar
mol
Posts: 2853
Joined: Thu Sep 11, 2008 5:31 am
Location: Myszków, Poland
Has thanked: 124 times
Been thanked: 65 times
Contact:

Post by mol » Tue Mar 14, 2017 7:01 pm

You should parse downloaded html code to find <table> <row> <cell> tags.

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Post by edk » Tue Mar 14, 2017 7:55 pm

mol wrote:
Tue Mar 14, 2017 7:01 pm
You should parse downloaded html code to find <table> <row> <cell> tags.
+1

RPC, you don't need read a html file.
All datas are at recv variable.
Just find <div id='csvContentDiv' style='display:none;'> and cut string to </div>
Char : is record's separator (new line in csv format)

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Post by edk » Wed Mar 15, 2017 2:51 pm

RPC
Change in the code line:

Code: Select all

strfile(recv,'Method1.html')
to:

Code: Select all

csv:=MakeCSV(recv)
strfile(csv,STRTRAN(iType,' ','_')+qDate(fromDate)+"-"+qDate(toDate)+".csv")
and add this function:

Code: Select all

Function makeCSV( cString )
Local StartTag:="<div id='csvContentDiv' style='display:none;'>"
Local StopTag:="</div>"
Local PosTag:=AT(StartTag, cString)
Local csv:=""

IF PosTag>0
	csv:=SUBSTR(cString,PosTag+LEN(StartTag))
	csv:=SUBSTR(csv,1,AT(StopTag,csv)-1)
	csv:=STRTRAN(csv,":",CRLF)
	csv:=STRTRAN(csv,'"','')
	csv:=STRTRAN(csv,',',';')
ENDIF
RETURN csv
Edward.

RPC
Posts: 95
Joined: Fri Feb 10, 2017 4:12 am
Has thanked: 22 times
Been thanked: 3 times

Post by RPC » Wed Mar 15, 2017 5:59 pm

Hi Edward
Great ! I got the info in csv file.
Many Thanks. :D
Further I want to know about following two variables :?
Query:="?indexType="+STRTRAN(iType,' ','%20')+"&"+"fromDate="+qDate(fromDate)+"&"+"toDate="+qDate(toDate)
Can this be used to download from any website, say I want to download from another website-www.bseindia.com-similar information. can i use the same query ? or each website has different format of query ?
cGet:="/products/dynaContent/equities/indices/historicalindices.jsp"+Query
the website's address is as follows
https://www.nseindia.com/products/conte ... x_data.htm
why did you put "dynaContent" instead of "Content" and "historicalindices.jsp" instead of "historical_index_data.htm"
Please guide.
Thanks
RPC

edk
Posts: 168
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland
Has thanked: 50 times
Been thanked: 99 times

Post by edk » Wed Mar 15, 2017 8:18 pm

The queries depends of host. Different query is for Google and others for Yahoo, and others for each host.
You need to analyze the code html web pages for which you want to query. Or look for information from the webmaster to send queries.

If you examine the code html https://www.nseindia.com/products/conte ... x_data.htm, you'll see what doing the "get data" button :-)

Apart from that, some hosts may need to define headers. Look for tools for html packet capture and see what the browser sends to the host.

Edward.

Indexes list:
p4.png
p4.png (75.62 KiB) Viewed 307 times
Dates:
p3.png
p3.png (67.67 KiB) Viewed 307 times
"Get Data":
p1.png
p1.png (83.94 KiB) Viewed 307 times
The query:
p2a.png
p2a.png (89.4 KiB) Viewed 307 times
Note the & sign.
In Clipper you can not use it directly because it will be interpreted as a macro. Therefore, it occurs separately: "&" + "fromDate" instead of "&fromDate".

RPC
Posts: 95
Joined: Fri Feb 10, 2017 4:12 am
Has thanked: 22 times
Been thanked: 3 times

Post by RPC » Thu Mar 16, 2017 8:38 pm

Hi Edward
Brilliantly explained ! :D
I have understood quite a lot from the way you have explained it. 8-)
I would like to learn more about this so that I can download from another website.
Could you tell me what should I read on the internet, perhaps a quick tutorial on "querying", which would teach me. :?:
If it is handy, could you please point out to some links. :idea:
Many thanks once again.

Post Reply