Download file from WWW

HMG Samples and Enhancements

Moderator: Rathinagiri

edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

hbtip library itself does not support https.
You need to add support for SSL to be able to use https protocol.
edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

mol wrote: Sun Mar 12, 2017 6:06 am Can you add to your code lines, which changes http AGENT to Mozilla or IE?
Marku add after oSoap:open

Code: Select all

oSoap:setRequestHeader ("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)")
edk.
edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

Hi RPC.
RPC wrote: Sat Mar 11, 2017 6:49 pm (...) Could you please send a program to download the info in a CSV/text file. I would be very grateful. (...)
Here is a program that retrieves data from the NSEIndia.
NSE.7z
(1.82 MiB) Downloaded 302 times
Unfortunately I can't download the file "CSV" (from the link in the corner), because the link executes the script on the server, I can't run it.
However, I'm thinking that the downloaded html table by my program will allow you to easily obtain the datas.
nse.png
nse.png (19.83 KiB) Viewed 6505 times
PS. Especially in the program I used two methods to obtain data to show (teach) how this can be done.
The first method uses OLE, the second uses hbTIP.

Edward.
RPC
Posts: 281
Joined: Fri Feb 10, 2017 4:12 am
DBs Used: DBF

Re: Download file from WWW

Post by RPC »

Hi Edward,
Many thanks for the program. You are just great !. :ugeek:
I can download the index information from nseindia website now. :D
The information is in html format. Can you please tell me a way to read this file and import the information in preferably CSV file. :?:
Thanks
RPC
User avatar
mol
Posts: 3718
Joined: Thu Sep 11, 2008 5:31 am
Location: Myszków, Poland
Contact:

Re: Download file from WWW

Post by mol »

You should parse downloaded html code to find <table> <row> <cell> tags.
edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

mol wrote: Tue Mar 14, 2017 7:01 pm You should parse downloaded html code to find <table> <row> <cell> tags.
+1

RPC, you don't need read a html file.
All datas are at recv variable.
Just find <div id='csvContentDiv' style='display:none;'> and cut string to </div>
Char : is record's separator (new line in csv format)
edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

RPC
Change in the code line:

Code: Select all

strfile(recv,'Method1.html')
to:

Code: Select all

csv:=MakeCSV(recv)
strfile(csv,STRTRAN(iType,' ','_')+qDate(fromDate)+"-"+qDate(toDate)+".csv")
and add this function:

Code: Select all

Function makeCSV( cString )
Local StartTag:="<div id='csvContentDiv' style='display:none;'>"
Local StopTag:="</div>"
Local PosTag:=AT(StartTag, cString)
Local csv:=""

IF PosTag>0
	csv:=SUBSTR(cString,PosTag+LEN(StartTag))
	csv:=SUBSTR(csv,1,AT(StopTag,csv)-1)
	csv:=STRTRAN(csv,":",CRLF)
	csv:=STRTRAN(csv,'"','')
	csv:=STRTRAN(csv,',',';')
ENDIF
RETURN csv
Edward.
RPC
Posts: 281
Joined: Fri Feb 10, 2017 4:12 am
DBs Used: DBF

Re: Download file from WWW

Post by RPC »

Hi Edward
Great ! I got the info in csv file.
Many Thanks. :D
Further I want to know about following two variables :?
Query:="?indexType="+STRTRAN(iType,' ','%20')+"&"+"fromDate="+qDate(fromDate)+"&"+"toDate="+qDate(toDate)
Can this be used to download from any website, say I want to download from another website-www.bseindia.com-similar information. can i use the same query ? or each website has different format of query ?
cGet:="/products/dynaContent/equities/indices/historicalindices.jsp"+Query
the website's address is as follows
https://www.nseindia.com/products/conte ... x_data.htm
why did you put "dynaContent" instead of "Content" and "historicalindices.jsp" instead of "historical_index_data.htm"
Please guide.
Thanks
RPC
edk
Posts: 909
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Download file from WWW

Post by edk »

The queries depends of host. Different query is for Google and others for Yahoo, and others for each host.
You need to analyze the code html web pages for which you want to query. Or look for information from the webmaster to send queries.

If you examine the code html https://www.nseindia.com/products/conte ... x_data.htm, you'll see what doing the "get data" button :-)

Apart from that, some hosts may need to define headers. Look for tools for html packet capture and see what the browser sends to the host.

Edward.

Indexes list:
p4.png
p4.png (75.62 KiB) Viewed 5329 times
Dates:
p3.png
p3.png (67.67 KiB) Viewed 5329 times
"Get Data":
p1.png
p1.png (83.94 KiB) Viewed 5329 times
The query:
p2a.png
p2a.png (89.4 KiB) Viewed 5329 times
Note the & sign.
In Clipper you can not use it directly because it will be interpreted as a macro. Therefore, it occurs separately: "&" + "fromDate" instead of "&fromDate".
RPC
Posts: 281
Joined: Fri Feb 10, 2017 4:12 am
DBs Used: DBF

Re: Download file from WWW

Post by RPC »

Hi Edward
Brilliantly explained ! :D
I have understood quite a lot from the way you have explained it. 8-)
I would like to learn more about this so that I can download from another website.
Could you tell me what should I read on the internet, perhaps a quick tutorial on "querying", which would teach me. :?:
If it is handy, could you please point out to some links. :idea:
Many thanks once again.
Post Reply