Ticket #2190 (new enhancement)

Opened 6 years ago

Last modified 6 years ago

Add LFTP-like HTTP filesystem to VFS

Reported by: harvie Owned by:
Priority: trivial Milestone: Future Releases
Component: mc-vfs Version: master
Keywords: HTTP LFTP VFS mirror recursive Cc: gotar@…
Blocked By: Blocking:
Branch state: Votes for changeset:


Hello, LFTP client have nice feature which i like to see in MC. LFTP is capable of transfering files (or directories) over almost any protocol... especialy i found usefull feature which can browse, download or recursively mirror links on some HTTP URL. In case of apache filelisting (or similar sites) this means browsing and mirroring directory tree in same way as read-only ftp.

I think such VFS can't be too hard to implement (just parse all links on page, handle relative/absolute links well and add make sure it will work well when recursively downloading whole directories).

another good feature is optional continuing of interrupted downloads...

maybe you can use LFTP binary for this (not sure if it comply with your rules).

here is example of how it looks in lftp for http://temp.harvie.cz/ listing:

[11:33:12] 0 ;) harvie@harvie-ntb ~ $ lftp http://temp.harvie.cz/
cd uspělo, cwd=/          
lftp temp.harvie.cz:/> ls
-rw-r--r--  --  icons/blank.png            
-rw-r--r--  --  icons/folder.png
drwxr-xr-x            -  2009-04-29 23:17  autedosk-acedomya
drwxr-xr-x            -  2008-03-04 22:04  deb
drwxr-xr-x            -  2007-10-21 22:10  doh
drwxr-xr-x            -  2008-02-26 17:17  firmwhore
-rw-r--r--         8.3K  2008-08-02 00:33  skillz.html
-rw-r--r--          313  2008-08-02 00:33  skoda.txt
-rw-r--r--         1.2M  2009-04-14 21:25  tablo.jpg
-rw-r--r--         4.0M  2009-04-19 19:58  tablo.svg
-rw-r--r--         6.8M  2009-04-19 20:40  tablo.zip
-rw-r--r--         4.1M  2009-05-12 22:46  tablo2.svg
-rw-r--r--          435  2010-03-24 11:10  zaver.txt
lftp temp.harvie.cz:/> exit

Change History

comment:1 Changed 6 years ago by andrew_b

  • Priority changed from major to trivial
  • Version changed from version not selected to master
  • Milestone changed from 4.7 to Future Releases

comment:2 Changed 6 years ago by gotar

  • Cc gotar@… added

comment:3 follow-up: ↓ 4 Changed 6 years ago by harvie

BTW LFTP It self provides great VFS very similar to one in mc... and there is also lftp filesystem for mounting remote servers based on fuse and using original lftp binary.

i think it can be really good idea to (re-)implement whole VFS in mc using LFTP. this can add support for multiple protocols at once... if you don't like 3rd party binaries you can fork lftp for yourself. only caveat is that right now lftp can't handle SMB which is one of really usefull features in mc when you don't have any samba client with GUI installed on your box...

see man lftp:

lftp is a file transfer program that allows sophisticated ftp, http and other connections to other hosts. If site is specified then lftp will connect to that site otherwise a connection has to be established with the open command.

lftp can handle several file access methods - ftp, ftps, http, https, hftp, fish, sftp and file (https and ftps are only available when lftp is compiled with GNU TLS or OpenSSL library). You can specify the method to use in open URL' command, e.g. open http://www.us.kernel.org/pub/linux'. hftp is ftp-over-http-proxy protocol. It can be used automatically instead of ftp if ftp:proxy is set to `http://proxy[:port]'. Fish is a protocol working over an ssh connection to a unix account. SFtp is a protocol implemented in ssh2 as sftp subsystem.

comment:4 in reply to: ↑ 3 ; follow-up: ↓ 5 Changed 6 years ago by andrew_b

Replying to harvie:

i think it can be really good idea to (re-)implement whole VFS in mc using LFTP.

Patches are welcome!

you can fork lftp for yourself.

No, thanks.

comment:5 in reply to: ↑ 4 Changed 6 years ago by harvie

Replying to andrew_b:

Patches are welcome!

i wanted to add http protocol to extfs.d script, but i've found that such script needs to know whole tree which is ok for archives, tarballs, packages and similar stuff, but unusable for implementing remote filesystems like http(s)...

http can be simply implemented if the script will be passed everything after it's name. right now extfs.d scripts have no support for paths like this one:

cd /#http:user:password@domain.tld/directory
my script will be passed only
http list /

Note: See TracTickets for help on using tickets.