Ticket #2072 (new defect)

Opened 6 years ago

Last modified 6 weeks ago

high cpu usage during directory navigation and UTF-8 locale

Reported by: LwarX Owned by:
Priority: major Milestone: Future Releases
Component: mc-core Version: 4.8.2
Keywords: Cc: onlyjob@…, baltazar.bz@…
Blocked By: Blocking:
Branch state: no branch Votes for changeset:

Description

Probably this is not an MC fault, but anyway this is what I see while entering to/from deeply nested directory (residing on local filesystem):

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
11675 lwarx     20   0 20980 14256  3780 S  0.0  1.4  0:02.79  `- urxvtd -f -q -o
12244 lwarx     20   0  6180  1764  1428 S  0.0  0.2  0:00.39  |   `- -bash
12544 lwarx     20   0  9132  3060  2176 S  1.0  0.3  0:00.77  |   |   `- /usr/bin/mc -P /tmp/mc-lwarx/mc.pwd.12244
12546 lwarx     20   0  6164  1672  1352 R 68.0  0.2  0:28.90  |   |       `- bash -rcfile .bashrc

The more nested directory is, the higher CPU usage. Delay is also noticeable - on my 1200Mhz machine it can be up to one second. With LANG=C this slowdown goes away. I'm on Linux, bash version 3.2.

Attachments

test.list (2.3 KB) - added by LwarX 6 years ago.
result of ls -lR

Change History

comment:1 Changed 6 years ago by LwarX

  • Priority changed from major to minor

I've narrowed this issue down to the underlying shell. First, I upgraded bash to 4.0.35 (with no success). Then I tried zsh and there is no slowdown at all!
Probably this is something similar to grep issue with multibyte locales: https://bugs.launchpad.net/ubuntu/+source/grep/+bug/7906 https://bugzilla.redhat.com/show_bug.cgi?id=194471 http://savannah.gnu.org/bugs/?14472

My guess is that MC triggers some bash operations which are slow on multibyte strings. Maybe it is somehow related to cwd length. Details are up to you, guys!

P.S. Funny thing: this is 2010 now and I am (while being reluctant and conservative for years) finally decided to switch my system to UTF-8. Nah! It is still not yet ready!

comment:2 Changed 6 years ago by angel_il

please do:
ls -lR > test.list
and attach this file.

comment:3 Changed 6 years ago by LwarX

I'm not sure it is useful:

$ pwd
/mnt/500/160G-hdd-onecopy/My Music/CD's/Fashionweek-fall winter20032004

$ ls -lR > /tmp/test.list
... see attached file

If I launch MC with LANG=C, filenames are unreadable but directory entering/quitting is fast.

Changed 6 years ago by LwarX

result of ls -lR

comment:4 Changed 6 years ago by angel_il

mc -V

comment:5 Changed 6 years ago by LwarX

$ mc -V
GNU Midnight Commander, версия 4.7.1
Виртуальная файловая система: tarfs, extfs, cpiofs, ftpfs, fish, undelfs
Со встроенным редактором
С установленной в системе библиотекой S-Lang с базой данных terminfo
C поддержкой внутренней командной оболочки
С поддержкой фоновых операций
С поддержкой мыши в xterm и консоли Linux
С поддержкой интернационализации
С поддержкой многих кодировок
Data types: char 8 int 32 long 32 void * 32 off_t 64 ecs_char 8

comment:6 Changed 6 years ago by angel_il

i'm execute script

#!/bin/sh
for i in `seq 5000`; do
touch "2009 01 Дорожка $i.wma"
done

i dont have trouble...

comment:7 Changed 6 years ago by angel_il

$locale

comment:8 Changed 6 years ago by LwarX

$ locale
LANG=ru_RU.UTF-8
LC_CTYPE="ru_RU.UTF-8"
LC_NUMERIC="ru_RU.UTF-8"
LC_TIME="ru_RU.UTF-8"
LC_COLLATE="ru_RU.UTF-8"
LC_MONETARY="ru_RU.UTF-8"
LC_MESSAGES="ru_RU.UTF-8"
LC_PAPER="ru_RU.UTF-8"
LC_NAME="ru_RU.UTF-8"
LC_ADDRESS="ru_RU.UTF-8"
LC_TELEPHONE="ru_RU.UTF-8"
LC_MEASUREMENT="ru_RU.UTF-8"
LC_IDENTIFICATION="ru_RU.UTF-8"
LC_ALL=

I launched your script and the time it takes to complete does not depend on locale settings.
Can you please elaborate a bit more on what MC does internally when user enters/leaves some directory? Is there any bash operations invoked in subshell?

comment:9 Changed 6 years ago by LwarX

New fact: without subshell support (-u switch) directory switching is fast!

My guess: when MC enters any directory, it sends "cd" command to underlying subshell and probably this slowdown caused by escaping or something similar (unicode character space is huge). Can we construct benchmark to verify this?

comment:10 Changed 5 years ago by andrew_b

  • Branch state set to no branch
  • Milestone changed from 4.7 to Future Releases

comment:11 Changed 4 years ago by onlyjob

  • Cc onlyjob@… added
  • Priority changed from minor to major

This is more serious than it looks.

Create nested folder structure with the following command:

  for n in $(seq 100); do DN="${DN}/notsolongdirnametotestnested_$n"; done; mkdir -p "./${DN}"

and then enter as deep as possible using mc. Around No.25 I feel slowdown, but entering one between No.34 and No.40 breaks the panels. Then MC hang when I press "Ctrl+o" to hide/restore panels.

Sometimes it print out "Warning: Cannot change to..."

Last edited 4 years ago by onlyjob (previous) (diff)

comment:12 Changed 4 years ago by egmont

About the performance: I have the same problems with bash 3.2.48, tracing mc shows it spends way too much time in feed_subshell(). I cannot reproduce the speed problem with bash 4.0.38, 4.1 or 4.2, they are all very fast.

The "cannot change to..." is indeed another problem, I guess we're hitting the OS limit on path length or something like that.

comment:13 Changed 4 years ago by onlyjob

If you try to enter long nested directories with MC with subshell enabled (bash 4.2) you will feel slowdown. Perhaps it was worse with older bash but most recent bash didn't make this problem go away.

It has nothing to do with OS limit and apparently only a problem with bash. (I couldn't reproduce with other shells).
I think partially this problem because MC changing current working directory by passing full path to subshell.

comment:14 Changed 4 years ago by onlyjob

  • Version changed from 4.7.1 to 4.8.2

setting version to 4.8.2, since latest release is affected as well.

I can not change to 33rd directory if hierarchy created as described in comment 11.
Hiding panels hang MC.

Just a thought - perhaps 'cd' algorithm can be optimised to change only from current working directory rather than from root (which implies using full path for every 'cd' operation).

Last edited 4 years ago by onlyjob (previous) (diff)

comment:15 Changed 4 months ago by pytnik89

I have the same problem with version 4.8.15. MC enters in directories very slow, but without subshell (mc -u) all works perfectly. Is the way to solve it?

comment:16 follow-up: ↓ 17 Changed 4 months ago by zaytsev

Try zsh? Use mc -u?

comment:17 in reply to: ↑ 16 Changed 4 months ago by pytnik89

Replying to zaytsev:

Try zsh? Use mc -u?

Of course I did. I use zsh. And with 'mc -u' all is perfect.

comment:18 follow-up: ↓ 19 Changed 4 months ago by zaytsev

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

comment:19 in reply to: ↑ 18 Changed 4 months ago by pytnik89

Replying to zaytsev:

Oh, if you already use zsh, then it's sad. I'm not sure what we can do about that at mc side. The best solution I think would be to profile the shells to see why they are so slow at cd'ing and try to fix this. The alternative suggestion of using relative paths sounds like something that's going to be difficult to get right and will probably work unreliably.

Thx for u reply, Yes it's a problem of zsh, and specialy with it zsh-suggestion plugin.

comment:20 Changed 6 weeks ago by balta2ar

  • Cc baltazar.bz@… added
Note: See TracTickets for help on using tickets.