Checking file integrity (ie files acceptable to mpd) on Linux

[rwankar]

Hi,

I'm using Volumio on an Allo Digione Signature player. When I connect an external USB disk it detects only a few files. The Volumio team asked me to check the mpd logs and I could see errors on many files. I wanted to know if...

1. It is possible to do an integrity check on all files using a Linux laptop? I did use shntool info and it reports a lot of information for the WAV files but I can't tell if the file is corrupt.

2. The weird thing is that although mpd reported many files as "bad" it still proceeded to scan the disk. However it seems to get stuck scanning some WAV files but no errors are shown.

Thanks for the help.

[MaxKellermann]

1. What does "integrity check" mean, other than the information you already have?
2. contains no question, so what do you want to know here?

[MaxKellermann]

btw., Volumio ships with a very old MPD version AFAIK. We cannot help with any MPD shipped with Volumio due to that. Anything related to Volumio is unsupported (until they decide to use the latest version).
Read https://www.musicpd.org/help/ before asking for help.

[rwankar]

While scanning the disk mpd logs each file as it is scanned. Sometimes I see a whole bunch of lines like this..

Oct 04 10:14 : audiofile: unrecognized audio file format
Oct 04 10:14 : ffmpeg/mp3: Header missing
Oct 04 10:14 : ffmpeg/mp3: Header missing
....
Oct 04 10:14 : ffmpeg/mp3: Estimating duration from bitrate, this may be inaccurate
Oct 04 10:14 : audiofile: unrecognized audio file format
Oct 04 10:14 : audiofile: unrecognized audio file format
Oct 04 10:14 : audiofile: unrecognized audio file format

1. I suppose the file shown preceding this log is the offending file. Because the "Header missing" lines repeat multiple times but no file name is shown so I don't know if they are for the same file.

2. What generally seems to be the belief is that mpd is finicky about the file and can hang/stop (I'm not sure) when it hits a "bad" file. There is a ticket raised update or rescan commands crash mpd without an error message when a file can't be accessed #1251 which looks like a similar issue.

3. Right now mpd seems to get stuck on some WAV files and no error is reported. So I've been trying to move one file away at a time and rescan. To Rescan I have to restart mpd and then hit Rescan. This is too slow.

I'm asking if there is a way to check all files on a Linux laptop either using mpd (or any other tool which won't hang) so I know mpd won't have a problem when I plug the disk back into the Allo.

Does that make sense?

[rwankar]

I didn't see your reply till I posted. I'm an end user so I don't know which version Volumio uses. They pointed me here. I'll go back to them :)

[MaxKellermann]

The bug report #1251 came without any logs or crash reports. Therefore I cannot help with that.

If you see crashes or hangs with the current MPD version, I'll do my best to fix it, of course. But all you have is a 7 years old MPD version.

Don't get fooled by Volumio people - if they decide to ship a MPD version that's 7 years old, and they decide to ignore 7 years of bug fixes, it's only their responsibility to provide support.

See https://github.com/volumio/Build/tree/master/scripts/packages/mpd - last updated on Jan 6, 2015, i.e. 7 years ago. They ship MPD 0.19.7 which was released on Dec 17, 2014. I mean, WTF!

No Volumio support here, sorry.

[volumio]

@rwankar Sorry to hear that you've been pointed here, that should not have happened. As Max say, it is our job to handle support for Volumio (and filtering before it gets here, so only if appropriate). Please get in touch with me at info at volumio dot org so we take care of you and correct the procedure that brought you here.

@MaxKellermann you're right we shall not send you user's support requests, if not, eventually, properly contextualized and filtered by us (as happened sometimes in the past) to actually help improve mpd. Trying to find out how it happened and fix that procedure.
BTW, what you pointed at is a leftover configuration. We actually ship more recent MPD versions with Volumio (20.18, to which we are stuck cause of dependencies), and looking to get on par with master with Volumio 3.X.
Sources are here FYI: https://github.com/volumio/volumio-mpd (we use branches for versions used to prepare .deb packages for Volumio)

UPDATE: Found the culprit. https://community.volumio.org/t/scanning-from-usb-drive-stops-after-about-4755-files/50009/5
seems like it was a polite suggestion to ask here for any "mpd file validity checker", not any support on the specific issue.
Clarifying since I value other's people time as I do mine.

[MaxKellermann]

MPD 0.20.18 (not 20.18) is "only" 3.5 years old (released on Feb 24th, 2018). And it's not even the latest bug-fix release in that (obsolete) branch, that would be 0.20.23 released in Oct 29th, 2018. You could have adopted those bugfixes easily, without any additional dependencies, but you chose not to.
Your repository https://github.com/volumio/volumio-mpd is also managed clumsily; it's not a fork, but some huge unmanageable commits.

[volumio]

Noted. One of the goals with Volumio 3.X is to do a better job with being on par with releases, mpd included.

If you have suggestions on how to better manage mpd versioning will gladly accept them.

[MaxKellermann]

Suggestions? Always use the latest version.
Sounds trivial, and should be, but I don't know what difficulties you faced, what made you have those strange choices which seem clumsy to me.
For example, why do you have this weird "volumio-mpd" repisitory at all? Did you patch the sources? If yes, why didn't you submit those patches for merging? Why is that repository not a fork, but instead a dumb import of the MPD source tarball, including all the generated files? This appears to make no sense, and adds a lot of difficulty to your task of packaging MPD for Volumio. You must have a very strong reason for this choice, but I can't imagine it; without knowing the reason for those weird choices, I can't make suggestions.

[volumio]

Yeah, I am with you on this one. It seems unnecessarily clumsy, but that's the best approach I've found considering our goals. Note that we use vanilla mpd, no patches or exotic stuff. We just tweak options to not include some stuff we don't need.

The goals we need to achieve are:

• Predictable builds. We prefer to stick with a specific version and not rely on debian repos. This is because multiple times an updated package introduced some regressions\bugs. So we want to build a specific version, make it a .deb an install it. Periodically we check newer versions and they all pass our tests (and don't introduce bugs) we change the .deb in our build script to the new one.
  So we compile manually each build (from the source on the repo, which is taken from debian sources where possible, and debianized from debian sources) and upload the .deb to be installed during build time.
  Forking and using releases would, at least in my opinion, not offer the same flexibility and will introduce another level of complexity.
• This also leads us to maintain all versions that were ever used in builds, for two reasons. First one is GPL compliance, we do our best to respect licenses and we are required to provide the exact source along with build scripts. Then, in case we find regresions, we want to be able to switch back to an older\newer versions. Since we manage multiple architectures, its convenient to have the very same source + build script to generate a new .deb for a new arch (this happened for example with x64 builds, which we introduced recently).
• Having the full source + build script is convenient since we are still on Jessie (I know... sigh) and compiling recent packages requires (when its possible at all) some trick... Replicating them is convenient, when needed, and this repo guarantees that for us.

So this repo serves our purposes in the following way:

• We have snapshot of each source + buildscript used to generate a .deb
• We comply with GPL by offering the exact sources
• We can replicate builds easily and be able to test quickly new versions

I am with you that it might be unnecessarily cumbersome, but it has served up well until now. But really if there is a better management of packages that fits our requirements I would love to jump the fence and embrace it.

Anyway, we're quite OT. Open to discuss this also somewhere else, if you're interested.