diff --git a/README.md b/README.md index af69556..7896715 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ in concert with file tagging software, e.g. *autotagical* may be most easily installed with *pip* by running: -``` +```bash pip install autotagical ``` @@ -27,38 +27,39 @@ to install the following requirements, e.g. with *pip*: ### Usage -``` +```bash autotagical [-h] [-V] [-C ] [-H] [-i ] [-I ] [-R] [-o ] [-O] - [-g ] [-s ] [-A] [-F] [-k] [-m] - [-M] [-n] [-N] [-t] [--debug] [-l ] [-L] [-P] - [-q] [-v] [--force] [--yes] + [-g ] [-s ] [-A] [--cleanin] + [--cleanout] [-c] [-F] [-k] [-m] [-M] [-n] [-N] [-t] + [--debug] [-l ] [-L] [-P] [-q] [-v] [--force] + [--yes] ``` ### Help Options These options display helpful information and exit. -* [-h/--help] -- Display a help/usage message and exit. -* [-V/--version] -- Display the current version and information about known -file formats and exit. +* `[-h/--help]` -- Display a help/usage message and exit. +* `[-V/--version]` -- Display the current version and information about known + file formats and exit. ### Configuration Options -* [-C/--config ] -- Loads the config file at the specified path. +* `[-C/--config ]` -- Loads the config file at the specified path. ### Input Options These options determine behavior loading in files to be moved and/or renamed. At least one input folder must be specified. -* [-H/--hidden] -- Process hidden files (and directories, if -R is specified). -* [-i/--input ] -- Path to a folder with input files. May be -specified more than once. -* [-I/--ignore ] -- Path to file patterns (regex format) to ignore -(each on new line). May be specified more than once. -* [-R/--recursive] -- Load files recursively from input folders, i.e. descend -into subfolders. +* `[-H/--hidden]` -- Process hidden files (and directories, if -R is specified). +* `[-i/--input ]` -- Path to a folder with input files. May be + specified more than once. +* `[-I/--ignore ]` -- Path to file patterns (regex format) to + ignore (each on new line). May be specified more than once. +* `[-R/--recursive]` -- Load files recursively from input folders, i.e. descend + into subfolders. ### Output Options @@ -66,10 +67,10 @@ These options determine behavior in outputting move/renamed files (but do not specify the rules by which they are to be moved and/or renamed). At least one output folder must be specified (by one option or the other). -* [-o/--output] -- Path to a root folder to output files to. May be specified -more than once (output will be duplicated to each). -* [-O/--organize] -- Organize files in place (i.e. use the first input folder -for output). Often used with -R. +* `[-o/--output]` -- Path to a root folder to output files to. May be specified + more than once (output will be duplicated to each). +* `[-O/--organize]` -- Organize files in place (i.e. use the first input folder + for output). Often used with -R. ### Schema Options These options specify the rules for moving/renaming files and are the heart of @@ -77,83 +78,92 @@ These options specify the rules for moving/renaming files and are the heart of in the [Tag Group Format](#tag-group-format) and [Schema Format](#schema-format) sections. -* [-g/--groups ] -- Path to a file to read tag groups from. -May be specified more than once (groups will be combined). Files may be in -either the *[TagSpaces](https://github.com/tagspaces/tagspaces)* or the -*autotagical* format. -* [-s/--schema ] -- Path to a schema file to move/rename files -based on. May be specified more than once, in which case rules are prioritized -in the order files are specified. +* `[-g/--groups ]` -- Path to a file to read tag groups from. + May be specified more than once (groups will be combined). Files may be in + either the *[TagSpaces](https://github.com/tagspaces/tagspaces)* or the + *autotagical* format. +* `[-s/--schema ]` -- Path to a schema file to move/rename files + based on. May be specified more than once, in which case rules are + prioritized in the order files are specified. ### Functionality Options These options tweak the default functioning of *autotagical* as specified below, typically adjusting how various circumstances are dealt with. -* [-A/--allmatchroot] -- Makes the root of an output folder match all tags, -i.e. every single file will be moved to the output folder, even if it does not -match anywhere more specifically. Use of this option is bad practice (consider -using the `/*|` operator as a root filter instead), but it is provided for the -user's convenience. This option does **not** imply `-M`, i.e. files that could -not be renamed will not be moved to the root folder just because `-A` is set. -* [-F/--failforcerename] -- This option flags failing to rename a -manually-named file that is being forcibly renamed due to the `-N` option -should be considered a failure to name the file. That sounds complicated, but -consider these cases: - * [-F] -- Normal behavior (option will be ignored). - * [-N] -- Force rename manually-named files, but manual names are "good -enough" and manually-named files that cannot be renamed will be moved. - * [-F -N] -- Force rename manually-named files and treat failures as -failures. Manually-named files that cannot be renamed will **not** be moved. - * [-N -M] -- Force rename manually-named files. All files will be moved. - * [-F -N -M] -- Equivalent to `-N -M`. `-F` has no effect. -* [-k/--keep] -- Keep original files in the input folders untouched, i.e copy -files to their new destinations rather than move them. -* [-m/--move] -- Only move files into a directory structure, do not try to -rename them. -* [-M/--moveall] -- Move all files, not only ones that are manually-named/ -successfully renamed. -* [-n/--name] -- Only rename files, do not try to move them into any directory -structure. All files will be placed in the root of the output folder. -* [-N/--renamemanual] -- Forcibly try to rename manually-named files, not just -unnamed ones. -* [-t/--trial] -- Trial run. Do not actually move or rename files, just log -what would happen. Combine with `-v` to check output before live run. **Using -this is good practice,** especially after making any changes to a schema or -options. The `-t` option ensures no changes will be reflected to disk -whatsoever. +* `[-A/--allmatchroot]` -- Makes the root of an output folder match all tags, + i.e. every single file will be moved to the output folder, even if it does not + match anywhere more specifically. Use of this option is bad practice + (consider using the `/*|` operator as a root filter instead), but it is + provided for the user's convenience. This option does **not** imply `-M`, + i.e. files that could not be renamed will not be moved to the root folder + just because `-A` is set. +* `[--cleanin]` -- Clean up (delete) all empty folders in the input folder/s. + This *will* recurse, whether or not the `-R` flag is set. +* `[--cleanout]` -- Clean up (delete) all empty folders in the output folder. + This *will* recurse, whether or not the `-R` flag is set. +* `[-c/--clean]` -- Clean up (delete) all empty folders in both input and output + folders. +* `[-F/--failforcerename]` -- This option flags failing to rename a + manually-named file that is being forcibly renamed due to the `-N` option + should be considered a failure to name the file. That sounds complicated, but + consider these cases: + * `[-F]` -- Normal behavior (option will be ignored). + * `[-N]` -- Force rename manually-named files, but manual names are "good + enough" and manually-named files that cannot be renamed will be moved. + * `[-F -N]` -- Force rename manually-named files and treat failures as + failures. Manually-named files that cannot be renamed will **not** be moved. + * `[-N -M]` -- Force rename manually-named files. All files will be moved. + * `[-F -N -M]` -- Equivalent to `-N -M`. `-F` has no effect. +* `[-k/--keep]` -- Keep original files in the input folders untouched, i.e copy + files to their new destinations rather than move them. +* `[-m/--move]` -- Only move files into a directory structure, do not try to + rename them. +* `[-M/--moveall]` -- Move all files, not only ones that are manually-named/ + successfully renamed. +* `[-n/--name]` -- Only rename files, do not try to move them into any directory + structure. All files will be placed in the root of the output folder. +* `[-N/--renamemanual]` -- Forcibly try to rename manually-named files, not just + unnamed ones. +* `[-t/--trial]` -- Trial run. Do not actually move or rename files, just log + what would happen. Combine with `-v` to check output before live run. + **Using this is good practice,** especially after making any changes to a + schema or options. The `-t` option ensures no changes will be reflected to + disk whatsoever. ### Logging Options These options tweak what sorts of messages are displayed when *autotagical* is run (and whether to save them or just print them to the console). -* [--debug] -- Display absolutely everything. You should probably never use -this. -* [-l/--log ] -- Output messages to the specified file rather than -just the console. By default, messages will be appended to the end of the file. -* [-L/--overwritelog] -- Overwrite the specified log file, rather than append -to it. Has no effect without `-l`. -* [-P/--posix] -- Silence warnings specific to Windows. Use this **only** if -the files are never to be used with Windows systems (which are pickier about -what file names can contain). -* [-q/--quiet] -- Silence all warnings and only display actual errors. Use of -this is **not recommended,** as warnings are typically printed for good reason. -* [-v/--verbose] -- Print all actions taken. This will list every file -movement/renaming, rather than merely warning about failures. Most useful when -combined with `-t` to check that a schema is doing what one wants before -running it for real. +* `[--debug]` -- Display absolutely everything. You should probably never use + this. +* `[-l/--log ]` -- Output messages to the specified file rather than + just the console. By default, messages will be appended to the end of the + file. +* `[-L/--overwritelog]` -- Overwrite the specified log file, rather than append + to it. Has no effect without `-l`. +* `[-P/--posix]` -- Silence warnings specific to Windows. Use this **only** if + the files are never to be used with Windows systems (which are pickier about + what file names can contain). +* `[-q/--quiet]` -- Silence all warnings and only display actual errors. Use of + this is **not recommended,** as warnings are typically printed for good + reason. +* `[-v/--verbose]` -- Print all actions taken. This will list every file + movement/renaming, rather than merely warning about failures. Most useful + when combined with `-t` to check that a schema is doing what one wants before + running it for real. ### Unsafe Options Do not use these unless you have very good reason to. **Data loss can occur.** -* [--force] -- Forcibly move/rename files, even if there is a file or directory -in the way. **This will clobber files;** use at your own risk, as data loss -can occur. -* [--yes] -- Assume "yes" for all user prompts. This implies `--force` and -**will clobber files and directories.** Use at your own risk, as data loss can -occur. +* `[--force]` -- Forcibly move/rename files, even if there is a file or + directory in the way. **This will clobber files;** use at your own risk, as + data loss can occur. +* `[--yes]` -- Assume "yes" for all user prompts. This implies `--force` and + **will clobber files and directories.** Use at your own risk, as data loss + can occur. ### Setting Priority @@ -177,7 +187,7 @@ will have no effect if set in a config file. This is to prevent data loss without explicit user input. Whitespace and newlines are ignored, e.g. one might write: -``` +```bash -H -P @@ -189,35 +199,145 @@ in a config file to process hidden and ignore Windows-specific warnings. *autotagical* is capable of reading the JSON files produced by exporting tag groups from [TagSpaces](https://github.com/tagspaces/tagspaces). Alternately, -tag groups may be defined in a more simple, human-readable fashion in JSON as -follows: +tag groups may be defined in a more simple, somewhat more human-readable +fashion in JSON. This *autotagical* tag group format supports additional +features not available in the TagSpaces format, detailed below. -``` +```json { - "file_type" : "autotagical_tag_groups", - "tag_group_file_version" : "1.0", - "tag_groups" : [ + "file_type": "autotagical_tag_groups", + "tag_group_file_version": "1.1", + "tag_groups": [ { - "name" : "tag group name", - "tags" : ["tag1", "tag2",...] + "name": "tag group 1", + "tags": ["tag1", "tag2"...] + }, + { + "name": "tag group 2", + "tags": ["tag3", "/G|tag group 1", "/RE|regex 1",...] }, - ... ] } ``` +### Inheritance + +In the *autotagical* format, tag groups support simple inheritance, where a +child tag group may be defined in terms of one or more parent tag groups (as +well as any additional tags). Inheritance is indicated by prefixing a group +name with `/G|` in the `tags` property of a group. In such cases, the child +group will inherit all tags in any of the parent groups. This is useful for +simplicity; it ensures that one only has to manually define the most "leaf" tag +groups while still writing filters based on more broad groups, all of which +will update if the more refined groups are updated. Consider the following +simple use case: + +```json +{ + "file_type": "autotagical_tag_groups", + "tag_group_file_version": "1.1", + "tag_groups": [ + { + "name": "American Styles", + "tags": [ + "ipa", + "dipa", + "pale_ale" + ] + }, + { + "name": "Belgian Styles", + "tags": [ + "witbier", + "dubbel", + "tripel" + ] + }, + { + "name": "Beer", + "tags": [ + "/G|American Styles", + "/G|Belgian Styles" + ] + } + ] +} +``` + +Here, we have defined the `Beer` group in terms of `American Styles` and +`Belgian Styles`, rather than having to specify all 6 tags that should fall +under it. Not only is this quicker to write, but it prevents errors of +oversight when one decides to add tags later. If, for example, `quadrupel` +is added to the `Belgian Styles` group, it will be part of the `Beer` group +without one having to remember to manually add it in two places. While this +simple case doesn't seem that unmanageable without inheritance, the number of +places where duplicate tags have to be added can quickly spiral out of control +with multiple levels of tag group. + +Keep in mind the following details about inheritance: +* **Multiple inheritance** -- Tag groups can inherit from multiple parent + group (as shown in the example). +* **Multilevel inheritance** -- Tag groups support multilevel inheritance, + i.e. an `Alcohol` group might inherit from `Beer` which inherits from + `American Styles`. +* **The (lack of) diamond problem** -- As there is no overriding in tag group + inheritance, the diamond problem (where a grandparent is inherited via two + different routes) is handled without problem. +* **Circular inheritance** -- It is okay for two groups to inherit from each + other (whether via intermediaries or not). Each inheritance path will + only be followed once, i.e. a tag group will stop "following" an + inheritance path if it is instructed to inherit from itself. +* **Flexible ordering** -- Tag group inheritance is flexible in the order that + groups are defined. There is no need for a tag group to be located after + a group it inherits from. In fact, tag groups can inherit from groups in + completely different tag group files, regardless of the order they are loaded + in (so long as they are both loaded). + +### Regex Tag Groups + +The *autotagical* tag group format supports defining tag groups in terms of +regexes. This can be incredibly powerful in certain situations. For example, +consider tagged bank statements. Rather than having to manually add every +account number tag to an "Account Number" group, one can instead define a tag +group as follows: + +```json +{ + "name": "Account Number", + "tags": ["/RE|(?:xx|\\*\\*)[0-9]{4}"] +} + +``` + +This will make any tag of the form `**1234` or `xx1234` match the "Account +Number" tag group. If one deals with a large number of tags with a standardized +form, this can be extremely valuable, especially if they change frequently. + +Keep in mind the following details about regex tag groups: +* **Full match** -- Regexes are used with `re.fullmatch()` and as such must + match 100% of the tag name. +* **Inheritance** -- Regexes will be inherited as with any other tags in a +group. +* **Mixed and matched** -- A tag group can be defined in terms of any + combination of inheritance, normal tags, or regexes. They do not need to be + specified solely in terms of regexes. +* **Tag in group (/?TIG|) operator** -- Regex tag groups work normally with + the tag in group format string operator. The first tag to match the regex + will be returned for it. + ## Schema Format + A schema is defined in a human-readable fashion in JSON and should consist of a single object as follows: -``` +```json { - "file_type" : "autotagical_schema", - "schema_file_version" : "1.0", - "tag_formats" : [], - "unnamed_patterns" : [], - "renaming_schemas" : [], - "movement_schema" : [] + "file_type": "autotagical_schema", + "schema_file_version": "1.1", + "tag_formats": [], + "unnamed_patterns": [], + "renaming_schemas": [], + "movement_schema": [] } ``` @@ -226,11 +346,11 @@ is described in the following sections. ### tag_formats -``` -"tag_formats" : [ +```json +"tag_formats": [ { - "tag_pattern" : "Regex containing groups: file, raw_tags, tags, and extension.", - "tag_split_pattern" : "Regex to split tags with" + "tag_pattern": "Regex containing groups: file, raw_tags, tags, and extension.", + "tag_split_pattern": "Regex to split tags with" }, ... ] @@ -252,11 +372,11 @@ and will be used with `re.split()`. An example is provided below, which matches the tag format used by [TagSpaces](https://github.com/tagspaces/tagspaces): -``` -"tag_formats" : [ +```json +"tag_formats": [ { - "tag_pattern" : "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", - "tag_split_pattern" : "\\s+" + "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", + "tag_split_pattern": "\\s+" } ] ``` @@ -267,10 +387,10 @@ one run. ### unnamed_patterns -``` -"unnamed_patterns" : [ +```json +"unnamed_patterns": [ "regex pattern 1", - "rege pattern 2", + "regex pattern 2", ... ] ``` @@ -283,8 +403,8 @@ above, so do not attempt to match against anything not captured by those two groups. An example is provided below, which might match PDF files with timestamps produced by two different scanners: -``` -"unnamed_patterns" : [ +```json +"unnamed_patterns": [ "[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]{2}_[0-9]{2}_[0-9]{2}\\s*.pdf", "(Pages\\sfrom)?\\s*XScan_[0-9]{14}\\s*.pdf" ] @@ -292,11 +412,11 @@ timestamps produced by two different scanners: ### renaming_schema -``` -"renaming_schemas" : [ +```json +"renaming_schemas": [ { "filter": ["condition 1", "condition 2", ...], - "format_string" : "file name format string" + "format_string": "file name format string" }, ... ] @@ -311,74 +431,24 @@ of renaming. See the [Filters](#filters) section for information on filters, conditions, and the various operators one may include in them. "format_string" defines how to rename files matching the filter. It cannot -contain the `/` character except where it denotes the following operators (as +contain the `/` character except where it denotes operators (as file names cannot contain `/`). At its simplest, it is simply a string that the file will be renamed to, but that string may include any number/combination -of the following operators: - -* `/EXT|` -- Anywhere it is put in the format string, `/EXT|` will be replaced -with the original extension of the file, as defined by the `extension` group in -the "tag_formats" regex that matched the file. This is obviously useful if -you're renaming multiple types of file and want to preserve extensions. You -will almost always want to end your format string with `/EXT|`. -* `/FILE|` -- Anywhere it is put in the format string, `/FILE|` will be -replaced with the original name of the file, as defined by the `file` group in -the "tag_formats" regex that matched the file. -* `/TAGS|` -- Anywhere it is put in the format string, `/TAGS|` will be -replaced with the tags on the original file. **This is necessary to avoid your -renamed files becoming de-tagged** and should almost always be included in a -format string. -* `/?|/T|/F|/E?|` -- The conditional operator -`/?|` allows for conditional naming. If `` is matched, the entire -expression will be replaced with ``; if it does not match, the -entire expression will be replaced with ``. Either text (or both, -but why would you) may be empty. The conditional operator can take anything -that can be in a filter condition. See the [Filters](#filters) section for -information on filters, conditions, and the various operators one may include in -them. `` and `` may contain other operators, i.e. -`/EXT|`, `/FILE|`, `/TAGS|`, and `/ITER|`. Note: conditional operators -**cannot** be nested or contain `/?T|/|` or `/?G|/|` within -replacement text. -* `/?T|/|` -- The tag conditional operator `/?T|` will insert the literal -name of the tag `` if it is present on the file. Note that this is -equivalent to `/?|/T|//F|/E?|`; it is merely a shortcut. -* `/?G|/|` -- The tag group conditional operator `/?G|` will insert -the literal name of the tag group `` if one of its tags is present -on the file. Note that this is equivalent to -`/?|/G|/T|/F|/E?|`; it is merely a shortcut. -* `/ITER|/#|/EITER|` -- The `/ITER|` operator is complicated -but important. It is invoked only in the event that multiple files are going -to be renamed to the same name. In this case, the text is placed in the file -name, along with `/#|` being replaced by the n-th file that this is that has -had the same name. Othrwise, the entirety of the `/ITER|` operator is ignored. -In essence, the `/ITER|` tag "counts" how many times the same file name has -been produced. It is good practice to always include an `/ITER|` operator in -your schema to avoid files not being renamed due to potential clobbering. -`/#|` may appear more than once in an `/ITER|` operator, but there is usually -no need to. The `/ITER|` operator *may* contain any other operator, including -the conditional operator `/?|`, but may not be nested. Note that the `/ITER|` -operator **will not be used** if files end up with the **same name but -different output directories**. It will only appear if necessary to avoid -clobbering. An example will make this easier to understand. Consider the -format string `Widget/ITER| /#|/EITER|` in the following cases: - * 1 matching file -- The file will be named `Widget`. - * 3 matching files in same folder -- The files will be named `Widget 1`, -`Widget 2`, and `Widget 3`. - * 3 matching files, each in a different folder -- The files will all be named -`Widget`. +of operators. See the [Format Strings](#format-strings) section for +information on format strings and the various operators one may include in them. ### movement_schema -``` -"movement_schema" : [ +```json +"movement_schema": [ { "filter": ["condition 1", "condition 2", ...], - "subfolder" : "", - "sublevels" : [ + "subfolder": "", + "sublevels": [ { "filter": ["condition 3", "condition 4", ...], - "subfolder" : "", - "sublevels" : [...] + "subfolder": "", + "sublevels": [...] }, ... ] @@ -397,14 +467,16 @@ to `tag2`. This allows one to define priorities of sorting. See the [Filters](#filters) section for information on filters, conditions, and the various operators one may include in them. -If "subfolder" contains a string, that folder will be added to the hierarchy -the file will be placed in (and further sorted based on "sublevels"); if it is -left blank `""`, files will be placed in the current (in the hierarchy) -directory without further sorting. If "sublevels" is left empty `[]`, files -will be placed in the specified subfolder without further sorting. Note that a -movement schema does not have to have a path for every possible file. Files -that fail to "find a home" will be left in the input folder and a warning will -be printed (unless `-A` is specified). +If "subfolder" contains a format string, it will be interpreted and added to +the folder hierarchy that the file will be placed in (and further sorted based +on "sublevels"); if it is left blank `""`, files will be placed in the current +(in the hierarchy) directory without further sorting. If "sublevels" is left +empty `[]`, files will be placed in the specified subfolder without further +sorting. Note that a movement schema does not have to have a path for every +possible file. Files that fail to "find a home" will be left in the input +folder and a warning will be printed (unless `-A` is specified). See the +[Format Strings](#format-strings) section for information on format strings and +the various operators one may include in them. Additionally, note that complete hierarchies (i.e. those that terminate with explicitly placing the file in a folder) will be preferred over partial @@ -420,29 +492,108 @@ A filter (wherever it might show up in a schema) is defined by an array of condition sets. These condition sets are combined in the logical sense by *inclusive or*, i.e. matching at least one condition set is necessary and sufficient to match the overall filter. At their simplest, a condition set may -simply be a tag, e.g. `"filter" : ["tag1", "tag2"]` will match any file with +simply be a tag, e.g. `"filter": ["tag1", "tag2"]` will match any file with either `tag1` or `tag2` (or both) on it. However, the following operators may be used to construct more complex condition sets (whether in filters or in the conditional `/?|` operator): * `/G|` -- The prefix `/G|` is used to denote a tag group instead of a tag -name, e.g. `"filter" : ["/G|Group 1", "tag2"]` will match any file with at -least one tag in `Group 1` or the tag `tag2` (or both). + name, e.g. `"filter": ["/G|Group 1", "tag2"]` will match any file with at + least one tag in `Group 1` or the tag `tag2` (or both). * `/*|` -- The all operator `/*|` matches all files, regardless of how they are -tagged. + tagged. * `/&|` -- `/&|` is a logical "and" operator, requiring matching both -conditions, e.g. `"filter" : ["tag1/&|tag2"]` will match files that have *both* -`tag1` and `tag2`. A condition may contain any number of `/&|` operators, e.g. -one may create the condition `"tag1/&|tag2/&|/G|Group 1"`. + conditions, e.g. `"filter": ["tag1/&|tag2"]` will match files that have *both* + `tag1` and `tag2`. A condition may contain any number of `/&|` operators, + e.g. one may create the condition `"tag1/&|tag2/&|/G|Group 1"`. * `/!|` -- The not prefix `/!|` negates the next condition. **This prefix must -come before any others logically,** i.e. you must write `/!|/G|` rather -than `/G|/!|` or `/!|/*|` rather than `/*|/!|`. The `/!|` operator -*can* follow the logical "and" operator `/&|`, e.g. `"/&|/!|"`, -which will match any file that has `tag1` and does not have `tag2`. + come before any others logically,** i.e. you must write `/!|/G|` rather + than `/G|/!|` or `/!|/*|` rather than `/*|/!|`. The `/!|` operator + *can* follow the logical "and" operator `/&|`, e.g. `"/&|/!|"`, + which will match any file that has `tag1` and does not have `tag2`. There are no (realistic) limits on the degree to which these operators may be combined or how many condition sets a filter might have. +### Format Strings + +A format string is simply a string that may or may not contain various +operators. These operators will be replaced with the corresponding data when +the format string is interpreted. + +* `/EXT|` -- Anywhere it is put in the format string, `/EXT|` will be replaced + with the original extension of the file, as defined by the `extension` group + in the "tag_formats" regex that matched the file. This is obviously useful if + you're renaming multiple types of file and want to preserve extensions. You + will almost always want to end your format string with `/EXT|`. +* `/FILE|` -- Anywhere it is put in the format string, `/FILE|` will be + replaced with the original name of the file, as defined by the `file` group in + the "tag_formats" regex that matched the file. +* `/TAGS|` -- Anywhere it is put in the format string, `/TAGS|` will be + replaced with the tags on the original file. **This is necessary to avoid + your renamed files becoming de-tagged** and should almost always be included + in a renaming format string. +* `/?|/T|/F|/E?|` -- The conditional operator + `/?|` allows for conditional naming. If `` is matched, the entire + expression will be replaced with ``; if it does not match, the + entire expression will be replaced with ``. Either text (or both, + but why would you) may be empty. The conditional operator can take anything + that can be in a filter condition. See the [Filters](#filters) section for + information on filters, conditions, and the various operators one may include + in them. `` and `` may contain other operators, i.e. + `/EXT|`, `/FILE|`, `/TAGS|`, and `/ITER|`. Note: conditional operators + **cannot** be nested or contain `/?T|/|` or `/?G|/|` within + replacement text. +* `/?T|/|` -- The tag conditional operator `/?T|` will insert the literal + name of the tag `` if it is present on the file. Note that this is + equivalent to `/?|/T|//F|/E?|`; it is merely a shortcut. +* `/?G|/|` -- The tag group conditional operator `/?G|` will insert + the literal name of the tag group `` if one of its tags is present + on the file. Note that this is equivalent to `/?|/G|/T|/F|/E?|`; it is merely a shortcut. +* `/?TIG|/|` -- The "tag in group" operator is a special operator + that will insert a tag on the file that is in the specified tag group. For + example, `/?TIG|group1?/|` will resolve to `tag1` if `tag1` is on the file and + in `group1`, or `tag2` if `tag2` is on the file and in `group1`. If multiple + tags on the file are in the group, the first (in tagging order, left-to-right) + will be substituted in. If the file lacks a tag in the group, the entire + operator will simply be blank. This "tag in group" operator is most useful + for writing flexible schema where the exact tags cannot be predicted in + advance. For example, if one wanted to sort bank statements into folders + based on what account they are associated with, one could use the "tag in + group" operator with an "Account Number" group and then only have to update + the tag group with the various account number tags, rather than having to + specify every account number in the schema. As an example, consider the + following cases, with `Group 1 = tag1, tag3` and + `format_string = "Tag: /?TIG|Group 1/|"`: + * `File [tag1, tag2]` -- "Tag: tag1" + * `File [tag2, tag4]` -- "Tag: " + * `File [tag1, tag2, tag3]` -- "Tag: tag2" +* `/ITER|/#|/EITER|` -- The `/ITER|` operator is complicated + but important. It is invoked only in the event that multiple files are going + to be renamed to the same name. In this case, the text is placed in the file + name, along with `/#|` being replaced by the n-th file that this is that has + had the same name. Otherwise, the entirety of the `/ITER|` operator is + ignored. **The /ITER| operator should not be used in folder name format + strings. It will be ignored.** + In essence, the `/ITER|` tag "counts" how many times the same file name has + been produced. It is good practice to always include an `/ITER|` operator in + your schema to avoid files not being renamed due to potential clobbering. + `/#|` may appear more than once in an `/ITER|` operator, but there is usually + no need to. The `/ITER|` operator *may* contain any other operator, including + the conditional operator `/?|`, but may not be nested. Note that the `/ITER|` + operator **will not be used** if files end up with the **same name but + different output directories**. It will only appear if necessary to avoid + clobbering. An example will make this easier to understand. Consider the + format string `Widget/ITER| /#|/EITER|` in the following cases: + * 1 matching file -- The file will be named `Widget`. + * 3 matching files in same folder -- The files will be named `Widget 1`, + `Widget 2`, and `Widget 3`. + * 3 matching files, each in a different folder -- The files will all be named + `Widget`. + + + ## Known Issues * Only POSIX hidden files are considered hidden, i.e. those that begin with a @@ -454,7 +605,7 @@ Fix**. `autotagical` may be tested by cloning this repository and running: -``` +```bash python setup.py test ``` diff --git a/autotagical/__init__.py b/autotagical/__init__.py index 3b6788a..553d72d 100644 --- a/autotagical/__init__.py +++ b/autotagical/__init__.py @@ -15,9 +15,10 @@ ----- autotagical [-h] [-V] [-C ] [-H] [-i ] [-I ] [-R] [-o ] [-O] - [-g ] [-s ] [-A] [-F] [-k] [-m] - [-M] [-n] [-N] [-t] [--debug] [-l ] [-L] [-P] - [-q] [-v] [--force] [--yes] + [-g ] [-s ] [-A] [--cleanin] + [--cleanout] [-c] [-F] [-k] [-m] [-M] [-n] [-N] [-t] + [--debug] [-l ] [-L] [-P] [-q] [-v] [--force] + [--yes] """ -__version__ = '1.0.0' # Define the current version +__version__ = '1.1.0' # Define the current version diff --git a/autotagical/file_handler.py b/autotagical/file_handler.py index 58ee03c..a0ff40b 100644 --- a/autotagical/file_handler.py +++ b/autotagical/file_handler.py @@ -11,6 +11,8 @@ --------- Functions --------- +clean_folder(folder_path, trial_run=False): + Removes all empty directories/subdirectories from the specified folder. check_windows_compat(name, full_path) Checks a file name and path for Windows-unsafe characters. check_output_location(settings) @@ -39,15 +41,51 @@ import logging +def clean_folder(folder_path, trial_run=False): + """ + Removes all empty directories/subdirectories from the specified folder. + + Parameters + ---------- + folder_path: str + Path to folder to clean. + trial_run: bool + If True, do not actually delete folders, only log that they will be. + + Returns + ------- + bool + True if successful, False if errors were encountered. + """ + successful = True + for root, dirs, _ in os.walk(top=folder_path, topdown=False): + for directory in dirs: + full_path = os.path.join(root, directory) + try: + if not os.listdir(full_path): + logging.info('Cleaning (deleting) folder at %s', + full_path) + if not trial_run: + os.rmdir(full_path) + else: + logging.info('Skipping cleaning non-empty directory at: ' + '%s', full_path) + except OSError as err: + logging.warning('Could not clean directory at %s:\n%s', + full_path, str(err)) + successful = False + return successful + + def check_windows_compat(name, full_path): """ Checks a file name and path for Windows-unsafe characters. Parameters ---------- - name : str + name: str Name of file (no directories). - full_path : str + full_path: str Full path of the file Returns @@ -192,10 +230,10 @@ def move_files(move_list, settings): Parameters ---------- - move_list : list of AutotagicalFile + move_list: list of AutotagicalFile A list of AutotagicalFile objects, each representing a file to be moved/renamed. - settings : AutotagicalSettings + settings: AutotagicalSettings An AutotagicalSettings object holding the settings for the movement. Returns @@ -252,25 +290,25 @@ class AutotagicalFile: # pylint: disable=R0902 Attributes ---------- - dest_folder : str + dest_folder: str The folder the file is to be moved to. - extension : str + extension: str The extension of the original file. - move_failed : bool + move_failed: bool True is no applicable movement schema was found, False otherwise. - name : str + name: str Original file name, less tags and extension. - original_path : str + original_path: str Complete path to original file, including directories and file name. - output_name : str + output_name: str The file name the file is to be renamed to. - raw_name : str + raw_name: str The original file name with tags and extension. - rename_failed : bool + rename_failed: bool True if no applicable renaming schema was found, False otherwise. - tags : str + tags: str Complete tags on the original file, including any delimiters. - tag_array : list of str + tag_array: list of str List of strings, each a tag on the original file. Class Methods @@ -297,17 +335,17 @@ def __init__(self, name, tags, extension, tag_array, raw_name, Parameters ---------- - name : str + name: str Original file name, less tags and extension. - tags : str + tags: str Complete tags on the original file, including any delimiters. - extension : str + extension: str The extension of the original file. - tag_array : list of str + tag_array: list of str List of strings, each a tag on the original file. - raw_name : str + raw_name: str The original file name with tags and extension. - original_path : str + original_path: str Complete path to original file, including all directories and file name. """ @@ -351,17 +389,17 @@ def load_file(cls, name, path, tag_patterns, ignore_patterns): Parameters ---------- - name : str + name: str The full name of the file to load. - path : str + path: str Complete path to original file, including directories and file name - tag_patterns : list of dict + tag_patterns: list of dict List of dictionaries, each of the following form: { - 'tag_pattern' : Regular Expression Object, - 'tag_split_pattern' : Regular Expression Object + 'tag_pattern': Regular Expression Object, + 'tag_split_pattern': Regular Expression Object } - ignore_patterns : list of Regular Expression Objects + ignore_patterns: list of Regular Expression Objects A list of compiled regexes full-matching file patterns to ignore. Returns @@ -404,14 +442,14 @@ class AutotagicalFileHandler: Instance Attributes ------------------- - __file_list : list of AutotagicalFile - __ignore_patterns : list of Regular Expression Objects + __file_list: list of AutotagicalFile + __ignore_patterns: list of Regular Expression Objects A list of compiled regexes full-matching file patterns to ignore. - __tag_patterns : list of dict + __tag_patterns: list of dict List of dictionaries, each of the following form: { - 'tag_pattern' : Regular Expression Object, - 'tag_split_pattern' : Regular Expression Object + 'tag_pattern': Regular Expression Object, + 'tag_split_pattern': Regular Expression Object } Methods @@ -446,10 +484,18 @@ def __init__(self, tag_formats): """ self.__file_list = [] self.__ignore_patterns = [] - self.__tag_patterns = [{ - 'tag_pattern': re.compile(pattern['tag_pattern']), - 'tag_split_pattern': re.compile(pattern['tag_split_pattern']) - } for pattern in tag_formats] + self.__tag_patterns = [] + # Try to compile tag patterns as a regex or warn + for pattern in tag_formats: + try: + self.__tag_patterns.append({ + 'tag_pattern': re.compile(pattern['tag_pattern']), + 'tag_split_pattern': + re.compile(pattern['tag_split_pattern']) + }) + except re.error as err: + logging.warning('Regex error in tag format: %s\n%s', + pattern, str(err)) def load_ignore_file(self, path): """ @@ -458,7 +504,7 @@ def load_ignore_file(self, path): Parameters ---------- - path : str + path: str The complete path to the file, including folders and the complete file name. @@ -496,9 +542,9 @@ def check_new(self, name, path): Parameters ---------- - name : str + name: str The full name of the file. - path : str + path: str The path to the file (including file name) Returns @@ -522,9 +568,9 @@ def load_file(self, name, path): Parameters ---------- - name : str + name: str The full name of the file. - path : str + path: str The path to the file (including file name) Returns @@ -549,11 +595,11 @@ def load_folder(self, input_folder, recurse=False, process_hidden=False): Parameters ---------- - input_folder : str + input_folder: str Path to folder to load files from. - recurse : bool + recurse: bool If True, will descend into subfolders recursively. - process_hidden : bool + process_hidden: bool If True, will process files beginning with '.' and (if recurse is True) descend into directories beginning with '.' diff --git a/autotagical/filtering.py b/autotagical/filtering.py index bf8b2f6..cedf80b 100644 --- a/autotagical/filtering.py +++ b/autotagical/filtering.py @@ -99,14 +99,14 @@ def check_condition(tag_array, condition, tag_groups): Parameters ---------- - tag_array : list of str + tag_array: list of str A list of strings, each holding a tag. These tags will be considered as a whole against the provided condition. It should be in the form: ['tag1', 'tag2', ...] - condition : str + condition: str A string with the condition against which the tags will be matched. This may contain various special operators. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve tag group operators. @@ -138,12 +138,9 @@ def check_condition(tag_array, condition, tag_groups): # If it's a tag group if match.group('tag_group'): - # Load what tags are in that tag_group - group_tags = tag_groups.get_tags_in_group(match.group('tag_group')) - # See if any tag is in the tag gorup and return accordingly - for tag in tag_array: - if tag in group_tags: - return match_found + # See if tags match the group + if tag_groups.tag_in_group(tag_array, match.group('tag_group')): + return match_found return not match_found # If here, something went horribly wrong, throw an error @@ -160,15 +157,15 @@ def check_against_condition_set(tag_array, condition_set, tag_groups): Parameters ---------- - tag_array : list of str + tag_array: list of str A list of strings, each holding a tag. These tags will be considered as a whole against the provided condition set. It should be in the form: ['tag1', 'tag2', ...] - condition_set : str + condition_set: str A string with the condition set against which the tags will be matched. This may contain various special operators, and may be multiple conditions concatenated with the AND operator /&|. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve group operators. @@ -204,15 +201,15 @@ def check_against_filter(tag_array, check_filter, tag_groups): Parameters ---------- - tag_array : list of str + tag_array: list of str A list of strings, each holding a tag. These tags will be considered as a whole against the provided filter. It should be in the form: ['tag1', 'tag2', ...] - filter : list of str + filter: list of str A list of strings, each containing a condition set against which the tags will be matched. Each may contain various special operators and may be multiple conditions concatenated with the AND operator /&|. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve group operators. diff --git a/autotagical/groups.py b/autotagical/groups.py index 745b0c0..7767952 100644 --- a/autotagical/groups.py +++ b/autotagical/groups.py @@ -18,10 +18,12 @@ import json import sys import os +import re from jsonschema import validate, ValidationError from packaging import version +# pylint: disable=R0902 class AutotagicalGroups: """ Holds tag groups, produces them, and knows how to load them from various @@ -38,6 +40,21 @@ class AutotagicalGroups: Instance Attributes ------------------- + __input_data: dict + A dictionary with strings as keys, representing the raw data loaded + from tag group files. Calling process_groups() populates __group_data + from this. + __inheritance: dict + A dicitonary with strings as keys and sets as values, representing + groups and the parent groups they inherit from. Used by + __resolve_inheritance() and __inherit(). + __to_compile: dict + A dictionary with strings as keys and sets as values, representing + regex groups to compile. + __loaded_autotagical_format + Whether any group files in the autotagical format were loaded (meaning + that more detailed processing is going to be required when + process_groups() is called. __group_data: dict A dictionary with strings as keys, represnting tag group names, and values that are sets of strings, each representing a tag. It should be @@ -48,9 +65,12 @@ class AutotagicalGroups: 'group 2': set of str {'tag3', 'tag4', ...}, } + __regex_group_data: dict + A dictionary with strings as keys, represnting tag group names, and + values that are lists of compiled regexes. tag_group_schema: dict The JSON schema against which to validate autotagical tag group files. - tagspaces_schema : dict + tagspaces_schema: dict The JSON schema against which to validate TagSpaces tag group files. Methods @@ -58,10 +78,19 @@ class AutotagicalGroups: __init__() Constructor, initializes tag group dictionary and loads validation schemas. - __repr__(self) + __repr__() Pretty print tag group data. Only for debugging. - get_tags_in_group(group) - Returns all tags in a specified tag group. + __inherit(child, parents): + Returns all inherited tags from parents by operating recursively and + updates the tags in self.__group_data. + __resolve_inheritance(): + Resolves inheritance notes stored in self.__inheritance. + process_groups() + Perform all necessary setup once groups are loaded (e.g. resolving + inheritance). + tag_in_group(tag_array, group) + Determines if a tag array matches the specified group and returns the + first matching tag. load_autotagical_format(json_input, append=False) Loads tag groups from JSON data in the *autotagical* format. This does not validate the data, as this should be handled upstream. @@ -79,7 +108,7 @@ class AutotagicalGroups: known schemas to ensure the data structure is correct. """ - TAG_GROUP_FILE_VERSION = "1.0" + TAG_GROUP_FILE_VERSION = "1.1" TAGSPACES_SETTINGS_VERSION = 3 TAGSPACES_APP_VERSION = "3.1.4" @@ -89,8 +118,12 @@ def __init__(self): schemas. """ - self.__group_data = dict() # Initialize tag group dictionary - + self.__input_data = dict() # Initialize tag group dictionary + self.__group_data = dict() + self.__inheritance = dict() + self.__to_compile = dict() + self.__regex_group_data = dict() + self.__loaded_autotagical_format = False # Load autotagical tag group validation schema or fail with message try: with open(os.path.join(os.path.dirname(__file__), 'json_schema', @@ -137,35 +170,186 @@ def __repr__(self): """ to_return = '-----Tag Groups----\n' for group_name, group_tags in sorted(self.__group_data.items()): - to_return += (' ' + group_name + ': ' + str(sorted(group_tags)) + + group_tags = sorted(group_tags) + # Append regexes + if group_name in self.__regex_group_data: + group_tags.extend(sorted( + ('/RE|' + pattern.pattern for pattern + in self.__regex_group_data[group_name]))) + to_return += (' ' + group_name + ': ' + str(group_tags) + '\n') to_return += '-----End Tag Groups-----' return to_return - def get_tags_in_group(self, group): + def __inherit(self, child, parents): """ - Returns all tags in a specified tag group. + Returns all inherited tags from parents by operating recursively and + updates the tags in self.__group_data. Parameters ---------- - group: str - A string consisting of the tag group name to return tags for. + child: str + Name of child group to inherit to. + parents: set + Parents to inherit from. + path: list + Path of recursion, ensuring that circular inheritance doesn't + happen. + + Returns + ------- + (set of str, set of str) + A tuple of sets of strings, the first tags that were inherited, the + second regex patterns that were inherited. + """ + tags_to_return = set() + patterns_to_return = set() + for parent in parents: + # If inheriting from a group with inheritance, get grandparents. + if parent in self.__inheritance: + grandparent_tags, grandparent_patterns = \ + self.__inherit(parent, self.__inheritance.pop(parent)) + logging.debug('Child group %s inheriting tags %s and patterns ' + '%s from grandparents.', child, grandparent_tags, + grandparent_patterns) + tags_to_return.update(grandparent_tags) + self.__group_data[child].update(grandparent_tags) + patterns_to_return.update(grandparent_patterns) + self.__to_compile[child].update(grandparent_patterns) + # If inheriting from a non-existent group, warn about it. + elif parent not in self.__group_data: + logging.warning('Attempt to inherit from group that was ' + 'not loaded: %s. Skipping this.', parent) + continue + # Inherit from parent. + logging.debug('Child group %s inheriting tags %s and patterns %s ' + 'from parent %s', child, self.__group_data[parent], + self.__to_compile[parent], parent) + tags_to_return.update(self.__group_data[parent]) + self.__group_data[child].update(self.__group_data[parent]) + patterns_to_return.update(self.__to_compile[parent]) + self.__to_compile[child].update(self.__to_compile[parent]) + + return (tags_to_return, patterns_to_return) + + def __resolve_inheritance(self): + """ + Resolves inheritance notes stored in self.__inheritance. + + Parameters + ---------- + None Returns ------- - set - A set of tags associated with the tag group. It will be in the - form: - {'tag1', 'tag2', 'tag3', ...} + None + """ + # If anything left to resolve + while self.__inheritance: + # Pick a group to resolve + group, inherits_from = self.__inheritance.popitem() + self.__inherit(group, inherits_from) + + def __compile_regexes(self): + """ + Compiles all regexes in self.__to_compile to self.__group_data. + + Parameters + ---------- + + Returns + ------- + None + """ + for group in self.__to_compile: + # Leave key out of regexes if nothing would go in it + if self.__to_compile[group]: + self.__regex_group_data[group] = [] + for pattern in self.__to_compile[group]: + try: + self.__regex_group_data[group].append( + re.compile(pattern)) + except re.error as err: + logging.warning('Regex error in tag group regex: %s\n%s', + pattern, str(err)) + + def process_groups(self): """ + Perform all necessary setup once groups are loaded (e.g. resolving + inheritance). - # Check if group exists, print error if it doesn't and return empty. + Parameters + ---------- + None + + Returns + ------- + None + """ + # TagSpaces format requires no processing + if not self.__loaded_autotagical_format: + self.__group_data = self.__input_data + else: + for group, entries in self.__input_data.items(): + self.__group_data[group] = set() + self.__to_compile[group] = set() + for entry in entries: + # If it's shorter than "/G|a" or doesn't start with '/G|' + # or '/RE|' it has to be a simple tag. + # pylint: disable=C0330 + if len(entry) < 4 \ + or (entry[:3] != '/G|' + and (len(entry) < 5 + or entry[:4] != '/RE|')): + self.__group_data[group].add(entry) + continue + # Otherwise, check for special groups and add them to be + # processed + if entry[:3] == '/G|': + if group not in self.__inheritance: + self.__inheritance[group] = set() + self.__inheritance[group].add(entry[3:]) + elif entry[:4] == '/RE|': + if group not in self.__to_compile: + self.__to_compile[group] = set() + self.__to_compile[group].add(entry[4:]) + # Process special groups + self.__resolve_inheritance() + self.__compile_regexes() + + def tag_in_group(self, tag_array, group): + """ + Determines if a tag array matches the specified group and returns the + first matching tag. + + Parameters + ---------- + tag_array: list of str + A list of strings, each representing a tag to be checked. + group: str + The group name to check. + + Returns + ------- + str + Returns the first tag that is in the group or an empty string if + none do. + """ + # Check if group exists, print error if it doesn't and return None. if group not in self.__group_data: logging.warning('Malformed condition encountered: Tag group "%s" ' 'is not among loaded tag groups', str(group)) - return set() - # Otherwise return the tag group data. - return self.__group_data[group] + return '' + has_regexes = group in self.__regex_group_data + for tag in tag_array: + if tag in self.__group_data[group]: + return tag + # Only check regexes if the group has them. + if has_regexes: + for pattern in self.__regex_group_data[group]: + if pattern.fullmatch(tag): + return tag + return '' def load_autotagical_format(self, json_input, append=False): """ @@ -174,21 +358,21 @@ def load_autotagical_format(self, json_input, append=False): Parameters ---------- - json_data : dict + json_data: dict A complex dictionary produced from parsing JSON. It should be in the form: { - "file_type" : "autotagical_tag_groups", - "tag_group_file_version" : 1.0, - "tag_groups" : [ + "file_type": "autotagical_tag_groups", + "tag_group_file_version": 1.1, + "tag_groups": [ { - "name" : "", - "tags" : ["", "",...] + "name": "", + "tags": ["", "/G|",...] }, ... ] } - append : bool (default False) + append: bool (default False) If True, tag group data is added to previously loaded groups. Otherwise, loaded tag groups are wiped first and then replaced. @@ -208,27 +392,34 @@ def load_autotagical_format(self, json_input, append=False): 'Update autotagical to continue!') return False + # Indicate that tag groups will require processing + self.__loaded_autotagical_format = True + # If not told to append, wipe out extant tag groups if not append: + self.__input_data = dict() self.__group_data = dict() + self.__inheritance = dict() + self.__to_compile = dict() + self.__regex_group_data = dict() for group in json_input['tag_groups']: # Bad practice to have the same group multiple times, so warn but # don't fail - if group['name'] in self.__group_data: + if group['name'] in self.__input_data: logging.warning('Tag group "%s" multiply defined! Already ' 'have: "%s", appending to this: "%s"', group['name'], - str(self.__group_data[group['name']]), + str(self.__input_data[group['name']]), str(group['tags'])) # Append if already seen - self.__group_data[group['name']].update(group['tags']) + self.__input_data[group['name']].update(group['tags']) else: # Otherwise add tag group - self.__group_data[group['name']] = set(group['tags']) + self.__input_data[group['name']] = set(group['tags']) logging.debug('Loaded tag groups from autotagical format:\n%s', - str(self.__group_data)) + str(self.__input_data)) # Loading was successful, so return True return True @@ -262,25 +453,30 @@ def load_tagspaces_format(self, json_input, append=False): # If not told to append, wipe out extant groups if not append: + self.__input_data = dict() self.__group_data = dict() + self.__inheritance = dict() + self.__to_compile = dict() + self.__regex_group_data = dict() + self.__loaded_autotagical_format = False try: for group in json_input['tagGroups']: # It's bad practice to have the same tag group multiple times, # so warn but don't fail - if group['title'] in self.__group_data: + if group['title'] in self.__input_data: logging.warning('Tag group "%s" multiply defined! Already' ' have: "%s", appending to this: "%s"', group['title'], - str(self.__group_data[group['title']]), + str(self.__input_data[group['title']]), str([tag['title'] for tag in group['children']])) # Append if already seen - self.__group_data[group['title']].update( + self.__input_data[group['title']].update( {tag['title'] for tag in group['children']}) else: # Otherwise add group - self.__group_data[group['title']] = \ + self.__input_data[group['title']] = \ {tag['title'] for tag in group['children']} except KeyError: # This may happen if format changes @@ -289,7 +485,7 @@ def load_tagspaces_format(self, json_input, append=False): return False logging.debug('Loaded tag groups from TagSpaces format:\n%s', - str(self.__group_data)) + str(self.__input_data)) # Loading was successful, so return True return True @@ -300,22 +496,22 @@ def load_tag_groups(self, json_input, append=False): Parameters ---------- - json_input : dict + json_input: dict This consists of the (usually complicated) dict object produced from parsing JSON. It may be in the TagSpace format or the autotagical format, which is again noted here: { - "file_type" : "autotagical_tag_groups", - "tag_group_file_version" : "1.0", - "tag_groups" : [ + "file_type": "autotagical_tag_groups", + "tag_group_file_version": "1.1", + "tag_groups": [ { - "name" : "", - "tags" : ["", "",...] + "name": "", + "tags": ["", "/G|",...] }, ... ] } - append : bool + append: bool Whether this should replace all known tag groups or be added to them. @@ -360,9 +556,9 @@ def load_tag_groups_from_string(self, json_string, append=False): Parameters ---------- - json_string : string + json_string: string String containing JSON data to load - append : bool + append: bool Whether this should replace all known tag groups or be added to them. @@ -392,9 +588,9 @@ def load_tag_groups_from_file(self, file_path, append=False): Parameters ---------- - file_path : string + file_path: string Path to the file to load. - append : bool + append: bool Whether this should replace all known tag groups or be added to them. diff --git a/autotagical/json_schema/schema_file_schema.json b/autotagical/json_schema/schema_file_schema.json index 8224d71..aed11ba 100644 --- a/autotagical/json_schema/schema_file_schema.json +++ b/autotagical/json_schema/schema_file_schema.json @@ -1,7 +1,7 @@ { "$schema": "http://json-schema.org/schema#", "$id": "https://raw.githubusercontent.com/SiriusStarr/autotagical/stable/autotagical/json_schema/schema_file_schema.json", - "$comment": "SCHEMA_FILE_VERSION = 1.0", + "$comment": "SCHEMA_FILE_VERSION = 1.1", "type": "object", "definitions": { "string_array": { diff --git a/autotagical/moving.py b/autotagical/moving.py index 7538488..3f8ccd6 100644 --- a/autotagical/moving.py +++ b/autotagical/moving.py @@ -25,10 +25,11 @@ import logging import os from autotagical.filtering import check_against_filter +from autotagical.naming import substitute_operators, strip_iters from autotagical.schema import SchemaError -def process_filter_level(tag_array, filter_level, tag_groups): +def process_filter_level(file, filter_level, tag_groups): """ Takes a file's tag array and a sfilter level and determines the contribution to path that it and all sublevels contribute by operating @@ -36,55 +37,58 @@ def process_filter_level(tag_array, filter_level, tag_groups): Parameters ---------- - tag_array : list of str - A list of strings, each holding a tag. These tags will be considered - as a whole against the provided filter. It should be in the form: - ['tag1', 'tag2', ...] - filter_level : dict + file: AutotagicalFile + An AutotagicalFile object, representing the file being moved. + filter_level: dict A dictionary with strings for keys, representing a filter level. It should have the following form: { - "filter" : list of str + "filter": list of str ["condition set 1", "condition set 2", ...] - "subfolder" : str - "name of the subfolder (left in parent directory if empty)" - "sublevels" : list of dict + "subfolder": str + "format string (left in parent directory if empty)" + "sublevels": list of dict [ { - "filter" : list of str, - "subfolder" : str - "sublevels" : list of dict + "filter": list of str, + "subfolder": str + "sublevels": list of dict }, ... ] } - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve group operators. Returns ------- - (full_match, path) : (bool, str) - full_match : bool + (full_match, path): (bool, str) + full_match: bool Whether or not a positive match was found at the lowest level. Returns False if the tag array did not determine a specific location. - path : str + path: str The constructed file path from the current level down. Empty if no matches whatsoever. """ # Check if it matches the filter at this level - if check_against_filter(tag_array, filter_level['filter'], tag_groups): - # If there isn't a subfolder name, it's supposed to go in the "current" - # (previous) folder - if not filter_level['subfolder']: - return (True, '') + if check_against_filter(file.tag_array, filter_level['filter'], + tag_groups): + + # Interpret format string + if filter_level['subfolder']: + subfolder_name = substitute_operators(strip_iters( + filter_level['subfolder']), file, tag_groups) + else: + subfolder_name = '' # If there are no sub filter levels, it's supposed to terminate in the - # specified subfolder - if not filter_level['sublevels']: - return (True, filter_level['subfolder']) + # specified subfolder; if there isn't a subfolder name, it's supposed + # to go in the "current" (previous) folder + if not filter_level['sublevels'] or not filter_level['subfolder']: + return (True, subfolder_name) # partial_sort becomes true if some further matching is found but not a # conclusive location @@ -93,16 +97,15 @@ def process_filter_level(tag_array, filter_level, tag_groups): for level in filter_level['sublevels']: # Recurse to check the entire filter tree - output = process_filter_level(tag_array, level, tag_groups) + output = process_filter_level(file, level, tag_groups) # If a positive match was found at a lower level, join the output # and return True if output[0]: # Only concatenate if there is actually a subfolder if output[1]: - return (True, os.path.join(filter_level['subfolder'], - output[1])) + return (True, os.path.join(subfolder_name, output[1])) # Otherwise return current path with a conclusive match - return (True, filter_level['subfolder']) + return (True, subfolder_name) # If no positive match was found, but a partial match was, set # partial_sort. This only gets set ONCE because filter order @@ -112,60 +115,57 @@ def process_filter_level(tag_array, filter_level, tag_groups): # If no exact match was found but a partial was, use the first partial # (because priority) if partial_sort: - return (False, os.path.join(filter_level['subfolder'], - partial_sort)) + return (False, os.path.join(subfolder_name, partial_sort)) # Otherwise, no matches were found at lower directories - return (False, filter_level['subfolder']) + return (False, subfolder_name) # If we got here, it means that the file didn't even match current filters return (False, '') -def generate_path(tag_array, movement_schema, tag_groups): +def generate_path(file, movement_schema, tag_groups): """ - Takes a file's tag array and determines how to organize it according to a - movement schema. + Takes a file and determines how to organize it according to a movement + schema. Parameters ---------- - tag_array : list of str - A list of strings, each holding a tag. These tags will be considered - as a whole against the provided schema. It should be in the form: - ['tag1', 'tag2', ...] - movement_schema : list of dict + file: AutotagicalFile + An AutotagicalFile object, representing the file to be moved. + movement_schema: list of dict A list of dictionaries, each with strings for keys, representing a single filter level. It should have the following form: [ { - "filter" : list of str + "filter": list of str ["filter1", "filter2", ...] - "subfolder" : str + "subfolder": str "name of subfolder (left in parent directory if empty)" - "sublevels" : list of dict + "sublevels": list of dict [ { - "filter" : list of str, - "subfolder" : str - "sublevels" : list of dict + "filter": list of str, + "subfolder": str + "sublevels": list of dict }, ... ] }, ... ] - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve group operators. Returns ------- - (move_failed, path) : (bool, str) - move_failed : bool + (move_failed, path): (bool, str) + move_failed: bool Whether or not any amount of matching was found. Returns False if the tag array did not determine any location whatsoever and shouldn't be moved. - path : str + path: str The constructed file path. Empty if no matches whatsoever. """ # partial_sort becomes true if some matching is found but not a conclusive @@ -180,10 +180,11 @@ def generate_path(tag_array, movement_schema, tag_groups): # For every movement schema for filter_level in movement_schema: # Try to sort according to it - output = process_filter_level(tag_array, filter_level, tag_groups) + output = process_filter_level(file, filter_level, tag_groups) # If a positive match was found, done (because filter priority) if output[0]: - logging.debug('Good match for moving tags: %s', str(tag_array)) + logging.debug('Good match for moving tags: %s', + str(file.tag_array)) return (False, output[1]) # A partial sorting was found; will be used if no exact match is found if output[1] and not partial_sort: @@ -195,7 +196,7 @@ def generate_path(tag_array, movement_schema, tag_groups): '\n%s\nSorted only to:\n%s\nThis is bad practice. ' 'Add a /*| operator filter at that level if you ' 'intend to catch all files at that point.', - str(tag_array), partial_sort) + str(file.tag_array), partial_sort) return (False, partial_sort) # Absolutely no matching whatsoever, so return that it failed @@ -209,12 +210,12 @@ def determine_destination(file_list, movement_schema, tag_groups): Parameters ---------- - file_list : list of AutotagicalFile + file_list: list of AutotagicalFile A list of AutotagicalFile objects, each representing a file to be moved - movement_schema : list + movement_schema: list A list of movement schema in the form provided by AutotagicalSchema.movement_schema - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object that will be used to resolve group operators. @@ -232,7 +233,7 @@ def determine_destination(file_list, movement_schema, tag_groups): # Loop through all provided files for file in file_list: # Determine the appropriate path - output = generate_path(file.tag_array, movement_schema, tag_groups) + output = generate_path(file, movement_schema, tag_groups) # Set values accordingly file.dest_folder = output[1] diff --git a/autotagical/naming.py b/autotagical/naming.py index 54a11e7..7be90a8 100644 --- a/autotagical/naming.py +++ b/autotagical/naming.py @@ -11,6 +11,10 @@ --------- Functions --------- +_tig_operator_sub(tag_array, tag_groups, match_obj) + Takes a tag array, tag groups, and a regex match object (from a TIG + operator match) and returns the string to subsitute it with. For use with + re.sub(). simplify_to_conditionals(format_string) Takes a format string and simplifies the "convenience" operators to simply be conditionals. @@ -35,11 +39,35 @@ import logging import re +import sys from autotagical.filtering import check_against_filter, \ check_against_condition_set from autotagical.schema import SchemaError +def _tig_operator_sub(tag_array, tag_groups, match_obj): + """ + Takes a tag array, tag groups, and a regex match object (from a TIG + operator match) and returns the string to subsitute it with. For use with + re.sub(). + + Parameters + ---------- + tag_array: list of str + List of strings, each representing a tag on the file. + tag_groups: AutotagicalGroups + An AutotagicalGroups object representing known tag groups. + match_obj: Regex Match Object + The match being subsituted. + + Returns + ------- + str + The tag in group that matches. + """ + return tag_groups.tag_in_group(tag_array, match_obj.group('group')) + + def simplify_to_conditionals(format_string): """ Takes a format string and simplifies the "convenience" operators to simply @@ -47,7 +75,7 @@ def simplify_to_conditionals(format_string): Parameters ---------- - format_string : str + format_string: str The format string may have tag /?T|, group /?G|, conditional /?|, file name /FILE|, tags /TAGS|, and/or extension /EXT| operators in it. It may not have the /ITER| operator, as this is handled separately. @@ -73,13 +101,13 @@ def evaluate_conditionals(format_string, tag_array, tag_groups): Parameters ---------- - format_string : str + format_string: str The format string may have conditional /?|, file name /FILE|, tags /TAGS|, and/or extension /EXT| operators in it. It may not have the /ITER| operator, as this is handled separately. - tag_array : list of str + tag_array: list of str A list of strings, each representing a tag on the file being evaluated. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object, representing known tag groups groups. Returns @@ -109,7 +137,7 @@ def strip_iters(format_string): Parameters ---------- - format_string : str + format_string: str The format string may have the tag /?T|, group /?G|, conditional /?|, file name /FILE|, tags /TAGS|, extension /EXT|, and/or /ITER| operators in it. @@ -137,11 +165,11 @@ def evaluate_iters(format_string, occurrence): Parameters ---------- - format_string : str + format_string: str The format string may have the tag /?T|, group /?G|, conditional /?|, file name /FILE|, tags /TAGS|, extension /EXT|, and/or /ITER| operators in it. - occurrence : int + occurrence: int Number of times the file name has been produced (will be inserted for the /#| occurrence operator) @@ -171,13 +199,13 @@ def substitute_operators(format_string, file, tag_groups): Parameters ---------- - format_string : str + format_string: str The format string may have tag /?T|, group /?G|, conditional /?|, file name /FILE|, tags /TAGS|, and/or extension /EXT| operators in it. It may not have the /ITER| operator, as this is handled separately. - file : AutotagicalFile + file: AutotagicalFile The file to be considered. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object, representing known tag groups. Returns @@ -195,6 +223,13 @@ def substitute_operators(format_string, file, tag_groups): to_return = evaluate_conditionals(simplify_to_conditionals(format_string), file.tag_array, tag_groups) + # Handle tag in group operator + # Have to use a lambda function to wrap _tig_operator_sub, since re.sub + # can only pass one argument. + to_return = AutotagicalNamer.tig_regex.sub( + lambda a: _tig_operator_sub(file.tag_array, tag_groups, a), + to_return) + # Warn if no extension or no tags if '/TAGS|' not in to_return: logging.warning('Renamed a file without preserving tags! This will ' @@ -211,10 +246,11 @@ def substitute_operators(format_string, file, tag_groups): # Check if there are any /'s left, as this is a very bad sign. if '/' in to_return: - logging.warning('A "/" is still in the format string after reducing ' - 'all operators! This probably means there was a ' - 'problem with the format string!\nFormat String: ' + - format_string + '\nOutput: ' + to_return) + logging.error('A "/" is still in the format string after reducing ' + 'all operators! This probably means there was a ' + 'problem with the format string!\nFormat String: %s\n' + 'Output: %s', format_string, to_return) + sys.exit() # If the operators evaluated to a completely blank string, this is bad, # because can't use that @@ -233,43 +269,43 @@ class AutotagicalNamer: Class Attributes ---------------- - tag_regex : Regular Expression Object + tag_regex: Regular Expression Object Compiled regex for finding /?T| tag operators - group_regex : Regular Expression Objects + group_regex: Regular Expression Objects Compiled regex for finding /?G| group operators - conditional_regex : Regular Expression Objects + conditional_regex: Regular Expression Objects Compiled regex for finding /?| conditional operators - iter_regex : Regular Expression Objects + iter_regex: Regular Expression Objects Compiled regex for finding /ITER| operators Instance Attributes ------------------- - __produced_names : dict + __produced_names: dict A dictionary with strings for keys and dictionaries for values, containing filenames that have been produced (used for iter operators). It should be of the following form: { - "iterless filename" : dict + "iterless filename": dict A dictionary representing a single file name that has been produced. It should be of the following form: { - 'occurrences' : int, + 'occurrences': int, Number of times this name has been produced - 'first_occurrence' : str + 'first_occurrence': str Full path to the first file to be named this } ... } - __unnamed_patterns : list of Regular Expression Objects + __unnamed_patterns: list of Regular Expression Objects List of compiled regexes that define unnamed files. - __renaming_schemas : list of dict + __renaming_schemas: list of dict List of dictionaries, each with strings as keys. It will be of the form: [ { - 'filter' : list of str + 'filter': list of str ['filter1', 'filter2', ...] - 'format_string' : str + 'format_string': str 'renaming format string containing operators' } ] @@ -299,6 +335,7 @@ class AutotagicalNamer: r'/T\|(?P.*?)' r'/F\|(?P.*?)/E\?\|)') iter_regex = re.compile(r'(?P/ITER\|(?P.*?)/EITER\|)') + tig_regex = re.compile(r'/\?TIG\|(?P[^/]+?)/\|') def __init__(self, renaming_schemas, unnamed_patterns): """ @@ -306,9 +343,9 @@ def __init__(self, renaming_schemas, unnamed_patterns): Parameters ---------- - renaming_schemas : list + renaming_schemas: list List of the sort stored in AutotagicalSchema.renaming_schemas - unnamed_patterns : list + unnamed_patterns: list List of the sort stored in AutotagicalSchema.unnamed_patterns Returns @@ -330,7 +367,11 @@ def __init__(self, renaming_schemas, unnamed_patterns): # Store schema and compile regexes self.__renaming_schemas = renaming_schemas for pattern in unnamed_patterns: - self.__unnamed_patterns.append(re.compile(pattern)) + try: + self.__unnamed_patterns.append(re.compile(pattern)) + except re.error as err: + logging.warning('Regex error unnamed pattern: %s\n%s', + pattern, str(err)) def check_if_unnamed(self, file_name): """ @@ -339,7 +380,7 @@ def check_if_unnamed(self, file_name): Parameters ---------- - file_name : str + file_name: str The file name to check against the patterns. This should not contain tags. @@ -361,10 +402,10 @@ def find_format_string(self, tag_array, tag_groups): Parameters ---------- - tag_array : list of str + tag_array: list of str A list of strings, each representing a tag on the file. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups Known tag groups. Returns @@ -388,19 +429,19 @@ def determine_names(self, file_list, tag_groups, force_name=False, Parameters ---------- - file_list : list of AutotagicalFile + file_list: list of AutotagicalFile A list of AutotagicalFile objects, each representing a file to be named. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object representing known tag groups. - force_name : bool (False by default) + force_name: bool (False by default) Whether to try to rename files that do not match the unnamed pattern. - force_fail_bad : bool (False by default) + force_fail_bad: bool (False by default) Whether a manually named file (being named because of force_name) not matching any naming schema should be considered a failure to name the file. - clear_occurrences : bool (False by default) + clear_occurrences: bool (False by default) Whether to reset produced file names before the run (resetting iter operators). diff --git a/autotagical/schema.py b/autotagical/schema.py index 833bd75..f7962c4 100644 --- a/autotagical/schema.py +++ b/autotagical/schema.py @@ -35,10 +35,10 @@ def _repr_filter_tree(filter_level, indent=0): Parameters ---------- - filter_level : dict + filter_level: dict The filter level to print. Should have keys 'filter', 'subfolder', and 'sublevels'. - indent : int + indent: int The number of spaces to begin indenting by. Should be a multiple of 2. Returns @@ -85,42 +85,42 @@ class AutotagicalSchema: Instance Attributes ------------------- - tag_formats : list of dict + tag_formats: list of dict A list of dictionaries, each with strings as keys. It will be of the form: [ { - 'tag_pattern' : str + 'tag_pattern': str 'regex containing groups "file", "raw_tags", "tags", and "extension"' - 'tag_split_pattern' : str + 'tag_split_pattern': str 'regex to be used with re.split() to separate tags' } ] - unnamed_patterns : list + unnamed_patterns: list List of strings, each a regex pattern that defines unnamed files. - renaming_schemas : list of dict + renaming_schemas: list of dict List of dictionaries, each with strings as keys. It will be of the form: [ { - 'filter' : list of str + 'filter': list of str ['condition set 1', 'condition set 2', ...] - 'format_string' : str + 'format_string': str 'renaming format string containing operators, e.g /?|tag1/T|true/F|false/E?|' } ] - movement_schema : list of dict + movement_schema: list of dict List of dictionaries, each with strings as keys keys. It will be of the form: [ { - 'filter' : list of str + 'filter': list of str ['condition set 1', 'condition set 2', ...] - 'subfolder' : str + 'subfolder': str 'subfolder name' - 'sublevels' : list of dict + 'sublevels': list of dict [ {additional dict with filter, subfolder, sublevels} ] @@ -144,7 +144,7 @@ class AutotagicalSchema: formats to ensure the data structure is correct. """ - SCHEMA_FILE_VERSION = "1.0" + SCHEMA_FILE_VERSION = '1.1' def __init__(self): """ @@ -206,12 +206,12 @@ def load_schema(self, json_input, append=False): Parameters ---------- - json_input : dict + json_input: dict A complex dictionary produced from parsing JSON. It should be in the form: { 'file_type': 'autotagical_schema', - 'schema_file_version': '1.0', + 'schema_file_version': '1.1', 'tag_formats': [ { 'tag_pattern': 'pattern', diff --git a/autotagical/settings.py b/autotagical/settings.py index eb80ed1..ff93fa1 100644 --- a/autotagical/settings.py +++ b/autotagical/settings.py @@ -47,15 +47,15 @@ def _init_logging(verbose, no_warn, debug, log_file=None, overwrite_log=False): Parameters ---------- - verbose : bool + verbose: bool Sets log level to at least INFO if True - no_warn : bool + no_warn: bool Sets log level to ERROR if True and no other flags - debug : bool + debug: bool Sets log level to DEBUG if True - log_file : str (default None) + log_file: str (default None) Path to log file to output to - overwrite_log : bool (default False) + overwrite_log: bool (default False) Whether to overwrite or append to specified log file. Returns @@ -151,55 +151,57 @@ class AutotagicalSettings: # pylint: disable=R0902, R0903 Class Attributes ---------------- - yes_regex : Regular Expression Object + yes_regex: Regular Expression Object Matches "yes"; used to parse user input - no_regex : Regular Expression Object + no_regex: Regular Expression Object Matches "no"; used to parse user input Instance Attributes ------------------- - all_match_root : bool + all_match_root: bool Whether to move files that did not match any movement schema, using the root output folder for such cases. - answer_yes : bool + answer_yes: bool Whether to assume "yes" for all user prompts. - tag_groups : AutotagicalGroups + tag_groups: AutotagicalGroups An AutotagicalGroups object, representing known tag groups. - clobber : bool + clean_folders: list of str + List of folders to clean (delete empty directories from) after run. + clobber: bool Whether to clobber files in the output folder (overwrite them without prompt). - copy : bool + copy: bool Whether to copy files from input folder or move them out of it. - force_move : bool + force_move: bool Whether to move files that could not be renamed. - force_name : bool + force_name: bool Whether to try to rename even files that have been manually named. - force_name_fail_bad : bool + force_name_fail_bad: bool Whether failing to rename a manually-named file counts as failing to name. - ignore_files : list of str + ignore_files: list of str List of paths to files containing patterns of files to ignore. - input_folders : list of str + input_folders: list of str List of paths to directories to parse files from. - move_only : bool + move_only: bool Whether to only move files, not rename them. - output_folders : list of str + output_folders: list of str List of paths to directories to output files to. Files will be copied to each. - process_hidden : bool + process_hidden: bool Whether to include hidden files and directories (those begining with ".") in input folders. - recurse : bool + recurse: bool Whether to descend into subdirectories looking for input files. - rename_only : bool + rename_only: bool Whether to only rename files, not move them. - schema : AutotagicalSchema + schema: AutotagicalSchema A representation of loaded movement/renaming schemas. - silence_windows : bool + silence_windows: bool Whether to silence warnings about unsafe characters in file names for Windows. - trial_run : bool + trial_run: bool Whether to only print actions rather than execute them. Methods @@ -232,6 +234,7 @@ def __init__(self): self.tag_groups = None self.schema = None self.all_match_root = True + self.clean_folders = [] self.copy = True self.move_only = True self.rename_only = True @@ -253,9 +256,9 @@ def get_yes_no(self, msg, default_to): Parameters ---------- - msg : string + msg: string The message to prompt the user with. - default_to : bool + default_to: bool Whether to default to "yes" (True) of "no" (False) Returns @@ -292,10 +295,10 @@ def __interpret_args(self, cl_args, file_args): Parameters ---------- - cl_args : Namespace + cl_args: Namespace Namespace produced by arparse's parse_args() function on the command line. - file_args : Namespace + file_args: Namespace Namespace produced by arparse's parse_args() function on a config file. @@ -382,6 +385,9 @@ def __interpret_args(self, cl_args, file_args): else: _load_files(file_args.tag_group_files, self.tag_groups.load_tag_groups_from_file) + + # Process groups to resolve fancy features. + self.tag_groups.process_groups() logging.debug('Tag Groups:\n%s', str(self.tag_groups)) self.schema = AutotagicalSchema() @@ -402,6 +408,22 @@ def __interpret_args(self, cl_args, file_args): self.all_match_root = False logging.debug('All match at root: %s', str(self.all_match_root)) + # Clean input only if told to on command line or told in config file + # and no cleaning on command line. + # pylint: disable=C0330 + if (cl_args.clean_both + or cl_args.clean_input_folders + or (not cl_args.clean_output_folders + and (file_args.clean_input_folders + or file_args.clean_both))): + self.clean_folders += self.input_folders + if (cl_args.clean_both + or cl_args.clean_output_folders + or (not cl_args.clean_input_folders + and (file_args.clean_output_folders + or file_args.clean_both))): + self.clean_folders += self.output_folders + # Keep originals to false only if neither set it. if not cl_args.copy and not file_args.copy: self.copy = False @@ -578,6 +600,22 @@ def __load_args_and_config(self): 'all tags, i.e. every file will be ' 'moved to output folder, even if it ' 'does not match more specifically.') + function_args.add_argument('--cleanin', + dest='clean_input_folders', + action='store_true', + help='Clean up (delete) all empty folders ' + 'in input folders (will recurse).') + function_args.add_argument('--cleanout', + dest='clean_output_folders', + action='store_true', + help='Clean up (delete) all empty folders ' + 'in output folders (will recurse).') + function_args.add_argument('-c', '--clean', + dest='clean_both', + action='store_true', + help='Clean up (delete) all empty folders ' + 'in input and output folders (will ' + 'recurse).') function_args.add_argument('-F', '--failforcerename', dest='force_name_fail_bad', action='store_true', diff --git a/bin/autotagical b/bin/autotagical index 5c9b5c9..15b7d52 100755 --- a/bin/autotagical +++ b/bin/autotagical @@ -16,16 +16,18 @@ Usage ----- autotagical [-h] [-V] [-C ] [-H] [-i ] [-I ] [-R] [-o ] [-O] - [-g ] [-s ] [-A] [-F] [-k] [-m] - [-M] [-n] [-N] [-t] [--debug] [-l ] [-L] [-P] - [-q] [-v] [--force] [--yes] + [-g ] [-s ] [-A] [--cleanin] + [--cleanout] [-c] [-F] [-k] [-m] [-M] [-n] [-N] [-t] + [--debug] [-l ] [-L] [-P] [-q] [-v] [--force] + [--yes] """ import sys import logging from autotagical.settings import AutotagicalSettings from autotagical import __version__ as version -from autotagical.file_handler import AutotagicalFileHandler, move_files +from autotagical.file_handler import AutotagicalFileHandler, \ + move_files, clean_folder from autotagical.moving import determine_destination from autotagical.naming import AutotagicalNamer @@ -85,3 +87,7 @@ if __name__ == '__main__': # Actually move files move_files(files_out, SETTINGS) + + # Clean up if told to + for folder in SETTINGS.clean_folders: + clean_folder(folder, SETTINGS.trial_run) diff --git a/tests/file_handler b/tests/file_handler index db87ec7..f3264b7 100644 --- a/tests/file_handler +++ b/tests/file_handler @@ -23,7 +23,7 @@ Set up a basic tag groups object. >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -44,12 +44,8 @@ Set up a basic tag groups object. ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" +... "/G|American Styles", +... "/G|Belgian Styles" ... ] ... }, ... { @@ -63,6 +59,7 @@ Set up a basic tag groups object. ... ] ... }) True +>>> test_groups.process_groups() Test Schema ----------- @@ -72,7 +69,7 @@ Set up test schemas. >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", @@ -248,8 +245,8 @@ Test Tag Patterns ----------------- >>> import re ->>> test_tag_patterns = [{'tag_pattern' : re.compile(pattern['tag_pattern']), -... 'tag_split_pattern' : re.compile(pattern['tag_split_pattern'])} for pattern in test_schema.tag_formats] +>>> test_tag_patterns = [{'tag_pattern': re.compile(pattern['tag_pattern']), +... 'tag_split_pattern': re.compile(pattern['tag_split_pattern'])} for pattern in test_schema.tag_formats] Test Ignore Patterns -------------------- diff --git a/tests/files/bad_test_schema.json b/tests/files/bad_test_schema.json index c85dfbf..57f79bb 100644 --- a/tests/files/bad_test_schema.json +++ b/tests/files/bad_test_schema.json @@ -1 +1 @@ -{"file_type":"bad","schema_file_version":"1.0","tag_formats":[{"tag_pattern":"(?P.+)(?P\\[(?P.+?)\\])(?P.*?)","tag_split_pattern":"\\s+"},{"tag_pattern":"","tag_split_pattern":""}],"unnamed_patterns":["Test[0-9]{4}\\.txt","also this\\.txt"],"renaming_schemas":[{"filter":["/G|nogroup"],"format_string":"empty"},{"filter":["/G|American Styles","/G|Belgian Styles"],"format_string":"Beer Bottle/ITER| /#|/EITER|"}],"movement_schema":[{"filter":["/G|Beer"],"subfolder":"Beer Bottles","sublevels":[{"filter":["/G|Belgian Styles"],"subfolder":"","sublevels":[]},{"filter":["dipa"],"subfolder":"DIPAs","sublevels":[{"filter":["motueka"],"subfolder":"Motueka","sublevels":[]}]}]},{"filter":["/G|Whisky"],"subfolder":"Whisky Bottles","sublevels":[{"filter":["highland","islay"],"subfolder":"Scotch Bottles","sublevels":[{"filter":["islay"],"subfolder":"Islay","sublevels":[]}]},{"filter":["bourbon"],"subfolder":"Bourbon Bottles","sublevels":[]}]}]} +{"file_type":"bad","schema_file_version":"1.1","tag_formats":[{"tag_pattern":"(?P.+)(?P\\[(?P.+?)\\])(?P.*?)","tag_split_pattern":"\\s+"},{"tag_pattern":"","tag_split_pattern":""}],"unnamed_patterns":["Test[0-9]{4}\\.txt","also this\\.txt"],"renaming_schemas":[{"filter":["/G|nogroup"],"format_string":"empty"},{"filter":["/G|American Styles","/G|Belgian Styles"],"format_string":"Beer Bottle/ITER| /#|/EITER|"}],"movement_schema":[{"filter":["/G|Beer"],"subfolder":"Beer Bottles","sublevels":[{"filter":["/G|Belgian Styles"],"subfolder":"","sublevels":[]},{"filter":["dipa"],"subfolder":"DIPAs","sublevels":[{"filter":["motueka"],"subfolder":"Motueka","sublevels":[]}]}]},{"filter":["/G|Whisky"],"subfolder":"Whisky Bottles","sublevels":[{"filter":["highland","islay"],"subfolder":"Scotch Bottles","sublevels":[{"filter":["islay"],"subfolder":"Islay","sublevels":[]}]},{"filter":["bourbon"],"subfolder":"Bourbon Bottles","sublevels":[]}]}]} diff --git a/tests/files/bad_test_tag_groups.json b/tests/files/bad_test_tag_groups.json index e5ab9de..40a27ca 100644 --- a/tests/files/bad_test_tag_groups.json +++ b/tests/files/bad_test_tag_groups.json @@ -1 +1 @@ -{"tag_group_file_version":"1.0","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]},{"name":"Belgian Styles","tags":["witbier","dubbel","tripel"]},{"name":"Beer","tags":["ipa","dipa","pale_ale","witbier","dubbel","tripel"]},{"name":"Whisky","tags":["bourbon","rye","scotch"]}]} +{"tag_group_file_version":"1.1","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]},{"name":"Belgian Styles","tags":["witbier","dubbel","tripel"]},{"name":"Beer","tags":["/G|American Styles","/G|Belgian Styles"]},{"name":"Whisky","tags":["bourbon","rye","scotch"]}]} diff --git a/tests/files/test_schema.json b/tests/files/test_schema.json index ca57acc..c4c1e92 100644 --- a/tests/files/test_schema.json +++ b/tests/files/test_schema.json @@ -1 +1 @@ -{"file_type":"autotagical_schema","schema_file_version":"1.0","tag_formats":[{"tag_pattern":"(?P.+)(?P\\[(?P.+?)\\])(?P.*?)","tag_split_pattern":"\\s+"},{"tag_pattern":"","tag_split_pattern":""}],"unnamed_patterns":["Test[0-9]{4}\\.txt","also this\\.txt"],"renaming_schemas":[{"filter":["/G|nogroup"],"format_string":"empty"},{"filter":["/G|American Styles","/G|Belgian Styles"],"format_string":"Beer Bottle/ITER| /#|/EITER|"}],"movement_schema":[{"filter":["/G|Beer"],"subfolder":"Beer Bottles","sublevels":[{"filter":["/G|Belgian Styles"],"subfolder":"","sublevels":[]},{"filter":["dipa"],"subfolder":"DIPAs","sublevels":[{"filter":["motueka"],"subfolder":"Motueka","sublevels":[]}]}]},{"filter":["/G|Whisky"],"subfolder":"Whisky Bottles","sublevels":[{"filter":["highland","islay"],"subfolder":"Scotch Bottles","sublevels":[{"filter":["islay"],"subfolder":"Islay","sublevels":[]}]},{"filter":["bourbon"],"subfolder":"Bourbon Bottles","sublevels":[]}]}]} +{"file_type":"autotagical_schema","schema_file_version":"1.1","tag_formats":[{"tag_pattern":"(?P.+)(?P\\[(?P.+?)\\])(?P.*?)","tag_split_pattern":"\\s+"},{"tag_pattern":"","tag_split_pattern":""}],"unnamed_patterns":["Test[0-9]{4}\\.txt","also this\\.txt"],"renaming_schemas":[{"filter":["/G|nogroup"],"format_string":"empty"},{"filter":["/G|American Styles","/G|Belgian Styles"],"format_string":"Beer Bottle/ITER| /#|/EITER|"}],"movement_schema":[{"filter":["/G|Beer"],"subfolder":"Beer Bottles","sublevels":[{"filter":["/G|Belgian Styles"],"subfolder":"","sublevels":[]},{"filter":["dipa"],"subfolder":"DIPAs","sublevels":[{"filter":["motueka"],"subfolder":"Motueka","sublevels":[]}]}]},{"filter":["/G|Whisky"],"subfolder":"Whisky Bottles","sublevels":[{"filter":["highland","islay"],"subfolder":"Scotch Bottles","sublevels":[{"filter":["islay"],"subfolder":"Islay","sublevels":[]}]},{"filter":["bourbon"],"subfolder":"Bourbon Bottles","sublevels":[]}]}]} diff --git a/tests/files/test_tag_groups.json b/tests/files/test_tag_groups.json index 749dc9c..cff3221 100644 --- a/tests/files/test_tag_groups.json +++ b/tests/files/test_tag_groups.json @@ -1 +1 @@ -{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.0","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]},{"name":"Belgian Styles","tags":["witbier","dubbel","tripel"]},{"name":"Beer","tags":["ipa","dipa","pale_ale","witbier","dubbel","tripel"]},{"name":"Whisky","tags":["bourbon","rye","scotch"]}]} +{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.1","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]},{"name":"Belgian Styles","tags":["witbier","dubbel","tripel"]},{"name":"Beer","tags":["/G|American Styles","/G|Belgian Styles"]},{"name":"Whisky","tags":["bourbon","rye","scotch"]}]} diff --git a/tests/filtering b/tests/filtering index 6aa908a..e152b8c 100644 --- a/tests/filtering +++ b/tests/filtering @@ -22,7 +22,7 @@ Set up a basic tag groups object. >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -43,12 +43,8 @@ Set up a basic tag groups object. ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" +... "/G|American Styles", +... "/G|Belgian Styles" ... ] ... }, ... { @@ -62,6 +58,7 @@ Set up a basic tag groups object. ... ] ... }) True +>>> test_groups.process_groups() Test Tags --------- diff --git a/tests/groups b/tests/groups index a7c0135..1e8a183 100644 --- a/tests/groups +++ b/tests/groups @@ -29,10 +29,10 @@ As the constructor loads JSON schema files, it needs to be tested. This inciden >>> test_groups = AutotagicalGroups() -AutotagicalGroups.get_tags_in_group(tag_group) -================================================= +AutotagicalGroups.tag_in_group(self, tag_array, group) +====================================================== -Returns all tags in a specified tag group. +Determines if a tag array matches the specified group and returns the first matching tag. Base Case --------- @@ -40,7 +40,7 @@ Base Case >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -61,12 +61,8 @@ Base Case ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" +... "/G|American Styles", +... "/G|Belgian Styles" ... ] ... }, ... { @@ -80,8 +76,17 @@ Base Case ... ] ... }) True ->>> sorted(test_groups.get_tags_in_group('Belgian Styles')) -['dubbel', 'tripel', 'witbier'] +>>> test_groups.process_groups() +>>> test_groups.tag_in_group(['tripel'], 'Belgian Styles') +'tripel' +>>> test_groups.tag_in_group(['dipa', 'tripel'], 'Belgian Styles') +'tripel' +>>> test_groups.tag_in_group(['dipa', 'witbier', 'tripel'], 'Belgian Styles') +'witbier' +>>> test_groups.tag_in_group(['dipa', 'tripel', 'witbier'], 'Belgian Styles') +'tripel' +>>> test_groups.tag_in_group(['dipa', 'ctz'], 'Belgian Styles') +'' Wrong Cases ----------- @@ -92,7 +97,7 @@ Wrong Cases >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -113,9 +118,63 @@ Wrong Cases ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", +... "/G|American Styles", +... "/G|Belgian Styles" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> test_groups.tag_in_group(['dipa'], 'Sours') +'' +>>> test_groups.tag_in_group([], 'Whisky') +'' +>>> test_groups.tag_in_group([], 'Sours') +'' + +Regex Groups +------------ +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Account Number", +... "tags": [ +... "/RE|(?:xx)[0-9]{4}", +... "/RE|(?:\\*\\*)[0-9]{4}", +... "acct", +... "acct2" +... ] +... }, +... { +... "name": "More Accounts", +... "tags": [ +... "/G|Account Number", +... "acct3" +... ] +... }, +... { +... "name": "Even More Accounts", +... "tags": [ +... "/G|More Accounts", +... "acct4" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ ... "witbier", ... "dubbel", ... "tripel" @@ -132,8 +191,31 @@ Wrong Cases ... ] ... }) True ->>> test_groups.get_tags_in_group('Sours') -set() +>>> test_groups.process_groups() +>>> test_groups.tag_in_group([], 'Whisky') +'' +>>> test_groups.tag_in_group(['whisk', 'rye', 'scotch'], 'Whisky') +'rye' +>>> test_groups.tag_in_group(['whisk', 'rye', 'scotch'], 'Account Number') +'' +>>> test_groups.tag_in_group(['whisk', 'xx0117', 'scotch'], 'Account Number') +'xx0117' +>>> test_groups.tag_in_group(['whisk', 'xx0117', '**5324'], 'Account Number') +'xx0117' +>>> test_groups.tag_in_group(['whisk', '**5324', 'xx0117'], 'Account Number') +'**5324' +>>> test_groups.tag_in_group(['whisk', 'xx0117', 'scotch'], 'Even More Accounts') +'xx0117' +>>> test_groups.tag_in_group(['whisk', 'xx0117', '**5324'], 'Even More Accounts') +'xx0117' +>>> test_groups.tag_in_group(['whisk', '**5324', 'xx0117'], 'Even More Accounts') +'**5324' +>>> test_groups.tag_in_group(['whisk', 'acct3', '**5324', 'xx0117'], 'Even More Accounts') +'acct3' +>>> test_groups.tag_in_group(['whisk', '**5324', 'acct3', 'xx0117'], 'Even More Accounts') +'**5324' +>>> test_groups.tag_in_group(['whisk', 'acct3'], 'Even More Accounts') +'acct3' AutotagicalGroups.load_autotagical_format(json_input, append=False) ======================================================================= @@ -149,10 +231,11 @@ Base Cases >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- -----End Tag Groups----- @@ -163,7 +246,7 @@ True >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -182,17 +265,6 @@ True ... ] ... }, ... { -... "name": "Beer", -... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" -... ] -... }, -... { ... "name": "Whisky", ... "tags": [ ... "bourbon", @@ -203,10 +275,10 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] - Beer: ['dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] Belgian Styles: ['dubbel', 'tripel', 'witbier'] Whisky: ['bourbon', 'rye', 'scotch'] -----End Tag Groups----- @@ -222,7 +294,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -235,6 +307,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -242,7 +315,7 @@ True >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "Whisky", @@ -259,6 +332,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['american_light_lager'] @@ -271,7 +345,7 @@ True >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -284,6 +358,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -291,7 +366,7 @@ True >>> test_groups.load_autotagical_format( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "Whisky", @@ -308,6 +383,7 @@ True ... ] ... }, True) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['american_light_lager', 'dipa', 'ipa', 'pale_ale'] @@ -353,6 +429,511 @@ False ... }) False +Inheritance Tests +----------------- + +* Simple case + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "American Styles", +... "tags": [ +... "ipa", +... "dipa", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Beer", +... "tags": [ +... "/G|American Styles", +... "witbier" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Beer: ['dipa', 'ipa', 'pale_ale', 'witbier'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Multiple inheritance + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "American Styles", +... "tags": [ +... "ipa", +... "dipa", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Beer", +... "tags": [ +... "/G|American Styles", +... "/G|Belgian Styles", +... "english_ipa" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Beer: ['dipa', 'dubbel', 'english_ipa', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Multilevel inheritance + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Hoppy Styles", +... "tags": [ +... "ipa", +... "dipa" +... ] +... }, +... { +... "name": "American Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Beer", +... "tags": [ +... "/G|American Styles", +... "/G|Belgian Styles" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Beer: ['dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Hoppy Styles: ['dipa', 'ipa'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Flexible ordering + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Beer", +... "tags": [ +... "/G|American Styles", +... "/G|Belgian Styles" +... ] +... }, +... { +... "name": "American Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... }, +... { +... "name": "Hoppy Styles", +... "tags": [ +... "ipa", +... "dipa" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Beer: ['dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Hoppy Styles: ['dipa', 'ipa'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Diamond problem + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Beer", +... "tags": [ +... "/G|American Styles", +... "/G|Refrigerated Styles", +... "/G|Belgian Styles" +... ] +... }, +... { +... "name": "American Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... }, +... { +... "name": "Refrigerated Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "american_light_lager" +... ] +... }, +... { +... "name": "Hoppy Styles", +... "tags": [ +... "ipa", +... "dipa" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Beer: ['american_light_lager', 'dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Hoppy Styles: ['dipa', 'ipa'] + Refrigerated Styles: ['american_light_lager', 'dipa', 'ipa'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Circular inheritance + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Good Beer", +... "tags": [ +... "/G|American Styles", +... "/G|Belgian Styles", +... "/G|Refrigerated Styles" +... ] +... }, +... { +... "name": "American Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "pale_ale" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... }, +... { +... "name": "Refrigerated Styles", +... "tags": [ +... "/G|Hoppy Styles", +... "american_light_lager" +... ] +... }, +... { +... "name": "Hoppy Styles", +... "tags": [ +... "ipa", +... "dipa", +... "/G|Good Beer", +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + American Styles: ['dipa', 'ipa', 'pale_ale'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Good Beer: ['american_light_lager', 'dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Hoppy Styles: ['american_light_lager', 'dipa', 'dubbel', 'ipa', 'pale_ale', 'tripel', 'witbier'] + Refrigerated Styles: ['american_light_lager', 'dipa', 'ipa'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +Regex Tag Groups +---------------- + +* "Pure" Regex group loading + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Account Number", +... "tags": [ +... "/RE|(?:xx|\\*\\*)[0-9]{4}" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + Account Number: ['/RE|(?:xx|\\*\\*)[0-9]{4}'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Mixed group loading + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Account Number", +... "tags": [ +... "/RE|(?:xx)[0-9]{4}", +... "/RE|(?:\\*\\*)[0-9]{4}", +... "acct", +... "acct2" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + Account Number: ['acct', 'acct2', '/RE|(?:\\*\\*)[0-9]{4}', '/RE|(?:xx)[0-9]{4}'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + +* Inheritance of regex loading + +>>> test_groups = AutotagicalGroups() +>>> test_groups.load_autotagical_format( +... { +... "file_type": "autotagical_tag_groups", +... "tag_group_file_version": "1.1", +... "tag_groups": [ +... { +... "name": "Account Number", +... "tags": [ +... "/RE|(?:xx)[0-9]{4}", +... "/RE|(?:\\*\\*)[0-9]{4}", +... "acct", +... "acct2" +... ] +... }, +... { +... "name": "More Accounts", +... "tags": [ +... "/G|Account Number", +... "acct3" +... ] +... }, +... { +... "name": "Even More Accounts", +... "tags": [ +... "/G|More Accounts", +... "acct4" +... ] +... }, +... { +... "name": "Belgian Styles", +... "tags": [ +... "witbier", +... "dubbel", +... "tripel" +... ] +... }, +... { +... "name": "Whisky", +... "tags": [ +... "bourbon", +... "rye", +... "scotch" +... ] +... } +... ] +... }) +True +>>> test_groups.process_groups() +>>> print(test_groups) +-----Tag Groups---- + Account Number: ['acct', 'acct2', '/RE|(?:\\*\\*)[0-9]{4}', '/RE|(?:xx)[0-9]{4}'] + Belgian Styles: ['dubbel', 'tripel', 'witbier'] + Even More Accounts: ['acct', 'acct2', 'acct3', 'acct4', '/RE|(?:\\*\\*)[0-9]{4}', '/RE|(?:xx)[0-9]{4}'] + More Accounts: ['acct', 'acct2', 'acct3', '/RE|(?:\\*\\*)[0-9]{4}', '/RE|(?:xx)[0-9]{4}'] + Whisky: ['bourbon', 'rye', 'scotch'] +-----End Tag Groups----- + AutotagicalGroups.load_tagspaces_format(json_input, append=False) ======================================================================= @@ -372,6 +953,7 @@ Base Cases ... "tagGroups": [] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- -----End Tag Groups----- @@ -619,6 +1201,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -658,6 +1241,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -685,6 +1269,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- Whisky: ['bourbon', 'rye', 'scotch'] @@ -716,6 +1301,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -743,6 +1329,7 @@ True ... ] ... }, True) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -763,6 +1350,7 @@ Incorrect Cases ... "tagGroups": [] ... }) True +>>> test_groups.process_groups() * Out of date settings version @@ -775,6 +1363,7 @@ True ... "tagGroups": [] ... }) True +>>> test_groups.process_groups() Bad Case -------- @@ -807,7 +1396,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -820,6 +1409,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -827,7 +1417,7 @@ True >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "Whisky", @@ -840,6 +1430,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- Whisky: ['bourbon', 'rye', 'scotch'] @@ -851,7 +1442,7 @@ True >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -864,6 +1455,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -871,7 +1463,7 @@ True >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "Whisky", @@ -884,6 +1476,7 @@ True ... ] ... }, True) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -897,18 +1490,19 @@ Multiply defined tag groups should work (but warn using *logging.warning*). >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "name" : "American Styles", -... "tags" : ["dipa", "ipa", "pale_ale"] +... "name": "American Styles", +... "tags": ["dipa", "ipa", "pale_ale"] ... }, ... { -... "name" : "American Styles", -... "tags" : ["american_light_lager"] +... "name": "American Styles", +... "tags": ["american_light_lager"] ... }] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['american_light_lager', 'dipa', 'ipa', 'pale_ale'] @@ -933,11 +1527,11 @@ False >>> test_groups.load_tag_groups({ ... "file_type": "wrong", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "name" : "American Styles", -... "tags" : ["dipa", "ipa", "pale_ale"] +... "name": "American Styles", +... "tags": ["dipa", "ipa", "pale_ale"] ... }] ... }) False @@ -947,10 +1541,10 @@ False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", ... "tag_group_file_version": "0.9", -... "tag_groups" : [ +... "tag_groups": [ ... { -... "name" : "American Styles", -... "tags" : ["dipa", "ipa", "pale_ale"] +... "name": "American Styles", +... "tags": ["dipa", "ipa", "pale_ale"] ... }] ... }) False @@ -958,10 +1552,10 @@ False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", ... "tag_group_file_version": "99.9", -... "tag_groups" : [ +... "tag_groups": [ ... { -... "name" : "American Styles", -... "tags" : ["dipa", "ipa", "pale_ale"] +... "name": "American Styles", +... "tags": ["dipa", "ipa", "pale_ale"] ... }] ... }) False @@ -970,50 +1564,50 @@ False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : "string" +... "tag_group_file_version": "1.1", +... "tag_groups": "string" ... }) False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [] +... "tag_group_file_version": "1.1", +... "tag_groups": [] ... }) False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "wrong_key" : "is wrong", +... "wrong_key": "is wrong", ... }] ... }) False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "name" : "American Styles", +... "name": "American Styles", ... }] ... }) False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "tags" : ["dipa", "ipa", "pale_ale"] +... "tags": ["dipa", "ipa", "pale_ale"] ... }] ... }) False >>> test_groups.load_tag_groups({ ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", -... "tag_groups" : [ +... "tag_group_file_version": "1.1", +... "tag_groups": [ ... { -... "name" : "American Styles", -... "tags" : [1, 2] +... "name": "American Styles", +... "tags": [1, 2] ... }] ... }) False @@ -1276,6 +1870,7 @@ False ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1315,6 +1910,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1342,6 +1938,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- Whisky: ['bourbon', 'rye', 'scotch'] @@ -1373,6 +1970,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1401,6 +1999,7 @@ True ... ] ... }, True) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1436,6 +2035,7 @@ Incorrect Cases ... ] ... }) True +>>> test_groups.process_groups() * Out of date settings version @@ -1463,6 +2063,7 @@ True ... ] ... }) True +>>> test_groups.process_groups() * Error-handling for key errors ensures it will actually fail if an updated format changes things. @@ -1484,14 +2085,16 @@ Loads tag groups from JSON data in a string. This validates against known schem * Without append >>> test_groups = AutotagicalGroups() ->>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.0","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]}]}') +>>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.1","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]}]}') True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] -----End Tag Groups----- ->>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.0","tag_groups":[{"name":"Whisky","tags":["bourbon","rye","scotch"]}]}') +>>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.1","tag_groups":[{"name":"Whisky","tags":["bourbon","rye","scotch"]}]}') True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- Whisky: ['bourbon', 'rye', 'scotch'] @@ -1500,14 +2103,16 @@ True * With append >>> test_groups = AutotagicalGroups() ->>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.0","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]}]}') +>>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.1","tag_groups":[{"name":"American Styles","tags":["ipa","dipa","pale_ale"]}]}') True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] -----End Tag Groups----- ->>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.0","tag_groups":[{"name":"Whisky","tags":["bourbon","rye","scotch"]}]}', True) +>>> test_groups.load_tag_groups_from_string('{"file_type":"autotagical_tag_groups","tag_group_file_version":"1.1","tag_groups":[{"name":"Whisky","tags":["bourbon","rye","scotch"]}]}', True) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1527,6 +2132,7 @@ Normal Usage >>> test_groups = AutotagicalGroups() >>> test_groups.load_tag_groups_from_file(os.path.join(os.path.dirname(sys.path[0]), 'tests', 'files', 'test_tag_groups.json')) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] @@ -1540,6 +2146,7 @@ True >>> test_groups = AutotagicalGroups() >>> test_groups.load_tag_groups_from_file(os.path.join(os.path.dirname(sys.path[0]), 'tests', 'files', 'test_tagspaces_tag_library.json')) True +>>> test_groups.process_groups() >>> print(test_groups) -----Tag Groups---- American Styles: ['dipa', 'ipa', 'pale_ale'] diff --git a/tests/moving b/tests/moving index 32bf88c..2d6490f 100644 --- a/tests/moving +++ b/tests/moving @@ -1,6 +1,6 @@ -============================ +================== autotagical.moving -============================ +================== Setup ===== @@ -23,7 +23,7 @@ Set up a basic tag groups object. >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -44,12 +44,8 @@ Set up a basic tag groups object. ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" +... "/G|American Styles", +... "/G|Belgian Styles" ... ] ... }, ... { @@ -63,17 +59,7 @@ Set up a basic tag groups object. ... ] ... }) True - -Test Tags ---------- -Set up basic tags. - ->>> test_tags_1 = ['dipa', 'ale', 'refrigerated', 'simcoe', 'ctz', 'centennial'] ->>> test_tags_2 = ['pale_ale', 'ale', 'refrigerated', 'cascade'] ->>> test_tags_3 = ['tripel', 'ale', 'refrigerated'] ->>> test_tags_4 = ['scotch', 'laphroaig', 'islay'] ->>> test_tags_5 = ['bourbon', '23_year', 'pappys'] ->>> test_tags_6 = ['non-alcoholic', 'refrigerated'] +>>> test_groups.process_groups() Test Files ---------- @@ -128,6 +114,14 @@ Test Files ... tag_array=['non-alcoholic', 'refrigerated'] ... ) >>> files = [test_file_1, test_file_2, test_file_3, test_file_4, test_file_5, test_file_6] +>>> test_file_empty = AutotagicalFile( +... name='', +... raw_name='', +... original_path='', +... extension='', +... tags='', +... tag_array=[] +... ) Test Filters ------------ @@ -135,21 +129,21 @@ Set up filters for testing. >>> filter_1 = { ... "filter": ["/G|Beer"], -... "subfolder" : "Beer Bottles", -... "sublevels" : [ +... "subfolder": "Beer Bottles", +... "sublevels": [ ... { ... "filter": ["/G|Belgian Styles"], -... "subfolder" : "", -... "sublevels" : [] +... "subfolder": "", +... "sublevels": [] ... }, ... { ... "filter": ["dipa"], -... "subfolder" : "DIPAs", -... "sublevels" : [ +... "subfolder": "DIPAs", +... "sublevels": [ ... { ... "filter": ["motueka"], -... "subfolder" : "Motueka", -... "sublevels" : [] +... "subfolder": "Motueka", +... "sublevels": [] ... } ... ] ... } @@ -157,92 +151,92 @@ Set up filters for testing. ... } >>> filter_2 = { ... "filter": ["/G|Whisky"], -... "subfolder" : "Whisky Bottles", -... "sublevels" : [ +... "subfolder": "Whisky Bottles", +... "sublevels": [ ... { ... "filter": ["highland", "islay"], -... "subfolder" : "Scotch Bottles", -... "sublevels" : [ +... "subfolder": "Scotch Bottles", +... "sublevels": [ ... { ... "filter": ["islay"], -... "subfolder" : "Islay", -... "sublevels" : [] +... "subfolder": "Islay", +... "sublevels": [] ... } ... ] ... }, ... { ... "filter": ["bourbon"], -... "subfolder" : "Bourbon Bottles", -... "sublevels" : [] +... "subfolder": "Bourbon Bottles", +... "sublevels": [] ... } ... ] ... } >>> filter_3 = { ... "filter": ["/G|Beer"], -... "subfolder" : "Beers", -... "sublevels" : [ +... "subfolder": "Beers", +... "sublevels": [ ... { ... "filter": ["/G|Belgian Styles"], -... "subfolder" : "Belgians", -... "sublevels" : [] +... "subfolder": "Belgians", +... "sublevels": [] ... }, ... { ... "filter": ["tripel"], -... "subfolder" : "Tripels", -... "sublevels" : [ +... "subfolder": "Tripels", +... "sublevels": [ ... { ... "filter": ["american"], -... "subfolder" : "", -... "sublevels" : [] +... "subfolder": "", +... "sublevels": [] ... }, ... ] ... }, ... { ... "filter": ["dipa"], -... "subfolder" : "DIPAs", -... "sublevels" : [] +... "subfolder": "DIPAs", +... "sublevels": [] ... } ... ] ... } >>> filter_4 = { ... "filter": ["/G|Beer"], -... "subfolder" : "Beers", -... "sublevels" : [ +... "subfolder": "Beers", +... "sublevels": [ ... { ... "filter": ["tripel"], -... "subfolder" : "Tripels", -... "sublevels" : [ +... "subfolder": "Tripels", +... "sublevels": [ ... { ... "filter": ["american"], -... "subfolder" : "", -... "sublevels" : [] +... "subfolder": "", +... "sublevels": [] ... }, ... ] ... }, ... { ... "filter": ["/G|Belgian Styles"], -... "subfolder" : "Belgians", -... "sublevels" : [] +... "subfolder": "Belgians", +... "sublevels": [] ... }, ... { ... "filter": ["/G|American Styles"], -... "subfolder" : "American Styles", -... "sublevels" : [ +... "subfolder": "American Styles", +... "sublevels": [ ... { ... "filter": ["american_light_lager"], -... "subfolder" : "ALL", -... "sublevels" : [] +... "subfolder": "ALL", +... "sublevels": [] ... } ... ] ... }, ... { ... "filter": ["dipa"], -... "subfolder" : "DIPA Styles", -... "sublevels" : [ +... "subfolder": "DIPA Styles", +... "sublevels": [ ... { ... "filter": ["motueka"], -... "subfolder" : "Motueka", -... "sublevels" : [] +... "subfolder": "Motueka", +... "sublevels": [] ... } ... ] ... } @@ -250,37 +244,37 @@ Set up filters for testing. ... } >>> filter_5 = { ... "filter": ["/G|Beer"], -... "subfolder" : "Beers", -... "sublevels" : [ +... "subfolder": "Beers", +... "sublevels": [ ... { ... "filter": ["tripel"], -... "subfolder" : "Tripels", -... "sublevels" : [] +... "subfolder": "Tripels", +... "sublevels": [] ... }, ... { ... "filter": ["/G|Belgian Styles"], -... "subfolder" : "Belgians", -... "sublevels" : [] +... "subfolder": "Belgians", +... "sublevels": [] ... }, ... { ... "filter": ["dipa"], -... "subfolder" : "DIPA Styles", -... "sublevels" : [ +... "subfolder": "DIPA Styles", +... "sublevels": [ ... { ... "filter": ["motueka"], -... "subfolder" : "Motueka", -... "sublevels" : [] +... "subfolder": "Motueka", +... "sublevels": [] ... } ... ] ... }, ... { ... "filter": ["/G|American Styles"], -... "subfolder" : "American Styles", -... "sublevels" : [ +... "subfolder": "American Styles", +... "sublevels": [ ... { ... "filter": ["american_light_lager"], -... "subfolder" : "ALL", -... "sublevels" : [] +... "subfolder": "ALL", +... "sublevels": [] ... } ... ] ... } @@ -288,37 +282,37 @@ Set up filters for testing. ... } >>> filter_6 = { ... "filter": ["/G|Beer"], -... "subfolder" : "Beers", -... "sublevels" : [ +... "subfolder": "Beers", +... "sublevels": [ ... { ... "filter": ["/G|Belgian Styles"], -... "subfolder" : "Belgians", -... "sublevels" : [] +... "subfolder": "Belgians", +... "sublevels": [] ... }, ... { ... "filter": ["tripel"], -... "subfolder" : "Tripels", -... "sublevels" : [] +... "subfolder": "Tripels", +... "sublevels": [] ... }, ... { ... "filter": ["dipa"], -... "subfolder" : "DIPA Styles", -... "sublevels" : [ +... "subfolder": "DIPA Styles", +... "sublevels": [ ... { ... "filter": ["motueka"], -... "subfolder" : "Motueka", -... "sublevels" : [] +... "subfolder": "Motueka", +... "sublevels": [] ... } ... ] ... }, ... { ... "filter": ["/G|American Styles"], -... "subfolder" : "American Styles", -... "sublevels" : [ +... "subfolder": "American Styles", +... "sublevels": [ ... { ... "filter": ["american_light_lager"], -... "subfolder" : "ALL", -... "sublevels" : [] +... "subfolder": "ALL", +... "sublevels": [] ... } ... ] ... } @@ -340,44 +334,42 @@ Edge Cases * Empty Filter (should throw an exception) ->>> process_filter_level(test_tags_1, {"filter": [""], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_1, {"filter": [""], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> process_filter_level(test_tags_1, {"filter": [], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_1, {"filter": [], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! * Empty Tags ->>> process_filter_level([], filter_1, test_groups) +>>> process_filter_level(test_file_empty, filter_1, test_groups) (False, '') * Both Empty ->>> process_filter_level([], {"filter": [""], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_empty, {"filter": [""], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> process_filter_level([], {"filter": [], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_empty, {"filter": [], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! ->>> process_filter_level([""], {"filter": [""], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_empty, {"filter": [""], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> process_filter_level([""], {"filter": [], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_empty, {"filter": [], "subfolder": "", "sublevels": []}, test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! * Empty Tags with All (/\*|) operator ->>> process_filter_level([], {"filter": ["/*|"], "subfolder" : "", "sublevels" : []}, test_groups) -(True, '') ->>> process_filter_level([""], {"filter": ["/*|"], "subfolder" : "", "sublevels" : []}, test_groups) +>>> process_filter_level(test_file_empty, {"filter": ["/*|"], "subfolder": "", "sublevels": []}, test_groups) (True, '') Base Cases @@ -385,44 +377,123 @@ Base Cases * No matches at all ->>> process_filter_level(test_tags_1, filter_2, test_groups) +>>> process_filter_level(test_file_1, filter_2, test_groups) (False, '') ->>> process_filter_level(test_tags_2, filter_2, test_groups) +>>> process_filter_level(test_file_2, filter_2, test_groups) (False, '') ->>> process_filter_level(test_tags_3, filter_2, test_groups) +>>> process_filter_level(test_file_3, filter_2, test_groups) (False, '') ->>> process_filter_level(test_tags_4, filter_1, test_groups) +>>> process_filter_level(test_file_4, filter_1, test_groups) (False, '') ->>> process_filter_level(test_tags_4, filter_3, test_groups) +>>> process_filter_level(test_file_4, filter_3, test_groups) (False, '') ->>> process_filter_level(test_tags_5, filter_1, test_groups) +>>> process_filter_level(test_file_5, filter_1, test_groups) (False, '') ->>> process_filter_level(test_tags_5, filter_3, test_groups) +>>> process_filter_level(test_file_5, filter_3, test_groups) (False, '') ->>> process_filter_level(test_tags_6, filter_1, test_groups) +>>> process_filter_level(test_file_6, filter_1, test_groups) (False, '') ->>> process_filter_level(test_tags_6, filter_2, test_groups) +>>> process_filter_level(test_file_6, filter_2, test_groups) (False, '') ->>> process_filter_level(test_tags_6, filter_3, test_groups) +>>> process_filter_level(test_file_6, filter_3, test_groups) (False, '') * Fully specified without subfolder ->>> process_filter_level(test_tags_3, filter_1, test_groups) == (True, os.path.join('Beer Bottles')) +>>> process_filter_level(test_file_3, filter_1, test_groups) == (True, os.path.join('Beer Bottles')) True * Fully specified with subfolder ->>> process_filter_level(test_tags_4, filter_2, test_groups) == (True, os.path.join('Whisky Bottles', 'Scotch Bottles', 'Islay')) +>>> process_filter_level(test_file_4, filter_2, test_groups) == (True, os.path.join('Whisky Bottles', 'Scotch Bottles', 'Islay')) True ->>> process_filter_level(test_tags_5, filter_2, test_groups) == (True, os.path.join('Whisky Bottles', 'Bourbon Bottles')) +>>> process_filter_level(test_file_5, filter_2, test_groups) == (True, os.path.join('Whisky Bottles', 'Bourbon Bottles')) True * Not fully specified ->>> process_filter_level(test_tags_1, filter_1, test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) +>>> process_filter_level(test_file_1, filter_1, test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) +True +>>> process_filter_level(test_file_2, filter_1, test_groups) == (False, os.path.join('Beer Bottles')) +True + +Format Strings +-------------- + +* /FILE| operator + +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "A /FILE| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /FILE| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A Test1999 operator.', 'Another Test1999 operator.')) +True + +* /?TIG| operator + +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "A /?TIG|American Styles/| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /?TIG|Whisky/| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A dipa operator.', 'Another operator.')) +True + +* /EXT| operator + +>>> process_filter_level(test_file_2, {"filter": ["/*|"], "subfolder": "A /EXT| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /EXT| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A .txt operator.', 'Another .txt operator.')) +True + +* /TAGS| operator + +>>> process_filter_level(test_file_2, {"filter": ["/*|"], "subfolder": "A /TAGS| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /TAGS| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A [pale_ale ale refrigerated cascade] operator.', 'Another [pale_ale ale refrigerated cascade] operator.')) +True + +* Tag conditionals + +>>> process_filter_level(test_file_4, {"filter": ["/*|"], "subfolder": "A /?T|scotch/| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /?T|dipa/| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A scotch operator.', 'Another operator.')) +True + +* Group conditionals + +>>> process_filter_level(test_file_5, {"filter": ["/*|"], "subfolder": "A /?G|Whisky/| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /?G|American Styles/| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A Whisky operator.', 'Another operator.')) +True + +* Conditional operator + +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "A /?|scotch/T|true/F|false/E?| operator.", "sublevels": [{"filter": ["/*|"], "subfolder": "Another /?|dipa/T|true/F|false/E?| operator.", "sublevels": []}]}, test_groups) == (True, os.path.join('A false operator.', 'Another true operator.')) +True + +* Complex conditions + +>>> complex_filter = '/G|American Styles/&|/!|dipa' +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|y/F|n/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a n conditional operator.')) +True + +>>> process_filter_level(test_file_2, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|y/F|n/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a y conditional operator.')) +True + +>>> process_filter_level(test_file_4, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|y/F|n/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a n conditional operator.')) +True + +* Combination of conditionals + +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "This is a /?G|American Styles/| group and /?T|dipa/| tag operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a American Styles group and dipa tag operator.')) True ->>> process_filter_level(test_tags_2, filter_1, test_groups) == (False, os.path.join('Beer Bottles')) + +>>> process_filter_level(test_file_2, {"filter": ["/*|"], "subfolder": "This is a /?G|American Styles/| group and /?T|dipa/| tag operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a American Styles group and tag operator.')) +True + +>>> process_filter_level(test_file_4, {"filter": ["/*|"], "subfolder": "This is a /?G|American Styles/| group and /?T|dipa/| tag operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a group and tag operator.')) +True + + +* Complex Combinations + +>>> complex_filter = '/G|American Styles/&|/!|dipa' +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|/TAGS|/F|/EXT|/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a .txt conditional operator.')) +True + +>>> process_filter_level(test_file_2, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|/EXT|/F|/FILE|/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a .txt conditional operator.')) +True + +>>> process_filter_level(test_file_4, {"filter": ["/*|"], "subfolder": "This is a /?|" + complex_filter + "/T|/TAGS|/F|/EXT|/E?| conditional operator.", "sublevels": []}, test_groups) == (True, os.path.join('This is a .txt conditional operator.')) +True + +* Bad Case + +>>> process_filter_level(test_file_1, {"filter": ["/*|"], "subfolder": "This is a bad /ITER|bad /#| bad/EITER| format string.", "sublevels": []}, test_groups) == (True, os.path.join('This is a bad format string.')) True Filter Priority @@ -432,23 +503,23 @@ Confirm filter priority works * Full match over partial ->>> process_filter_level(test_tags_3, filter_3, test_groups) == (True, os.path.join('Beers', 'Belgians')) +>>> process_filter_level(test_file_3, filter_3, test_groups) == (True, os.path.join('Beers', 'Belgians')) True ->>> process_filter_level(test_tags_3, filter_4, test_groups) == (True, os.path.join('Beers', 'Belgians')) +>>> process_filter_level(test_file_3, filter_4, test_groups) == (True, os.path.join('Beers', 'Belgians')) True * For full matches ->>> process_filter_level(test_tags_3, filter_5, test_groups) == (True, os.path.join('Beers', 'Tripels')) +>>> process_filter_level(test_file_3, filter_5, test_groups) == (True, os.path.join('Beers', 'Tripels')) True ->>> process_filter_level(test_tags_3, filter_6, test_groups) == (True, os.path.join('Beers', 'Belgians')) +>>> process_filter_level(test_file_3, filter_6, test_groups) == (True, os.path.join('Beers', 'Belgians')) True * For partial matches ->>> process_filter_level(test_tags_1, filter_4, test_groups) == (False, os.path.join('Beers', 'American Styles')) +>>> process_filter_level(test_file_1, filter_4, test_groups) == (False, os.path.join('Beers', 'American Styles')) True ->>> process_filter_level(test_tags_1, filter_5, test_groups) == (False, os.path.join('Beers', 'DIPA Styles')) +>>> process_filter_level(test_file_1, filter_5, test_groups) == (False, os.path.join('Beers', 'DIPA Styles')) True generate_path(tag_array, movement_schema, tag_groups) @@ -461,43 +532,43 @@ Edge Cases * Empty Filter (should throw an exception) ->>> generate_path(test_tags_1, [{"filter": [""], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_1, [{"filter": [""], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> generate_path(test_tags_1, [{"filter": [], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_1, [{"filter": [], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! * Empty Tags ->>> generate_path([], [filter_1, filter_2], test_groups) +>>> generate_path(test_file_empty, [filter_1, filter_2], test_groups) (True, '') * Both Empty ->>> generate_path([], [{"filter": [""], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": [""], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> generate_path([], [{"filter": [], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": [], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! ->>> generate_path([""], [{"filter": [""], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": [""], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed condition set encountered: Completely empty! ->>> generate_path([""], [{"filter": [], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": [], "subfolder": "", "sublevels": []}], test_groups) Traceback (most recent call last): ... autotagical.filtering.FilterError: Malformed filter encountered: Completely empty! ->>> generate_path([""], [], test_groups) +>>> generate_path(test_file_empty, [], test_groups) Traceback (most recent call last): ... autotagical.schema.SchemaError: Completely empty movement schema! ->>> generate_path([], [], test_groups) +>>> generate_path(test_file_empty, [], test_groups) Traceback (most recent call last): ... autotagical.schema.SchemaError: Completely empty movement schema! @@ -505,9 +576,9 @@ autotagical.schema.SchemaError: Completely empty movement schema! * Empty Tags with All (/\*|) operator ->>> generate_path([], [{"filter": ["/*|"], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": ["/*|"], "subfolder": "", "sublevels": []}], test_groups) (False, '') ->>> generate_path([""], [{"filter": ["/*|"], "subfolder" : "", "sublevels" : []}], test_groups) +>>> generate_path(test_file_empty, [{"filter": ["/*|"], "subfolder": "", "sublevels": []}], test_groups) (False, '') Base Cases @@ -515,38 +586,38 @@ Base Cases * No matches at all ->>> generate_path(test_tags_1, [filter_2], test_groups) +>>> generate_path(test_file_1, [filter_2], test_groups) (True, '') ->>> generate_path(test_tags_2, [filter_2], test_groups) +>>> generate_path(test_file_2, [filter_2], test_groups) (True, '') ->>> generate_path(test_tags_3, [filter_2], test_groups) +>>> generate_path(test_file_3, [filter_2], test_groups) (True, '') ->>> generate_path(test_tags_4, [filter_1], test_groups) +>>> generate_path(test_file_4, [filter_1], test_groups) (True, '') ->>> generate_path(test_tags_5, [filter_1], test_groups) +>>> generate_path(test_file_5, [filter_1], test_groups) (True, '') ->>> generate_path(test_tags_6, [filter_1], test_groups) +>>> generate_path(test_file_6, [filter_1], test_groups) (True, '') ->>> generate_path(test_tags_6, [filter_2], test_groups) +>>> generate_path(test_file_6, [filter_2], test_groups) (True, '') * Fully specified without subfolder ->>> generate_path(test_tags_3, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles')) +>>> generate_path(test_file_3, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles')) True * Fully specified with subfolder ->>> generate_path(test_tags_4, [filter_1, filter_2], test_groups) == (False, os.path.join('Whisky Bottles', 'Scotch Bottles', 'Islay')) +>>> generate_path(test_file_4, [filter_1, filter_2], test_groups) == (False, os.path.join('Whisky Bottles', 'Scotch Bottles', 'Islay')) True ->>> generate_path(test_tags_5, [filter_1, filter_2], test_groups) == (False, os.path.join('Whisky Bottles', 'Bourbon Bottles')) +>>> generate_path(test_file_5, [filter_1, filter_2], test_groups) == (False, os.path.join('Whisky Bottles', 'Bourbon Bottles')) True * Not fully specified ->>> generate_path(test_tags_1, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) +>>> generate_path(test_file_1, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) True ->>> generate_path(test_tags_2, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles')) +>>> generate_path(test_file_2, [filter_1, filter_2], test_groups) == (False, os.path.join('Beer Bottles')) True Filter Priority @@ -556,23 +627,23 @@ Confirm filter priority works * Full match over partial ->>> generate_path(test_tags_1, [filter_1, filter_2, filter_3], test_groups) == (False, os.path.join('Beers', 'DIPAs')) +>>> generate_path(test_file_1, [filter_1, filter_2, filter_3], test_groups) == (False, os.path.join('Beers', 'DIPAs')) True ->>> generate_path(test_tags_1, [filter_3, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'DIPAs')) +>>> generate_path(test_file_1, [filter_3, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'DIPAs')) True * For full matches ->>> generate_path(test_tags_3, [filter_1, filter_2, filter_3], test_groups) == (False, os.path.join('Beer Bottles')) +>>> generate_path(test_file_3, [filter_1, filter_2, filter_3], test_groups) == (False, os.path.join('Beer Bottles')) True ->>> generate_path(test_tags_3, [filter_3, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'Belgians')) +>>> generate_path(test_file_3, [filter_3, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'Belgians')) True * For partial matches ->>> generate_path(test_tags_1, [filter_1, filter_2, filter_4], test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) +>>> generate_path(test_file_1, [filter_1, filter_2, filter_4], test_groups) == (False, os.path.join('Beer Bottles', 'DIPAs')) True ->>> generate_path(test_tags_1, [filter_4, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'American Styles')) +>>> generate_path(test_file_1, [filter_4, filter_2, filter_1], test_groups) == (False, os.path.join('Beers', 'American Styles')) True determine_destination(file_list, movement_schema, tag_groups) diff --git a/tests/naming b/tests/naming index 511cf31..eb31a8f 100644 --- a/tests/naming +++ b/tests/naming @@ -23,7 +23,7 @@ Set up a basic tag groups object. >>> test_groups.load_tag_groups( ... { ... "file_type": "autotagical_tag_groups", -... "tag_group_file_version": "1.0", +... "tag_group_file_version": "1.1", ... "tag_groups": [ ... { ... "name": "American Styles", @@ -44,12 +44,8 @@ Set up a basic tag groups object. ... { ... "name": "Beer", ... "tags": [ -... "ipa", -... "dipa", -... "pale_ale", -... "witbier", -... "dubbel", -... "tripel" +... "/G|American Styles", +... "/G|Belgian Styles" ... ] ... }, ... { @@ -63,6 +59,7 @@ Set up a basic tag groups object. ... ] ... }) True +>>> test_groups.process_groups() Test Schema ----------- @@ -72,7 +69,7 @@ Set up test schemas. >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", @@ -171,7 +168,7 @@ True >>> empty_schema = AutotagicalSchema() >>> empty_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": [], @@ -605,6 +602,14 @@ Conditionals ... test_file_3, test_groups) 'This is a group and tag operator.' +* Tag In Group operator + +>>> substitute_operators('This is a /?TIG|American Styles/| operator.', test_file_1, test_groups) +'This is a dipa operator.' +>>> substitute_operators('This is a /?TIG|American Styles/| operator.', test_file_2, test_groups) +'This is a pale_ale operator.' +>>> substitute_operators('This is a /?TIG|American Styles/| operator.', test_file_3, test_groups) +'This is a operator.' Complex Combinations -------------------- @@ -616,19 +621,12 @@ Complex Combinations 'This is a Test1532 conditional operator.' >>> substitute_operators('This is a /?|' + complex_filter + '/T|/FILE|/F|/TAGS|/E?| conditional operator.', test_file_4, test_groups) 'This is a [scotch laphroaig islay] conditional operator.' - -Bad Cases ---------- - -* Bad operators (throws a warning but doesn't fail) - ->>> substitute_operators('This is a string with a / messed up operator. /FILE|', test_file_1, test_groups) -'This is a string with a / messed up operator. Test1999' - -* Nested conditionals (not supported) - ->>> substitute_operators('So /?|tag/T|nested/?|tag2/T|conditionals/F|are/E?|bad/F|but/E?| do not fail./FILE|', test_file_1, test_groups) -'So arebad/F|but/E?| do not fail.Test1999' +>>> substitute_operators('This is a /?|' + complex_filter + '/T|/?TIG|Whisky/|/F|/?TIG|American Styles/|/E?| conditional operator.', test_file_1, test_groups) +'This is a dipa conditional operator.' +>>> substitute_operators('This is a /?|' + complex_filter + '/T|/?TIG|Whisky/|/F|/?TIG|American Styles/|/E?| conditional operator.', test_file_2, test_groups) +'This is a conditional operator.' +>>> substitute_operators('This is a /?|' + complex_filter + '/T|/?TIG|American Styles/|/F|/?TIG|Whisky/|/E?| conditional operator.', test_file_4, test_groups) +'This is a scotch conditional operator.' AutotagicalNamer(renaming_schema, unnamed_patterns) ================================================================= diff --git a/tests/schema b/tests/schema index 051ddc8..25ced21 100644 --- a/tests/schema +++ b/tests/schema @@ -63,7 +63,7 @@ Base Cases >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": [], @@ -77,7 +77,7 @@ True >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", @@ -232,7 +232,7 @@ With the append parameter set to True, we shouldn't overwrite the old tags. >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", @@ -314,7 +314,7 @@ True -----End Schema----- >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "", @@ -402,7 +402,7 @@ True >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "(?P.+)(?P\\[(?P.+?)\\])(?P.*?)", @@ -484,7 +484,7 @@ True -----End Schema----- >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [ ... { ... "tag_pattern": "", @@ -654,7 +654,7 @@ False >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "wrong", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": [], @@ -667,7 +667,7 @@ False >>> test_schema = AutotagicalSchema() >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": [] @@ -675,7 +675,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "movement_schema": [] @@ -683,7 +683,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "renaming_schemas": [], ... "movement_schema": [] @@ -691,7 +691,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "unnamed_patterns": [" "], ... "renaming_schemas": [], ... "movement_schema": [] @@ -699,7 +699,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": [], @@ -708,7 +708,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": [" "], ... "renaming_schemas": "string", @@ -717,7 +717,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": [{"tag_pattern": " ", "tag_split_pattern": " "}], ... "unnamed_patterns": "string", ... "renaming_schemas": [], @@ -726,7 +726,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": ["string"], ... "unnamed_patterns": [" "], ... "renaming_schemas": [], @@ -735,7 +735,7 @@ False False >>> test_schema.load_schema({ ... "file_type": "autotagical_schema", -... "schema_file_version": "1.0", +... "schema_file_version": "1.1", ... "tag_formats": "string", ... "unnamed_patterns": [" "], ... "renaming_schemas": [],