NER model is not training at all

[toadle]

Hey guys,

I'm new to spaCy and would like the train a NER model to recognize manufacturer-pids in product-names.
I have a training dataset of 100k products and their labeled entities. An Example would look like this:

{"text":"SMC AW30-F03DE4-18RZA-B Filterregler","spans":[{"start":4,"end":23,"label":"manufacturer_pid","text":"AW30-F03DE4-18RZA-B"},{"start":0,"end":3,"label":"manufacturer_name","text":"SMC"}]}

I have written a convert.py that converts this JSONL-data into a .spacy file like this:

filename = sys.argv[1]
targetFile = sys.argv[2]

lines = read_jsonl_file(filename)

nlp = spacy.blank("en") # load a new spacy model
db = DocBin() # create a DocBin object

count = 0
for line in tqdm(lines):
  text = line['text']
  doc = nlp.make_doc(text)
  ents = []

  print(doc)

  for entity in line['spans']:
    start = int(entity['start'])
    end = int(entity['end'])
    label = entity['label'].upper()
    span_text = entity['text'] 

    span = doc.char_span(start, end, label=label)
    print(span, label, start, end, span_text)
    
    if span is None:
      msg = f"Skipping entity [{start}, {end}, {label}] in the following text because the character span '{doc.text[start:end]}' does not align with token boundaries:{repr(text)}"
      print(msg)
      continue
    else:
      print("Adding entity")
      count += 1
      ents.append(span)

    db.add(doc)

print(db)

db.to_disk(targetFile)

print(count)

I successfully get a train.spacy file from this.

My config.cfg looks like this, after generating it with spacy init config -p ner -G configs/config.cfg -F:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
vectors = {"@vectors":"spacy.Vectors.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

When I try and train this model with python -m spacy train configs/config.cfg --output training/ --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --training.eval_frequency 100 --training.patience 1000 --gpu-id ${vars.gpu_id} I only get this output:

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00      0.00    0.00    0.00    0.00    0.00
  0     100          0.00      0.00    0.00    0.00    0.00    0.00
  0     200          0.00      0.00    0.00    0.00    0.00    0.00
  0     300          0.00      0.00    0.00    0.00    0.00    0.00
  0     400          0.00      0.00    0.00    0.00    0.00    0.00
  0     500          0.00      0.00    0.00    0.00    0.00    0.00
  0     600          0.00      0.00    0.00    0.00    0.00    0.00
  0     700          0.00      0.00    0.00    0.00    0.00    0.00
  0     800          0.00      0.00    0.00    0.00    0.00    0.00
  0     900          0.00      0.00    0.00    0.00    0.00    0.00
  1    1000          0.00      0.00    0.00    0.00    0.00    0.00
✔ Saved pipeline to output directory
training/model-last

So it does not do anything useful and I'm lost on the "why?".
I thought I set-up everything according to https://github.com/explosion/projects/tree/v3/pipelines/ner_demo , but that seems to be wrong.

Any help would be greatly appreciated!

[svlandeg]

Hm, odd. What is the output if you run the command debug data with this same config file and .spacy files?

[toadle]

The output looks like this:

spacy debug data ./configs/config.cfg --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --training.eval_frequency 100 --training.patience 1000

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, ner
12490 training docs
127 evaluation docs
✔ No overlap between training and evaluation data

============================== Vocab & Vectors ==============================
ℹ 140480 total word(s) in the data (28164 unique)
ℹ No word vectors present in the package

========================== Named Entity Recognition ==========================
ℹ 0 label(s)
0 missing value(s) (tokens with '-' label)
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
✔ No entities consisting of or starting/ending with whitespace
✔ No entities crossing sentence boundaries

================================== Summary ==================================
✔ 7 checks passed

I suppose the 0 label(s) is not right?

[toadle]

@svlandeg Thanks for the quick reply btw!

[toadle]

OK, well - that was stupid.
I actually just missing adding the entities to the doc. 🤦🏼‍♂️

Now it works:

python -m spacy train configs/config.cfg --output training/ --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy
ℹ Saving to output directory: training
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     49.64    0.00    0.00    0.00    0.00
  0     200        225.93   3545.72   69.75   74.77   65.35    0.70
  0     400        159.60   1798.19   79.22   78.91   79.53    0.79
  0     600        167.25   1941.50   75.80   90.22   65.35    0.76
  1     800        182.67   1876.76   81.60   82.93   80.31    0.82
  1    1000        234.85   1600.80   81.54   79.70   83.46    0.82
  1    1200        332.96   1898.73   79.34   83.48   75.59    0.79
  2    1400        299.54   1305.89   83.05   89.91   77.17    0.83
  3    1600        348.71   1182.99   81.93   83.61   80.31    0.82
  4    1800        384.46    996.23   82.05   89.72   75.59    0.82
  5    2000        458.76    869.51   83.13   87.07   79.53    0.83
  6    2200        460.60    762.19   85.36   91.07   80.31    0.85
  8    2400        443.90    683.28   85.47   93.46   78.74    0.85
 10    2600        501.44    603.37   84.55   87.39   81.89    0.85
 12    2800        444.59    563.61   86.44   93.58   80.31    0.86
 14    3000        462.38    502.08   84.17   89.38   79.53    0.84
 15    3200        437.74    457.85   82.64   86.96   78.74    0.83
 17    3400        397.54    362.20   82.85   88.39   77.95    0.83
 19    3600        484.39    405.32   80.69   88.68   74.02    0.81
 21    3800        540.85    408.67   82.20   88.99   76.38    0.82
 22    4000        421.54    329.42   82.55   89.81   76.38    0.83
 24    4200        584.13    383.31   81.36   88.07   75.59    0.81
 26    4400        506.31    323.87   82.30   86.21   78.74    0.82

[svlandeg]

Thanks for reporting back, and happy you were able to find the problem! The debug CLI command is your friend ;-)