gerar matriz de confusão para parsing

[leoalenc]

• criar script para gerar matriz de confusão para parsing
• gerar arquivo com o parsing do UDPipe 1.4 conforme o modelo de LAS mais baixo com toquenização ouro e etiquetas ouro do exp2 de melhorias diversas nos scripts de processamento dos resultados de parsing #874
• aplicar o script para gerar matriz de confusão
• melhorar gráficos
• implementar outro script para gerar outro relatório com os vários erros do parser
• subir esses relatórios para exp2
• construir script para gerar saída do parser para todas as outras nove execuções
• produzir versão do script deprel_confusion.py que leve em conta todos os erros de todas as 10 execuções
• produzir versão análoga de parser_diagnostics.py
• implementar script para extração de sentenças com mismatches , analogamente à função find_mismatches abaixo
• eliminar eventuais redundâncias nos scripts deprel_confusion.py e parser_diagnostics.py, i.e., arquivos de saída em comum

[leoalenc]

O modelo 6 no teste 6 produziu o resultado pior, ou seja, LAS: 83.08%:

gr LAS gold-tok-tags-results-*.txt
gold-tok-tags-results-10.txt:Parsing from gold tokenization with gold tags - forms: 2655, UAS: 89.60%, LAS: 86.21%
gold-tok-tags-results-1.txt:Parsing from gold tokenization with gold tags - forms: 2823, UAS: 88.10%, LAS: 83.32%
gold-tok-tags-results-2.txt:Parsing from gold tokenization with gold tags - forms: 2679, UAS: 89.25%, LAS: 86.34%
gold-tok-tags-results-3.txt:Parsing from gold tokenization with gold tags - forms: 2840, UAS: 87.39%, LAS: 83.91%
gold-tok-tags-results-4.txt:Parsing from gold tokenization with gold tags - forms: 2660, UAS: 87.82%, LAS: 84.25%
gold-tok-tags-results-5.txt:Parsing from gold tokenization with gold tags - forms: 2799, UAS: 88.14%, LAS: 84.17%
gold-tok-tags-results-6.txt:Parsing from gold tokenization with gold tags - forms: 2636, UAS: 86.57%, LAS: 83.08%
gold-tok-tags-results-7.txt:Parsing from gold tokenization with gold tags - forms: 2548, UAS: 88.85%, LAS: 84.34%
gold-tok-tags-results-8.txt:Parsing from gold tokenization with gold tags - forms: 2498, UAS: 87.83%, LAS: 83.99%
gold-tok-tags-results-9.txt:Parsing from gold tokenization with gold tags - forms: 2647, UAS: 87.61%, LAS: 83.30%

[leoalenc]

Para gerar o CoNNL-U do parser:

udpipe --parse model-6.output test-6.conllu > test-6.out.conllu

[leoalenc]

$ deprel_confusion.py test-6.conllu test-6.out.conllu
=== Confusion matrix for DEPREL ===

Predicted    acl:relcl  advcl  advmod  amod  appos  aux  case  ...  obj  obl  parataxis  punct  root  vocative  xcomp
Gold                                                           ...                                                   
acl                  0      2       0     0      0    0     0  ...    0    0          0      0     0         0      0
acl:relcl           25      0       0     0      0    0     0  ...    2    0          2      0     3         0      0
advcl                0     54       1     0      0    0     0  ...    0    1          3      0     6         0      0
advcl:relcl          0      0       0     0      0    0     0  ...    0    0          0      0     3         0      0
advmod               1      1     327     0      0    0     0  ...    0    0          1      0     1         0      0
amod                 0      0       0    15      0    0     0  ...    0    0          1      0     3         0      0
appos                0      0       0     0      0    0     0  ...    0    0          1      0     0         0      0
aux                  0      0       0     0      0   63     0  ...    0    0          0      0     0         0      0
case                 0      0       0     0      0    0   161  ...    0    0          0      0     0         0      0
cc                   0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
ccomp                0      3       3     0      0    0     0  ...    0    2          9      0     4         0      2
conj                 0      1       0     0      0    0     0  ...    0    0          6      0     0         0      0
cop                  0      0       0     0      0    2     0  ...    0    0          0      0     0         0      0
csubj                0      0       1     0      0    0     0  ...    0    0          0      0     0         0      0
det                  0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
discourse            0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
dislocated           0      0       0     0      0    0     0  ...    0    0          1      0     0         0      0
expl                 0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
fixed                0      0       0     0      0    0     0  ...    0    0          0      1     0         0      0
goeswith             0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
iobj                 0      0       0     0      0    0     0  ...    1    4          0      0     0         0      0
mark                 0      0       0     0      0    0     0  ...    0    0          0      0     0         0      0
nmod                 0      0       1     0      0    0     0  ...    1    6          0      0     0         0      0
nmod:poss            0      0       0     0      0    0     0  ...    6    0          0      0     0         0      0
nsubj                0      0       0     0      1    0     0  ...   18    0          0      0     3         0      0
nummod               0      0       0     0      0    0     0  ...    0    1          0      0     0         0      0
obj                  0      0       0     0      0    0     0  ...  148    1          0      0     0         0      0
obl                  0      0       0     0      0    0     1  ...    4  119          0      0     0         0      0
parataxis            0      0       3     1      0    0     0  ...    1    0         59      0     1         0      4
punct                0      0       0     0      0    0     0  ...    0    0          0    533     0         0      0
reparandum           0      0       0     0      0    0     0  ...    1    0          0      0     0         0      0
root                 0      3       3     1      0    0     0  ...    1    0         13      0   257         1      0
vocative             0      0       0     0      0    0     0  ...    4    0          0      0     1         8      0
xcomp                0      2       1     0      0    0     0  ...    2    0          5      0     1         0     11

[34 rows x 29 columns]

=== Most frequent confusions (gold -> predicted) ===

nsubj           -> obj             : 18
obj             -> nsubj           : 15
root            -> parataxis       : 13
ccomp           -> parataxis       : 9
nmod            -> obl             : 6
advcl           -> root            : 6
conj            -> parataxis       : 6
nmod:poss       -> obj             : 6
nsubj           -> nmod:poss       : 5
xcomp           -> parataxis       : 5
nmod:poss       -> nsubj           : 4
iobj            -> obl             : 4
parataxis       -> xcomp           : 4
vocative        -> obj             : 4
advcl           -> ccomp           : 4
xcomp           -> ccomp           : 4
obl             -> obj             : 4
ccomp           -> root            : 4
obj             -> nmod:poss       : 4
ccomp           -> advcl           : 3

Total comparable tokens: 2636
Correct DEPREL labels:   2391
Incorrect DEPREL labels: 245

Saved mismatch list to deprel_mismatches.tsv
Saved confusion matrix to deprel_confusion_matrix.tsv
Saved row-normalized confusion matrix to deprel_confusion_matrix_normalized.tsv

[leoalenc]

@dominickmaia , veja esta issue.

[leoalenc]

@dominickmaia , veja como extrair as sentenças com erros de deprel:

>>> gold="/home/leonel/complin/nheengatu/data/experiments/issue874/exp2/test-6.conllu"
>>> test="/home/leonel/complin/nheengatu/data/experiments/issue874/exp2/test-6.out.conllu"
>>> import Yauti
>>> gold_sents=Yauti.extractConlluSents(gold)
>>> test_sents=Yauti.extractConlluSents(test)
>>> def find_mismatches(gold_deprel,test_deprel):
	i=0
	while(i<len(gold_sents)):
		gold_sent=gold_sents[i]
		test_sent=test_sents[i]
		nsubjs=gold_sent.filter(deprel='nsubj')
		for gold_token in nsubjs:
			test_token=test_sent.filter(id=gold_token['id'])[0]
			if test_token['deprel'] == test_deprel:
				print(test_sent.metadata['sent_id'],gold_token['id'])
		i+=1

		
>>> find_mismatches('nsubj','obj')
MooreFP1994:0:0:16 10
MooreFP1994:0:0:27 4
Navarro2016:4:9:176 6
Avila2021:0:0:143 12
Avila2021:0:0:213 16
Avila2021:0:0:401 7
Avila2021:0:0:522 11
Avila2021:0:0:629 29
Casasnovas2006:3:7:31 11
Casasnovas2006:9:31:115 15
Casasnovas2006:11:19:150 2
Casasnovas2006:11:23:154 12
Casasnovas2006:12:7:173 19
Casasnovas2006:0:0:24 1
Amorim1928:6-1:145:145 11
Hartt1938:0:0:493 6
Aguiar1898:21-1:3:147 2
Aguiar1898:21-1:3:147 6
>>> 

[leoalenc]

@dominickmaia , veja aqui um exemplo do erro obj em vez de nsubj, que se refere ao nó 10:

# sent_id = MooreFP1994:0:0:16
# text = Ayururé se manha upitá arama yané rendawa upé se ratiwa uxari waá yandé arã.
# text_eng = I asked my mother to stay in our farm that my granpa left for us.
# text_por = Pedi para minha mãe ficar na nossa fazenda que meu avô deixou para nós.
# text_source = p. 108
# text_orig = a-yururé se-mẫỹã u-pitá arãma iane-rẽndá upé [se-ratíwa u-šári waʔá yãndé arã]S'Rel
# text_annotator = Leonel Figueiredo de Alencar
# reviewer1 = Hélio Leonam Barroso Silva
1	Ayururé	yururé	VERB	V	Mood=Ind|Number=Sing|Person=1|VerbForm=Fin	0	root	_	TokenRange=0:8
2	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	3	nmod:poss	_	TokenRange=9:11
3	manha	manha	NOUN	N	Number=Sing	1	obj	_	TokenRange=12:17
4	upitá	pitá	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	1	advcl	_	TokenRange=18:23
5	arama	arama	SCONJ	SCONJ	_	4	mark	_	TokenRange=24:29
6	yané	yané	PRON	PRON2	Case=Gen|Number=Plur|Person=1|Poss=Yes|PronType=Prs	7	nmod:poss	_	TokenRange=30:34
7	rendawa	tendawa	NOUN	N	Number=Sing|Rel=Cont	4	obl	_	TokenRange=35:42
8	upé	upé	ADP	ADP	AdpType=Post	7	case	_	TokenRange=43:46
9	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	10	nmod:poss	_	TokenRange=47:49
10	ratiwa	tatiwa	NOUN	N	Number=Sing|Rel=Cont	4	obj	_	TokenRange=50:56
11	uxari	xari	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	10	acl:relcl	_	TokenRange=57:62
12	waá	waá	PRON	REL	Number=Sing|PronType=Rel	11	obj	_	TokenRange=63:66
13	yandé	yandé	PRON	PRON	Case=Acc,Nom|Number=Plur|Person=1|PronType=Prs	11	iobj	_	TokenRange=67:72
14	arã	arã	ADP	ADP	AdpType=Post	13	case	_	SpaceAfter=No|TokenRange=73:76
15	.	.	PUNCT	PUNCT	_	1	punct	_	SpaceAfter=No|TokenRange=76:77

Compare com o arquivo ouro:

# sent_id = MooreFP1994:0:0:16
# text = Ayururé se manha upitá arama yané rendawa upé se ratiwa uxari waá yandé arã.
# text_eng = I asked my mother to stay in our farm that my granpa left for us.
# text_por = Pedi para minha mãe ficar na nossa fazenda que meu avô deixou para nós.
# text_source = p. 108
# text_orig = a-yururé se-mẫỹã u-pitá arãma iane-rẽndá upé [se-ratíwa u-šári waʔá yãndé arã]S'Rel
# text_annotator = Leonel Figueiredo de Alencar
# reviewer1 = Hélio Leonam Barroso Silva
1	Ayururé	yururé	VERB	V	Mood=Ind|Number=Sing|Person=1|VerbForm=Fin	0	root	_	TokenRange=0:8
2	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	3	nmod:poss	_	TokenRange=9:11
3	manha	manha	NOUN	N	Number=Sing	1	iobj	_	TokenRange=12:17
4	upitá	pitá	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	1	ccomp	_	TokenRange=18:23
5	arama	arama	SCONJ	SCONJ	_	4	mark	_	TokenRange=24:29
6	yané	yané	PRON	PRON2	Case=Gen|Number=Plur|Person=1|Poss=Yes|PronType=Prs	7	nmod:poss	_	TokenRange=30:34
7	rendawa	tendawa	NOUN	N	Number=Sing|Rel=Cont	4	obl	_	TokenRange=35:42
8	upé	upé	ADP	ADP	AdpType=Post	7	case	_	TokenRange=43:46
9	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	10	nmod:poss	_	TokenRange=47:49
10	ratiwa	tatiwa	NOUN	N	Number=Sing|Rel=Cont	11	nsubj	_	TokenRange=50:56
11	uxari	xari	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	7	acl:relcl	_	TokenRange=57:62
12	waá	waá	PRON	REL	Number=Sing|PronType=Rel	11	obj	_	TokenRange=63:66
13	yandé	yandé	PRON	PRON	Case=Acc,Nom|Number=Plur|Person=1|PronType=Prs	11	iobj	_	TokenRange=67:72
14	arã	arã	ADP	ADP	AdpType=Post	13	case	_	SpaceAfter=No|TokenRange=73:76
15	.	.	PUNCT	PUNCT	_	1	punct	_	SpaceAfter=No|TokenRange=76:77

[leoalenc]

@dominickmaia , veja esta versão melhorada da funçào, que exibe o comprimento das sentenças:

>>> def find_mismatches(gold_deprel,test_deprel):
	i=0
	while(i<len(gold_sents)):
		gold_sent=gold_sents[i]
		test_sent=test_sents[i]
		nsubjs=gold_sent.filter(deprel='nsubj')
		for gold_token in nsubjs:
			test_token=test_sent.filter(id=gold_token['id'])[0]
			if test_token['deprel'] == test_deprel:
				print(f"{test_sent.metadata['sent_id']},node: {gold_token['id']}, sentence length {len(gold_sent)}")
		i+=1

		
>>> find_mismatches('nsubj','obj')
MooreFP1994:0:0:16,node: 10, sentence length 15
MooreFP1994:0:0:27,node: 4, sentence length 8
Navarro2016:4:9:176,node: 6, sentence length 9
Avila2021:0:0:143,node: 12, sentence length 21
Avila2021:0:0:213,node: 16, sentence length 30
Avila2021:0:0:401,node: 7, sentence length 13
Avila2021:0:0:522,node: 11, sentence length 12
Avila2021:0:0:629,node: 29, sentence length 46
Casasnovas2006:3:7:31,node: 11, sentence length 22
Casasnovas2006:9:31:115,node: 15, sentence length 16
Casasnovas2006:11:19:150,node: 2, sentence length 19
Casasnovas2006:11:23:154,node: 12, sentence length 13
Casasnovas2006:12:7:173,node: 19, sentence length 25
Casasnovas2006:0:0:24,node: 1, sentence length 4
Amorim1928:6-1:145:145,node: 11, sentence length 23
Hartt1938:0:0:493,node: 6, sentence length 10
Aguiar1898:21-1:3:147,node: 2, sentence length 10
Aguiar1898:21-1:3:147,node: 6, sentence length 10
>>> 

Predominam sentenças mais longas, nas quais é mais comum termos mais de um verbo. Vejamos a única sentença curta no padrão-ouro:

# sent_id = Casasnovas2006:0:0:24
# text = Maã taá remunhã?
# text_eng = TODO
# text_por = O que estás fazendo?
# text_source = p. 19
# text_orig = Maá taá (mata) remunhã?
# text_annotator = Leonel Figueiredo de Alencar
# text_orig_transcriber = Juliana Lopes Gurgel
# text_por_modernizer = Juliana Lopes Gurgel
# inputline = Maã/int taá remunhã?
1	Maã	maã	PRON	INT	PronType=Int	3	nsubj	_	TokenRange=0:3
2	taá	taá	PART	CQ	PartType=Int	3	advmod	_	TokenRange=4:7
3	remunhã	munhã	VERB	V	Mood=Ind|Number=Sing|Person=2|VerbForm=Fin	0	root	_	SpaceAfter=No|TokenRange=8:15
4	?	?	PUNCT	PUNCT	_	3	punct	_	SpaceAfter=No|TokenRange=15:16

Este dado é interessantíssimo! O parser está certo e o padrão-ouro, errado!

[leoalenc]

@dominickmaia , neste exemplo, o parser analisou o sujeito de "dormir" como se fosse o objeto do verbo da subordinada adverbial. Talvez a especificação da valência pudesse evitar esse tipo de erro, uma vez que "chegar" é intransitivo e, como tal, não admite objeto direto:

# sent_id = MooreFP1994:0:0:27
# text = Asika ramé se mbira-itá ukiri ana uikú.
# text_eng = When I arrive, my children are already sleeping.
# text_por = Quando chego, meus filhos já estão dormindo.
# text_source = p. 110
# text_orig = [[a-síka ramẽ]VP]S' sẽ-mbíra-itá [u-kíri-ãna u-ikú]VP
# text_annotator = Leonel Figueiredo de Alencar
1	Asika	sika	VERB	V	Mood=Ind|Number=Sing|Person=1|VerbForm=Fin	5	advcl	_	TokenRange=0:5
2	ramé	ramé	SCONJ	SCONJ	_	1	mark	_	TokenRange=6:10
3	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	4	nmod:poss	_	TokenRange=11:13
4	mbira-itá	mbira	NOUN	N	Number=Plur	1	obj	_	TokenRange=14:23
5	ukiri	kiri	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	0	root	_	TokenRange=24:29
6	ana	ana	PART	PFV	Aspect=Perf	5	advmod	_	TokenRange=30:33
7	uikú	ikú	AUX	AUXFS	Mood=Ind|Person=3|VerbForm=Fin	5	aux	_	SpaceAfter=No|TokenRange=34:38
8	.	.	PUNCT	PUNCT	_	5	punct	_	SpaceAfter=No|TokenRange=38:39

A correta é esta análise:

# sent_id = MooreFP1994:0:0:27
# text = Asika ramé se mbira-itá ukiri ana uikú.
# text_eng = When I arrive, my children are already sleeping.
# text_por = Quando chego, meus filhos já estão dormindo.
# text_source = p. 110
# text_orig = [[a-síka ramẽ]VP]S' sẽ-mbíra-itá [u-kíri-ãna u-ikú]VP
# text_annotator = Leonel Figueiredo de Alencar
1	Asika	sika	VERB	V	Mood=Ind|Number=Sing|Person=1|VerbForm=Fin	5	advcl	_	TokenRange=0:5
2	ramé	ramé	SCONJ	SCONJ	_	1	mark	_	TokenRange=6:10
3	se	se	PRON	PRON2	Case=Gen|Number=Sing|Person=1|Poss=Yes|PronType=Prs	4	nmod:poss	_	TokenRange=11:13
4	mbira-itá	mbira	NOUN	N	Number=Plur	5	nsubj	_	TokenRange=14:23
5	ukiri	kiri	VERB	V	Mood=Ind|Person=3|VerbForm=Fin	0	root	_	TokenRange=24:29
6	ana	ana	PART	PFV	Aspect=Perf	5	advmod	_	TokenRange=30:33
7	uikú	ikú	AUX	AUXFS	Mood=Ind|Person=3|VerbForm=Fin	5	aux	_	SpaceAfter=No|TokenRange=34:38
8	.	.	PUNCT	PUNCT	_	5	punct	_	SpaceAfter=No|TokenRange=38:39