Performances (résultat)
Copy reference -
Report abuse
- View options
- Truncate descriptions
Dataset principal
@alam Après avoir lancé Tenet sur le corpus principal et corrigé les fichiers ODRL obtenus pour qu'ils soient lisibles, les résultats obtenus sont les suivants :
############ STATS ############
Number of modalities:
odrl:permission: 37
odrl:obligation: 36
odrl:prohibition: 27
Number of actions:
odrl:sell: 7
odrl:play: 15
odrl:derive: 3
cc:Distribution: 22
odrl:run: 7
odrl:use: 8
odrl:display: 19
odrl:modify: 11
cc:Reproduction: 9
odrl:copy: 9
Number of and: 13
Number of not: 45
############ SCORES ###########
# Modality scores:
permission:
TP = 28, FP = 5, FN = 9
Precision = 0.8484848484848485, Recall = 0.7567567567567568
obligation:
TP = 18, FP = 0, FN = 18
Precision = 1.0, Recall = 0.5
prohibition:
TP = 18, FP = 13, FN = 9
Precision = 0.5806451612903226, Recall = 0.6666666666666666
Total precision = 0.7804878048780488
Total recall = 0.64
# Actions scores:
http://www.w3.org/ns/odrl/2/sell:
TP = 6, FP = 0, FN = 1
Precision = 1.0, Recall = 0.8571428571428571
http://www.w3.org/ns/odrl/2/play:
TP = 11, FP = 0, FN = 4
Precision = 1.0, Recall = 0.7333333333333333
http://www.w3.org/ns/odrl/2/derive:
TP = 3, FP = 1, FN = 0
Precision = 0.75, Recall = 1.0
http://creativecommons.org/ns#Distribution:
TP = 19, FP = 0, FN = 3
Precision = 1.0, Recall = 0.8636363636363636
http://www.w3.org/ns/odrl/2/run:
TP = 5, FP = 0, FN = 2
Precision = 1.0, Recall = 0.7142857142857143
http://www.w3.org/ns/odrl/2/use:
TP = 8, FP = 0, FN = 0
Precision = 1.0, Recall = 1.0
http://www.w3.org/ns/odrl/2/modify:
TP = 8, FP = 0, FN = 3
Precision = 1.0, Recall = 0.7272727272727273
http://www.w3.org/ns/odrl/2/display:
TP = 16, FP = 0, FN = 3
Precision = 1.0, Recall = 0.8421052631578947
http://creativecommons.org/ns#Reproduction:
TP = 5, FP = 0, FN = 4
Precision = 1.0, Recall = 0.5555555555555556
http://www.w3.org/ns/odrl/2/copy:
TP = 9, FP = 0, FN = 0
Precision = 1.0, Recall = 1.0
http://creativecommons.org/ns#Derivation:
TP = 0, FP = 0, FN = 1
Precision = None, Recall = 0.0
Total precision = 0.989010989010989
Total recall = 0.8108108108108109
# Global scores:
global:
TP = 63, FP = 0, FN = 37
Precision = 1.0, Recall = 0.63
Total precision = 1.0
Total recall = 0.63
Je ne sais pas à quels résultats tu t'attendais mais personnellement je trouve ça un peu décevant pour les modalités. Je soupçonne notamment AMRLib de parfois produire des AMR incorrects lorsqu'il y a des négations.
Dataset annoté (issu de vraies licences)
Sur le dataset des 10 phrases issues de licences, Tenet s'en sort plutôt très bien pour les modalités et plutôt très mal pour les actions. Pour les actions il a tendance à utiliser des URI du namespace rdf:
(que j'ai aussi ajouté en préfixe pour que les graphes soient corrects).
# Modality scores:
permission:
TP = 2, FP = 1, FN = 3
Precision = 0.6666666666666666, Recall = 0.4
obligation:
TP = 4, FP = 0, FN = 2
Precision = 1.0, Recall = 0.6666666666666666
prohibition:
TP = 1, FP = 0, FN = 1
Precision = 1.0, Recall = 0.5
Total precision = 0.875
Total recall = 0.5384615384615384
# Actions scores:
http://creativecommons.org/ns#DerivativeWorks:
TP = 0, FP = 0, FN = 3
Precision = None, Recall = 0.0
http://creativecommons.org/ns#Distribution:
TP = 0, FP = 0, FN = 4
Precision = None, Recall = 0.0
http://creativecommons.org/ns#Attribution:
TP = 0, FP = 0, FN = 2
Precision = None, Recall = 0.0
http://www.w3.org/1999/02/22-rdf-syntax-ns#make:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://www.w3.org/ns/odrl/2/run:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://www.w3.org/1999/02/22-rdf-syntax-ns#propagate:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://creativecommons.org/ns#Reproduction:
TP = 0, FP = 0, FN = 2
Precision = None, Recall = 0.0
http://creativecommons.org/ns#Copyleft:
TP = 0, FP = 0, FN = 1
Precision = None, Recall = 0.0
http://www.w3.org/1999/02/22-rdf-syntax-ns#right:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://www.w3.org/1999/02/22-rdf-syntax-ns#exercise:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://creativecommons.org/ns#CommercialUse:
TP = 0, FP = 0, FN = 2
Precision = None, Recall = 0.0
http://www.w3.org/1999/02/22-rdf-syntax-ns#credit:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://creativecommons.org/ns#Notice:
TP = 1, FP = 1, FN = 1
Precision = 0.5, Recall = 0.5
http://www.w3.org/ns/odrl/2/aggregate:
TP = 0, FP = 0, FN = 1
Precision = None, Recall = 0.0
http://www.w3.org/1999/02/22-rdf-syntax-ns#include:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://www.w3.org/1999/02/22-rdf-syntax-ns#license:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://creativecommons.org/ns#ShareAlike:
TP = 0, FP = 0, FN = 1
Precision = None, Recall = 0.0
http://www.w3.org/1999/02/22-rdf-syntax-ns#give:
TP = 0, FP = 1, FN = 0
Precision = 0.0, Recall = None
http://creativecommons.org/ns#SourceCode:
TP = 0, FP = 0, FN = 1
Precision = None, Recall = 0.0
Total precision = 0.09090909090909091
Total recall = 0.05263157894736842
# Global scores:
global:
TP = 0, FP = 0, FN = 10
Precision = None, Recall = 0.0
Total precision = None
Total recall = 0.0