Skip to content
Snippets Groups Projects
Closed Performances (résultat)
  • View options
  • Performances (résultat)

    • View options
  • Closed created by Ghost User

    Dataset principal

    @alam Après avoir lancé Tenet sur le corpus principal et corrigé les fichiers ODRL obtenus pour qu'ils soient lisibles, les résultats obtenus sont les suivants :

    ############ STATS ############
    
    Number of modalities:
    	odrl:permission: 37
    	odrl:obligation: 36
    	odrl:prohibition: 27
    Number of actions:
    	odrl:sell: 7
    	odrl:play: 15
    	odrl:derive: 3
    	cc:Distribution: 22
    	odrl:run: 7
    	odrl:use: 8
    	odrl:display: 19
    	odrl:modify: 11
    	cc:Reproduction: 9
    	odrl:copy: 9
    Number of and: 13
    Number of not: 45
    
    
    ############ SCORES ###########
    
    # Modality scores:
    permission:
    	TP = 28, FP = 5, FN = 9
    	Precision = 0.8484848484848485, Recall = 0.7567567567567568
    obligation:
    	TP = 18, FP = 0, FN = 18
    	Precision = 1.0, Recall = 0.5
    prohibition:
    	TP = 18, FP = 13, FN = 9
    	Precision = 0.5806451612903226, Recall = 0.6666666666666666
    Total precision = 0.7804878048780488
    Total recall = 0.64
    
    # Actions scores:
    http://www.w3.org/ns/odrl/2/sell:
    	TP = 6, FP = 0, FN = 1
    	Precision = 1.0, Recall = 0.8571428571428571
    http://www.w3.org/ns/odrl/2/play:
    	TP = 11, FP = 0, FN = 4
    	Precision = 1.0, Recall = 0.7333333333333333
    http://www.w3.org/ns/odrl/2/derive:
    	TP = 3, FP = 1, FN = 0
    	Precision = 0.75, Recall = 1.0
    http://creativecommons.org/ns#Distribution:
    	TP = 19, FP = 0, FN = 3
    	Precision = 1.0, Recall = 0.8636363636363636
    http://www.w3.org/ns/odrl/2/run:
    	TP = 5, FP = 0, FN = 2
    	Precision = 1.0, Recall = 0.7142857142857143
    http://www.w3.org/ns/odrl/2/use:
    	TP = 8, FP = 0, FN = 0
    	Precision = 1.0, Recall = 1.0
    http://www.w3.org/ns/odrl/2/modify:
    	TP = 8, FP = 0, FN = 3
    	Precision = 1.0, Recall = 0.7272727272727273
    http://www.w3.org/ns/odrl/2/display:
    	TP = 16, FP = 0, FN = 3
    	Precision = 1.0, Recall = 0.8421052631578947
    http://creativecommons.org/ns#Reproduction:
    	TP = 5, FP = 0, FN = 4
    	Precision = 1.0, Recall = 0.5555555555555556
    http://www.w3.org/ns/odrl/2/copy:
    	TP = 9, FP = 0, FN = 0
    	Precision = 1.0, Recall = 1.0
    http://creativecommons.org/ns#Derivation:
    	TP = 0, FP = 0, FN = 1
    	Precision = None, Recall = 0.0
    Total precision = 0.989010989010989
    Total recall = 0.8108108108108109
    
    # Global scores:
    global:
    	TP = 63, FP = 0, FN = 37
    	Precision = 1.0, Recall = 0.63
    Total precision = 1.0
    Total recall = 0.63

    Je ne sais pas à quels résultats tu t'attendais mais personnellement je trouve ça un peu décevant pour les modalités. Je soupçonne notamment AMRLib de parfois produire des AMR incorrects lorsqu'il y a des négations.

    Dataset annoté (issu de vraies licences)

    Sur le dataset des 10 phrases issues de licences, Tenet s'en sort plutôt très bien pour les modalités et plutôt très mal pour les actions. Pour les actions il a tendance à utiliser des URI du namespace rdf: (que j'ai aussi ajouté en préfixe pour que les graphes soient corrects).

    # Modality scores:
    permission:
    	TP = 2, FP = 1, FN = 3
    	Precision = 0.6666666666666666, Recall = 0.4
    obligation:
    	TP = 4, FP = 0, FN = 2
    	Precision = 1.0, Recall = 0.6666666666666666
    prohibition:
    	TP = 1, FP = 0, FN = 1
    	Precision = 1.0, Recall = 0.5
    Total precision = 0.875
    Total recall = 0.5384615384615384
    
    # Actions scores:
    http://creativecommons.org/ns#DerivativeWorks:
    	TP = 0, FP = 0, FN = 3
    	Precision = None, Recall = 0.0
    http://creativecommons.org/ns#Distribution:
    	TP = 0, FP = 0, FN = 4
    	Precision = None, Recall = 0.0
    http://creativecommons.org/ns#Attribution:
    	TP = 0, FP = 0, FN = 2
    	Precision = None, Recall = 0.0
    http://www.w3.org/1999/02/22-rdf-syntax-ns#make:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://www.w3.org/ns/odrl/2/run:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://www.w3.org/1999/02/22-rdf-syntax-ns#propagate:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://creativecommons.org/ns#Reproduction:
    	TP = 0, FP = 0, FN = 2
    	Precision = None, Recall = 0.0
    http://creativecommons.org/ns#Copyleft:
    	TP = 0, FP = 0, FN = 1
    	Precision = None, Recall = 0.0
    http://www.w3.org/1999/02/22-rdf-syntax-ns#right:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://www.w3.org/1999/02/22-rdf-syntax-ns#exercise:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://creativecommons.org/ns#CommercialUse:
    	TP = 0, FP = 0, FN = 2
    	Precision = None, Recall = 0.0
    http://www.w3.org/1999/02/22-rdf-syntax-ns#credit:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://creativecommons.org/ns#Notice:
    	TP = 1, FP = 1, FN = 1
    	Precision = 0.5, Recall = 0.5
    http://www.w3.org/ns/odrl/2/aggregate:
    	TP = 0, FP = 0, FN = 1
    	Precision = None, Recall = 0.0
    http://www.w3.org/1999/02/22-rdf-syntax-ns#include:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://www.w3.org/1999/02/22-rdf-syntax-ns#license:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://creativecommons.org/ns#ShareAlike:
    	TP = 0, FP = 0, FN = 1
    	Precision = None, Recall = 0.0
    http://www.w3.org/1999/02/22-rdf-syntax-ns#give:
    	TP = 0, FP = 1, FN = 0
    	Precision = 0.0, Recall = None
    http://creativecommons.org/ns#SourceCode:
    	TP = 0, FP = 0, FN = 1
    	Precision = None, Recall = 0.0
    Total precision = 0.09090909090909091
    Total recall = 0.05263157894736842
    
    # Global scores:
    global:
    	TP = 0, FP = 0, FN = 10
    	Precision = None, Recall = 0.0
    Total precision = None
    Total recall = 0.0
    Edited by Ghost User

    Activity

    • All activity
    • Comments only
    • History only
    • Newest first
    • Oldest first