Experimental Results

Amilcare has been tested on a number of corpora. Here are the results on two well known corpora, i.e. the CMU seminar announcement proposed by Freitag and the Job Postings proposed by M. E. Califf. For results on teh Pascal challenge corpus, see here.

Parameters used for Seminar Announcements and Job Postings

Parameter	Value
Pattern Length	4
Consider EOL	YES
Use Ontology	NO
MinMatches for Tagging Rules	1
Error Threshold	0.95

Results for Seminar Announcements

(Cross-validation on 5 random 50/50 splits of the data)

Slot	Partial Matches			Exact Matches
Slot	P	R	F 1	P	R	F 1
Speaker	90.38	87.95	89.11	86.64	84.33	85.44
Location	91.78	75.52	82.83	85.51	70.37	77.18
Stime	96.07	93.52	94.77	94.59	92.08	93.32
Etime	98.13	96.92	97.52	97.11	95.91	96.50
Macro-average	94.09	88.48	91.06	90.96	85.67	88.11
Micro-average	93.83	88.38	90.90	90.70	85.56	87.94

Results for Job Postings

(Cross-validation on 5 random 50/50 splits of the data)

Slot	Partial Matches			Exact Matches
Slot	P	R	F 1	P	R	F 1
id	99.20	99.47	99.33	98.80	99.07	98.93
title	73.08	49.48	58.89	56.75	38.38	45.71
company	85.09	80.85	82.73	80.09	76.07	77.86
salary	88.10	68.92	77.21	80.13	62.71	70.24
recruiter	88.02	77.53	82.39	83.52	73.50	78.15
state	95.30	99.11	97.16	94.78	98.57	96.63
city	95.74	98.56	97.13	93.79	96.55	95.15
country	98.22	98.72	98.47	98.16	98.66	98.41
language	84.01	82.03	82.98	76.65	74.83	75.70
platform	77.90	74.23	75.93	69.18	65.89	67.41
application	82.36	83.23	82.74	76.29	77.07	76.64
area	69.29	54.21	60.78	59.23	46.33	51.95
req_yrs_exp	85.10	86.90	85.84	82.40	84.15	83.12
des_yrs_exp	89.76	93.55	91.51	87.28	90.96	88.97
req_degree	93.83	87.92	90.76	92.88	87.02	89.84
des_degree	89.39	51.17	64.55	85.64	48.67	61.55
post_date	97.98	100.00	98.97	97.98	100.00	98.97
Macro-average	87.79	81.52	83.96	83.15	77.55	79.72
Micro-average	84.64	79.55	81.67	78.65	74.37	76.15

Last updated: November 24, 2002