-
Notifications
You must be signed in to change notification settings - Fork 6
Expand file tree
/
Copy pathst2024.html
More file actions
227 lines (189 loc) · 10.4 KB
/
Copy pathst2024.html
File metadata and controls
227 lines (189 loc) · 10.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
<!DOCTYPE HTML>
<!--
Imagination by TEMPLATED
templated.co @templatedco
Released for free under the Creative Commons Attribution 3.0 license (templated.co/license)
-->
<html>
<head>
<title>SIGTYP -- 2024 Shared Task</title>
<link rel="shortcut icon" type="image/x-icon" href="images/favicon.ico" />
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta name="description" content="" />
<meta name="keywords" content="SIGTYP, Shared Task, typology, computational,lingiustics" />
<link href='http://fonts.googleapis.com/css?family=Raleway:400,100,200,300,500,600,700,800,900' rel='stylesheet' type='text/css'>
<!--[if lte IE 8]><script src="js/html5shiv.js"></script><![endif]-->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
<script src="js/skel.min.js"></script>
<script src="js/skel-panels.min.js"></script>
<script src="js/init.js"></script>
<noscript>
<link rel="stylesheet" href="css/skel-noscript.css" />
<link rel="stylesheet" href="css/style.css" />
<link rel="stylesheet" href="css/style-desktop.css" />
</noscript>
</head>
<body>
<div id="header-wrapper">
<!-- Header -->
<div id="header">
<div class="container">
<!-- Logo -->
<div id="logo">
<a href="index.html"><img src="images/sigtyp1.jpg" style="width:70px" alt="SIGTYP"></a>
<!--h1><a href="#"><font color=red>S</font>IG<font color=red>T</font>YP</a></h1-->
</div>
<!-- Nav -->
<nav id="nav">
<ul>
<li><a href="index.html">Homepage</a></li>
<li><a href="constitution.html">Constitution</a></li>
<li><a href="members.html">Members</a></li>
<li class="active"><a href="workshop.html">Workshop</a></li>
<li><a href="lectures.html">Lectures</a></li>
<li><a href="blog.html">Blog</a></li>
</ul>
</nav>
</div>
</div>
<!-- Header -->
<!-- Banner -->
<div id="banner">
<div class="container">
<section>
<!--span class="fa fa-cubes"></span-->
<header>
<h2 > </h2>
<h2 > </h2>
<h2 > </h2>
<span class="byline"> </span>
</header>
<!--a href="#" class="button medium">Fusce ultrices fringilla</a-->
</section>
</div>
</div>
<!-- /Banner -->
</div>
<!-- Main -->
<div id="main">
<div class="container">
<div class="row">
<div class="9u skel-cell-important">
<section>
<header>
<h2> Word Embedding Evaluation for<br/> <br/> Ancient and Historical Languages </h2> </header>
<br/>
<p style="font-size:17px">
In 2024, <font color="red">S</font>IG<font color="red">T</font>YP is hosting a <strong>Word Embedding Evaluation for Ancient and Historical Languages.</strong> Since the rise of word embeddings, their evaluation has been considered a challenging task that sparked considerable debate regarding the optimal approach. The two major strategies that researchers have developed over the years are intrinsic and extrinsic evaluation. The first amounts to solving specially designed problems like semantic proportions or “odd one out”, or comparing word/sentence similarity scores yielded by a model to human judgement. The second one focuses on solving downstream NLP tasks, such as sentiment analysis or question answering, probing word or sentence representations in real-world applications.
<br/><br/>
In recent years, sets of downstream tasks called benchmarks have become a very popular, if not default, method to evaluate general-purpose word and sentence embeddings. Starting with decaNLP (McCann et al., 2018) and SentEval (Conneau & Kiela, 2018), multitask benchmarks for NLU keep appearing and improving every year. However, even the largest multilingual benchmarks, such as XGLUE, XTREME, XTREME-R or XTREME-UP (Hu et al., 2020; Liang et al., 2020; Ruder et al., 2021, 2023), only include modern languages. When it comes to ancient and historical languages, scholars mostly adapt/translate intrinsic evaluation datasets from modern languages or create their own diagnostic tests. We argue that there is a need for a universal evaluation benchmark for embeddings learned from ancient and historical language data and view this shared task as a proving ground for it.
<br/><br/>
The shared task involves solving the following problems for 12+ ancient and historical languages that belong to 4 language families and use 6 different scripts.
</p>
<header><h2 style="font-size:27px">Subtasks</h2></header>
<p style="font-size:17px">
<ul>
<li><strong>A.Constrained</strong>
<ul>
<li> 1. POS-tagging</li>
<li> 2. Full morphological annotation</li>
<li> 3. Lemmatisation</li>
</ul>
</li>
<li><strong>B. Unconstrained</strong>
<ul>
<li> 1. POS-tagging</li>
<li> 2. Full morphological annotation</li>
<li> 3. Lemmatisation</li>
<li> 4. Filling gaps (word-level, character-level)</li>
</ul>
</li>
</ul>
</p>
<p style="font-size:17px">
For subtask A, participants are not allowed to use any additional data; however, they can reduce and balance provided training datasets if they see fit. For subtask B, participants are allowed to use any additional data in any language, including pre-trained embeddings and LLMs.
</p>
<p style="font-size:17px">
Participants will be invited to describe their system in a paper for the SIGTYP workshop proceedings. The task organizers will write an overview paper that describes the task and summarizes the different approaches taken, and analyzes their results.
</p>
<header><h2 style="font-size:27px">Important Links</h2></header>
<p style="font-size:17px">
↣ <a href="https://github.com/sigtyp/ST2024">Shared Task GIT Page (provides all the details and regular updates)</a>!
</p>
<p style="font-size:17px">
↣ <a href="https://docs.google.com/forms/d/e/1FAIpQLSdINgMfzzZGIZ-uBVQhvyndB6yeaaj-wT7v45A6UB4F2h6QBQ/viewform?usp=sf_link">Register for the Task</a>!
</p>
<header><h2 style="font-size:27px">Important Dates</h2></header>
<p style="font-size:17px">
Training data Release: ↣ 5 November 2023<br/>
Test data Release: ↣ 2 January 2024<br/>
Submissions Due: ↣ 8 January 2024<br/>
Notifications of Results: ↣ 13 January 2024<br/>
System Descriptions Due: ↣ 20 January 2024<br/>
Notifications of Acceptance: ↣ 27 January 2024<br/>
Camera-ready Due: ↣ 3 February 2024<br/>
Video Recordings Due: ↣ 15 March 2024<br/>
Workshop: ↣ 21/22 March 2024<br/>
</p>
<header>
<h2 style="font-size:27px">Task Organizers</h2>
</header>
<p style="font-size:15px">
<strong>Oksana Dereza</strong>, Insight SFI Research Centre for Data Analytics, Data Science Institute, University of Galway <br/>
<strong>Priya Rani</strong>, SFI Centre for Research and Training in AI, Data Science Institute, University of Galway <br/>
<strong>Atul Kr. Ojha</strong>, Insight SFI Research Centre for Data Analytics, Data Science Institute, University of Galway <br/>
<strong>Adrian Doyle</strong>, Insight SFI Research Centre for Data Analytics, Data Science Institute, University of Galway<br/>
<strong>Pádraic Moran</strong>, School of Languages, Literatures and Cultures, Moore Institute, University of Galway<br/>
<strong>John P. McCrae</strong>, Insight SFI Research Centre for Data Analytics, Data Science Institute, University of Galway<br/>
</p>
<header>
<h2 style="font-size:27px">Contact</h2>
</header>
<p style="font-size:17px"> Please contact <a href="mailto:oksana.dereza@insight-centre.org"> oksana.dereza@insight-centre.org </a> or <a href="mailto:priya.rani@insight-centre.org"> priya.rani@insight-centre.org </a> if you have any questions </p>
<div align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSdINgMfzzZGIZ-uBVQhvyndB6yeaaj-wT7v45A6UB4F2h6QBQ/viewform?usp=sf_link" class="button medium">Register!</a></div>
</section>
</div>
<div class="3u">
<section class="sidebar">
<header>
<h2>Workshops</h2>
</header>
<ul class="default">
<li><a href="ws2024-mrl.html">» MRL 2024</a></li>
<li><a href="ws2024-sigtyp.html">» SIGTYP 2024 </a></li>
<li><a href="ws2023-sigtyp.html">» SIGTYP 2023</a></li>
<li><a href="ws2023-mrl.html">» MRL 2023</a></li>
<li><a href="ws2022-sigtyp.html">» SIGTYP 2022</a></li>
<li><a href="ws2022-mrl.html">» MRL 2022</a></li>
<li><a href="ws2021.html">» SIGTYP 2021</a></li>
<li><a href="ws2020.html">» SIGTYP 2020</a></li>
<li><a href="ws2019.html">» TyP-NLP 2019</a></li>
</ul>
<br/><br/>
<header>
<h2>Shared Tasks</h2>
</header>
<ul class="default">
<li><a href="st2026-mrl.html">» MRL 2026 Shared Task on ECLeKTic-style Multimodal and Multilingual Benchmark</a></li>
<li><a href="st2024.html">» <font color="red">2024: Word Embedding Evaluation for Ancient and Historical Languages</font></a></li>
<li><a href="st2023.html">» 2023: Cognate and Derivative Detection for Low-Resourced Languages</a></li>
<li><a href="st2023-mrl.html">» MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrieval</a></li>
<li><a href="st2022.html">» 2022: Prediction of Cognate Reflexes</a></li>
<li><a href="st2022-mrl.html">» 2022 (MRL): Multilingual Clause-level Morphology</a></li>
<li><a href="st2021.html">» 2021: Robust Language ID from Speech</a></li>
<li><a href="st2020.html">» 2020: Prediction of Typological Features</a></li>
</ul>
</section>
</div>
</div>
</div>
</div>
<!-- Main -->
<!-- Copyright -->
<div id="copyright">
<div class="container">
Contact Email: <a href="mailto:mattis_list@eva.mpg.de">Mattis List</a> WebMaster: <a href="http://kat.academy">EKATERINA VYLOMOVA</a> Design: <a href="http://templated.co">TEMPLATED</a>
</div>
</div>
</body>
</html>