Repository navigation
Expand file tree
/
Copy pathTextSimilarityService.php
More file actions
293 lines (250 loc) · 9.53 KB
/
Copy pathTextSimilarityService.php
File metadata and controls
293 lines (250 loc) · 9.53 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
<?php
/**
Author:Arda Baran
Description:This class provides text similarity analysis for iss chatbot.This service compares user messages with predefined chatbot messages,commands, and intents to determine the most relevant textual match.
In order to do that, user input is normalized to handle common variations such as capitalization, Turkish characters, punctuation, extra spaces, and minor spelling differences.
Then performs similarity analysis by using these approaches:
1)Levenshtein similarity for detecting character-level differences and common spelling mistakes.
2)Jaro-Winkler similarity for identifying closely related strings, particularly when differences occur near the beginning of words.
3)Cosine similarity for comparing the word composition and frequency of two messages.
All of these similarity metrics are combined using weighted coefficients in order to produce a single hybrid similarity score between 0 and 1 so that higher score indicates that the user's message is more similar to
the target message, allowing the ISS Chatbot to handle different variations of the same user request more effectively.
Summary , this service is intended to improve chatbot message matching,intent detection, command recognition, and tolerance to user spelling variations.
*/
namespace App\Services\bot;
class TextSimilarityService
{
public function normalize(string $text): string
/**
* ------------------------------
* Summary:
* This function converts user input into a comparable standard form.
*
* It converts uppercase letters to lowercase, replaces Turkish-specific
* characters with their ASCII equivalents, removes unnecessary characters,
* normalizes multiple spaces, and trims leading and trailing spaces.
*
* This process ensures that different representations of the same user input
* can be compared more reliably by the similarity algorithms.
* ------------------------------
*//
{
$map = [
'ş' => 's', 'ç' => 'c', 'ğ' => 'g', 'ü' => 'u', 'ö' => 'o', 'ı' => 'i', 'İ' => 'i',
];
$text = strtr(mb_strtolower($text), $map);
$text = preg_replace('/[^a-z0-9\s]/u', ' ', $text);
$text = preg_replace('/\s+/', ' ', $text);
return trim($text);
}
public function tokenize(string $text): array
{
/**
* ------------------------------
* Summary:
* This function splits the normalized user input into individual words
* called tokens.
*
* First, the input text is normalized to ensure consistent formatting.
* If the resulting text is empty, an empty array is returned.
* Otherwise, the text is split by spaces and each word is returned
* as a separate token.
*
* These tokens are mainly used by the Cosine Similarity algorithm
* to compare the word composition and frequency of two messages.
* ------------------------------
*/
$text = $this->normalize($text);
if ($text === '') return [];
return explode(' ', $text);
}
public function levenshteinSimilarity(string $a, string $b): float
{
/**
* ------------------------------
* Summary:
* This function calculates the similarity between two strings
* using the Levenshtein distance algorithm.
*
* Levenshtein distance measures how many character-level changes
* are required to transform one string into another.
*
* These changes may include:
* - Inserting a character
* - Deleting a character
* - Replacing a character
*
* The calculated distance is normalized according to the length
* of the longer string and converted into a similarity score
* between 0 and 1.
*
* A score of 1 means that both strings are identical,
* while a score closer to 0 means that the strings are more different.
*
* This method is particularly useful for detecting spelling mistakes,
* missing characters, extra characters, and small typing differences.
* ------------------------------
*/
$max = max(strlen($a), strlen($b));
if ($max === 0) return 1;
return 1 - (levenshtein($a, $b) / $max);
}
public function jaroWinkler(string $a, string $b): float
{
/**
* ------------------------------
* Summary:
* This function calculates the textual similarity between two strings
* using PHP's built-in similar_text() function.
*
* The similar_text() function calculates the percentage of similarity
* between two strings based on their matching character sequences.
*
* The resulting percentage value is converted from a range of 0-100
* into a normalized similarity score between 0 and 1.
*
* A score of 1 means that the strings are considered fully similar,
* while a score closer to 0 means that they have fewer matching parts.
*
* This method is useful for detecting general textual similarity
* between two messages, especially when their character sequences
* are partially similar.
* ------------------------------
*/
similar_text($a, $b, $percent);
return $percent / 100;
}
public function cosineSimilarity(string $a, string $b): float
{
/**
* ------------------------------
* Summary:
* This function calculates the Cosine Similarity between two strings.
*
* First, both strings are normalized and divided into individual
* words called tokens.
*
* The frequency of each token is then counted in both strings.
* These token frequencies are used to create numerical vectors
* representing the content of each message.
*
* The function then calculates:
*
* - Dot product:
* Measures how much the two vectors overlap.
*
* - Vector magnitude:
* Measures the size of each vector.
*
* Finally, the Cosine Similarity formula is applied:
*
* similarity = dot product / (magnitude A * magnitude B)
*
* The resulting score is between 0 and 1. A score of 1 means that both messages have the same word distribution, while a score closer to 0 means that they have little or no word overlap.
*
* Unlike character-based similarity methods, Cosine Similarity focuses primarily on the words contained in the messages.
*
* It also does not consider word order.
* For example:
*
* "bugün hava güzel"
* "güzel hava bugün"
*
* contain the same words and therefore produce a very high Cosine Similarity score.
* ------------------------------
*/
$tokensA = array_count_values($this->tokenize($a));
$tokensB = array_count_values($this->tokenize($b));
$allTokens = array_unique(array_merge(
array_keys($tokensA),
array_keys($tokensB)
));
$vecA = [];
$vecB = [];
foreach ($allTokens as $token) {
$vecA[] = $tokensA[$token] ?? 0;
$vecB[] = $tokensB[$token] ?? 0;
}
$dot = 0;
$normA = 0;
$normB = 0;
for ($i = 0; $i < count($vecA); $i++) {
$dot += $vecA[$i] * $vecB[$i];
$normA += $vecA[$i] ** 2;
$normB += $vecB[$i] ** 2;
}
if ($normA == 0 || $normB == 0) {
return 0;
}
return $dot / (sqrt($normA) * sqrt($normB));
}
public function hybridScore(string $a, string $b): float
{
/**
* ------------------------------
* Summary:
* This function combines multiple text similarity algorithms
* into a single hybrid similarity score.
*
* First, both input strings are normalized to ensure that
* capitalization, Turkish characters, punctuation, and whitespace
* differences do not negatively affect the comparison.
*
* The function then calculates three different similarity scores:
*
* 1) Levenshtein Similarity:
* Measures character-level differences and helps detect
* spelling mistakes, missing characters, and typing errors.
*
* 2) Jaro-Winkler Similarity:
* Measures the general textual similarity between the two strings.
*
* 3) Cosine Similarity:
* Compares the words and their frequencies in both messages.
*
* Each similarity method focuses on a different aspect of the text. Therefore, combining them provides a more balanced similarity measurement than relying on a single algorithm.
* The final score is calculated using weighted coefficients:
*
* Cosine Similarity -> 55%
* Jaro-Winkler Similarity -> 30%
* Levenshtein Similarity -> 15%
*
* Cosine Similarity receives the highest weight because word-level
* similarity is considered the primary factor when comparing
* chatbot messages.
*
* Levenshtein Similarity receives a lower weight because character-level
* similarity is mainly intended to compensate for spelling mistakes
* and small typing differences.
*
* The final result is a normalized score between 0 and 1.
*
* A score closer to 1 indicates that the two messages are highly
* similar, while a score closer to 0 indicates that they are
* significantly different.
*
* Example:
*
* User input:
* "internte baglantim yok"
*
* Expected chatbot message:
* "internet bağlantım yok"
*
* In this case:
* - Levenshtein Similarity helps detect the spelling differences.
* - Jaro-Winkler / similar_text helps measure general character similarity.
* - Cosine Similarity compares the common words between the messages.
*
* Combining these results allows the chatbot to recognize that
* both messages represent a very similar user request.
* ------------------------------
*/
$a = $this->normalize($a);
$b = $this->normalize($b);
$lev = $this->levenshteinSimilarity($a, $b);
$jaro = $this->jaroWinkler($a, $b);
$cos = $this->cosineSimilarity($a, $b);
return ($cos * 0.55) + ($jaro * 0.30) + ($lev * 0.15);
}
}