Generates a single string which can be used to train or test the data from the given JSON array
Parameters:
Name |
Type |
Description |
data |
InputData
|
Data object containing labels and training data |
config |
object
|
Module config |
Returns:
string
- Converted string
Examples
Generate labelled string
const FastTextTools = require('fasttext-node/tools');
const data = [
{
text: 'Custard Pudding tasting like raw eggs',
labels: ['egg', 'custard', 'pudding'],
},
{
text: 'Is Himalayan pink salt the same as the pink salt used for curing?',
labels: ['salt', 'curing', 'pink', 'usage'],
},
];
const labelledData = FastTextTools.generateLabelString(data);
The following generated output can be then written to a file and the file can be used to train or test the model
__label__egg __label__custard __label__pudding Custard Pudding tasting like raw eggs
__label__salt __label__curing __label__pink __label__usage Is Himalayan pink salt the same as the pink salt used for curing?
Sanitizes a string for training. This effectively removes all dates, special characters, digits, timestamps etc.
It also converts the string to lower case.
Parameters:
Name |
Type |
Description |
input |
string
|
String to be sanitized |
Returns:
string
- Sanitized string
Examples
Sanitize string
const FastTextTools = require('fasttext-node/tools');
const data = "Thank you for your request for Rs123 recharge, benefit will be credited to your account";
const sanitizedString = FastTextTools.sanitizeString(data);
The following sanitized string can be then written to a file and the file can be used to train or test the model
thank you for your request for recharge benefit will be credited to your account