Create ML Trouble Loading CSV to Train Word Tagger With Commas in Training Data

Question

Created 8h

Replies 1

Boosts 0

Participants 1

I'm using Numbers to build a spreadsheet that I'm exporting as a CSV. I then import this file into Create ML to train a word tagger model. Everything has been working fine for all the models I've trained so far, but now I'm coming across a use case that has been breaking the import process: commas within the training data. This is a case that none of Apple's examples show.

My project takes Navajo text that has been tokenized by syllables and labels the parts-of-speech.

Case that works...

Raw text:

Naaltsoos yídéeshtah.

Tokens column:

Naal,tsoos, ,yí,déesh,tah,.

Labels column:

NObj,NObj,Space,Verb,Verb,VStem,Punct

Case that breaks...

Raw text:

óola, béésh łigaii, tłʼoh naadą́ą́ʼ, wáin, akʼah, dóó á,shįįh

Tokens column with tokenized text (commas quoted):

óo,la,",", ,béésh, ,łi,gaii,",", ,tłʼoh, ,naa,dą́ą́ʼ,",", ,wáin,",", ,a,kʼah,",", ,dóó, ,á,shįįh

(Create ML reports mismatched columns)

Tokens column with tokenized text (commas escaped):

óo,la,\,, ,béésh, ,łi,gaii,\,, ,tłʼoh, ,naa,dą́ą́ʼ,\,, ,wáin,\,, ,a,kʼah,\,, ,dóó, ,á,shįįh

(Create ML reports mismatched columns)

Tokens column with tokenized text (commas escape-quoted):

óo,la,\",\", ,béésh, ,łi,gaii,\",\", ,tłʼoh, ,naa,dą́ą́ʼ,\",\", ,wáin,\",\", ,a,kʼah,\",\", ,dóó, ,á,shįįh

(record not detected by Create ML)

Tokens column with tokenized text (commas escape-quoted):

óo,la,"","", ,béésh, ,łi,gaii,"","", ,tłʼoh, ,naa,dą́ą́ʼ,"","", ,wáin,"","", ,a,kʼah,"","", ,dóó, ,á,shįįh

(Create ML reports mismatched columns)

Labels column:

NSub,NSub,Punct,Space,NSub,Space,NSub,NSub,Punct,Space,NSub,Space,NSub,NSub,Punct,Space,NSub,Punct,Space,NSub,NSub,Punct,Space,Conj,Space,NSub,NSub

Sample From Spreadsheet

Screenshot 2024-12-22 at 4.43.08 PM.png

Solution Needed

It's simple enough to escape commas within CSV files, but the format needed by Create ML essentially combines entire CSV records into single columns, so I'm ending up needing a CSV record that contains a mixture of commas to use for parsing and ones to use as character literals. That's where this gets complicated.

For this particular use case (which seems like it would frequently arise when training a word tagger model), how should I properly escape a comma literal?

Boost

Answer 1

HullBreach OP

2h

In the (hopefully) short-term, I am able to export the Numbers spreadsheet as a TSV and have written up a crude converter to generate JSON from it that Create ML can properly handle. However, that adds an extra step that I would hope could be eliminated by directly exporting from Numbers for use in Create ML.

0