Squirrel SQL: Maximum column length of 100,000 exceeded

While trying to import a large dataset into my database using Squirrel I encountered the following error.


[Dataimport Executor Thread] ERROR net.sourceforge.squirrel_sql.plugins.dataimport.ImportDataIntoTableExecutor - Error while reading file
java.io.IOException: Maximum column length of 100,000 exceeded in column 3 in record 519,064. Set the SafetySwitch property to false if you're expecting column lengths greater than 100,000 characters to avoid this error.
 at com.csvreader.CsvReader.readRecord(CsvReader.java:841)
 at net.sourceforge.squirrel_sql.plugins.dataimport.importer.csv.CSVFileImporter.next(CSVFileImporter.java:123)
 at net.sourceforge.squirrel_sql.plugins.dataimport.ImportDataIntoTableExecutor._execute(ImportDataIntoTableExecutor.java:160)
 at net.sourceforge.squirrel_sql.plugins.dataimport.ImportDataIntoTableExecutor.access$000(ImportDataIntoTableExecutor.java:51)
 at net.sourceforge.squirrel_sql.plugins.dataimport.ImportDataIntoTableExecutor$1.run(ImportDataIntoTableExecutor.java:108)

I tried looking for this SafetySwitch property and found out that it was somewhere hidden in the source-code. I checked out the source code and recompiled the plugin with the SafetySwitch off. Additionally, if there is a SQL exception,the query will be outputted to the logger. I’ve tested the updated plugin in Squirrel-3.4.0-Optional and it worked flawlessly:

Squirrel Data Import
Fixed plugin for large rows

You can download the jar here (use at own risk).

I spent some time doing it the right way. I’ve added a checkbox to enable or disable the SafetySwitch:

Screenshot showing the new features in DataImport pluging

You can download the update jar here.

To install this plugin, simply replace the dataimport.jar in the plugins folder from SquirrelSQL and then restart.

 

I’ve filed the changes to this plugin as a patch so maybe it’ll get included in the next release 🙂 .

A man can dream 🙂

Advertisements

2 thoughts on “Squirrel SQL: Maximum column length of 100,000 exceeded

  1. hehe, sorry, the safety switch logic made sense to me at the time. I didn’t really try to hide it. it’s there so people don’t accidentally try to parse a 10 gb exe as csv, which will buffer into memory into a single column, and probably result in crashed servers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s