Storage file size exceeds Java technical IO limitations #321

Bios-Marcel · 2024-10-29T16:25:21Z

Similar to #319 I am getting an issue on start-up. The thing that wonders me, is that EclipseStore happily writes to the file and "breaks" the database, resulting in not being able to access the data anymore. What are the steps to recover or avoid this altogether.

i assume this happened because I wrote one giant transaction (store and commit)?

I am getting the following issue on startup now:

Caused by: org.eclipse.store.storage.exceptions.StorageException: Problem in channel #0
	at org.eclipse.store.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:106) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:168) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageSystem$Default.startThreads(StorageSystem.java:332) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageSystem$Default.internalStartUp(StorageSystem.java:520) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageSystem$Default.start(StorageSystem.java:608) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageSystem$Default.start(StorageSystem.java:71) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:250) ~[storage-embedded-2.0.0.jar:?]
	at org.eclipse.store.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:96) ~[storage-embedded-2.0.0.jar:?]
	at de.ispinsoft.kkg.server.persistence.StorageImpl.<init>(StorageImpl.java:47) ~[classes/:?]
	at de.ispinsoft.kkg.server.persistence.StorageConfiguration.storageImpl(StorageConfiguration.java:61) ~[classes/:?]
	at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$0.CGLIB$storageImpl$1(<generated>) ~[classes/:?]
	at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$FastClass$$1.invoke(<generated>) ~[classes/:?]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) ~[spring-core-6.1.6.jar:6.1.6]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-6.1.6.jar:6.1.6]
	at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$0.storageImpl(<generated>) ~[classes/:?]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1335) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1165) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:225) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1323) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1284) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:486) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:341) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:334) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1252) ~[spring-context-6.1.6.jar:6.1.6]
	at de.ispinsoft.kkg.server.KkgServer.start(KkgServer.java:196) ~[classes/:?]
	at de.ispinsoft.kkg.server.KkgServer.<init>(KkgServer.java:174) ~[classes/:?]
	at de.ispinsoft.kkg.server.KkgServer$$SpringCGLIB$$0.<init>(<generated>) ~[classes/:?]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:208) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:111) ~[spring-beans-6.1.6.jar:6.1.6]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315) ~[spring-beans-6.1.6.jar:6.1.6]
	... 18 more
Caused by: org.eclipse.store.storage.exceptions.StorageExceptionIoReading: Storage file size exceeds Java technical IO limitations: storage$channel_0$channel_0_2.dat
	at org.eclipse.store.storage.types.StorageEntityInitializer$Default.determineLargestFileSize(StorageEntityInitializer.java:304) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityInitializer$Default.allocateInitializationBuffer(StorageEntityInitializer.java:253) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:95) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:85) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:48) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageFileManager$Default.initializeForExistingFiles(StorageFileManager.java:1002) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageFileManager$Default.initializeStorage(StorageFileManager.java:886) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.initializeStorage(StorageChannel.java:782) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:200) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:34) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:78) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:126) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:260) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.work(StorageChannel.java:453) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.run(StorageChannel.java:536) ~[storage-2.0.0.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]

Is this even still a real limitation?

The text was updated successfully, but these errors were encountered:

Bios-Marcel · 2024-10-29T16:38:41Z

Given microstream-one/microstream#611 is still open, I assume this is the same bug?

hg-ms · 2025-01-10T06:44:58Z

Issue #320 is caused by the problematic store operation described in #319

Prevent storage corruption if storage files become lager than 2GB. This fix throws a PersistenceExceptionCommitSizeExceeded exception if a single channels commit size is to large before data is written or it creates a new storage file if appending to the current head file would exceed the size limit.

hg-ms added the bug Something isn't working label Jan 10, 2025

hg-ms mentioned this issue Feb 6, 2025

Store size limit #319 #321 #371

Merged

hg-ms linked a pull request Feb 6, 2025 that will close this issue

Store size limit #319 #321 #371

Merged

hg-ms closed this as completed in #371 Feb 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage file size exceeds Java technical IO limitations #321

Storage file size exceeds Java technical IO limitations #321

Storage file size exceeds Java technical IO limitations #321

Storage file size exceeds Java technical IO limitations #321

Comments