Out of Memory crash on JsonAdapter when parsing very large json array

[hannah-smudge]

Hi! I would like to request if there is a solution for an out of memory issue when parsing a very large json array.
This is the part where it crashes:

 java.lang.OutOfMemoryError: Failed to allocate a 48 byte allocation with 2431568 free bytes and 2374KB until OOM, target footprint 268435456, growth limit 268435456; giving up on allocation because <1% of heap free after GC.
        at com.squareup.moshi.LinkedHashTreeMap.find(LinkedHashTreeMap.java:179)
        at com.squareup.moshi.LinkedHashTreeMap.put(LinkedHashTreeMap.java:97)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:74)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:30)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.StandardJsonAdapters$ObjectJsonAdapter.fromJson(StandardJsonAdapters.java:346)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.Moshi$Lookup.fromJson(Moshi.java:409)
        at com.squareup.moshi.CollectionJsonAdapter.fromJson(CollectionJsonAdapter.java:81)
        at com.squareup.moshi.CollectionJsonAdapter$2.fromJson(CollectionJsonAdapter.java:55)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.StandardJsonAdapters$ObjectJsonAdapter.fromJson(StandardJsonAdapters.java:343)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.Moshi$Lookup.fromJson(Moshi.java:409)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:73)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:30)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.StandardJsonAdapters$ObjectJsonAdapter.fromJson(StandardJsonAdapters.java:346)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.CollectionJsonAdapter.fromJson(CollectionJsonAdapter.java:81)
        at com.squareup.moshi.CollectionJsonAdapter$2.fromJson(CollectionJsonAdapter.java:55)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:73)
        at com.squareup.moshi.MapJsonAdapter.fromJson(MapJsonAdapter.java:30)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at com.test.TestJsonAdapter.fromJson(TestJsonAdapter.kt:52)
        at com.test.TestJsonAdapter.fromJson(TestJsonAdapter.kt:23)
        at com.squareup.moshi.internal.NullSafeJsonAdapter.fromJson(NullSafeJsonAdapter.java:41)
        at retrofit2.converter.moshi.MoshiResponseBodyConverter.convert(MoshiResponseBodyConverter.java:45)
        at retrofit2.converter.moshi.MoshiResponseBodyConverter.convert(MoshiResponseBodyConverter.java:27)
        at retrofit2.OkHttpCall.parseResponse(OkHttpCall.java:225)
        at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:121)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923) 

Appreciate your help. Thanks! Currently using the 2.6.0 version of retrofit.

Or is there a better way to handle large amount of json using MoshiResponseBodyConverter.convert? Thanks

[lebedynskyi]

Same issue for me. Server just give some big JSON file. Need something like Sax Parsser or some adapter to convert stream of bytes to stream of objects

Fatal Exception: java.lang.OutOfMemoryError
Failed to allocate a 46610248 byte allocation with 25165824 free bytes and 40MB until OOM, target footprint 116962944, growth limit 134217728

java.lang.StringFactory.newStringFromBytes (StringFactory.java:249)
okio.Buffer.readString (Buffer.kt:315)
okio.Buffer.readString (Buffer.kt:304)
okio.RealBufferedSource.readString (RealBufferedSource.kt:98)
okhttp3.ResponseBody.string (ResponseBody.kt:72)
com.jakewharton.retrofit2.converter.kotlinx.serialization.Serializer$FromString.fromResponseBody (Serializer.kt:29)
com.jakewharton.retrofit2.converter.kotlinx.serialization.DeserializationStrategyConverter.convert (DeserializationStrategyConverter.kt:11)
com.jakewharton.retrofit2.converter.kotlinx.serialization.DeserializationStrategyConverter.convert (DeserializationStrategyConverter.kt:7)
retrofit2.OkHttpCall.parseResponse (OkHttpCall.java:243)
retrofit2.OkHttpCall$1.onResponse (OkHttpCall.java:153)
okhttp3.internal.connection.RealCall$AsyncCall.run (RealCall.kt:535)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
java.lang.Thread.run (Thread.java:923)```

[JakeWharton]

kotlinx.serialization does not arbitrarily support streaming response bodies and its textual formats require charset decoding (hence the call to ResponseBody.string() in the trace) which are both bad for memory. If you are only doing JSON, there potentially will be streaming support in #4166, once I decide where to put it (here or external somewhere). All of the other serialization converters will already stream the response body, and ones like Moshi decode UTF-8 directly on the byte stream without needing a separate charset decoding step.

[JakeWharton]

There's nothing Retrofit can do to avoid OOMs–these are occurring because your heap is full and is likely fragmented. It's possible there's things like memory leaks and other large allocations occupying the heap and preventing new objects to be created. The stack trace of an OOM is almost always never the underlying cause. It's merely the straw the broke the camels back. You can use heap analysis tools and leak analysis tools to try and identify where your memory is going, but Retrofit was just the unlucky code that happened to be running when you ran out of memory. Any allocation could have failed at any point in the app.