Laravel: Working with large JSON responses as streams

I recently proposed a new feature for the Laravel framework that I believe will make working with stream resources much more straightforward. Let's dive into the details of the new resource() method and how we can leverage it to efficiently handle large JSON responses.

The Problem: Verbose Stream Handling

When dealing with large JSON responses, loading the entire response into memory can be inefficient and sometimes impossible due to memory constraints. Streaming the response is a better approach, but the implementation can be verbose and complex. That's where our new macro comes in handy.

The Solution: A Lazy Collection Macro

Here's the macro I've developed to simplify working with large JSON responses:

use Generator;
use Illuminate\Http\Client\Response;
use Illuminate\Support\LazyCollection;
use JsonMachine\Items;
use JsonMachine\JsonDecoder\ExtJsonDecoder;

/**
 * Get the JSON decoded body of the response as a lazyCollection. The pointer is optional, and should follow the
 * JSON Pointer RFC 6901 syntax. See link below for more information on how to use.
 *
 * @link https://github.com/halaxa/json-machine?tab=readme-ov-file#json-pointer
 */
Response::macro('lazy', fn (?string $key = null): LazyCollection => new LazyCollection(function () use ($key): Generator {
    $options = [
        'decoder' => new ExtJsonDecoder(true), // Cast objects to associative arrays
        'pointer' => $key ?? '',
    ];

    /** @var Response $this */
    rewind($resource = $this->resource());
    foreach (Items::fromStream($resource, $options) as $arrayKey => $item) {
        yield $arrayKey => $item;
    }
}));

This macro adds a lazy() method to the Response class, which returns a LazyCollection. To implement this in your Laravel application, you should add this macro in the boot() method of your AppServiceProvider or any other service provider of your choice.

The macro leverages the json-machine library to efficiently stream and parse large JSON responses. Make sure to install this library in your project using Composer:

composer require halaxa/json-machine

By adding this macro, you're extending Laravel's HTTP Client capabilities to handle large JSON responses more efficiently, without loading the entire response into memory at once.

Key features of this macro

1. Memory Efficiency: It processes the JSON response as a stream, avoiding the need to load the entire response into memory.
2. Flexibility: The optional $key parameter allows you to start streaming from a specific key in the JSON structure, following the JSON Pointer RFC 6901 syntax.
3. Integration: It seamlessly integrates with Laravel's existing LazyCollection for a familiar API.

Using the LazyCollection Macro

Here's how you can use the lazy() method in your Laravel application:

$response = Http::get('https://api.example.com/large-data');

$response->lazy()->each(function ($value, $key) {
    // Process each item without loading the entire response into memory
});

You can also use the JSON Pointer feature to start streaming from a specific key. In the example below I'll demonstrate how you can map the items to a custom data object:

$response = Http::get('https://api.example.com/large-data');

$items = $response->lazy('/results/items')
    ->map(fn (array $item): ItemData => ItemData::fromArray($item));

Note: Read more about the JSON Pointer here. It is a little bit different from the one that the response's collect() method uses, which relies on data_get() under the hood.

Testing the Macro

To ensure the reliability of our macro, I've created a comprehensive test suite. Here's a snippet from the test class:

class ResponseLazyCollectionMacroTest extends TestCase
{
    #[Test]
    #[DataProvider('responseFixturesProvider')]
    public function test_client_response(string $jsonContent): void
    {
        Http::fake([
            'example.com/*' => Http::response($jsonContent, 200),
        ]);

        $response = Http::get('example.com/');
        $this->assertEquals($response->collect()->all(), $response->lazy()->all());
    }

    public static function responseFixturesProvider(): array
    {
        return [
            'simple object' => [
                '{"results": {"apple": {"color": "red"}, "pear": {"color": "yellow"}}}',
            ],
            'nested array' => [
                '{"results": [{"name": "apple", "color": "red"}, {"name": "pear", "color": "yellow"}]}',
            ],
            'multiple subtrees' => [
                '{"berries": [{"name": "strawberry", "color": "red"}, {"name": "raspberry", "color": "red"}],"citruses": [{"name": "orange", "color": "orange"}, {"name": "lime", "color": "green"}]}',
            ],
            'null value' => [
                '{"nullValue": null}',
            ],
        ];
    }

}

This test suite covers various JSON structures to ensure our macro works correctly in different scenarios.

Benefits of Using the Lazy Collection Macro

1. Memory Efficiency: Process large JSON responses without exhausting your server's memory.
2. Improved Performance: Start working with the data as soon as it starts streaming, rather than waiting for the entire response.
3. Cleaner Code: Simplify your codebase by using a consistent, easy-to-understand API for handling large JSON responses.
4. Flexibility: Easily navigate complex JSON structures using JSON Pointer syntax.

Conclusion

By leveraging the new resource() method and this custom lazy() macro, we can significantly improve how we handle large JSON responses in Laravel applications. This approach combines the power of streaming with the elegance of Laravel's collections, resulting in more efficient and maintainable code.

Remember, when working with large datasets, it's crucial to consider both performance and memory usage. This lazy loading approach offers a great balance between the two, allowing you to process vast amounts of data with a minimal memory footprint.

Next time you're faced with a large JSON API response, give this macro a try and see how it can improve your data processing workflows!

Happy coding!