Stream a file chunk by chunk

[vincentLock21]

Hello,

I am currently using ZipStream in my project, but I encountered a fairly specific case and I wanted to know if there was a solution available to me.

I use the addFileFromStream function because I indeed need to stream, but the problem is that I am not recovering whole files but chunks from S3. So I would have liked to know If it was possible to stream a file chunk by chunk using ZipStream ?

For the options configuration :

$options = new Archive();
$options->setContentType('application/zip');
$options->setContentDisposition('attachment; filename="test.zip"');
$options->setZeroHeader(true);
$options->setComment('test zip file.');
$options->setSendHttpHeaders(true);

For now, I'm going through temporary in-memory storage :

$stream = fopen('php://memory', 'ab+');

The chunk being encrypted, I decrypt it and transform the returned character string into a resource to be able to pass it to the function.

$streamRead = \openssl_decrypt(
  stream_get_contents(fopen($s3path, 'rb')),
  /* ... */
);
fwrite($stream, $streamRead);
rewind($stream);
$zip->addFileFromStream($fileName, $stream);

Unfortunately, by doing this each chunk overwrites the previous one instead of ending at the end of the file.

Thanks for your help !

[maennchen]

It should probably be possible to implement a custom class implementing PSR7 StreamInterface to do this.

EDIT:

This could work something like that (completely untested pseudo code):

use Psr7\AppendStream;
use Psr7\Utils;

$zip = new ZipStream('test.zip', $options);

foreach ($s3keys as $key => $value) {
  $fileName = $value;

  // $chunks = get chunks from S3;

  $fileStream = new AppendStream(array_map(function($chunk) using($bucket, $key, $value) {
    $path = sprintf('s3://%s/%s/%s/%s', $bucket, $key, $chunk->getChunk(), basename($value));

    return Utils::streamFor(openssl_decrypt(fopen($path, 'rb'), /* ... */));
  }, $chunks));

  $zip->addFileFromPsr7Stream($fileName, $fileStream);
}

$zip->finish();

[dudz94]

Hello @maennchen !

Thanks a lot for your help. Code you provide works perfectly.
The only things is the file is write before on disk before the download start. I have a lot of big files (100Go+) and the filesystem cannot write it before send to the browser.

Do you know if there is a way to send the file chunk by chunk without get the all chunks in the server ?

Thanks again for your help !

[maennchen]

Could you try with $options->setFlushOutput(true);

[vincentLock21]

Hello @maennchen ! Thanks for all !

I've tried with $options->setFlushOutput(true); but nothing. I think this is because the ZipStream outpout is fopen('php://output', 'wb'); by default , and Utils::streamFor write on fopen('php://temp', 'r+');.

I tried $options->setOutputStream(fopen('php://temp', 'r+')) and :
$stream = $zip->opt->getOutputStream();
fwrite($stream, $resource);
fseek($stream, 0);
But the result is an empty archive.

I also tested to use, to overcome this, the send() function of ZipStream, which writes to the ZipStream's default outpout, and there I have the chunks which are streamed one after the other. When I open my Mac Finder I can see the zip with the correct size. But impossible to open it.

Here is my code :

        $options = new Archive();
        $options->setFlushOutput(true);
        $options->setSendHttpHeaders(true);

        $zip = new ZipStream('test.zip', $options);

        foreach ($s3keys as $key => $value) {
            $fileName = $value;
            $chunks = //get chunks s3;

            $fileStream = new AppendStream(array_map(function($chunk) use ($key, $value, $zip) {
                $s3path = XXX;

                $str = \openssl_decrypt(
                    stream_get_contents(fopen($s3path, 'rb')),
                    XXX,
                    ....
                );
                $zip->send($str);

                return $zip->opt->getOutputStream();

            }, $chunks));

            $zip->addFileFromPsr7Stream($fileName, $fileStream);
        }

        $zip->finish();`

Did I understand the use of the send() function correctly? Chunks are well streamed one by one but the archive may not be correctly reconstituted?

Thank you again for your time!

[maennchen]

@vincentLock21 You do not need ZipStream::send. This is to send data to the output stream and I'm not even sure right now why it even is public...

ZipStream will start writing into the output stream (default php://output) as soon as you start calling addFile* functions.

ZipStream doe not save anything to disk by itself. That only happens if the output stream is saved to the disk (for example if the output is not flushed or if specifying another target stream) or if the input streams are on disk.

In this case, I would guess that this is something the S3 SDK does.

My time is quite limited at the moment. So to dig around the S3 SDK I do not have the resources to help, sorry.

[vincentLock21]

Ok I'll watch this ! Thanks for all ! 😃