Skip to content

Commit

Permalink
Merge pull request #3 from mateuszbieniek/page-fieldtype-cleanup-command
Browse files Browse the repository at this point in the history
Implement 'page-fieldtype-cleanup' command
  • Loading branch information
mateuszbieniek authored Jan 22, 2021
2 parents f269b27 + d4bcacd commit 12e75ad
Show file tree
Hide file tree
Showing 7 changed files with 263 additions and 19 deletions.
25 changes: 20 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,26 @@
## Description
This bundle allows you to check your database against know database corruption and fixes them.
This bundle allows you to check your database against know database corruption and fix them.
Also, you can perform a Smoke Test on your project to determine if all Contents are accessible (ignoring permissions).
The additional functionality is cleaning up your database from leftovers left by `ezplatform-page-fieldtype` bundle.

### Supported database corruptions:
- Content without version (fixed by removing corrupted content)
- Content without attributes (fixed by removing corrupted content)
- Content with duplicated attributes (fixed by removing duplicated attributes)
- Page FieldType related records which are unnecessary and cause flooding

## Usage
The following bundle introduces two commands: `ezplatform:database-health-check` and `ezplatform:page-fieldtype-cleanup`.

*Fixing corruptions will modify your database! Always perform the database backup before running those commands!*

After running those commands it is recommended to [regenerate URL aliases](https://doc.ezplatform.com/en/2.5/guide/url_management/#regenerating-url-aliases), clear persistence cache and [reindex](https://doc.ezplatform.com/en/2.5/guide/search/#reindexing).

### ezplatform:database-health-check
Bundle adds `db-checker` SiteAccess with `cache_pool` set to [NullAdapter](https://github.com/symfony/symfony/blob/3.4/src/Symfony/Component/Cache/Adapter/NullAdapter.php)
so no SPI cache is used when retrieving Content from the database during Smoke Test.
If corruption is found, you will be asked if you want to fix it.

*Fixing corruption will modify your database! Always perform the database backup before running this command!*

All Content's location will be checked for subitems, before removing it. In the case of existing subitems, you will be
presented with an option to swap location with a different one, so subitems are preserved (Content won't be deleted after
swap so script has to be re-run if you wish to delete corrupted Content).
Expand All @@ -23,7 +30,15 @@ php -d memory_limit=-1 bin/console ezplatform:database-health-check --siteaccess
```
Please note that Command may run for a long time (depending on project size). You can speed it up by skipping Smoke Testing with `--skip-smoke-test` option.

After running command is recommended to [regenerate URL aliases](https://doc.ezplatform.com/en/2.5/guide/url_management/#regenerating-url-aliases), clear persistence cache and [reindex](https://doc.ezplatform.com/en/2.5/guide/search/#reindexing).
### ezplatform:page-fieldtype-cleanup
*Warning! This command is only available for Enterprise versions of the platform.*

This command searches your database for `ezpage_*` records which are leftovers from https://issues.ibexa.co/browse/EZEE-3430
and deletes them if necessary to prevent uncontrolled growth of the database.

```
php -d memory_limit=-1 bin/console ezplatform:page-fieldtype-cleanup
```

## Installation
### Requirements
Expand All @@ -32,7 +47,7 @@ This bundle requires eZ Platform 2.5+
### 1. Enable `EzPlatformDatabaseHealthCheckerBundle`
Edit `app/AppKernel.php`, and add
```
$bundles[] = new MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\EzPlatformDatabaseHealthCheckerBundle();
$bundles[] = new MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\EzPlatformDatabaseHealthCheckerBundle();
```
at the end of list of bundles in `dev` environment.

Expand Down
19 changes: 7 additions & 12 deletions src/bundle/Command/DatabaseHealthCheckCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@

namespace MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\Command;

use Doctrine\DBAL\Connection;
use eZ\Bundle\EzPublishCoreBundle\ApiLoader\RepositoryConfigurationProvider;
use eZ\Publish\API\Repository\ContentService;
use eZ\Publish\API\Repository\LocationService;
use eZ\Publish\API\Repository\PermissionResolver;
Expand Down Expand Up @@ -59,18 +57,14 @@ class DatabaseHealthCheckCommand extends Command
/** @var \Symfony\Component\Console\Style\SymfonyStyle */
private $io;

/** @var \eZ\Bundle\EzPublishCoreBundle\ApiLoader\RepositoryConfigurationProvider */
private $repositoryConfigurationProvider;

public function __construct(
ContentGateway $contentGateway,
ContentService $contentService,
LocationService $locationService,
SiteAccess $siteAccess,
PermissionResolver $permissionResolver,
Handler $handler,
Repository $repository,
Connection $connection
Repository $repository
) {
$this->contentGateway = $contentGateway;
$this->contentService = $contentService;
Expand All @@ -79,7 +73,6 @@ public function __construct(
$this->permissionResolver = $permissionResolver;
$this->persistenceHandler = $handler;
$this->repository = $repository;
$this->connection = $connection;

parent::__construct();
}
Expand Down Expand Up @@ -128,27 +121,27 @@ protected function initialize(InputInterface $input, OutputInterface $output): v
parent::initialize($input, $output);
}

protected function execute(InputInterface $input, OutputInterface $output): void
protected function execute(InputInterface $input, OutputInterface $output): ?int
{
$this->io->title('eZ Platform Database Health Checker');
$this->io->text(
sprintf('Using database: <info>%s</info>', $this->connection->getDatabase())
sprintf('Using database: <info>%s</info>', $this->contentGateway->connection->getDatabase())
);

$this->io->warning(
'Fixing corruption will modify your database! Always perform the database backup before running this command!'
);

if (!$this->io->confirm('Are you sure that you want to proceed?', false)) {
return;
return 0;
}

if ($this->siteAccess->name !== 'db-checker') {
if (!$this->io->confirm(
'It is recommended to run this command in "db-checker" SiteAccess. Are you sure that you want ' .
'to continue?',
false)) {
return;
return 0;
}
}

Expand All @@ -161,6 +154,8 @@ protected function execute(InputInterface $input, OutputInterface $output): void
$this->checkDuplicatedAttributes();

$this->io->success('Done');

return 0;
}

private function checkContentWithoutAttributes(InputInterface $input, OutputInterface $output)
Expand Down
123 changes: 123 additions & 0 deletions src/bundle/Command/PageFieldTypeCleanupCommand.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
<?php

declare(strict_types=1);

namespace MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\Command;

use MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway\PageFieldTypeGatewayInterface as Gateway;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;

class PageFieldTypeCleanupCommand extends Command
{
const PAGE_LIMIT = 100;

/** @var \Symfony\Component\Console\Style\SymfonyStyle */
private $io;

/** @var Gateway */
private $gateway;

public function __construct(Gateway $gateway)
{
$this->gateway = $gateway;

parent::__construct();
}

protected function configure(): void
{
$this
->setName('ezplatform:page-fieldtype-cleanup')
->setDescription(
'This command allows you to search your database for orphaned page fieldtype related records
and clean them up.'
)
->setHelp(
<<<EOT
The command <info>%command.name%</info> allows you to check your database for orphaned records related to the Page Fieldtype
and clean those records if chosen to do so.
After running command it is recommended to regenerate URL aliases, clear persistence cache and reindex.
!As the script directly modifies the Database always perform a backup before running it!
EOT
);
}

protected function initialize(InputInterface $input, OutputInterface $output): void
{
$this->io = new SymfonyStyle($input, $output);

parent::initialize($input, $output);
}

protected function execute(InputInterface $input, OutputInterface $output): int
{
if ($this->gateway->pageFieldTypeGateway === null) {
$this->io->warning('Page FieldType bundle is missing. Cannot continue.');

return 0;
}

$this->io->title('eZ Platform Database Health Checker');
$this->io->text(
sprintf('Using database: <info>%s</info>', $this->gateway->connection->getDatabase())
);

$this->io->warning('Always perform the database backup before running this command!');

if (!$this->io->confirm(
'Are you sure that you want to proceed and that you have created the database backup?',
false)
) {
return 0;
}

if ($this->countOrphanedPageRelations() <= 0) {
return 0;
}

$this->deleteOrphanedPageRelations();

$this->io->success('Done');

return 0;
}

private function countOrphanedPageRelations(): int
{
$count = $this->gateway->countOrphanedPageRelations();

$count <= 0
? $this->io->success('Found: 0')
: $this->io->caution(sprintf('Found: %d orphaned pages', $count));

return $count;
}

private function deleteOrphanedPageRelations(): void
{
if (!$this->io->confirm(
sprintf('Are you sure that you want to proceed? The maximum number of pages that will be cleaned
in first iteration is equal to %d.', self::PAGE_LIMIT),
false)
) {
return;
}

$records = $this->gateway->getOrphanedPageRelations(self::PAGE_LIMIT);

$progressBar = $this->io->createProgressBar(count($records));

for ($i = 0; $i < self::PAGE_LIMIT; ++$i) {
if (isset($records[$i])) {
$progressBar->advance(1);
$this->gateway->removePage((int) $records[$i]);
}
}
}
}
12 changes: 11 additions & 1 deletion src/bundle/Resources/config/services.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ services:
$locationGateway: '@ezpublish.persistence.legacy.location.gateway'
$fieldHandler: '@ezpublish.persistence.legacy.field_handler'

MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway\PageFieldTypeDoctrineDatabase:
arguments:
$connection: '@ezpublish.persistence.connection'
$pageFieldTypeGateway: '@?EzSystems\EzPlatformPageFieldType\FieldType\Page\Storage\DoctrineGateway'

MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\Command\DatabaseHealthCheckCommand:
arguments:
$contentGateway: '@MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway\DoctrineDatabase'
Expand All @@ -15,10 +20,15 @@ services:
$permissionResolver: '@eZ\Publish\API\Repository\PermissionResolver'
$handler: '@ezpublish.api.storage_engine'
$repository: '@ezpublish.api.repository'
$connection: '@ezpublish.persistence.connection'
tags:
- { name: 'console.command', command: 'ezplatform:database-health-check' }

MateuszBieniek\EzPlatformDatabaseHealthCheckerBundle\Command\PageFieldTypeCleanupCommand:
arguments:
$gateway: '@MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway\PageFieldTypeDoctrineDatabase'
tags:
- { name: 'console.command', command: 'ezplatform:page-fieldtype-cleanup' }

cache.null:
class: Symfony\Component\Cache\Adapter\NullAdapter
arguments: [~]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ class DoctrineDatabase implements GatewayInterface
/**
* @var \Doctrine\DBAL\Connection
*/
protected $connection;
public $connection;

/**
* @var ContentGateway
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
<?php

declare(strict_types=1);

namespace MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway;

use Doctrine\DBAL\Connection;
use Doctrine\DBAL\FetchMode;
use EzSystems\EzPlatformPageFieldType\FieldType\Page\Storage\Gateway;

class PageFieldTypeDoctrineDatabase implements PageFieldTypeGatewayInterface
{
/** @var \Doctrine\DBAL\Connection */
public $connection;

/** @var \EzSystems\EzPlatformPageFieldType\FieldType\Page\Storage\Gateway|null */
public $pageFieldTypeGateway;

public function __construct(Connection $connection, ?Gateway $pageFieldTypeGateway = null)
{
$this->connection = $connection;
$this->pageFieldTypeGateway = $pageFieldTypeGateway;
}

public function countOrphanedPageRelations(): int
{
$pagesQuery = $this->connection->createQueryBuilder();
$pagesQuery = $pagesQuery->select('id')
->from('ezpage_pages')
->getSQL();

$countQuery = $this->connection->createQueryBuilder();
$countQuery->select('COUNT(page_id)')
->from('ezpage_map_zones_pages', 'p')
->where(
$countQuery->expr()->notIn(
'page_id',
$pagesQuery
)
);

return (int) $countQuery->execute()->fetch(FetchMode::NUMERIC)[0];
}

public function getOrphanedPageRelations(int $limit): array
{
$pagesQuery = $this->connection->createQueryBuilder();
$pagesQuery = $pagesQuery->select('id')
->from('ezpage_pages')
->getSQL();

$orphanedPagesQuery = $this->connection->createQueryBuilder();
$orphanedPagesQuery->select('page_id')
->from('ezpage_map_zones_pages', 'p')
->where(
$orphanedPagesQuery->expr()->notIn(
'page_id',
$pagesQuery
)
)
->setMaxResults($limit);

return $orphanedPagesQuery->execute()->fetchAll(FetchMode::COLUMN);
}

public function removePage(int $pageId): void
{
$removedBlocks = [];
$removedZones = [];

foreach ($this->pageFieldTypeGateway->loadAttributesAssignedToPage($pageId) as $attribute) {
$this->pageFieldTypeGateway->unassignAttributeFromBlock((int) $attribute['id'], (int) $attribute['block_id']);
$this->pageFieldTypeGateway->removeAttribute((int) $attribute['id']);

if (!\in_array($attribute['block_id'], $removedBlocks, true)) {
$this->pageFieldTypeGateway->unassignBlockFromZone((int) $attribute['block_id'], (int) $attribute['zone_id']);
$this->pageFieldTypeGateway->removeBlock((int) $attribute['block_id']);
$this->pageFieldTypeGateway->removeBlockDesign((int) $attribute['block_id']);
$this->pageFieldTypeGateway->removeBlockVisibility((int) $attribute['block_id']);
$removedBlocks[] = $attribute['block_id'];
}

if (!\in_array($attribute['zone_id'], $removedZones, true)) {
$this->pageFieldTypeGateway->unassignZoneFromPage((int) $attribute['zone_id'], $pageId);
$this->pageFieldTypeGateway->removeZone((int) $attribute['zone_id']);
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<?php

declare(strict_types=1);

namespace MateuszBieniek\EzPlatformDatabaseHealthChecker\Persistence\Legacy\Content\Gateway;

interface PageFieldTypeGatewayInterface
{
public function countOrphanedPageRelations(): int;

public function getOrphanedPageRelations(int $limit): array;
}

0 comments on commit 12e75ad

Please sign in to comment.