Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/initial implementation #1

Closed
wants to merge 95 commits into from
Closed
Show file tree
Hide file tree
Changes from 94 commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
702b9d2
feat: initial implementation
sitepark-veltrup Nov 29, 2023
e1439a7
refactor: Various changes following internal coordination
sitepark-veltrup Nov 30, 2023
ec432ba
feat: filter key is optional
sitepark-veltrup Dec 1, 2023
124de81
fix: only url field required
sitepark-veltrup Dec 4, 2023
68352eb
feat: add background indexer
sitepark-veltrup Dec 7, 2023
97b92c7
feat: indexer extended
sitepark-veltrup Dec 11, 2023
2b7fcc9
refactor: indexing by paths, aborter
sitepark-veltrup Dec 13, 2023
b49edc7
chore: composer update
sitepark-veltrup Dec 13, 2023
d65b54c
feat: add chunkSize parameter, remove by idList
sitepark-veltrup Dec 18, 2023
1bcebac
refactor: Rename ResourceSearchResult to SearchResult
sitepark-veltrup Dec 19, 2023
2de4c02
feat: sorting
sitepark-veltrup Jan 9, 2024
8c29ad8
chore: composer update
sitepark-veltrup Jan 9, 2024
3bb7e17
feat: and, or, not filter
sitepark-veltrup Jan 9, 2024
28aee3a
feat: query-filter
sitepark-veltrup Jan 9, 2024
1f9d75c
feat: translation-splitter
sitepark-veltrup Jan 17, 2024
183eab0
feat: IndexSchema2xDocument implementation
sitepark-veltrup Jan 18, 2024
242c169
refactor: DefaultSchema21DocumentEnricher -> DocumentEnricherSiteKitS…
sitepark-veltrup Jan 18, 2024
5ba12f4
feat: nullable strings for index-documents
sitepark-veltrup Jan 18, 2024
f4c1751
Merge branch 'main' into feature/initial-implementation
sitepark-veltrup Jan 19, 2024
e7021d6
feat: stability
sitepark-veltrup Jan 23, 2024
5f16d14
fix: error message
sitepark-veltrup Jan 23, 2024
086e90b
feat: cache error, to see the right location
sitepark-veltrup Jan 23, 2024
d735303
fix: indexing
sitepark-veltrup Jan 23, 2024
0707410
feat: Support translation information such as /path?loc=en_US
sitepark-veltrup Jan 24, 2024
14e0d3d
test: tests made runnable again
sitepark-veltrup Jan 29, 2024
377bb7e
fix: phpstan errors
sitepark-veltrup Jan 30, 2024
9611662
fix: phpcs errors
sitepark-veltrup Jan 30, 2024
2af8c75
docs: add phpstan badge
sitepark-veltrup Jan 30, 2024
06ccb04
docs: add php badge
sitepark-veltrup Jan 30, 2024
4d09f3a
docs: add php badge
sitepark-veltrup Jan 30, 2024
0800eaf
docs: update codecov badge
sitepark-veltrup Jan 30, 2024
aa54d36
test: fix test
sitepark-veltrup Jan 30, 2024
f877a2d
test: new tests for Index-Command
sitepark-veltrup Jan 31, 2024
52deb6b
test: add more tests
sitepark-veltrup Jan 31, 2024
41806d3
fix: phpsc error
sitepark-veltrup Jan 31, 2024
cf32291
test: fix test for ci-server
sitepark-veltrup Jan 31, 2024
52da0f6
feat: cli command atoolo:dump-index-document
sitepark-veltrup Feb 13, 2024
7f1ccbe
feat(indexer): use intro as description if description is not defined
sitepark-veltrup Feb 14, 2024
abcd512
feat: rename config/services.yml -> config/commands.yml
sitepark-veltrup Feb 15, 2024
c40d02d
feat: content matcher to index full-text-content
sitepark-veltrup Feb 15, 2024
6222660
feat: add category name to full-text content
sitepark-veltrup Feb 15, 2024
aaa78a0
refactor: simplify dto's
sitepark-veltrup Feb 20, 2024
8856dac
refactor: QueryDefaultOperator as backed enum
sitepark-veltrup Feb 20, 2024
e05ba48
test: add dto tests
sitepark-veltrup Feb 22, 2024
0b441b5
test: add more tests
sitepark-veltrup Feb 23, 2024
586301a
test: add more tests
sitepark-veltrup Feb 26, 2024
3d1062b
Resource base locator unified
sitepark-veltrup Feb 27, 2024
f1000ea
docs: update README.md
sitepark-veltrup Feb 27, 2024
d8530b6
docs: update class documentation for SolrIndexer
sitepark-veltrup Feb 27, 2024
0e927a8
fix: remove debug code
sitepark-veltrup Feb 27, 2024
6f71e7c
docs: update phpdoc
sitepark-veltrup Mar 1, 2024
358acfd
style: fix max line length
sitepark-veltrup Mar 1, 2024
441bea3
refactor: change filter constructor
sitepark-veltrup Mar 1, 2024
2920d51
feat: excludeFilter for facet dto is optional
sitepark-veltrup Mar 4, 2024
2b5aae2
fix: php warnings
sitepark-veltrup Mar 5, 2024
b32ddec
fix : IndexDocumentDumperBuilder must use ResourceBaseLocatorBuilder
sitepark-veltrup Mar 7, 2024
5f30c8b
feat: IndexSchema2xDocument::setMetaString must also accept arrays
sitepark-veltrup Mar 7, 2024
c1b2473
feat: add IndexSchema2xDocument::setMetaBool()
sitepark-veltrup Mar 7, 2024
299485a
refactor: extract SolrIndexService
sitepark-veltrup Mar 12, 2024
a644145
fix: analyser tests
sitepark-veltrup Mar 12, 2024
6c2a775
feat: date_from and date_to for DefaultSchema2xRceEventDocumentEnricher
sitepark-veltrup Mar 12, 2024
cc386ba
feat: add IndexSchema2xDocument::setMetaText()
sitepark-veltrup Mar 12, 2024
8f95a63
feat: add contenttype field for index-document
sitepark-veltrup Mar 13, 2024
7b744bb
ci: trigger rebuild
sitepark-veltrup Mar 15, 2024
0a37638
refactor: renmae QueryDefaultOperator to DefaultQueryOperator
sitepark-veltrup Mar 18, 2024
ab87006
refactor: renme DefaultQueryOperator to QueryOperator
sitepark-veltrup Mar 20, 2024
7513a47
docs: fix incorrect grammar
sitepark-veltrup Mar 21, 2024
a0cbf2d
style: fix code style
sitepark-veltrup Mar 21, 2024
acab5c3
style: fix indent
sitepark-veltrup Mar 21, 2024
5325c82
style: string concatenation
sitepark-veltrup Mar 21, 2024
cd9c8fd
refactore: text argument as string
sitepark-veltrup Mar 21, 2024
ca7e76b
refactore: terms argument as string
sitepark-veltrup Mar 21, 2024
913148b
refactor IndexerStatusState::INDEXED to IndexerStatusState::FINISHED
sitepark-veltrup Mar 21, 2024
52bd921
refactor: builder methods could return static
sitepark-veltrup Mar 21, 2024
26a5d7c
fix: exception message
sitepark-veltrup Mar 21, 2024
e1ea53a
refactor: default values for all members
sitepark-veltrup Mar 21, 2024
e0bc293
refactor: inherited and meta fields
sitepark-veltrup Mar 21, 2024
da8eef3
refactor: inherited and meta fields
sitepark-veltrup Mar 21, 2024
b07662d
style: fix empty line
sitepark-veltrup Mar 21, 2024
401ce2a
refactore: create dir recursive
sitepark-veltrup Mar 21, 2024
1c40c70
refactore: create the serializer in the constructor
sitepark-veltrup Mar 21, 2024
78541dc
refactore: aborter method names
sitepark-veltrup Mar 21, 2024
76f2980
refactore: rename method mapTranslationPaths() to normalizePaths()
sitepark-veltrup Mar 21, 2024
2b1ec13
refactore: ContentMatcher::match() return type and phpdoc added
sitepark-veltrup Mar 21, 2024
2637fbf
refactor: upate phpstan-type for QuoteSectionMatcher
sitepark-veltrup Mar 21, 2024
24f32ab
refactor: rename SolrIndexService::getAvailableIndexes to SolrIndexSe…
sitepark-veltrup Mar 21, 2024
6bee895
feat: add sp_id to external resources
sitepark-veltrup Mar 21, 2024
0ebb699
refactor: if $len is uneven we run into an error
sitepark-veltrup Mar 21, 2024
da65a68
refactor: Apply suggestions from code review
sitepark-veltrup Mar 21, 2024
7f0911d
test: fix tests
sitepark-veltrup Mar 21, 2024
f4bdbd2
refactor: reduce code
sitepark-veltrup Mar 21, 2024
f79b342
Merge branch 'main' into feature/initial-implementation
sitepark-veltrup Mar 22, 2024
9abecb7
docs: add param doc for MoreLikeThisQuery::fields
sitepark-veltrup Mar 25, 2024
083230c
refactor: reduce code
sitepark-veltrup Mar 25, 2024
0fae792
refactor: use named arguments
sitepark-veltrup Mar 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
!var/cache/.gitkeep
/var/log/*
!var/log/.gitkeep
/var/test/*
/tools
.phpactor.json
composer.lock
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
[![codecov](https://codecov.io/gh/sitepark/atoolo-resource/graph/badge.svg?token=QwvDRxKEa2)](https://codecov.io/gh/sitepark/atoolo-resource)
[![codecov](https://codecov.io/gh/sitepark/atoolo-search/graph/badge.svg?token=xBMwUzm34b)](https://codecov.io/gh/sitepark/atoolo-search)
![phpstan](https://img.shields.io/badge/PHPStan-level%209-brightgreen)
![php](https://img.shields.io/badge/PHP-8.1-blue)
![php](https://img.shields.io/badge/PHP-8.2-blue)
![php](https://img.shields.io/badge/PHP-8.3-blue)

# Atoolo search

Provides services with which a Solr index can be filled and searched for [resources](https://github.com/sitepark/atoolo-resource) via this index.
Provides services with which a Solr index can be filled and searched for [resources](https://github.com/sitepark/atoolo-resource) via a index.

[Documentation](https://sitepark.github.io/atoolo-docs/develop/components/search/)
20 changes: 20 additions & 0 deletions bin/console
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env php
<?php
// application.php

require __DIR__.'/../vendor/autoload.php';

use Atoolo\Search\Console\Application;
use Symfony\Component\Config\FileLocator;

$container = new Symfony\Component\DependencyInjection\ContainerBuilder();
$loader = new Symfony\Component\DependencyInjection\Loader\YamlFileLoader(
$container,
new FileLocator(__DIR__ . '/../config'));

$loader->load('commands.yml');
$container->compile();

$application = $container->get(Application::class);

$application->run();
22 changes: 14 additions & 8 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,26 @@
"prefer-stable": true,
"require": {
"php": ">=8.1 <8.4.0",
"atoolo/resource": "dev-feature/hierarchy-loader",
"atoolo/resource": "dev-main",
"solarium/solarium": "^6.3",
"symfony/config": "^6.3",
"symfony/console": "^6.3",
"symfony/dependency-injection": "^6.3",
"symfony/event-dispatcher": "^6.3",
"symfony/finder": "^6.3",
"symfony/yaml": "^6.3"
"symfony/config": "^6.3 | ^7.0",
"symfony/console": "^6.3 | ^7.0",
"symfony/dependency-injection": "^6.3 | ^7.0",
"symfony/event-dispatcher": "^6.3 | ^7.0",
"symfony/finder": "^6.3 | ^7.0",
"symfony/lock": "^6.3 | ^7.0",
"symfony/property-access": "^6.3 | ^7.0",
"symfony/serializer": "^6.3 | ^7.0",
"symfony/yaml": "^6.3 | ^7.0"
},
"require-dev": {
"dealerdirect/phpcodesniffer-composer-installer": "^1.0",
"infection/infection": "^0.27.6",
"phpcompatibility/php-compatibility": "^9.3",
"phpunit/phpunit": "^10.4",
"roave/security-advisories": "dev-latest",
"squizlabs/php_codesniffer": "^3.7"
"squizlabs/php_codesniffer": "^3.7",
"symfony/filesystem": "^6.3 | ^7.0"
},
"scripts": {

Expand Down Expand Up @@ -77,5 +81,7 @@
"*": "dist"
},
"sort-packages": true
},
"repositories": {
}
}
23 changes: 23 additions & 0 deletions config/commands.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
services:
_defaults:
autowire: true
autoconfigure: true
_instanceof:
Symfony\Component\Console\Command\Command:
tags: ['command']

Atoolo\Search\Console\:
resource: '../src/Console'

Atoolo\Search\Console\Command\Indexer:
arguments:
- !tagged_iterator { tag: 'atoolo.search.indexer.documentEnricher.schema2x' }

Atoolo\Search\Console\Command\DumpIndexDocument:
arguments:
- !tagged_iterator { tag: 'atoolo.search.indexer.documentEnricher.schema2x' }

Atoolo\Search\Console\Application:
public: true
arguments:
- !tagged command
9 changes: 7 additions & 2 deletions phpunit.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@
bootstrap="vendor/autoload.php"
executionOrder="random"
cacheResultFile="var/cache/.phpunit.result.cache"
cacheDirectory="var/cache/.phpunit.cache">
cacheDirectory="var/cache/.phpunit.cache"
displayDetailsOnTestsThatTriggerDeprecations="true"
displayDetailsOnTestsThatTriggerErrors="true"
displayDetailsOnTestsThatTriggerNotices="true"
displayDetailsOnTestsThatTriggerWarnings="true">

<coverage>
<coverage
disableCodeCoverageIgnore="false">
<report>
<clover outputFile="var/log/clover/clover.xml"/>
<html outputDirectory="var/log/clover"/>
Expand Down
22 changes: 22 additions & 0 deletions src/Console/Application.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<?php

declare(strict_types=1);

namespace Atoolo\Search\Console;

use Symfony\Component\Console\Application as BaseApplication;
use Symfony\Component\Console\Command\Command;

class Application extends BaseApplication
{
/**
* @param iterable<Command> $commands
*/
public function __construct(iterable $commands = [])
{
parent::__construct();
foreach ($commands as $command) {
$this->add($command);
}
}
}
80 changes: 80 additions & 0 deletions src/Console/Command/DumpIndexDocument.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
<?php

declare(strict_types=1);

namespace Atoolo\Search\Console\Command;

use Atoolo\Search\Console\Command\Io\TypifiedInput;
use Atoolo\Search\Service\Indexer\DocumentEnricher;
use Atoolo\Search\Service\Indexer\IndexDocument;
use JsonException;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

#[AsCommand(
name: 'atoolo:dump-index-document',
description: 'Dump a index document'
)]
class DumpIndexDocument extends Command
{
/**
* phpcs:ignore
sitepark-schaeper marked this conversation as resolved.
Show resolved Hide resolved
* @param iterable<DocumentEnricher<IndexDocument>> $documentEnricherList
*/
public function __construct(
private readonly iterable $documentEnricherList,
private readonly IndexDocumentDumperBuilder $indexDocumentDumperBuilder
) {
parent::__construct();
}

protected function configure(): void
{
$this
->setHelp('Command to dump a index-document')
->addArgument(
'resource-dir',
InputArgument::REQUIRED,
'Resource directory whose data is to be indexed.'
)
->addArgument(
'paths',
InputArgument::OPTIONAL | InputArgument::IS_ARRAY,
'Resources paths or directories of resources to be indexed.'
)
sitepark-schaeper marked this conversation as resolved.
Show resolved Hide resolved
;
}

/**
* @throws JsonException
*/
protected function execute(
InputInterface $input,
OutputInterface $output
): int {

$typedInput = new TypifiedInput($input);

$resourceDir = $typedInput->getStringArgument('resource-dir');

$dumper = $this->indexDocumentDumperBuilder
->resourceDir($resourceDir)
->documentEnricherList($this->documentEnricherList)
->build();

$paths = $typedInput->getArrayArgument('paths');
$dump = $dumper->dump($paths);

foreach ($dump as $fields) {
$output->writeln(json_encode(
$fields,
JSON_THROW_ON_ERROR | JSON_PRETTY_PRINT
));
}

return Command::SUCCESS;
}
}
56 changes: 56 additions & 0 deletions src/Console/Command/IndexDocumentDumperBuilder.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
<?php

declare(strict_types=1);

namespace Atoolo\Search\Console\Command;

use Atoolo\Resource\Loader\SiteKitLoader;
use Atoolo\Search\Service\Indexer\DocumentEnricher;
use Atoolo\Search\Service\Indexer\IndexDocument;
use Atoolo\Search\Service\Indexer\IndexDocumentDumper;

class IndexDocumentDumperBuilder
{
private string $resourceDir;
/**
* phpcs:ignore
* @var iterable<DocumentEnricher<IndexDocument>>
*/
private iterable $documentEnricherList;

public function __construct(
private readonly ResourceBaseLocatorBuilder $resourceBaseLocatorBuilder
) {
}

public function resourceDir(string $resourceDir): IndexDocumentDumperBuilder
{
$this->resourceDir = $resourceDir;
return $this;
}

/**
* phpcs:ignore
* @param iterable<DocumentEnricher<IndexDocument>> $documentEnricherList
*/
public function documentEnricherList(
iterable $documentEnricherList
): IndexDocumentDumperBuilder {
$this->documentEnricherList = $documentEnricherList;
return $this;
}

public function build(): IndexDocumentDumper
{
$resourceBaseLocator = $this->resourceBaseLocatorBuilder->build(
$this->resourceDir
);

$resourceLoader = new SiteKitLoader($resourceBaseLocator);

return new IndexDocumentDumper(
$resourceLoader,
$this->documentEnricherList
);
}
sitepark-schaeper marked this conversation as resolved.
Show resolved Hide resolved
}
Loading
Loading