Rework search indexing / crawling #981
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
leofeyer
pushed a commit
that referenced
this pull request
Dec 4, 2019
Description ----------- This is a sub PR of #981 only containing the CLI variant. It will allow us to focus more on the technical implementation first before we have all UI/CSS changes. Commits ------- 7bcce12 Implemented contao:crawl command c305881 Removed left-over controller 35e68cc Added unit tests for the Escargot factory 9a2f0ea CS 772ed1f Added tests for the subscriber result and fixed issues found along the way 0c8d30b Added unit tests for the searchindexsubscriber ceff452 CS e52c855 Added unit test for the EscargotSubscriberPass 749e00b Added unit tests for the extension as well as configuration 10f18aa CS 281e43b Adjusted to latest Escargot changes 452dfcc Added more command options and unit tests for the command 7d22b89 Merge branch 'master' into feature/crawl-command 072be02 CS b990477 Catch the new indexer exceptions in SearchIndexListener 06b6d9a Updated the deps 7d2df71 Fix the coding style 63b0fdf CS 369ad57 Merge branch 'master' into feature/crawl-command 2d95608 Add unit tests for the service definitions e714b07 Correctly name the interface
leofeyer
pushed a commit
to contao/core-bundle
that referenced
this pull request
Dec 4, 2019
Description ----------- This is a sub PR of contao/contao#981 only containing the CLI variant. It will allow us to focus more on the technical implementation first before we have all UI/CSS changes. Commits ------- 7bcce121 Implemented contao:crawl command c305881f Removed left-over controller 35e68cc2 Added unit tests for the Escargot factory 9a2f0ea5 CS 772ed1fb Added tests for the subscriber result and fixed issues found along the way 0c8d30b0 Added unit tests for the searchindexsubscriber ceff4529 CS e52c8550 Added unit test for the EscargotSubscriberPass 749e00b9 Added unit tests for the extension as well as configuration 10f18aa5 CS 281e43ba Adjusted to latest Escargot changes 452dfcc1 Added more command options and unit tests for the command 7d22b89c Merge branch 'master' into feature/crawl-command 072be02f CS b9904779 Catch the new indexer exceptions in SearchIndexListener 06b6d9a4 Updated the deps 7d2df716 Fix the coding style 63b0fdfc CS 369ad573 Merge branch 'master' into feature/crawl-command 2d956081 Add unit tests for the service definitions e714b074 Correctly name the interface
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements our https://github.com/terminal42/escargot library.
Escargot can crawl all Contao pages like any other search engine does.
Processing the results is achieved by subscribers and we are now not limited to the search indexer anymore. We may add additional subscribers for features such as checking broken links or accessibility checks (missing
altattributes on images etc.).However, replacing the existing search logic and providing the framework for additional subscribers is what this PR is all about.
In addtion, everything that can be done in the back end is now also possible using the new
contao:crawlcommand.I will need styling support here.
Let's bring Contao to the next level :)