Scraper parsing tests #41
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's Guide by SourceryThis pull request refactors the scraper module to improve modularity, readability, and type safety. It introduces new helper functions for extracting zone URLs and WHOIS servers, adds type annotations to existing functions, and implements comprehensive tests for the new functionality. The changes focus on enhancing the code structure and testability of the scraper module. File-Level Changes
Tips
|
There was a problem hiding this comment.
Hey @kgaughan - I've reviewed your changes and they look great!
Here's what I looked at during the review
- 🟢 General issues: all looks good
- 🟢 Security: all looks good
- 🟡 Testing: 3 issues found
- 🟢 Complexity: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.
|
@sourcery-ai review |
There was a problem hiding this comment.
Hey @kgaughan - I've reviewed your changes and they look great!
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟢 Security: all looks good
- 🟡 Testing: 4 issues found
- 🟢 Complexity: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.
|
@sourcery-ai review |
There was a problem hiding this comment.
Hey @kgaughan - I've reviewed your changes and they look great!
Here's what I looked at during the review
- 🟢 General issues: all looks good
- 🟢 Security: all looks good
- 🟡 Testing: 4 issues found
- 🟢 Complexity: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.
tests/test_scraper.py
Outdated
Comment on lines
1
to
37
| from os import path | ||
|
|
||
| import bs4 | ||
|
|
||
| from uwhoisd import scraper | ||
|
|
||
| HERE = path.dirname(__file__) | ||
|
|
||
|
|
||
| def test_extract_zone_urls(): | ||
| with open(path.join(path.dirname(__file__), "iana-root-zone.html"), encoding="utf-8") as fh: | ||
| body = bs4.BeautifulSoup(fh, "html.parser") | ||
| result = list(scraper.extract_zone_urls("http://example.com", body)) | ||
| # The test zone should not appear | ||
| assert result == [ | ||
| ("aaa", "http://example.com/domains/root/db/aaa.html"), | ||
| ("bt", "http://example.com/domains/root/db/bt.html"), | ||
| ("xxx", "http://example.com/domains/root/db/xxx.html"), | ||
| ] | ||
|
|
||
|
|
||
| def test_extract_zone_urls_edge_cases(): | ||
| empty_body = bs4.BeautifulSoup("", "html.parser") | ||
| assert list(scraper.extract_zone_urls("http://example.com", empty_body)) == [] | ||
|
|
||
|
|
||
| def test_extract_whois_server(): | ||
| with open(path.join(path.dirname(__file__), "zone-info-fragment.html"), encoding="utf-8") as fh: | ||
| body = bs4.BeautifulSoup(fh, "html.parser") | ||
| result = scraper.extract_whois_server(body) | ||
| assert result == "whois.nic.abc" | ||
|
|
||
|
|
||
| def test_extract_whois_server_not_found(): | ||
| body = bs4.BeautifulSoup("<html><body></body></html>", "html.parser") | ||
| result = scraper.extract_whois_server(body) | ||
| assert result is None |
There was a problem hiding this comment.
suggestion (testing): Add tests for scrape_whois_from_iana function
The tests cover the new helper functions well, but there are no tests for the main scrape_whois_from_iana function. Consider adding tests that mock the network requests and verify the function's behavior with different inputs and scenarios.
from unittest.mock import patch
import pytest
from uwhoisd import scraper
def test_scrape_whois_from_iana():
with patch('uwhoisd.scraper.requests.get') as mock_get:
mock_get.return_value.text = '<html></html>'
result = scraper.scrape_whois_from_iana()
assert isinstance(result, dict)
assert len(result) > 0
assert all(isinstance(k, str) and isinstance(v, str) for k, v in result.items())
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Refactor the scraper to improve code modularity by introducing helper functions for extracting zone URLs and WHOIS servers. Add comprehensive tests to validate the new functions and ensure robust handling of edge cases.
Enhancements:
scrape_whois_from_ianafunction to use helper functionsextract_zone_urlsandextract_whois_serverfor improved readability and modularity.Tests:
extract_zone_urlsandextract_whois_serverfunctions to ensure correct functionality and edge case handling.iana-root-zone.htmlandzone-info-fragment.htmlto support the new test cases.