8000 LookML Validation facing error · Issue #748 · spectacles-ci/spectacles · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

LookML Validation facing error #748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
danitp97 opened this issue Jan 2, 2024 · 11 comments
Open

LookML Validation facing error #748

danitp97 opened this issue Jan 2, 2024 · 11 comments

Comments

@danitp97
Copy link
danitp97 commented Jan 2, 2024

lookmlerror

Hi,
I'm facing this error. I didn't have any issue in previous PRs. I'm not sure what does the error mean.
Could somebody help?
Thx!
@DylanBaker
Copy link
Collaborator

Hi @danitp97. This looks like it is as a result of an HTTP request error. Typically these are transient. Are you getting this repeatedly on this PR?

@matt-lam-thinkific
Copy link

I'll add that I'm getting this as well, running Spectacles CLI from GitHub actions. This also looks similar to a previous issue, #725 ...

Seems like the same setup as above: Running something like

spectacles lookml --base-url [redacted] --project [redacted] --client-id $CLIENT_ID --client-secret $CLIENT_SECRET --branch $GITHUB_HEAD_REF --remote-reset

This generally resolves on retry for me, though this seems to be happening with increasing frequency.

@danitp97
Copy link
Author
danitp97 commented Jan 4, 2024

Hey! Yes, exactly I tried several times and finally worked out. Thanks!!

@camtr0n
Copy link
camtr0n commented Jan 9, 2024 •
edited
Loading

Hi this is happening intermittently on our Spectacles CLI github actions. Just started popping off about 3 weeks ago and was totally fine before that. This does not resolve on retry.
spectacles lookml --base-url ${LOOKER_BASE_URL} --client-id ${LOOKER_CLIENT_ID} --client-secret ${LOOKER_CLIENT_SECRET} --project replicated --severity error

Screenshot 2024-01-09 at 11 53 56 AM
Screenshot 2024-01-09 at 11 55 02 AM

@matt-lam-thinkific
Copy link

One thing I noticed is that every time (at least, every time I've looked at) we get this particular error, the GitHub action run duration is just over 5 minutes. Seems to me like there's some timeout somewhere -- and this also doesn't explain why it runs quickly (~10 seconds) without issue other times.

@Hoanglinh1201
Copy link

Hey, I also got this error and now it is becoming more and more unstable. Maybe 1/10 could be finished and when success, it is very quick, however the rest runs until timeout.

Anyone has any hacks or workarounds that I can use?

@camtr0n
Copy link
camtr0n commented Jan 11, 2024

@Hoanglinh1201 one workaround that seems to work consistently for us is, if I go to the Spectacles user in Looker that has api credentials and if I sudo as that user, and go into the dev branch that is struggling, and pull from remote -- it starts working again for that CI run. I've also done the same by pulling from Production to get our nightly job to pass on rerun.

@Hoanglinh1201
Copy link

thank @camtr0n for the suggestion. I tried but it doesn't seem to solve our problem. I feel we cannot perform lookml validation at all now with 100/100 jobs timed out.

@matt-lam-thinkific
Copy link

Out of curiosity, how complex are everyone's LookML projects who're experiencing this? We've got a pretty large and messy project (lots of include: *.view situations 🙈).

Based on the behaviour @camtr0n is describing, and our own experience, my hypothesis is that LookML validation is just taking longer than 5 minutes, and there's a rogue timeout somewhere in the client code for that long (even though the timeout is set to 7200 seconds...). LookML validation eventually finishes, though, so at some point when we retry the Spectacles run, we get a cache hit and so a retry eventually succeeds and returns results in seconds.

I think this problem is worse when there are lots of CI runs being initiated, all on different PRs on different change sets. In that case, you never get a cache hit because you're always validating a different LookML configuration. We certainly find that retries are likelier to succeed when there are fewer people working on the repo simultaneously.

Any ideas on how we can validate this?

@fredriv
Copy link
fredriv commented Apr 22, 2024

We're seeing the same issue on validations taking > 5 minutes. Here's a full stack trace from Spectacles (v2.4.8):

Completed validation in 5 minutes and 23 seconds.


Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 192, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 375, in main
    asyncio.run(
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/utils.py", line 65, in timed_function
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 826, in run_lookml
    results = await runner.validate_lookml(ref, severity, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/runner.py", line 523, in validate_lookml
    results = await validator.validate(self.project, severity, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/validators/lookml.py", line 43, in validate
    validation_results = await self.client.lookml_validation(project, timeout)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/client.py", line 873, in lookml_validation
    response = await self.post(url=url, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/client.py", line 1[62](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:63), in post
    return await self.request("POST", url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x[64](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:65)/lib/python3.11/site-packages/spectacles/client.py", line 156, in request
    return await self.async_client.request(method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1[67](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:68)4, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 3[70](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:71), in handle_async_request
    with map_httpcore_exceptions():
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

Encountered unexpected ReadError: ""
Full error traceback logged to file.

For support, please create an issue at https://github.com/spectacles-ci/spectacles/issues

Error: Child_process exited with error code 1

@DylanBaker
Copy link
Collaborator

Hey everyone! Just an update that we should be pushing some update retry and httpx code shortly that should hopefully tackle some of these read errors.

We don't yet have a solution to the long running LookML validation via the API, but continue to work with the team at Google/Looker on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fredriv @camtr0n @DylanBaker @Hoanglinh1201 @matt-lam-thinkific @danitp97
0