10000 Specifying a variant in rspec tests · Issue #198 · assaf/vanity · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
This repository was archived by the owner on Jan 1, 2024. It is now read-only.

Specifying a variant in rspec tests #198

Closed
humandoing opened this issue Mar 25, 2014 · 8 comments
Closed

Specifying a variant in rspec tests #198

humandoing opened this issue Mar 25, 2014 · 8 comments
Labels
feature testing

Comments

@humandoing
Copy link

Hey there,

We've started using vanity for A/B testing in a project, and one issue that we've run into is that a lot of the A/B tests we're running change copy or structure that our automated tests are using. This would be fine if we could force the A/B test to always choose a specific test variant (the control) -- but so far I have not been able to figure out how this works (in spite of reading the documentation and working on it for several hours).

Is there a proper way to do this that I simply have not found?

Thanks!
@phillbaker
Copy link
Collaborator

Yup! Please see https://github.com/assaf/vanity#testing. But please let me
know if that's not clear or does not work (I've read it a 100 times, so it
helps to have new eyes).

Closes #198

On Tue, Mar 25, 2014 at 6:16 PM, Daniel Wintschel
notifications@github.comwrote:

Hey there,

We've started using vanity for A/B testing in a project, and one issue
that we've run into is that a lot of the A/B tests we're running change
copy or structure that our automated tests are using. This would be fine if
we could force the A/B test to always choose a specific test variant (the
control) -- but so far I have not been able to figure out how this works
(in spite of reading the documentation and working on it for several hours).

Is there a proper way to do this that I simply have not found?

Thanks!

Reply to this email directly or view it on GitHubhttps://github.com//issues/198
.

@humandoing
Copy link
Author

Hi @phillbaker - I actually had already tried this approach, and ended up monkey around for a couple hours but to no avail. The problems start with Vanity complaining about there being no Vanity context. From there I tried creating a vanity context and specifying the test to choose using the syntax above -- the result was that there was no runtime error in terms of code execution, but Vanity is definitely not choosing the test that I specify per the syntax provided above.

Is there something else that needs to be done? Do the vanity test helpers need to be included in our spec_helper.rb somehow? Adding a "raise" into the vanity test_helper doesn't result in an error being raised when our tests are run, which leads me to believe that some setup that may be required by vanity is not being included.

Would love to know if you can point me a little further along the path, but if you need more info, I can definitely get you some more specifics.

Thanks.
-daniel

@phillbaker phillbaker reopened this Apr 2, 2014
@phillbaker
Copy link
Collaborator

@humandoing looks like Vanity isn't initializing. Definitely a bug - let me poke around and get back to you.

Also looks like #199 is related.

@phillbaker phillbaker mentioned this issue Apr 2, 2014
Closed
@phillbaker
Copy link
Collaborator

Sorry for the slow response @humandoing. How are you starting your tests? rake test, rspec or something else?

@humandoing
Copy link
Author

Hi @philbaker -- starting tests using rspec.

If I can provide any additional details please let me know.

On Apr 8, 2014, at 5:27 AM, Phill Baker notifications@github.com wrote:

Sorry for the slow response @humandoing. How are you starting your tests? rake test, rspec or something else?

—
Reply to this email directly or view it on GitHub.

@phillbaker
Copy link
Collaborator

I was able to reproduce this, thanks. Looks like a regression that the test
suite didn't catch. So I'll fix both of those issues, sorry for the
slowness on this.

On Tue, Apr 8, 2014 at 11:10 AM, Daniel Wintschel
notifications@github.comwrote:

Hi @philbaker -- starting tests using rspec.

If I can provide any additional details please let me know.

On Apr 8, 2014, at 5:27 AM, Phill Baker notifications@github.com wrote:

Sorry for the slow response @humandoing. How are you starting your
tests? rake test, rspec or something else?

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com//issues/198#issuecomment-39859833
.

@phillbaker
Copy link
Collaborator

Sorry this took so long to solve. Finally figured out a work around for controller specs. Please use a before filter with the following:

  before do
    Vanity.context = controller
  end

So a full spec would look like:

describe StaticController do

  before do
    Vanity.context = controller
  end

  describe "#home" do
    it "foo" do
      Vanity.playground.experiment(:price_options).chooses(19)
      ab_test(:price_options).should == 19
    end
  end
end

I'll probably extract this to a file to be included in the spec helper and update the documentation, but wanted to post it here for other folks with this problem.

@phillbaker phillbaker mentioned this issue Jun 9, 2014
Open
@phillbaker
Copy link
Collaborator

Closing for #209

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
feature testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@humandoing @phillbaker
0