8000 proposal: eager loading support · Issue #91 · ent/ent · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

proposal: eager loading support #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clic 8000 king “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
a8m opened this issue Oct 13, 2019 · 11 comments · Fixed by #263
Closed

proposal: eager loading support #91

a8m opened this issue Oct 13, 2019 · 11 comments · Fixed by #263
Labels
Proposal

Comments

@a8m
Copy link
Member
a8m commented Oct 13, 2019 •
edited
Loading

We open this task in order to get feedback for the generated-code API of eager loading.
The 3 requirements (maybe there are more) we thought about are:

  • Allow filter edges/relations (using .Where)
  • Get nested edges/relations
  • Limit/offset capabilities

A few examples here for what we thought about:

Add a With<EdgeName> method to the builder:

client.User.
  Query().
  Where(...).
  WithPets().                   // populate all user pets.
  WithGroups(group.HasAdmin()). // filter out groups without admin.
  All(ctx)

This example gets all users under a condition + their pets + their groups (with an admin).

What about getting all users + their groups (limit to X) + their admins (eager-loading with depth 2)?
I'm not sure how common this scenario, so I think either disable this, or add a With<Edge>Fn option for more flexible loading:

client.User.
  Query().
  Where(...).
  WithGroupsFn(func(q *ent.GroupQuery) {
    q.Where(...).Limit(X).WithAdmin()
  }).
  All(ctx)

Maybe we can change With<Edge> to behave like With<Edge>Fn and make the function to be an optional parameter.

The return type for eager loading:

We were thinking about the 2 options:

  • Add the schema edges as public fields in the generated struct (e.g. ent.User), and then populate the result into these fields.

  • Add another struct type (struct) for holding the schema edges. For example:

      type UserEdges struct {
     Pets []*Pet
     // ...
  }

    I'm not sure how this is supposed to handle nested eager-loading (User->Pets->Friends)?
@a8m a8m added the help wanted Extra attention is needed label Oct 13, 2019
This was referenced Oct 13, 2019
Open
Closed
@a8m
Copy link
Member Author
a8m commented Oct 15, 2019

Update:
After a discussion between @alexsn and I, the suggested changes to the API are:

  • Add a field named Edges to the generated type that will holds all type edges. For example: 
    type User struct {
    // user fields.
    
    // preloaded edges.
    Edges UserEdges {
      Pet []*Pet
      Card *Card
      // ...
    }
}
  • Add a With<Edge>(<Edge>QueryFunc...) to each builder to preload edges. For example: 
    // PetQueryFunc is a function type that applied on the PetQuery.
type PetQueryFunc func(*PetQuery)

 client.User.
     Query().
     // "groups" is an edge of a user. 
     WithGroups().
     // "pets" is an edge of a user. 
     WithPets(func (q *ent.PetQuery) {
             q.Where(pet.HasFriends()).Limit(5)
     })
     // nested eager loading. 
     WithFriends(func (q *ent.UserQuery) {
             q.WithPets()
     }).
     All(ctx)
    This way, reusable options can be shared. 
    func Limit(n int) PetQueryFunc { return func (q *ent.UserQuery) { q.Limit(n) } }
func Offset(n int) PetQueryFunc { return func (q *ent.UserQuery) { q.Offset(n) } }

// ..
WithPets(Limit(5), Offset(5))
// ..

@devig
Copy link
devig commented Oct 17, 2019

Will it be implemented with JOIN?

@alexsn
Copy link
Collaborator
alexsn commented Oct 17, 2019

Yes.

@a8m
Copy link
Member Author
a8m commented Oct 17, 2019

Will it be implemented with JOIN?

Like @alexsn said, JOINs will be used for eager loading. But, in cases of loading multiple and nested relations, additional queries will be executed as well.

For Gremlin, we didn't figure out yet what is the "best" way for doing this.

@marwan-at-work
Copy link
Contributor

By the same token, if I get a user and all of his/her cars in one eager query, will the return value have both the user's columns as well as the cars?

Right now, graph traversals only return the last edge but not the edges you've visited along the way.

@alexsn
Copy link
Collaborator
alexsn commented Oct 18, 2019 •
edited
Loading

By the same token, if I get a user and all of his/her cars in one eager query, will the return value have both the user's columns as well as the cars?

Yes, as an example:

users, err := client.User.
     Query().
     // eagerly load cars along with users
     WithCars().
     All(ctx)
if err != nil {
     // handle error
}
for _, user := range users {
    for _, car := range user.Edges.Cars {
        // do something with car
    }
}

@alexsn alexsn removed the help wanted Extra attention is needed label Oct 21, 2019
@htdvisser htdvisser mentioned this issue Nov 9, 2019
Closed
@a8m a8m added the Proposal label Dec 4, 2019
@a8m a8m mentioned this issue Dec 5, 2019
Closed
@a8m a8m mentioned this issue Jan 13, 2020
Merged
@a8m a8m closed this as completed in #263 Jan 13, 2020
@marwan-at-work
Copy link
Contributor

@a8m this is awesome 👏 👏 👏 -- gonna give it a try this week

@ansarizafar
Copy link

Multiple database queries for a single ent query means multiple database round trips and increased latency. Microsoft Entity framework 3 is now generating only one sql query for each LINQ query. Previously they were using ent approch but they had to change it due to performance and data consistency issues, https://www.youtube.com/watch?v=PXdgyPpfaz4&list=PLReL099Y5nRd04p81Q7p5TtyjCrj9tz1t&index=14&t=899s

@a8m
Copy link
Member Author
a8m commented Jan 13, 2020

Thanks for the reference @ansarizafar, I'll go over it.

I just want to mention that the development is never done, there's always room for performance improvements. We just discussed between us on how we can use JOIN in some specific cases to improve performance.

We have a working solution now (better state than before), we'll improve it gradually, and we'll be able to compare the performance in each iteration.

@ansarizafar
Copy link
ansarizafar commented Jan 14, 2020 •
edited
Loading

@a8m Please also take a look at Fauna Query Language https://docs.fauna.com/fauna/current/api/fql/ and https://github.com/volatiletech/sqlboiler

@ansarizafar
Copy link

@a8m This go project is can translate GraphQL queries into a single fast SQL query. We can use the same method for eager loading in ent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Proposal
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

entc/gen: add eager-loading support ent/ent
5 participants
@alexsn @devig @ansarizafar @a8m @marwan-at-work
0